Overview and High-Level Build Interfaces#

This developer guide chapter shows you how to use the repository’s high-level build interfaces to generate HIP Python’s Cython sources, the final packages, and the documentation. While this guide does not target the typical HIP Python user, an understanding of the HIP Python build and packaging process might still be of interest to advanced users that want to tailor the code generator or the autogenerated source code to their needs.

Requirement analysis details

  • Commit used for evaluation: 1db035

  • Date: 06/26/2023

HIP Python Repository Structure#

Listing 28 HIP Python Repository Structure. Main entry points and package folders are highlighted.#
 1 hip-python
 2 ├── _codegen
 3 │   ├── __init__.py
 4 │   ├── control.py
 5 │   ├── cparser.py
 6 │   ├── cython.py
 7 │   ├── doxyparser.py
 8 │   ├── test
 9 │   ├── tree.py
10 ├── _controls.py
11 ├── _cuda_interop_layer_gen.py
12 ├── _gitversion.py
13 ├── _parse_hipify_perl.py
14 ├── codegen_hip_python.py
15 ├── generate_hip_python_pkgs.sh
16 ├── requirements.txt
17 └───packages
18   ├── build_hip_python_pkgs.sh
19   ├── hip-python
20   │   ├── dist
21   │   ├── docs
22   │   ├── examples
23   │   ├── hip
24   │   ├── pyproject.toml
25   │   ├── requirements.txt
26   │   ├── setup.cfg
27   │   └── setup.py
28   └───hip-python-as-cuda
29         ├── cuda
30         ├── dist
31         ├── pyproject.toml
32         ├── requirements.txt
33         ├── setup.cfg
34         └── setup.py

Building from Source#

Building the HIP Python packages from the HIP C/C++ sources is achieved in two steps:

  1. Code generation via generate_hip_python_pkgs.sh.

  2. Package and docs building via packages/build_hip_python_pkgs.sh.

The former script can be used as follows:

Usage: ./generate_hip_python_pkgs.sh [OPTIONS]

Options:
  --rocm-path       Path to a ROCm™ installation, defaults to variable 'ROCM_PATH' if set or '/opt/rocm'.
  -l, --libs        HIP Python modules to generate as comma separated list without whitespaces, defaults to variable 'HIP_PYTHON_LIBS' if set or '*'.
  --pre-clean       Remove the virtual Python environment subfolder '_venv' --- if it exists --- before all other tasks.
  --post-clean      Remove the virtual Python environment subfolder '_venv' --- if it exists --- after all other tasks.
  -n, --no-venv     Do not create and use a virtual Python environment.
  -h, --help        Show this help message.

The latter can be used as shown below:

Usage: ./build_hip_python_pkgs.sh [OPTIONS]

Options:
  --rocm-path        Path to a ROCm installation, defaults to variable 'ROCM_PATH' if set or '/opt/rocm'.
  --libs             HIP Python libraries to build as comma separated list without whitespaces, defaults to variable 'HIP_PYTHON_LIBS' if set or '*'.
                     Add a prefix '^' to NOT build the comma-separated list of libraries that follows but all other libraries.
  --cuda-libs        HIP Python CUDA interop libraries to build as comma separated list without whitespaces, defaults to variable 'HIP_PYTHON_CUDA_LIBS' if set or '*'.
                     Add a prefix '^' to NOT build the comma-separated list of libraries that follows but all other libraries.
  --no-hip           Do not build package 'hip-python'.
  --no-cuda          Do not build package 'hip-python-as-cuda'.
  --no-docs          Do not build the docs of package 'hip-python'.
  --no-api-docs      Temporarily move the 'hip-python/docs/python_api' subfolder so that sphinx does not see it.
  --no-clean-docs    Do not generate docs from scratch, i.e. don't run sphinx with -E switch.
  --run-tests        Run the tests.
  -j,--num-jobs      Number of build jobs to use (currently only applied for building docs). Defaults to 1.
  --pre-clean        Remove the virtual Python environment subfolder '_venv' --- if it exists --- before all other tasks.
  --post-clean       Remove the virtual Python environment subfolder '_venv' --- if it exists --- after all other tasks.
  -n, --no-venv      Do not create and use a virtual Python environment.
  -h, --help         Show this help message.

By default, both scripts will create a virtual Python environment named _venv in their respective parent folder when run. All Python dependencies are automatically installed into these environments.

On systems with an ROCm™ installation installed to the standard location, it suffices to run:

Listing 29 Code generation and building (no removal of virtual Python environments).#
cd hip-python
./generate_hip_python_pkgs.sh
cd packages
./build_hip_python_pkgs.sh

To remove the virtual environments after running the script, additionally add the --post-clean flag as shown below:

Listing 30 Code generation and building plus removal of virtual Python environments.#
cd hip-python
./generate_hip_python_pkgs.sh --post-clean
cd packages
./build_hip_python_pkgs.sh --post-clean

Code Generation Step#

In the code generation step, the header files in a ROCm™ installation are parsed and the HIP Python’s Cython modules are generated. Moreover, content of the hipify-perl script is combined with the HIP header file parse trees in order to generate the CUDA® interoperability layer. Finally, package metadata and documentation files per HIP and CUDA interoperability layer module are generated.

The script ./generate_hip_python_pkgs.sh abstracts all these tasks plus the dependency management. The full script is printed below for a deeper inspection. Most options do not concern DevOps engineers, they are mainly intended for easing development work.

Requirements#

Note

All Python requirements mentioned here are also “baked into” the requirements.txt file at the top level of the repository.

  • Python >= 3.7

  • The following pip packages are required for running the code generation script

    • libclang (>= 14.0.1, != 15.0.3, != 15.0.6, <= 15.0.6.1)

      Used for parsing the ROCm™ header files.

      We tested the following versions:

      • 9.0.1 – fail

      • 10.0.1.0 – fail

      • 11.0.0 – fail

      • 11.0.1 – fail

      • 11.1.0 – fail

      • 12.0.0 – fail

      • 13.0.0 – fail

      • 14.0.1 – success

      • 14.0.6 – success

      • 15.0.3 – fail

      • 15.0.6 – fail

      • 15.0.6.1 – success

      • 16.0.0 – fail

    • Cython

      We use Cython’s Tempita template engine for code generation.

    • pyparsing

      Used for constructing the doxygen parser.

The following pip packages are optional:

  • levenshtein

    For giving suggestions if a HIP name could not be mapped to a CUDA name when constructing the CUDA interoperability modules.

Packaging and Docs Building Step#

In the second step, the generated Cython modules are compiled to Python modules before both are packaged via PyPA ~.build into a ~.pip wheel that can be installed directly or uploaded to a package repository such as PyPI. Finally, the docs are built in this step which require the previously build packages as dependencies.

The script ./packages/build_hip_python_pkgs.sh abstracts all these tasks plus the dependency management — aside from uploading the package. The full script is printed below for a deeper inspection. Most options do not concern DevOps engineers, they are mainly intended for easing development work.

Packaging Requirements#

Note

All Python requirements mentioned here are also “baked into” the requirements.txt files in the subfolders packages/hip-python and packages/hip-python-as-cuda.

  • Python >= 3.7

  • The following pip packages are required for running setup.py:

    • setuptools>=42

    • cython

    • wheel

    • build

Docs Building Requirements#

Note

All Python requirements mentioned here are also “baked into” the requirements.txt file in the subfolders packages/docs/hip-python.

  • Python >= 3.7

  • The following pip packages are required for running building the documentation:

    • rocm-docs-core

    • myst-parser[linkify]