Overview and High-Level Build Interfaces#
This developer guide chapter shows you how to use the repository’s high-level build interfaces to generate HIP Python’s Cython sources, the final packages, and the documentation. While this guide does not target the typical HIP Python user, an understanding of the HIP Python build and packaging process might still be of interest to advanced users that want to tailor the code generator or the autogenerated source code to their needs.
Requirement analysis details
Commit used for evaluation: 1db035
Date: 06/26/2023
HIP Python Repository Structure#
1 hip-python
2 ├── _codegen
3 │ ├── __init__.py
4 │ ├── control.py
5 │ ├── cparser.py
6 │ ├── cython.py
7 │ ├── doxyparser.py
8 │ ├── test
9 │ ├── tree.py
10 ├── _controls.py
11 ├── _cuda_interop_layer_gen.py
12 ├── _gitversion.py
13 ├── _parse_hipify_perl.py
14 ├── codegen_hip_python.py
15 ├── generate_hip_python_pkgs.sh
16 ├── requirements.txt
17 └───packages
18 ├── build_hip_python_pkgs.sh
19 ├── hip-python
20 │ ├── dist
21 │ ├── docs
22 │ ├── examples
23 │ ├── hip
24 │ ├── pyproject.toml
25 │ ├── requirements.txt
26 │ ├── setup.cfg
27 │ └── setup.py
28 └───hip-python-as-cuda
29 ├── cuda
30 ├── dist
31 ├── pyproject.toml
32 ├── requirements.txt
33 ├── setup.cfg
34 └── setup.py
Building from Source#
Building the HIP Python packages from the HIP C/C++ sources is achieved in two steps:
Code generation via
generate_hip_python_pkgs.sh
.Package and docs building via
packages/build_hip_python_pkgs.sh
.
The former script can be used as follows:
Usage: ./generate_hip_python_pkgs.sh [OPTIONS]
Options:
--rocm-path Path to a ROCm™ installation, defaults to variable 'ROCM_PATH' if set or '/opt/rocm'.
-l, --libs HIP Python modules to generate as comma separated list without whitespaces, defaults to variable 'HIP_PYTHON_LIBS' if set or '*'.
--pre-clean Remove the virtual Python environment subfolder '_venv' --- if it exists --- before all other tasks.
--post-clean Remove the virtual Python environment subfolder '_venv' --- if it exists --- after all other tasks.
-n, --no-venv Do not create and use a virtual Python environment.
-h, --help Show this help message.
The latter can be used as shown below:
Usage: ./build_hip_python_pkgs.sh [OPTIONS]
Options:
--rocm-path Path to a ROCm installation, defaults to variable 'ROCM_PATH' if set or '/opt/rocm'.
--libs HIP Python libraries to build as comma separated list without whitespaces, defaults to variable 'HIP_PYTHON_LIBS' if set or '*'.
Add a prefix '^' to NOT build the comma-separated list of libraries that follows but all other libraries.
--cuda-libs HIP Python CUDA interop libraries to build as comma separated list without whitespaces, defaults to variable 'HIP_PYTHON_CUDA_LIBS' if set or '*'.
Add a prefix '^' to NOT build the comma-separated list of libraries that follows but all other libraries.
--no-hip Do not build package 'hip-python'.
--no-cuda Do not build package 'hip-python-as-cuda'.
--no-docs Do not build the docs of package 'hip-python'.
--no-api-docs Temporarily move the 'hip-python/docs/python_api' subfolder so that sphinx does not see it.
--no-clean-docs Do not generate docs from scratch, i.e. don't run sphinx with -E switch.
--run-tests Run the tests.
-j,--num-jobs Number of build jobs to use (currently only applied for building docs). Defaults to 1.
--pre-clean Remove the virtual Python environment subfolder '_venv' --- if it exists --- before all other tasks.
--post-clean Remove the virtual Python environment subfolder '_venv' --- if it exists --- after all other tasks.
-n, --no-venv Do not create and use a virtual Python environment.
-h, --help Show this help message.
By default, both scripts will create a virtual Python environment named _venv
in their
respective parent folder when run. All Python dependencies are automatically installed
into these environments.
On systems with an ROCm™ installation installed to the standard location, it suffices to run:
cd hip-python
./generate_hip_python_pkgs.sh
cd packages
./build_hip_python_pkgs.sh
To remove the virtual environments after running the script, additionally add
the --post-clean
flag as shown below:
cd hip-python
./generate_hip_python_pkgs.sh --post-clean
cd packages
./build_hip_python_pkgs.sh --post-clean
Code Generation Step#
In the code generation step, the header files in a ROCm™ installation are parsed
and the HIP Python’s Cython modules are generated. Moreover, content of
the hipify-perl
script is combined with the HIP header file parse trees in order to generate
the CUDA® interoperability layer. Finally, package metadata and documentation files
per HIP and CUDA interoperability layer module are generated.
The script ./generate_hip_python_pkgs.sh
abstracts all these tasks plus the dependency management.
The full script is printed below for a deeper inspection.
Most options do not concern DevOps engineers, they are mainly intended for easing
development work.
Requirements#
Note
All Python requirements mentioned here are also “baked into” the requirements.txt
file at the top level of the repository.
Python >= 3.7
The following
pip
packages are required for running the code generation scriptlibclang
(>= 14.0.1, != 15.0.3, != 15.0.6, <= 15.0.6.1)Used for parsing the ROCm™ header files.
We tested the following versions:
9.0.1 – fail
10.0.1.0 – fail
11.0.0 – fail
11.0.1 – fail
11.1.0 – fail
12.0.0 – fail
13.0.0 – fail
14.0.1 – success
14.0.6 – success
15.0.3 – fail
15.0.6 – fail
15.0.6.1 – success
16.0.0 – fail
-
We use Cython’s Tempita template engine for code generation.
-
Used for constructing the
doxygen
parser.
The following pip
packages are optional:
-
For giving suggestions if a HIP name could not be mapped to a CUDA name when constructing the CUDA interoperability modules.
Packaging and Docs Building Step#
In the second step, the generated Cython modules are compiled to Python modules before
both are packaged via PyPA ~.build
into a ~.pip
wheel that can be
installed directly or uploaded to a package repository such as PyPI.
Finally, the docs are built in this step which require the previously build packages as dependencies.
The script ./packages/build_hip_python_pkgs.sh
abstracts all these tasks
plus the dependency management — aside from uploading the package.
The full script is printed below for a deeper inspection.
Most options do not concern DevOps engineers, they are mainly intended for easing
development work.
Packaging Requirements#
Note
All Python requirements mentioned here are also “baked into” the requirements.txt
files in
the subfolders packages/hip-python
and packages/hip-python-as-cuda
.
Python >= 3.7
The following
pip
packages are required for runningsetup.py
:setuptools>=42
cython
wheel
build
Docs Building Requirements#
Note
All Python requirements mentioned here are also “baked into” the requirements.txt
file in
the subfolders packages/docs/hip-python
.
Python >= 3.7
The following
pip
packages are required for running building the documentation:rocm-docs-core
myst-parser[linkify]