hipify-clang#
hipify-clang
is a clang-based tool for translating CUDA
sources into HIP
sources.
It translates CUDA
source into an abstract syntax tree (AST
), which is traversed by transformation matchers.
After applying all the matchers, the output HIP
source is produced.
Advantages:
hipify-clang
is a translator; any complex constructs will be parsed successfully, or an error will be reported.It supports
clang
options like-I
,-D
,--cuda-path
, etc.Seamless support of new
CUDA
versions, as theclang
front end is statically linked intohipify-clang
and does all the syntactical parsing of aCUDA
source to hipify.Ease of support as a compiler extension.
Disadvantages:
Ensure the input
CUDA
code is correct; incorrect code will not be translated toHIP
.CUDA
should be installed and provided in case of multiple installations by--cuda-path
option.All the includes and defines should be provided to transform code successfully.
hipify-clang: dependencies#
hipify-clang
requires:
LLVM+clang of at least version 4.0.0; the latest stable and recommended release: 18.1.1.
CUDA of at least version 7.0, the latest supported version is 12.3.2.
*
LLVM 3.x
is not supported anymore but might still work.
**
Download the patch and unpack it into your LLVM distributive directory
: a few header files will be overwritten; rebuilding of LLVM
is not needed.
***
Download the patch and unpack it into your LLVM source directory
: the file Cuda.cpp
will be overwritten; needs further rebuilding of LLVM
.
In most cases, you can get a suitable version of LLVM+clang
with your package manager.
Failing that or having multiple versions of LLVM
, you can download a release archive, build or install it, and set CMAKE_PREFIX_PATH so CMake
can find it; for instance: -DCMAKE_PREFIX_PATH=D:\LLVM\18.1.1\dist
hipify-clang: usage#
To process a file, hipify-clang
needs access to the same headers that would be required to compile it with clang
.
For example:
./hipify-clang square.cu --cuda-path=/usr/local/cuda-12.3 -I /usr/local/cuda-12.3/samples/common/inc
hipify-clang
arguments are given first, followed by a separator '--'
, and then the arguments you’d pass to clang
if you were compiling the input file. For example:
./hipify-clang cpp17.cu --cuda-path=/usr/local/cuda-12.3 -- -std=c++17
hipify-clang
also supports the hipification of multiple files that might be specified in a single command line with absolute or relative paths, for instance:
./hipify-clang cpp17.cu ../../square.cu /home/user/cuda/intro.cu --cuda-path=/usr/local/cuda-12.3 -- -std=c++17
To use a specific version of LLVM
during hipification, the hipify-clang
option --clang-resource-directory=
must be specified to point to the clang resource directory - the parent directory for the include
folder that contains __clang_cuda_runtime_wrapper.h
and other header files used during the hipification process, for instance:
./hipify-clang square.cu --cuda-path=/usr/local/cuda-12.3 --clang-resource-directory=/usr/llvm/18.1.1/dist/lib/clang/18
The Clang manual for compiling CUDA may be useful.
hipify-clang: using JSON Compilation Database#
For some hipification automation (starting from clang
8.0.0), it is also possible to provide a Compilation Database in JSON format in the compile_commands.json
file:
-p <folder containing compile_commands.json> or
-p=<folder containing compile_commands.json>
The compilation database should be provided in the compile_commands.json
file or generated by clang
based on CMake
; multiple source files can be specified as well.
Only clang
options must be provided in the compile_commands.json
file; hipify-clang
options can only be provided in the hipify-clang
command line.
Note: Do not use the options separator '--'
. A compilation error will occur if the hipify-clang
options are provided before the separator.
Here is an example of the compile_commands.json
usage: https://github.com/ROCm/HIPIFY/blob/amd-staging/tests/unit_tests/compilation_database/compile_commands.json.in
hipify-clang: hipification statistics#
There are options --print-stats
and --print-stats-csv
that help to see an overall picture of what is hipified and what is not and obtain the hipification statistics. For instance:
hipify-clang intro.cu -cuda-path="C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v12.3.2" --print-stats
[HIPIFY] info: file 'intro.cu' statistics:
CONVERTED refs count: 40
UNCONVERTED refs count: 0
CONVERSION %: 100.0
REPLACED bytes: 604
[HIPIFY] info: file 'intro.cu' statistics:
CONVERTED refs count: 40
UNCONVERTED refs count: 0
CONVERSION %: 100.0
REPLACED bytes: 604
TOTAL bytes: 5794
CHANGED lines of code: 34
TOTAL lines of code: 174
CODE CHANGED (in bytes) %: 10.4
CODE CHANGED (in lines) %: 19.5
TIME ELAPSED s: 0.41
[HIPIFY] info: CONVERTED refs by type:
error: 2
device: 2
memory: 16
event: 9
thread: 1
include_cuda_main_header: 1
type: 2
numeric_literal: 7
[HIPIFY] info: CONVERTED refs by API:
CUDA Driver API: 1
CUDA RT API: 39
[HIPIFY] info: CONVERTED refs by names:
cuda.h: 1
cudaDeviceReset: 1
cudaError_t: 1
cudaEventCreate: 2
cudaEventElapsedTime: 1
cudaEventRecord: 3
cudaEventSynchronize: 3
cudaEvent_t: 1
cudaFree: 4
cudaFreeHost: 3
cudaGetDeviceCount: 1
cudaGetErrorString: 1
cudaGetLastError: 1
cudaMalloc: 3
cudaMemcpy: 6
cudaMemcpyDeviceToHost: 3
cudaMemcpyHostToDevice: 3
cudaSuccess: 1
cudaThreadSynchronize: 1
hipify-clang intro.cu -cuda-path="C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v12.3.2" --print-stats-csv
The generated file with statistics is intro.cu.csv
:
In the case of multiple source files, the statistics will be provided per file and in total.
For a list of hipify-clang
options, run hipify-clang --help
.
hipify-clang: building#
Once the HIPIFY
repository is cloned (git clone https://github.com/ROCm/HIPIFY.git
), the following commands must be run from the HIPIFY
root folder.
cd .. \
mkdir build dist \
cd build
cmake \
-DCMAKE_INSTALL_PREFIX=../dist \
-DCMAKE_BUILD_TYPE=Release \
../hipify
make -j install
Having not found or multiple LLVM
instances, the root folder with LLVM
distributive must be specified in the CMake
command line to build hipify-clang
, for example:
-DCMAKE_PREFIX_PATH=/usr/llvm/18.1.1/dist
On Windows, the following option should be specified for CMake
in the first place: -G "Visual Studio 17 2022"
; the generated hipify-clang.sln
should be built by Visual Studio 17 2022
instead of Make.
Please, see hipify-clang: Windows for the supported tools for building.
Debug build type -DCMAKE_BUILD_TYPE=Debug
is also supported and tested; LLVM+clang
should be built in Debug
mode as well.
64-bit build mode (-Thost=x64
on Windows) is also supported; LLVM+clang
should be built in 64-bit mode as well.
The binary can then be found at ./dist/hipify-clang
or at the folder specified by the -DCMAKE_INSTALL_PREFIX
option.
hipify-clang: testing#
hipify-clang
has unit tests using LLVM
lit
/FileCheck
.
LLVM+clang
should be built from sources, pre-built binaries are not exhaustive for testing. Before building ensure that the software required for building is of an appropriate version.
LLVM <= 9.0.1:
build
LLVM+clang
:
cd .. \
mkdir build dist \
cd build
Linux:
cmake \
-DCMAKE_INSTALL_PREFIX=../dist \
-DLLVM_SOURCE_DIR=../llvm \
-DLLVM_TARGETS_TO_BUILD="X86;NVPTX" \
-DLLVM_INCLUDE_TESTS=OFF \
-DCMAKE_BUILD_TYPE=Release \
../llvm
make -j install
Windows:
cmake \
-G "Visual Studio 16 2019" \
-A x64 \
-Thost=x64 \
-DCMAKE_INSTALL_PREFIX=../dist \
-DLLVM_SOURCE_DIR=../llvm \
-DLLVM_TARGETS_TO_BUILD="NVPTX" \
-DLLVM_INCLUDE_TESTS=OFF \
-DCMAKE_BUILD_TYPE=Release \
../llvm
Run Visual Studio 16 2019
, open the generated LLVM.sln
, build all, and build the INSTALL
project.
LLVM >= 10.0.0:
download
LLVM project
sources;build
LLVM project
:
cd .. \
mkdir build dist \
cd build
Linux:
cmake \
-DCMAKE_INSTALL_PREFIX=../dist \
-DLLVM_TARGETS_TO_BUILD="" \
-DLLVM_ENABLE_PROJECTS="clang" \
-DLLVM_INCLUDE_TESTS=OFF \
-DCMAKE_BUILD_TYPE=Release \
../llvm-project/llvm
make -j install
Windows:
cmake \
-G "Visual Studio 17 2022" \
-A x64 \
-Thost=x64 \
-DCMAKE_INSTALL_PREFIX=../dist \
-DLLVM_TARGETS_TO_BUILD="" \
-DLLVM_ENABLE_PROJECTS="clang" \
-DLLVM_INCLUDE_TESTS=OFF \
-DCMAKE_BUILD_TYPE=Release \
../llvm-project/llvm
Run Visual Studio 17 2022
, open the generated LLVM.sln
, build all, build project INSTALL
.
Ensure
CUDA
of minimum version 7.0 is installed.
Having multiple
CUDA
installations to choose a particular version, theDCUDA_TOOLKIT_ROOT_DIR
option must be specified:Linux:
-DCUDA_TOOLKIT_ROOT_DIR=/usr/include
Windows:
-DCUDA_TOOLKIT_ROOT_DIR="C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v12.3"
-DCUDA_SDK_ROOT_DIR="C:/ProgramData/NVIDIA Corporation/CUDA Samples/v12.3"
Ensure
cuDNN
of the version corresponding toCUDA
version is installed.
Path to
cuDNN
should be specified by theCUDA_DNN_ROOT_DIR
option:Linux:
-DCUDA_DNN_ROOT_DIR=/usr/include
Windows:
-DCUDA_DNN_ROOT_DIR=D:/CUDA/cuDNN/8.9.7
Ensure
CUB
of the version corresponding toCUDA
version is installed.
Path to CUB should be specified by the
CUDA_CUB_ROOT_DIR
option:Linux:
-DCUDA_CUB_ROOT_DIR=/srv/git/CUB
Windows:
-DCUDA_CUB_ROOT_DIR=D:/CUDA/CUB/cub-2.1.0
Ensure
python
of minimum required version 2.7 is installed.Ensure
lit
andFileCheck
are installed - these are distributed withLLVM
.
Install
lit
intopython
:Linux:
python /usr/llvm/18.1.1/llvm-project/llvm/utils/lit/setup.py install
Windows:
python D:/LLVM/18.1.1/llvm-project/llvm/utils/lit/setup.py install
In case of errors, similar to
ModuleNotFoundError: No module named 'setuptools'
, upgrade thesetuptools
package:python -m pip install --upgrade pip setuptools
Starting with LLVM 6.0.1 path to
llvm-lit
python script should be specified by theLLVM_EXTERNAL_LIT
option:Linux:
-DLLVM_EXTERNAL_LIT=/usr/llvm/18.1.1/build/bin/llvm-lit
Windows:
-DLLVM_EXTERNAL_LIT=D:/LLVM/18.1.1/build/Release/bin/llvm-lit.py
FileCheck
:Linux: copy from
/usr/llvm/18.1.1/build/bin/
toCMAKE_INSTALL_PREFIX/dist/bin
Windows: copy from
D:/LLVM/18.1.1/build/Release/bin
toCMAKE_INSTALL_PREFIX/dist/bin
Or specify the path to
FileCheck
inCMAKE_INSTALL_PREFIX
option
To run OpenGL tests successfully on:
Linux: install at least essential GL headers (on Ubuntu by
sudo apt-get install mesa-common-dev
)Windows: nothing to do: all the required headers are shipped with Windows SDK
Set
HIPIFY_CLANG_TESTS
option turned on:-DHIPIFY_CLANG_TESTS=ON
.Build and run tests:
hipify-clang: Linux testing#
On Linux the following configurations are tested:
Ubuntu 14: LLVM 4.0.0 - 7.1.0, CUDA 7.0 - 9.0, cuDNN 5.0.5 - 7.6.5
Ubuntu 16-19: LLVM 8.0.0 - 14.0.6, CUDA 7.0 - 10.2, cuDNN 5.1.10 - 8.0.5
Ubuntu 20-21: LLVM 9.0.0 - 18.1.1, CUDA 7.0 - 12.3.2, cuDNN 5.1.10 - 8.9.7
Ubuntu 22-23: LLVM 13.0.0 - 18.1.1, CUDA 7.0 - 12.3.2, cuDNN 8.0.5 - 8.9.7
Minimum build system requirements for the above configurations:
CMake 3.16.8, GNU C/C++ 9.2, Python 2.7.
Recommended build system requirements:
CMake 3.28.3, GNU C/C++ 13.2, Python 3.12.2.
Here is an example of building hipify-clang
with testing support on Ubuntu 23.10.01
:
cmake
-DHIPIFY_CLANG_TESTS=ON \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_INSTALL_PREFIX=../dist \
-DCMAKE_PREFIX_PATH=/usr/llvm/18.1.1/dist \
-DCUDA_TOOLKIT_ROOT_DIR=/usr/local/cuda-12.3.2 \
-DCUDA_DNN_ROOT_DIR=/usr/local/cudnn-8.9.7 \
-DCUDA_CUB_ROOT_DIR=/usr/local/cub-2.1.0 \
-DLLVM_EXTERNAL_LIT=/usr/llvm/18.1.1/build/bin/llvm-lit \
../hipify
A corresponding successful output:
-- The C compiler identification is GNU 13.2.0
-- The CXX compiler identification is GNU 13.2.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- HIPIFY config:
-- - Build hipify-clang : ON
-- - Test hipify-clang : ON
-- - Is part of HIP SDK : OFF
-- Found ZLIB: /usr/lib/x86_64-linux-gnu/libz.so (found version "1.2.13")
-- Found LLVM 18.1.1:
-- - CMake module path : /usr/llvm/18.1.1/dist/lib/cmake/llvm
-- - Clang include path : /usr/llvm/18.1.1/dist/include
-- - LLVM Include path : /usr/llvm/18.1.1/dist/include
-- - Binary path : /usr/llvm/18.1.1/dist/bin
-- Linker detection: GNU ld
-- ---- The below configuring for hipify-clang testing only ----
-- Found Python: /usr/bin/python3.12 (found version "3.12.2") found components: Interpreter
-- Found lit: /usr/local/bin/lit
-- Found FileCheck: /GIT/LLVM/trunk/dist/FileCheck
-- Initial CUDA to configure:
-- - CUDA Toolkit path : /usr/local/cuda-12.3.2
-- - CUDA Samples path : OFF
-- - cuDNN path : /usr/local/cudnn-8.9.7
-- - CUB path : /usr/local/cub-2.1.0
-- Found CUDAToolkit: /usr/local/cuda-12.3.2/targets/x86_64-linux/include (found version "12.3.107")
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE
-- Found CUDA config:
-- - CUDA Toolkit path : /usr/local/cuda-12.3.2
-- - CUDA Samples path : OFF
-- - cuDNN path : /usr/local/cudnn-8.9.7
-- - CUB path : /usr/local/cub-2.1.0
-- Configuring done (0.5s)
-- Generating done (0.0s)
-- Build files have been written to: /usr/hipify/build
make test-hipify
A corresponding successful output:
Running HIPify regression tests
===============================================================
CUDA 12.3.107 - will be used for testing
LLVM 18.1.1 - will be used for testing
x86_64 - Platform architecture
Linux 6.5.0-15-generic - Platform OS
64 - hipify-clang binary bitness
64 - python 3.12.2 binary bitness
===============================================================
-- Testing: 102 tests, 12 threads --
Testing Time: 6.70s
Total Discovered Tests: 102
Passed: 102 (100.00%)
hipify-clang: Windows testing#
Tested configurations:
LLVM |
CUDA |
cuDNN |
Visual Studio |
CMake |
Python |
---|---|---|---|---|---|
4.0.0 - 5.0.2 |
7.0 - 8.0 |
5.1.10 - 7.1.4 |
2015.14.0, 2017.15.5.2 |
3.5.1 - 3.18.0 |
3.6.4 - 3.8.5 |
6.0.0 - 6.0.1 |
7.0 - 9.0 |
7.0.5 - 7.6.5 |
2015.14.0, 2017.15.5.5 |
3.6.0 - 3.18.0 |
3.7.2 - 3.8.5 |
7.0.0 - 7.1.0 |
7.0 - 9.2 |
7.6.5 |
2017.15.9.11 |
3.13.3 - 3.18.0 |
3.7.3 - 3.8.5 |
8.0.0 - 8.0.1 |
7.0 - 10.0 |
7.6.5 |
2017.15.9.15 |
3.14.2 - 3.18.0 |
3.7.4 - 3.8.5 |
9.0.0 - 9.0.1 |
7.0 - 10.1 |
7.6.5 |
2017.15.9.20, 2019.16.4.5 |
3.16.4 - 3.18.0 |
3.8.0 - 3.8.5 |
10.0.0 - 11.0.0 |
7.0 - 11.1 |
7.6.5 - 8.0.5 |
2017.15.9.30, 2019.16.8.3 |
3.19.2 |
3.9.1 |
11.0.1 - 11.1.0 |
7.0 - 11.2.2 |
7.6.5 - 8.0.5 |
2017.15.9.31, 2019.16.8.4 |
3.19.3 |
3.9.2 |
12.0.0 - 13.0.1 |
7.0 - 11.5.1 |
7.6.5 - 8.3.2 |
2017.15.9.43, 2019.16.11.9 |
3.22.2 |
3.10.2 |
14.0.0 - 14.0.6 |
7.0 - 11.7.1 |
8.0.5 - 8.4.1 |
2017.15.9.57*, 2019.16.11.17, 2022.17.2.6 |
3.24.0 |
3.10.6 |
15.0.0 - 15.0.7 |
7.0 - 11.8.0 |
8.0.5 - 8.8.1 |
2019.16.11.25, 2022.17.5.2 |
3.26.0 |
3.11.2 |
16.0.0 - 16.0.6 |
7.0 - 12.2.2 |
8.0.5 - 8.9.5 |
2019.16.11.29, 2022.17.7.1 |
3.27.3 |
3.11.4 |
17.0.1** - 18.1.1*** |
7.0 - 12.3.2 |
8.0.5 - 8.9.7 |
2019.16.11.34, 2022.17.9.0 |
3.28.3 |
3.12.2 |
19.0.0git |
7.0 - 12.3.2 |
8.0.5 - 8.9.7 |
2019.16.11.34, 2022.17.9.0 |
3.28.3 |
3.12.2 |
* LLVM 14.x.x is the latest major release supporting Visual Studio 2017. To build LLVM 14.x.x correctly by Visual Studio 2017, -DLLVM_FORCE_USE_OLD_TOOLCHAIN=ON should be added to a corresponding CMake command line. LLVM < 14.x.x can be built correctly by Visual Studio 2017 without the LLVM_FORCE_USE_OLD_TOOLCHAIN option. ** Note that LLVM 17.0.0 was withdrawn due to an issue; please use 17.0.1 or newer instead. *** Note that LLVM 18.0.0 has never been released; please use 18.1.0 or newer instead.
Building with testing support by Visual Studio 17 2022
on Windows 10
:
cmake
-G "Visual Studio 17 2022" \
-A x64 \
-Thost=x64 \
-DHIPIFY_CLANG_TESTS=ON \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_INSTALL_PREFIX=../dist \
-DCMAKE_PREFIX_PATH=D:/LLVM/18.1.1/dist \
-DCUDA_TOOLKIT_ROOT_DIR="C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v12.3" \
-DCUDA_SDK_ROOT_DIR="C:/ProgramData/NVIDIA Corporation/CUDA Samples/v12.3" \
-DCUDA_DNN_ROOT_DIR=D:/CUDA/cuDNN/8.9.7 \
-DCUDA_CUB_ROOT_DIR=D:/CUDA/CUB/cub-2.1.0 \
-DLLVM_EXTERNAL_LIT=D:/LLVM/18.1.1/build/Release/bin/llvm-lit.py \
../hipify
A corresponding successful output:
-- Selecting Windows SDK version 10.0.22621.0 to target Windows 10.0.19045.
-- The C compiler identification is MSVC 19.37.32824.0
-- The CXX compiler identification is MSVC 19.37.32824.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: C:/Program Files/Microsoft Visual Studio/2022/Community/VC/Tools/MSVC/14.37.32822/bin/Hostx64/x64/cl.exe - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: C:/Program Files/Microsoft Visual Studio/2022/Community/VC/Tools/MSVC/14.37.32822/bin/Hostx64/x64/cl.exe - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- HIPIFY config:
-- - Build hipify-clang : ON
-- - Test hipify-clang : ON
-- - Is part of HIP SDK : OFF
-- Found LLVM 18.1.1:
-- - CMake module path : D:/LLVM/18.1.1/dist/lib/cmake/llvm
-- - Clang include path : D:/LLVM/18.1.1/dist/include
-- - LLVM Include path : D:/LLVM/18.1.1/dist/include
-- - Binary path : D:/LLVM/18.1.1/dist/bin
-- ---- The below configuring for hipify-clang testing only ----
-- Found Python: C:/Users/TT/AppData/Local/Programs/Python/Python312/python.exe (found version "3.12.2") found components: Interpreter
-- Found lit: C:/Users/TT/AppData/Local/Programs/Python/Python312/Scripts/lit.exe
-- Found FileCheck: D:/LLVM/18.1.1/dist/bin/FileCheck.exe
-- Initial CUDA to configure:
-- - CUDA Toolkit path : C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v12.3
-- - CUDA Samples path : C:/ProgramData/NVIDIA Corporation/CUDA Samples/v12.3
-- - cuDNN path : D:/CUDA/cuDNN/8.9.7
-- - CUB path : D:/CUDA/CUB/cub-2.1.0
-- Found CUDAToolkit: C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v12.3/include (found version "12.3.52")
-- Found CUDA config:
-- - CUDA Toolkit path : C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v12.3
-- - CUDA Samples path : C:/ProgramData/NVIDIA Corporation/CUDA Samples/v12.3
-- - cuDNN path : D:/CUDA/cuDNN/8.9.7
-- - CUB path : D:/CUDA/CUB/cub-2.1.0
-- Configuring done (1.4s)
-- Generating done (0.1s)
-- Build files have been written to: D:/HIPIFY/build
Run Visual Studio 17 2022
, open the generated hipify-clang.sln
, build the project test-hipify
.