Building TransferBench from source

Building TransferBench from source#

Note

TransferBench is part of the ROCmValidationSuite (RVS) and is installed with it.

To build TransferBench from source, install the following required dependencies first:

Required dependencies#

  • ROCm stack to obtain HIP runtime.

    • The installed HIP version might impact support for some features, such as amd-smi pod membership detection, or UALoE support.

  • libnuma for allocating memory or spawning threads on correct NUMA nodes.

    • For Ubuntu/Debian:

      sudo apt install libnuma-dev

    • For RHEL/CentOS:

      sudo yum install numactl-devel

Optional dependencies#

Depending on your requirement, you can install these optional dependencies:

  • libibverbs: Required for enabling NIC Executor for RDMA transfers.

    • For Ubuntu/Debian:

      sudo apt install rdma-core libibverbs-dev ibverbs-utils

    • For RHEL/CentOS:

      sudo yum install rdma-core libibverbs libibverbs-devel

  • MPI installation (any of the following)

    • OpenMPI:

      • For Ubuntu/Debian:

        sudo apt install openmpi-bin libopenmpi-dev

      • For RHEL/CentOS:

        sudo yum install openmpi openmpi-devel

    • MPICH:

      • For Ubuntu/Debian:

        sudo apt-get install mpich libmpich-dev

      • For RHEL/CentOS:

        sudo yum install mpich mpich-devel

You can build TransferBench from source using two methods: Makefile and CMake.

Method 1: Building from source using Makefile#

To build TransferBench from source using Makefile, run:

git clone https://github.com/ROCm/TransferBench.git
cd TransferBench
make

Note

By default, make targets the GPU architecture detected on the build machine (GPU_TARGETS=native). To target specific architectures, set GPU_TARGETS. See Makefile environment variables.

Makefile environment variables#

To modify the Makefile behavior, use the following environment variables:

Category Environment variable Description Default value
Paths and compilers - To customize which compiler to use or the library to link against. ROCM_PATH ROCm installation path for HIP compiler, includes, and libs. /opt/rocm
CUDA_PATH CUDA installation path for NVCC when building TransferBench for CUDA. /usr/local/cuda
MPI_PATH MPI installation path (for mpi.h and MPI libraries). /usr/local/openmpi
HIPCC HIP compiler. Falls back to hipcc if not found. $(ROCM_PATH)/bin/amdclang++
NVCC CUDA compiler (for building TransferBench for CUDA). $(CUDA_PATH)/bin/nvcc
ROCM_DEVICE_LIB_PATH Path to amdgcn bitcode. Auto-detected from the ROCm layout. (auto)
HIPCONFIG Path to hipconfig, which is used to query the HIP version (for pod communication support check). hipconfig
Feature flags - To control enabling features that require compile-time support. By default, these are enabled under the right conditions. DISABLE_NIC_EXEC Disables NIC Executor support. 0
DISABLE_DMA_BUF Disables DMA-BUF for GPU Direct RDMA. Requires NIC Executor support. 1
DISABLE_MPI_COMM Disables MPI communication backend support for multinode TransferBench. 0
DISABLE_AMD_SMI Disables AMD-SMI pod membership checks. 0
DISABLE_POD_COMM Disables pod communication support (UALoE / MNNVL). 0
Build options SINGLE_KERNEL To compile with a single GFX kernel (faster build, but fewer kernel variants), set to 1. Used mostly for development and debug. 0
GPU_TARGETS Comma-separated GPU architecture targets such as gfx942, gfx950. native
DEBUG To build in debug mode with debug symbols (-O0, -g), set to 1. Runs otherwise in the release mode (-O3). 0

Note

The Makefile looks for MPI headers at $(MPI_PATH)/include (default: /usr/local/openmpi). System package managers typically install MPI to a different location, so you must set MPI_PATH when building with MPI support. To find the correct path, run mpicc --showme:incdirs after installing MPI, then pass it explicitly:

MPI_PATH=<path-to-mpi> make

Method 2: Building from source using CMake#

To build TransferBench from source using CMake, run:

git clone https://github.com/ROCm/TransferBench.git
cd TransferBench
mkdir build && cd build
cmake ..
make

CMake environment variables#

To modify the CMake behavior, use the following environment variables:

Category Environment variable Description Default value
Paths and compilers - To customize which compiler to use or the library to link against. ROCM_PATH ROCm installation path. /opt/rocm
CMAKE_TOOLCHAIN_FILE Toolchain file. Uses ROCM_PATH and CXX to select compiler. toolchain-linux.cmake
CXX C++ compiler. If not set, amdclang++ or hipcc is used. Taken from the toolchain
MPI_PATH Path to MPI installation. Takes priority over find_package(MPI).
Build options (ON/OFF) - Pass -DVAR=value to set BUILD_LOCAL_GPU_TARGET_ONLY Builds only for the GPUs detected on the given machine using rocm_agent_enumerator. OFF
ENABLE_NIC_EXEC Enables RDMA NIC Executor. OFF
ENABLE_MPI_COMM Enables MPI communicator as backbone for multinode TransferBench. OFF
ENABLE_DMA_BUF Enables DMA-BUF for GPU Direct RDMA (requires NIC). OFF
ENABLE_AMD_SMI Enables AMD-SMI pod membership queries. OFF
ENABLE_POD_COMM Enables pod communication (HIP >= 8.0). OFF
Build options SINGLE_KERNEL To compile with a single GFX kernel (faster build, but fewer kernel variants), set to 1. Used mostly for development and debug. 0
DEBUG To build in debug mode with debug symbols (-O0, -g), set to 1. Runs otherwise in release mode (-O3). 0
CMake cache variables GPU_TARGETS Semicolon-separated GPU architectures. Overridden if BUILD_LOCAL_GPU_TARGET_ONLY is ON gfx906;gfx908;gfx90a;gfx942;gfx950;gfx1030;gfx1100;gfx1101;gfx1102;gfx1150;gfx1151;gfx1200;gfx1201;gfx1250
AMD_SMI_EXECUTABLE Path to amd-smi for AMD-SMI version check. amd-smi
HIPCONFIG_EXECUTABLE Path to hipconfig for HIP version or pod check. hipconfig

Note

CMake requires optional features to be explicitly enabled (all default to OFF). Makefile enables features automatically when their dependencies are detected; use DISABLE_* flags to turn them off. To set cache variables, pass -DVAR=value to CMake.

Example: building with MPI and NIC support

git clone https://github.com/ROCm/TransferBench.git
cd TransferBench
mkdir build && cd build
cmake .. -DENABLE_NIC_EXEC=ON -DENABLE_MPI_COMM=ON
make

Troubleshooting common build errors#

Here are some commonly encountered build errors and their fixes:

  • Could not find /opt/rocm/bin/amdclang++ or /opt/rocm/bin/hipcc. Check if the path is correct if you want to build TransferBench

    Occurs if HIP isn’t installed correctly. If it is installed in a different directory, specify it using ROCM_PATH.

  • Could not find standard C++ header 'cmath'

    Normally occurs if the standard C++ headers aren’t installed. Try installing g++-12 or g++-14 based on the OS version. For example, apt-get install g++-12.

Building TransferBench for CUDA#

To build TransferBench on a system with CUDA installed, install the required dependencies first.

Required dependencies#

  • CUDA: The installed CUDA version might impact support for some features such as MNNVL support.

  • libnuma: Used for allocating memory or spawning threads on the right NUMA nodes. Here are the install instructions based on the OS:

    • Ubuntu/Debian:

      sudo apt install libnuma-dev

    • RHEL/CentOS:

      sudo yum install numactl-devel

Build steps#

To build TransferBench on a system with CUDA installed, run:

git clone https://github.com/ROCm/TransferBench.git
cd TransferBench
make TransferBenchCuda
Contents