.. meta:: :description: Instruction on how to install rocSHMEM. :keywords: rocSHMEM, ROCm, install, build, dependencies, MPI, UCX, Open MPI .. _install-rocshmem: -------------------------- Build rocSHMEM from source -------------------------- To build rocSHMEM as part of the ROCm Core SDK, see `TheRock build instructions `__. TheRock is the recommended way to build ROCm components from source. Alternatively, you can build rocSHMEM standalone using the following instructions. Requirements ------------ * ROCm 6.4.0 or later, including the :doc:`HIP runtime `. For more information, see `ROCm installation for Linux `_. * ROCm 7.2 or later is required for the VMM POSIX memory allocator (``ROCSHMEM_HEAP_ALLOCATOR_TYPE=vmm_posix``). * ROCm 7.14 or later with AMD SMI fabric handle support is required for the VMM Fabric memory allocator (``ROCSHMEM_HEAP_ALLOCATOR_TYPE=vmm_fabric``). * The following AMD GPUs have been fully tested for compatibility with rocSHMEM: * MI250X * MI300X, MI308X * MI325X * MI350X, MI355X (Requires ROCm 7.0 or later) * The following AMD GPUs have experimental support in rocSHMEM: * Radeon AI PRO9700, Radeon RX 9070XT, Radeon RX 9070 * Radeon Pro W7900, Radeon RX 7900XTX, Radeon RX 7900XT .. note:: Other AMD GPUs might function with unknown limitations. For the complete list of supported hardware, see `ROCm System Requirements `_. * The RO backend requires ROCm-aware Open MPI and UCX. When using the IPC or GDA backends, MPI is optional. For more information about installing ROCm-aware Open MPI and UCX, see :ref:`install-dependencies`. * Inter-node communication requires AMD Pollara IONIC, Broadcom Thor 2, or CX7 Infiniband NICs. Available network backends -------------------------- rocSHMEM supports the following network backends: * The **IPC (Inter-Process Communication)** backend enables fast communication between GPUs on the same host using ROCm inter-process mechanisms. It does not support inter-node communication. * The **RO (Reverse Offload)** backend enables communication between GPUs on different nodes through a NIC, using a host-based proxy to forward communication orders to and from the GPU. RO is built on an MPI-RMA compatibility layer. * The **GDA (GPU Direct Async)** backend enables communication between GPUs on different nodes through a NIC. In this backend, the GPU directly interacts with the NIC with no host (CPU) involvement in the critical path of communication. You can activate IPC, RO, and GDA backends in the same rocSHMEM build. .. note:: When RO + IPC is active, all atomic operations use the RO backend, even for intra-node communication. When GDA + IPC is active, all atomic operations use the GDA backend, even for intra-node communication. .. _install-dependencies: Building dependencies --------------------- GDA NIC dependencies ^^^^^^^^^^^^^^^^^^^^ - GDA on Mellanox NICs should work on any recent version of rdma-core. - GDA on Broadcom Thor requires driver version 233.2.108.0 and firmware version 233.2.104.0 or later. - GDA on AMD Pensando Pollara 400 AI NIC requires a newer driver and firmware version - contact AMD for the latest supported version. Building rocSHMEM with MPI (Optional) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ rocSHMEM requires ROCm-Aware Open MPI and UCX for the RO backend. MPI is optional with the IPC and GDA backends. Other MPI implementations, such as MPICH, have not been fully tested. UCX version 1.21.1 or newer is recommended for the RO backend. To build and configure ROCm-Aware UCX, run: .. code-block:: bash git clone https://github.com/ROCm/ucx.git -b v1.21.x cd ucx ./autogen.sh ./configure --prefix= --with-rocm= --enable-mt make -j 8 make -j 8 install To build Open MPI 5.0.7 or later with UCX support, run: .. code-block:: bash git clone --recursive https://github.com/open-mpi/ompi.git -b v5.0.x cd ompi ./autogen.pl ./configure --prefix= --with-rocm= --with-ucx= make -j 8 make -j 8 install Alternatively, you can use a script to install dependencies: .. code-block:: bash export BUILD_DIR=/path/to/not_rocshmem_src_or_build/dependencies /path/to/rocshmem_src/scripts/install_dependencies.sh .. note:: Configuration options vary by platform. Review the script to ensure it is compatible with your system. For more information about OpenMPI-UCX support, see `GPU-enabled Message Passing Interface `_. Installing from source -------------------------------- You can choose from three communication backends at build time for rocSHMEM: IPC, RO, and GDA. Backend can be combined during build time. MPI is not required to build rocSHMEM. To disable MPI, pass the following flag to the build configuration scripts ``-DUSE_EXTERNAL_MPI=OFF``. However, this will disable the functional and unit tests, as they require MPI to run. Memory allocator options ^^^^^^^^^^^^^^^^^^^^^^^^^ rocSHMEM provides several GPU memory allocator options that control how the symmetric heap is allocated. The allocator is selected at runtime via the ``ROCSHMEM_HEAP_ALLOCATOR_TYPE`` environment variable: * **finegrained** (default): GPU memory with fine-grained coherency. Provides CPU access to GPU memory with cache coherency. * **coarsegrained**: GPU memory with coarse-grained coherency. Better performance for GPU-only access patterns. * **uncached**: GPU memory in uncached mode (requires ROCm 5.5+). May provide better performance on some architectures. * **vmm_posix**: GPU memory using Virtual Memory Management (VMM) with POSIX file descriptor-based IPC (requires ROCm 7.2+). This allocator uses advanced HIP VMM APIs (``hipMemCreate``, ``hipMemAddressReserve``, ``hipMemMap``) and cross-process file descriptor sharing via Linux kernel syscalls (``pidfd_open``, ``pidfd_getfd``). .. note:: The ``vmm_posix`` allocator requires: * ROCm 7.2 or newer * Linux kernel 5.6 or newer * TCP Bootstrap-based initialization (not compatible with MPI-based initialization) * **vmm_fabric**: GPU memory using VMM with fabric handle-based IPC (requires ROCm 7.14+ with AMD SMI fabric handle support). This allocator is only supported on gfx1250 (MI455) GPUs. To select a non-default allocator, set the environment variable before launching the application. For example: .. code-block:: bash export ROCSHMEM_HEAP_ALLOCATOR_TYPE=coarsegrained mpiexec -np 2 ./my_rocshmem_app For a full description of accepted values and defaults, see :ref:`rocshmem-api-env-variables`. Profiling and tracing support ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ rocSHMEM can register its host-side API tables with `rocprofiler-register `_ so that rocprofiler-sdk tools, such as ``rocprofv3``, can trace rocSHMEM host-stream API calls. This is controlled by the ``USE_ROCPROFILER_REGISTER`` build option, which is enabled (``ON``) by default. When the option is enabled, the build searches for the ``rocprofiler-register`` package. If it is found, rocSHMEM is compiled with tracing support and the following host-stream APIs are made visible to rocprofiler-sdk tools: * ``rocshmem_barrier_all_on_stream`` * ``rocshmem_quiet_on_stream`` * ``rocshmem_sync_all_on_stream`` * ``rocshmem_alltoallmem_on_stream`` * ``rocshmem_broadcastmem_on_stream`` * ``rocshmem_getmem_on_stream`` * ``rocshmem_putmem_on_stream`` * ``rocshmem_putmem_signal_on_stream`` * ``rocshmem_signal_wait_until_on_stream`` If ``rocprofiler-register`` is not found, the build continues with tracing disabled and prints a status message. To build without tracing support, pass ``-DUSE_ROCPROFILER_REGISTER=OFF`` to CMake: .. code-block:: bash cd projects/rocshmem/build cmake .. -DUSE_ROCPROFILER_REGISTER=OFF cmake --build . --parallel 8 Once rocSHMEM is built with tracing support, the host-stream APIs listed above become visible to rocprofiler-sdk tools such as ``rocprofv3``, which can then collect and report rocSHMEM API activity alongside the rest of the application's ROCm trace. For more information about capturing traces, see the `rocprofiler-sdk documentation `_. All backends build ^^^^^^^^^^^^^^^^^^ To build and install rocSHMEM with all three backends, run: .. code-block:: bash git clone --no-checkout --filter=blob:none git@github.com:ROCm/rocm-systems.git cd rocm-systems git sparse-checkout set --cone projects/rocshmem git checkout develop cd projects/rocshmem mkdir build cd build ../scripts/build_configs/all_backends The build script passes configuration options to CMake to set up a canonical build. .. note:: This builds rocSHMEM with all backends. You can select IPC, RO, GDA, or any combination at runtime by setting an environment variable (see :doc:`Environment variables <./env_variables>` for more details). However, this portability can reduce performance, so the other build scripts are recommended if you need maximum performance. If no specific backend is requested by the user, the library will use the IPC backend if all PEs are on a single node. If the job spans multiple nodes, rocSHMEM will try to use the various GDA backends first, and fall back to the RO backend if neither of the GDA backends can be used. GDA backend build ^^^^^^^^^^^^^^^^^ To build and install rocSHMEM with the GDA backends, run: .. code-block:: bash git clone --no-checkout --filter=blob:none git@github.com:ROCm/rocm-systems.git cd rocm-systems git sparse-checkout set --cone projects/rocshmem git checkout develop cd projects/rocshmem mkdir build cd build # Choose one of the following scripts for your NIC vendor: ../scripts/build_configs/gda_bnxt # Broadcom ../scripts/build_configs/gda_ionic # AMD Pollara ../scripts/build_configs/gda_mlx5 # Mellanox The build script passes configuration options to CMake to set up a canonical build. RO and IPC backend build ^^^^^^^^^^^^^^^^^^^^^^^^ To build and install rocSHMEM with the hybrid RO (off-node) and IPC (on-node) backends, run: .. code-block:: bash git clone --no-checkout --filter=blob:none git@github.com:ROCm/rocm-systems.git cd rocm-systems git sparse-checkout set --cone projects/rocshmem git checkout develop cd projects/rocshmem mkdir build cd build ../scripts/build_configs/ro_ipc The build script passes configuration options to CMake to set up a canonical build. .. note:: The only officially supported configuration for the RO backend uses Open MPI and UCX with a CX7 InfiniBand adapter. For more information, see :ref:`install-dependencies`. Other configurations, such as MPI implementations that are thread-safe and support GPU buffers, might work but are considered experimental. IPC only backend build ^^^^^^^^^^^^^^^^^^^^^^ To build and install rocSHMEM with the IPC on-node, GPU-to-GPU backend, run: .. code-block:: bash git clone --no-checkout --filter=blob:none git@github.com:ROCm/rocm-systems.git cd rocm-systems git sparse-checkout set --cone projects/rocshmem git checkout develop cd projects/rocshmem mkdir build cd build ../scripts/build_configs/ipc_single The build script passes configuration options to CMake to setup a single-node build. This is similar to the default build in ROCm 6.4. .. note:: The default configuration changed from IPC only in ROCm 6.4 (built with the ``ipc_single`` script) to RO and IPC in ROCm 7.0 (built with the ``ro_ipc`` script), and then to GDA and RO and IPC in ROCm 7.1 (built with the ``all_backends`` script). Other experimental configuration scripts are available in ``./scripts/build_configs``, but only ``all_backends``, ``ipc_single`` and ``ro_ipc`` are officially supported. Targeting GPU architectures ^^^^^^^^^^^^^^^^^^^^^^^^^^^ By default, rocSHMEM builds device bitcode for a predefined set of supported GPU architectures (``gfx90a``, ``gfx942``, ``gfx950``, ``gfx1100``, ``gfx1201``, and ``gfx1250`` on ROCm 7 or later). To restrict the build to a specific subset - for example to reduce build time or when targeting a known deployment environment - pass ``GPU_TARGETS`` to CMake: .. code-block:: bash ../scripts/build_configs/all_backends -DGPU_TARGETS="gfx942;gfx950" Some GPU features, such as SRAM ECC and XNACK (hardware page-fault support), are encoded as feature suffixes in the architecture string (for example ``gfx950:sramecc+:xnack-``). These suffixes are embedded in the device bitcode metadata. If the suffixes in the bitcode do not match the active feature configuration of the GPU at runtime, the bitcode may fail to load. To embed the correct feature metadata, include the suffixes in ``GPU_TARGETS``: .. code-block:: bash ../scripts/build_configs/all_backends \ -DGPU_TARGETS="gfx942:sramecc+:xnack-;gfx950:sramecc+:xnack-" The feature strings for your GPU can be obtained by running ``rocminfo`` on the target machine and looking for the ``Name:`` field under each agent, for example ``amdgcn-amd-amdhsa--gfx950:sramecc+:xnack-``. .. note:: The ``xnack`` feature is runtime-configurable and can differ between machines. If a cluster is configured with ``xnack+`` (unified memory), specifying ``xnack-`` in ``GPU_TARGETS`` may cause device bitcode load failures at runtime. Match the suffix to the actual configuration of the target system. .. note:: Requesting more than one feature variant of the same base architecture (for example ``-DGPU_TARGETS="gfx90a:xnack+;gfx90a:xnack-"``) builds a fat binary containing both variants into the main library. However, the installed device bitcode (``librocshmem_device_gfx90a.bc``) is a single file and can only embed one variant's feature metadata. The chosen variant is reported at configure time, for example: ``Multiple feature variants requested for gfx90a (...); device bitcode will embed gfx90a:xnack+``. Installation prefix ^^^^^^^^^^^^^^^^^^^ By default, the build scripts install the library to ``~/rocshmem``. You can customize the installation path by adding the desired path through the ``INSTALL_PREFIX`` environment variable. For example, to relocate the default configuration: .. code-block:: bash INSTALL_PREFIX=/path/to/install ../scripts/build_configs/all_backends