.. meta:: :description: rocSHMEM environment variables reference :keywords: rocSHMEM, ROCm, API, environment variables, environment, reference .. _rocshmem-api-env-variables: ******************************************************************** rocSHMEM environment variables ******************************************************************** This section describes the important environment variables used to control the behavior of rocSHMEM. .. list-table:: :header-rows: 1 :widths: 35,14,51 * - **Environment variable** - **Default value** - **Value** * - | ``ROCSHMEM_DEBUG_LEVEL`` | Debug output level - ``ERROR`` - | Levels (from least to most verbose): | ``NONE``: Suppress all output. | ``ERROR``: Print error messages only (default). | ``WARN``: Print warnings and errors. | ``ENV``: Print modified environment variables at startup. | ``VERSION``: Print build/version information at startup. | ``INFO``: Print informational messages and above. | ``API``: Print API call tracing (requires ``BUILD_DEBUG_TRACE_HOST``/``BUILD_DEBUG_TRACE_DEVICE``). | ``TRACE``: Print all messages including internal traces (requires ``BUILD_DEBUG_TRACE_HOST``/``BUILD_DEBUG_TRACE_DEVICE``). | | Modifiers can be appended with ``:`` to suppress specific categories: | ``:noerror``, ``:nowarn``, ``:noenv``, ``:noversion``, ``:noinfo``, ``:noapi``, ``:notrace`` | ``:full`` or ``:all`` after ``env`` or ``:env`` modifier controls env print detail. | ``:color`` (default) or ``:nocolor`` enables/disables ANSI color output. | | Examples: ``trace:noversion``, ``env:full``, ``api:noenv``, ``trace:nocolor`` * - | ``ROCSHMEM_HEAP_ALLOCATOR_TYPE`` | Selects the GPU memory allocator used for the symmetric heap. - arch-dependent - | ``uncached``: GPU memory in uncached mode (default on most architectures). | ``finegrained``: Fine-grained coherent GPU memory (default on gfx1100, gfx1201). | ``coarsegrained``: Coarse-grained coherent GPU memory. | ``vmm_posix``: VMM with POSIX fd-based IPC (requires ROCm 7.2+, Linux 5.6+, TCP bootstrap). | ``vmm_fabric``: VMM with fabric handle IPC (requires ROCm 7.14+, gfx1250/MI455 only). | Values are case-insensitive. * - | ``ROCSHMEM_HEAP_SIZE`` | Defines the size of the rocSHMEM symmetric heap in bytes (per PE). - ``1073741824`` (1 GB) - | Size in bytes (per PE). | Note: the heap is on GPU memory. * - | ``ROCSHMEM_MAX_NUM_HOST_CONTEXTS`` | Maximum number of host-side communication contexts - ``1`` - Maximum number of host-side contexts. * - | ``ROCSHMEM_MAX_NUM_CONTEXTS`` | Defines the number of contexts an application can use. - ``32`` - Maximum number of contexts. * - | ``ROCSHMEM_MAX_NUM_TEAMS`` | Defines the number of teams an application can use. - ``40`` - Maximum number of teams. * - | ``ROCSHMEM_BACKEND`` | When rocSHMEM is compiled for all backends, this environment variable | selects which backend to execute. The default value is an empty string and rocSHMEM auto-selects the most appropriate backend. - `` `` - | ``ipc``: IPC Backend | ``ro``: Reverse Offload Backend | ``gda``: GPU Direct Async Backend * - | ``ROCSHMEM_UNIQUEID_WITH_MPI`` | Defines whether rocSHMEM is expected to use MPI internally when using the uniqueId based initialization. - ``0`` - | ``0``: Do not use MPI. | ``1``: Use MPI. * - | ``ROCSHMEM_DISABLE_MIXED_IPC`` | Defines whether to force using the network conduit even when IPC is available. - ``0`` - | ``0``: Use IPC when available. | ``1``: Force network conduit. * - | ``ROCSHMEM_USE_IB_HCA`` | Forces the NIC that this PE uses. When this value is set NIC auto-detection and mapping is disabled, the NIC specified in the variable | will be selected. The default value is an empty string and rocSHMEM auto-detects the most appropriate NIC. - `` `` - | Example value: ``bnxt_re0`` * - | ``ROCSHMEM_HCA_LIST`` | Comma separated list of NIC names that can be used by rocSHMEM. Unlike ``ROCSHMEM_USE_IB_HCA``, when this variable is set, | NIC auto-detection and mapping still executes, but NICs that are not in the list are discarded before auto-detection runs. | Prefixing the list with ``^`` turns the list in an *exclude* list, NICs that are in the list are discarded before auto-detection runs. | The default value is an empty string and rocSHMEM auto-detects the most appropriate NIC. - `` `` - | Example value: ``bnxt_re1,bnxt_re11``, ``^mlx5_0,mlx5_3`` * - | ``ROCSHMEM_BOOTSTRAP_SOCKET_IFNAME`` | Chooses the interface to bootstrap rocSHMEM with. | Only valid when not using MPI. | The default value is an empty string and rocSHMEM auto-detects the most appropriate interface. - `` `` - | Example value: ``eno8303`` * - | ``ROCSHMEM_GDA_PROVIDER`` | When rocSHMEM is compiled with support for multiple NIC vendors, | the environment variable selects the desired provider. | The default value is an empty string and rocSHMEM auto-detects the most appropriate NIC. - `` `` - | ``bnxt``: Broadcom Thor 2 | ``pensando``: AMD Pensando Pollara | ``ionic``: AMD Pensando Pollara (alias) | ``mlx5``: Mellanox ConnectX-7 * - | ``ROCSHMEM_GDA_ALTERNATE_QP_PORTS`` | Enables or disables alternating QP mappings across rocSHMEM contexts. - ``1`` - | ``0``: Disabled. | ``1``: Enabled. This helps saturate bandwidth on multiport bonded interfaces. * - | ``ROCSHMEM_GDA_TRAFFIC_CLASS`` | When using an NIC with an Ethernet link layer, this sets the traffic class for the QPs. - ``0`` - The traffic class number. * - | ``ROCSHMEM_GDA_PCIE_RELAXED_ORDERING`` | Enables PCIe Relaxed Ordering when registering the symmetric heap with the RDMA NICs. - ``0`` - | ``0``: Disabled. | ``1``: Enabled. * - | ``ROCSHMEM_GDA_ENABLE_DMABUF`` | Enable dmabuf support for memory registration. - ``0`` - | ``0``: Disabled. | ``1``: Enabled. * - | ``ROCSHMEM_GDA_ALLTOALLV_WG_ALGO`` | Selects between two algorithms to use for GDA based alltoallv. | The GET algorithm uses an initial round of alltoallv | communication to distribute displacements then a second round to | get transfer data. This algorithm has a higher latency but | has better performance for large messages. | The COPY algorithm does an alltoallv communication | pattern into a staging buffer then does a copy into the destination | buffers. This reduces latency but requires more memory, this | algorithm only works for small messages. - ``GET`` - | ``GET``: GET-based alltoallv algorithm | ``COPY``: Copy alltoallv algorithm * - | ``ROCSHMEM_GDA_OVERRIDE_NIC_FIRMWARE_CHECK`` | This environment variable should be used with caution. | It overrides the NIC firmware check if | a user wants to use an unsupported NIC firmware. | If the firmware check is disabled rocSHMEM is not guaranteed to work. - ``0`` - | ``0``: Disabled. | ``1``: Enabled. * - | ``ROCSHMEM_GDA_SQ_SIZE`` | This environment variable sets the length of the SQ for GDA. - ``1024`` - | Maximum number of Work Queue Entries (WQEs) posted on the Send Queue (SQ) * - | ``ROCSHMEM_GDA_NUM_QPS_PER_PE_DEFAULT_CTX`` | Sets the number of Queue Pairs (QPs) to create per PE for the default context. - ``1`` - Number of QPs per PE for the default context. * - | ``ROCSHMEM_GDA_NUM_QPS_PER_PE_USR_CTX`` | Sets the number of Queue Pairs (QPs) to create per PE for each user context. - ``1`` - Number of QPs per PE for each user context. * - | ``ROCSHMEM_GDA_NUM_USER_BUFFERS`` | GDA supports ``rocshmem_buffer_register`` and ``rocshmem_buffer_unregister`` | for user buffers. This variable sets the number of user buffers an | application may register when using the GDA backend. | If the application uses more user buffers than what is defined with | this variable, then the behavior is undefined. - ``4`` - Maximum number of user buffer registrations for GDA * - | ``ROCSHMEM_MAX_SYMM_REGIONS`` | rocSHMEM supports ``rocshmem_buffer_register_symmetric`` and | ``rocshmem_buffer_unregister_symmetric`` for symmetric user buffers. | This variable sets the maximum number of symmetric user buffers an | application may register concurrently. It is backend-agnostic and is | currently honored by the IPC backend. - ``4`` - Maximum number of symmetric user buffer registrations * - | ``ROCSHMEM_MAX_WF_BUFFERS`` | Maximum number of wavefront buffer arrays in default context (determines size of status, return, and atomic return buffers) - ``1024`` - * - | ``ROCSHMEM_BOOTSTRAP_TIMEOUT`` | Bootstrap initialization timeout in seconds - ``5`` - * - | ``ROCSHMEM_BOOTSTRAP_HOSTID`` | Override host identifier for bootstrap. Empty string uses hostname. - `` `` - * - | ``ROCSHMEM_BOOTSTRAP_SOCKET_FAMILY`` | Socket family for bootstrap (AF_UNSPEC, AF_INET, AF_INET6) - ``types::socket_family::UNSPEC`` - * - | ``ROCSHMEM_SDMA_ENABLED`` | Enable or disable the SDMA transport at runtime (requires ``USE_SDMA`` build option). - ``1`` - | ``0``: Disabled. All transfers use GPU load/store (IPC path). | ``1``: Enabled. Transfers at or above ``ROCSHMEM_SDMA_THRESHOLD`` use the SDMA engine. * - | ``ROCSHMEM_SDMA_THRESHOLD`` | Minimum transfer size in bytes to route through the SDMA engine. | Transfers smaller than this threshold use GPU load/store instead. - ``256`` - Size in bytes. * - | ``ROCSHMEM_SDMA_NUM_CHANNELS`` | Number of SDMA channels (ring buffers) allocated per destination PE. | More channels reduce CAS contention when many wavefronts submit concurrently, | at the cost of additional queue memory and SDMA engine resources. - ``1`` - Number of channels per destination PE. * - | ``ROCSHMEM_SDMA_SPREAD_CHANNELS`` | When enabled, each wavefront within a workgroup selects its SDMA channel | using an offset based on its wavefront index: | ``effective_channel = (ctx_channel + wf_id) % num_channels``. | This reduces CAS contention when multiple wavefronts in the same workgroup | target the same destination PE on a shared context. | By default, spreading is automatically enabled only for the default context | (``ctx_id=0``), which is shared by all workgroups when contexts are not | created per-workgroup. Per-workgroup contexts (created via | ``rocshmem_wg_ctx_create``) already distribute workgroups across channels | via their context ID; enabling spreading for them reshuffles contention | without reducing it. - ``0`` - | ``0``: Apply wf_id spreading only for the default (shared) context. | ``1``: Apply wf_id spreading for all contexts.