RCCL environment variables

This section describes the most important RCCL environment variables, which are grouped by functionality.

Configuration and setup

The configuration and setup environment variables for RCCL are collected in the following table.

Environment variable	Values
NCCL_CONF_FILE Specifies the path to the RCCL configuration file.	String path to configuration file. RCCL loads configuration from three sources (earlier files take precedence — values already set are not overwritten): 1. ${NCCL_CONF_FILE} when set, otherwise ~/.rccl.conf 2. /etc/rccl.conf — system-wide admin config 3. /var/run/rccl.conf — runtime config generated by platform or node setup tools (e.g. rccl_bcm_isl_discover.sh)
NCCL_IBVERBS_LIB Specifies the libibverbs shared object that RCCL loads at runtime for the InfiniBand/RoCE (IB verbs) transport. Use it when rdma-core is installed in a non-default prefix, such as inside a container or an HPC software module, where the loader cannot find the library by its default name. When the override is unset or fails to load, RCCL falls back to libibverbs.so and then libibverbs.so.1. NCCL_LIBIBVERBS_SO is accepted as an alias and is used when NCCL_IBVERBS_LIB is unset.	String path or soname of the libibverbs shared object Default: unset (loads libibverbs.so or libibverbs.so.1)
NCCL_HOSTID Sets the host identifier for multi-node communication.	String value for host identification Used for host hash generation
NCCL_BOOTSTRAP_BIDIR_ALLGATHER Enables the bidirectional ring AllGather (N/2 steps) on the socket OOB path during bootstrap. The unidirectional ring (N-1 steps) is kept as a fallback. Has no effect when net OOB is in use.	0: Force unidirectional ring. 1: Force bidirectional ring (default).
NCCL_CUMEM_ENABLE Enables cuMem virtual memory management (VMM) for RCCL allocations, which is required for ncclCommSuspend and ncclCommResume to release the physical GPU memory of a suspended communicator. See Suspending and resuming a communicator for the full prerequisites.	0: Disabled. 1: Enabled on any architecture. -2: Auto-detect (default); enable when the platform supports VMM. Auto-detect is limited to gfx1250, the only architecture where the VMM path is validated. Use 1 to force it on elsewhere.
NCCL_RMA_DISABLE Disables the RMA proxy, the network path for one-sided RMA. The proxy is never connected and windows are not registered with it.	0: RMA proxy enabled (default). 1: RMA proxy disabled.
NCCL_MIN_CTAS Minimum number of CTAs (channels) used for a collective. Overrides the minCTAs field of ncclConfig_t.	Positive integer (values <= 0 are ignored). Default: unset (uses the RCCL default).
NCCL_MAX_CTAS Maximum number of CTAs (channels) used for a collective. Overrides the maxCTAs field of ncclConfig_t.	Positive integer (values <= 0 are ignored). Default: unset (uses the RCCL default).
NCCL_ENV_PLUGIN Loads an external environment plugin that intercepts all RCCL parameter lookups. See Using the RCCL environment plugin API for full details.	Path to a plugin .so file, a bare name expanded to librccl-env-<name>.so, or none to disable. Default: unset (tries librccl-env.so, then reads from the process environment).
NCCL_ENV_JSON_FILE Path to a JSON configuration file used by librccl-env-json.so. Has no effect unless NCCL_ENV_PLUGIN points to that plugin.	Path to a flat JSON file mapping variable names to string values. Relative paths are resolved from the application’s working directory. Default: unset (falls back to getenv() for all lookups).
NCCL_ALLGATHERV_ENABLE Fuses grouped multi-root ncclBroadcast calls into a single AllGatherV ring kernel when two or more distinct roots appear in a group.	0: Disabled (default). 1: Enabled.
RCCL_HIERARCHICAL_LAZY_INIT Controls when a communicator of eight or more nodes builds the sub-communicators that hierarchical AllGather uses. Ignored when RCCL_HIERARCHICAL_REDUCE_SCATTER=1. All ranks in a communicator must use the same value.	0: Build them during communicator initialization (default). 1: Build them on the first AllGather eligible for hierarchical AllGather, outside graph capture.
NCCL_NVLS_ENABLE Controls NVLink SHARP (NVLS), an NVIDIA NVSwitch multicast feature. RCCL accepts this variable for compatibility with NCCL but does not implement NVLS, so it has no effect on AMD GPUs. Because RCCL never creates NVLS multicast buffers, the multicast bind failure that NCCL 2.29 and later reports as a WARN and an initialization error cannot occur in RCCL.	0, 1, 2: All values behave the same. Communicator initialization does not use NVLS and does not fail because of it. Default: 2

Logging and debugging

The logging and debugging environment variables for RCCL are collected in the following table.

Environment variable	Values
NCCL_DEBUG Controls debug logging in RCCL for troubleshooting and monitoring collective communication operations.	These are the logging levels in RCCL set via NCCL_DEBUG. Each logging level contains all logging for levels below it. The default logging level is ERROR. NONE: No logging is printed. ERROR: These messages report when a fatal condition has occurred in RCCL and the operation can’t continue. VERSION: librccl version info is printed during the initialization phase. WARN: Prints warnings about unusual conditions that could lead to unexpected results. ATTN: Prints WARN messages plus notices that need attention, such as a plugin named in NCCL_*_PLUGIN that could not be loaded or initialized. INFO also prints these notices. INFO: Prints standard logging messages about status and operations performed. ABORT: Unused. TRACE: Prints trace-level logging of function calls and parameters. Only active when librccl is built using ENABLE_TRACE.
NCCL_DEBUG_SUBSYS Controls which subsystems generate debug output.	These are the logging subsystems set via NCCL_DEBUG_SUBSYS. These can be set as a comma-separated list, and can be inverted using the ^ prefix. The default subsystem set is INIT, BOOTSTRAP, and ENV. INIT: Prints during the initialization phase. COLL: Prints during execution of collectives. P2P: Prints logs related to peer-to-peer setup or communication. SHM: Prints logs related to shared memory. NET: Prints logs related to network setup or communication. GRAPH: Prints logs related to parsing the topology of the network. TUNING: Prints logs related to the tuner plugin. ENV: Prints logs related to environment variables. ALLOC: Prints logs related to memory allocation. CALL: Prints logs for function calls (TRACE only). PROXY: Prints logs related to the proxy thread. NVLS: Not valid for AMD/RCCL. BOOTSTRAP: Prints logs related to the bootstrapping phase of initialization. REG: Prints logs related to registration and deregistration of transport initialization. PROFILE: Prints logs related to the profiling/timing info. RAS: Prints logs related to RAS. VERBS: Prints logs related to IB/Verbs. DESTROY: Prints logs related to communicator/plugin teardown (destroy, abort, revoke, plugin unload). ALL: Activates all logging subsystems.
NCCL_WARN_ENABLE_DEBUG_INFO Converts all WARN level logs to INFO level logs.	0: Default value. Variable is not enabled. 1: Enable the variable.
NCCL_DEBUG_TIMESTAMP_LEVELS The timestamp levels for NCCL_DEBUG.	A set of NCCL_DEBUG levels can have a timestamp prepended set as a comma-separated list which can be inverted using the ^ prefix. The default set is WARN and ATTN.
NCCL_DEBUG_TIMESTAMP_FORMAT The timestamp format for NCCL_DEBUG.	Set the format of the timestamp in printf style. The default format is "[%F %T] ".
NCCL_DEBUG_FILE Write logs to a file rather than stdout.	The filename can be formatted using %h for hostname, %p for pid, and %% to escape the % character. It is recommended to use %p to output to individual files per pid to avoid mixing or potentially overwriting the output. Example usage: NCCL_DEBUG_FILE=debugfile.%h.%p
NCCL_CHECK_MODE Selects how thoroughly RCCL validates the arguments of every collective call. Checking costs latency, so it is disabled by default and intended for development and bring-up. See Validating collective arguments for what each mode detects.	DEFAULT: No argument validation (default). DEBUG_LOCAL: Validate the buffer pointers locally on each rank. Replaces the deprecated NCCL_CHECK_POINTERS. DEBUG_GLOBAL: Also validate arguments across ranks, including symmetric buffer registration. Values other than DEBUG_LOCAL and DEBUG_GLOBAL leave the mode unchanged, so writing DEFAULT does not switch checking off again.
NCCL_CHECK_POINTERS Deprecated. Enables local validation of the buffer pointers passed to each collective.	0: Disabled (default). 1: Enabled, equivalent to NCCL_CHECK_MODE=DEBUG_LOCAL. Use NCCL_CHECK_MODE instead. When both are set, DEBUG_LOCAL or DEBUG_GLOBAL wins; any other NCCL_CHECK_MODE value keeps the mode selected by NCCL_CHECK_POINTERS=1.
RCCL_FABRIC_TELEMETRY_ENABLE Periodically logs UALoE fabric counters for the GPUs this process uses. Also requires RCCL_USE_AMD_SMI_LIB=1, since the counters are only reachable through the amd-smi library, and NCCL_DEBUG=INFO, since the reports are logged at INFO level under the INIT subsystem. See Fabric telemetry for what they contain.	0: Disabled (default). 1: Enabled.
RCCL_FABRIC_TELEMETRY_INTERVAL_MS Sampling period for RCCL_FABRIC_TELEMETRY_ENABLE, in milliseconds. The firmware republishes its datasets roughly once a second, so periods much shorter than that cost work without yielding more information.	Default 1000. Values below 100 are raised to 100. Values above 3600000 (one hour) are lowered to 3600000. 0 or less disables telemetry.

Validating collective arguments

NCCL_CHECK_MODE=DEBUG_LOCAL inspects only what a rank can see by itself: it verifies that the sendbuff and recvbuff arguments are valid device pointers that belong to the device the communicator was created on. Passing a host pointer or a pointer from another device makes the collective return ncclInvalidArgument instead of faulting inside the kernel.

NCCL_CHECK_MODE=DEBUG_GLOBAL adds cross-rank validation of symmetric buffer registration. The symmetric kernels require every rank to describe its buffers identically, because a rank addresses a peer’s buffer by applying its own offsets to the peer’s symmetric window. RCCL cannot verify that from a single rank, so at group launch the ranks exchange the identity of the windows backing their buffers and compare against rank 0. A collective is rejected with ncclInvalidArgument when:

• Some ranks pass buffers registered with NCCL_WIN_COLL_SYMMETRIC while others pass unregistered buffers.

• The ranks pass buffers from windows registered at different positions in the symmetric address space.

• The ranks pass buffers at different offsets inside their windows.

Each rejection is reported by rank 0 with a WARN message naming the collective, the message size, and the first rank that disagrees, so set NCCL_DEBUG=WARN when using this mode. Setting NCCL_DEBUG=INFO with NCCL_DEBUG_SUBSYS=COLL additionally prints a SymCheck line per rank with the window and user offsets that were compared.

Without this mode such a mismatch is not diagnosed: RCCL silently falls back to the general kernels for calls it cannot serve symmetrically, so the collective still produces correct results but loses the performance of the symmetric path. Enable DEBUG_GLOBAL when a workload registers symmetric windows yet does not reach the expected symmetric performance.

Note

DEBUG_GLOBAL adds a bootstrap all-gather to every group launch, which is far more expensive than the collective itself for small messages. Use it to diagnose a configuration, not in production.

Fabric telemetry

RCCL_FABRIC_TELEMETRY_ENABLE=1 starts a sampler that reads every amd-smi fabric telemetry category for the GPUs this process opens communicators on and logs the counters that moved. It is diagnostic only and off by default.

Three settings are needed for any output to appear. The first two are easy to omit, and leaving either out makes the feature silent rather than noisy:

RCCL_USE_AMD_SMI_LIB=1 RCCL_FABRIC_TELEMETRY_ENABLE=1 \
  NCCL_DEBUG=INFO NCCL_DEBUG_SUBSYS=INIT \
  mpirun -np 4 ./all_reduce_perf -b 8 -e 1G -f 2 -g 1

RCCL_USE_AMD_SMI_LIB=1 is required because the counters are only reachable through the amd-smi library; the sysfs fabric discovery path RCCL uses otherwise exposes no telemetry. Without it, RCCL warns that telemetry is disabled and initialization continues.

Note

VERSION is below WARN in the NCCL_DEBUG ordering, so running with NCCL_DEBUG=VERSION suppresses both the reports and the warning explaining why there are none. Enabling telemetry then looks exactly like not enabling it. Use NCCL_DEBUG=INFO to see the reports, or at least WARN to see why they are missing.

Each report names the GPU, the communicator that opened telemetry on it, that communicator’s rank, and the category, followed by the firmware generation count, how many counters moved out of how many exist, and the largest movers:

fabric telemetry GPU 1 comm 0xa01ec474f15d9f01 rank 1 NetPort: gen=28091
  changed=888/4104 top: NETPORT_RX_TOTAL_BYTES[netport6] +3887018566, ...

The rank is included because the amd-smi device index alone does not say which rank owns the GPU. The counters are per device, so when several communicators share a GPU the reports cover all of them but keep the identity of whichever opened it, and a line is logged when a later communicator joins.

Values are cumulative since firmware boot, so each report is a delta. A category whose generation count has not advanced is skipped rather than reported as unchanged traffic. When a device’s last communicator is destroyed it is reported once more, measured over the whole time it was sampled, so a run too short for the firmware to publish more than once still produces output.

Telemetry also needs access to the IFoE config character device, which ships root-only. It is a request/response channel rather than a readable file, so amd-smi opens it O_RDWR and read-only permission is not sufficient. Granting the render group read and write is enough:

# /etc/udev/rules.d/70-amdgpu-ualoe-telemetry.rules
SUBSYSTEM=="misc", KERNEL=="cbl-cfg-*", GROUP="render", MODE="0660"

Note

On a ROCm host render is every GPU user and container, and a file in /etc/udev/rules.d persists across reboots, so the rule outlives the run that motivated it. Install it to diagnose a fabric and remove it afterwards, rather than carrying it in a deployment.

Without that access amdsmi_get_fabric_telemetry_data returns NOT_SUPPORTED. RCCL logs which device it skipped and why, and initialization continues normally. That line is logged at INFO, so unlike the missing-RCCL_USE_AMD_SMI_LIB case above, WARN alone will not explain this one.

Algorithm and protocol control

The algorithm and protocol control environment variables for RCCL are collected in the following table.

Environment variable	Values
NCCL_ALGO Forces specific algorithm selection for collectives.	Algorithm name string Used to override automatic algorithm selection
NCCL_PROTO Forces specific protocol selection for communication.	Protocol name string Used to override automatic protocol selection
RCCL_DIRECT_ALLGATHER_DISABLE Controls the direct AllGather algorithm. Because the algorithm builds a full point-to-point mesh, its queue-pair footprint grows with the square of the job size.	-1: Automatic (default). Not selected on AINIC above 8 nodes. 0: Skips the automatic AINIC check. The size, architecture and CTA-policy gates in rcclUseAllGatherDirect still apply. Any other value: Disabled.
RCCL_HIERARCHICAL_ALLGATHER_MIN_BYTES_PER_RANK Sets the smallest AllGather, in bytes per rank (sendcount x type size), that can select hierarchical AllGather, so that small startup AllGathers such as PyTorch DDP’s 8-byte one stay on the default path. rccl-tests reports the total size, this value times the number of ranks. All ranks in a communicator must use the same value.	Bytes per rank Default: 16 0: No lower bound.
RCCL_HIERARCHICAL_REDUCE_SCATTER_MIN_BYTES_PER_RANK Sets the smallest ReduceScatter, in bytes per rank (recvcount x type size), that can select hierarchical ReduceScatter.	Bytes per rank Default: 16 0: No lower bound.
NCCL_TREE_COMPACT Builds the inter-node trees with a half-interleave node remapping. Intended for jobs whose nodes are ordered by network domain. Both trees keep cross-domain hops at the leaves only when the nodes split evenly into two domains; with an uneven split, the first tree does so when the first domain holds at least half of the nodes, and the second tree still crosses domains. All ranks in a communicator must use the same value; initialization fails otherwise.	0: Standard double binary tree (default) 1: Compact tree
RCCL_TUNER_EMBEDDED_CONFIG Controls whether the built-in CSV tuner uses the tuning configurations compiled into librccl.so from the source tree’s tuner/*.csv. These are the stock defaults, so no file has to be staged on each node. NCCL_TUNER_CONFIG_FILE and a CSV in <librccl.so dir>/tuner/ or ${ROCM_PATH}/share/rccl/tuner/ take precedence over them. When the GPU architecture is known, only an architecture-specific (rccl_tuner_<arch>.csv) or generic (rccl_tuner.csv) configuration is used; another architecture’s configuration is never applied. When the architecture cannot be determined, any embedded configuration may be used. To disable the tuner entirely, embedded or not, use NCCL_TUNER_PLUGIN=none.	1: Use the embedded configurations when no file is found (default). 0: Ignore the embedded configurations. File-based tuning is unaffected.

Network and topology

The network and topology environment variables for RCCL are collected in the following table.

Environment variable	Values
NCCL_IB_HCA Specifies InfiniBand device:port to use.	Device specification string Prefix with ^ for exclusion, = for exact match
NCCL_IB_GID_INDEX Defines the Global ID index used in RoCE mode.	Integer value (default: -1) See InfiniBand show_gids command for valid values
NCCL_IB_QUERY_PORT_SPEED Controls whether RCCL queries the port speed with ibv_query_port_speed (IBVERBS_1.16) when it initializes devices. Applies to both the IB and IB-CAST transports. If the installed libibverbs does not export this function, or the query fails, RCCL falls back to the port’s active_speed_ex (for XDR or newer rates) or active_speed, combined with active_width. Disabling it also turns off runtime speed-change detection, which only the IB transport supports.	1: Query the port speed (default). 0: Use active_speed_ex/active_speed and active_width only.
NCCL_IB_SUBNET_AWARE_ROUTING Enables subnet-aware device selection. When a peer’s GID subnet does not match the default device’s RoCE ports, RCCL searches other locally merged devices for one whose ports do match, so the connection uses a device on the same subnet as the peer. Only meaningfully exercised on a multi-subnet RoCE fabric (for example, behind an IB router); on a single-subnet cluster the default device already matches and this is a no-op.	0: Disabled (default). 1: Enabled.
NCCL_IB_SUBNET_PREFIX_LEN Prefix length, in bits, used when comparing two GIDs’ subnets for NCCL_IB_SUBNET_AWARE_ROUTING.	Integer value, bits (default: 24)
NCCL_IB_WQE_LATENCY_THRESHOLD_NS Enables per-queue-pair (QP) CPU-side WQE post-to-poll latency monitoring in the IB network transport (net_ib). When set above 0, each send-side QP measures the time from posting a signaled work request (ibv_post_send) to the CPU polling its completion, which includes any delay before RCCL polls the completion queue (CQ). It is not the hardware completion time. Each QP samples one signaled WQE at a time. The next one is sampled only after the previous sample completes, so under pipelined traffic the statistics cover a subset of WQEs. Samples longer than the threshold are counted as slow and logged as NET/IB: WQE slow lines. A sampled WQE that has been outstanding longer than the threshold without a completion is logged as NET/IB: WQE stall (no CQE). Both messages are logged at most once per second per QP, at INFO level with the NET subsystem (NCCL_DEBUG=INFO, NCCL_DEBUG_SUBSYS=NET). Each line identifies the local and remote QP number, device, port and LID. The GIDs are only filled in on RoCE (Ethernet link layer) devices and print empty on InfiniBand. Read once per process. Not applied by the IB-CAST transport.	Integer value, nanoseconds (default: 0, disabled)
NCCL_IB_WQE_LATENCY_REPORT Controls the per-QP latency summary logged when a send connection is closed while NCCL_IB_WQE_LATENCY_THRESHOLD_NS is enabled. The summary (NET/IB: WQE latency summary) reports the sample count, slow count, threshold and the post-to-poll mean, standard deviation, P50/P90/P99/P99.9 estimates and maximum. QPs with no samples are not reported.	1: Log a summary per QP on close (default). 0: Disabled. Slow and stall messages are still logged.
NCCL_PXN_C2C Allows PXN routing through a C2C link to reach a NIC attached to a peer GPU. The C2C path is NVIDIA-specific and is not currently applicable on AMD hardware.	0: Disabled (default). 1: Enabled.
NCCL_GIN_MLOPART Allows GPU-initiated networking (GIN) on a communicator that includes a rank on a GPU partition. Without this variable, one partitioned rank turns GIN off for the whole communicator. The default allows partitions on a single-node communicator only; set 1 to allow them across nodes. See Use the RCCL device API and GIN and NPS4 and CPX partition modes.	Unset: Allow GIN on partitions on a single-node communicator; disable GIN on a partitioned communicator spanning several nodes (default). The latter is reported through NCCL_DEBUG=INFO. 1: Allow GIN on partitions on any number of nodes. 0: Disable GIN whenever any rank runs on a partition. A communicator with no partitioned rank is unaffected by any value.
NCCL_SOCKET_IFNAME Specifies which IP interfaces to use for communication. When unset, RCCL auto-selects an interface in this order: ib* first; if none is found and NCCL_COMM_ID is set, an interface on the same subnet as that address; then any interface other than docker*, lo and virbr*; then docker*; then lo; and finally virbr*. Libvirt bridge interfaces (virbr*) are considered last because they serve host-to-VM (virtual machine) traffic and cannot reach a remote node.	Interface prefix string or list Multiple prefixes separated by , Prefix with ^ for exclusion, = for exact match Example: eth (all eth interfaces), =eth0 (exact match)
NCCL_SOCKET_FAMILY Forces IPv4/IPv6 interface selection.	AF_INET: Force IPv4 AF_INET6: Force IPv6 Unset: Use first available
NCCL_IGNORE_NET_MISMATCH Controls what happens when ranks report a different number of local network (NET) devices during communicator initialization. RCCL gathers each rank’s local NET device count and compares the minimum and maximum across the communicator. A mismatch usually means the job was launched with an inconsistent NIC selection (for example, an uneven NCCL_SOCKET_IFNAME/NCCL_IB_HCA per rank, or nodes with different NIC counts), which otherwise surfaces later as obscure transport failures. See Ranks with different NIC counts.	1: Detect and continue, logging the mismatch at INFO level (default). 0: Fail initialization with ncclSystemError and a warning on the mismatch.
NCCL_IGNORE_COLLNET_MISMATCH Same as NCCL_IGNORE_NET_MISMATCH but for the number of local CollNet devices reported by each rank.	0: Fail initialization with ncclSystemError and a warning on the mismatch (default). 1: Detect and continue, logging the mismatch at INFO level.
NCCL_IB_MERGE_NICS Enables RCCL to combine several physical IB NICs that are close to the same GPU into a single logical network device (NIC Fusion). This allows RCCL to aggregate the bandwidth of those NICs. Use NCCL_NET_MERGE_LEVEL and NCCL_NET_FORCE_MERGE to control which NICs are combined.	1: Enabled (default). 0: Disabled. On AINIC with the IB-CAST transport, merging is off unless this variable is explicitly set to 1.
NCCL_NET_MERGE_LEVEL Sets the maximum topological distance between two NICs that can be merged into a single logical device. NICs farther apart than this level are left separate.	LOC: Same device only, which disables merging. PORT: Two ports of the same NIC (default). PIX: Under the same PCIe switch. PXB: Multiple PCIe bridges, without crossing the PCIe host bridge. P2C, PXN: Accepted, with the same effect as PXB for NIC pairs. PHB: Under the same CPU socket. SYS: Anywhere in the node, including across NUMA nodes. The value is a string, so PATH_PORT is not valid. An unrecognized value falls back to LOC and disables merging.
NCCL_NET_FORCE_MERGE Merges the listed groups of NICs regardless of NCCL_NET_MERGE_LEVEL. NICs that are not listed are then merged automatically.	Semicolon-separated list of groups, each a comma-separated list of device names in NCCL_IB_HCA notation. Default: unset.
NCCL_NETDEVS_POLICY Controls how many of a GPU’s locally reachable NICs are used on the network path for send, recv, and all-to-all. The policy governs per-channel NIC selection (ncclTopoGetLocalNet); the per-peer network channel count is still bounded by available NIC bandwidth. Any unset, malformed, or out-of-range value falls back to AUTO.	AUTO (default): use ceil(localNetCount / localGpuCount) NICs, dividing the local NICs across the GPUs that share them. ALL: use every locally reachable NIC. MAX:N: use at most N NICs (clamped to the number reachable); N must be a positive integer.
RCCL_IB_SPLIT_DATA_THRESHOLD Minimum message size (in bytes) before the payload is split across multiple NICs/QPs. Smaller messages use one QP for data to reduce latency. This variable can be leveraged when NIC Fusion (NCCL_NET_MERGE_LEVEL) and/or data splitting on QPs (NCCL_IB_SPLIT_DATA_ON_QPS) is enabled.	Integer value in bytes (default: 128) N: Split only when message size >= N bytes
RCCL_IB_GDR_FLUSH_FLAG Replaces the GPUDirect RDMA flush of Simple-protocol receives in the IB network transport (net_ib). By default, the proxy fences each receive with an RDMA read on a loopback QP and waits for it before handing the data to the GPU. With this variable set, the sender writes a per-QP flag into GPU memory after the payload, on the same QP and without relaxed ordering, and the GPU waits for the flag instead. A communicator uses it only when every rank sets it and can use it: gfx942 or gfx950, the IB transport, and no adaptive routing (NCCL_IB_ADAPTIVE_ROUTING), out-of-order receive queues (NCCL_IB_OOO_RQ), ID-based receive matching (NCCL_IB_RECEIVER_SIDE_MATCHING_SCHEME=1) or port-failover resiliency (NCCL_IB_RESILIENCY_PORT_FAILOVER). Otherwise every rank keeps the RDMA read flush. A connection also keeps it when either side uses more than one NIC (NIC Fusion, including a fused vNIC chosen by NCCL_IB_SUBNET_AWARE_ROUTING), a request spans more than 8 QPs, or the flag memory cannot be registered. LL and LL128 receives never use the flag. On gfx950 with 2 to 8 nodes of 8 GPUs, a communicator using it also adjusts the ncclAllReduce algorithm and protocol choice from 16 MB per node upward, and switches ncclReduceScatter from LL128 to Simple above 6 MB per rank instead of 8 MB, unless NCCL_ALGO or NCCL_PROTO is set. Read once per process.	0: Disabled (default). 1: Enabled.
RCCL_IB_GDR_FLUSH_FLAG_TUNE Turns on additional collective tuning for communicators that use RCCL_IB_GDR_FLUSH_FLAG, on gfx950 with 8 GPUs per node on 2 to 8 or 16 nodes. It has no effect on other communicators. The LL128 buffer of ring and tree connections grows to 4 times its default size; send/recv connections keep the default. NCCL_LL128_BUFFSIZE disables the larger buffer. ncclAllReduce selects Ring/LL128 from 12 MB per node for as long as one ring loop can carry the data (about 98 MiB per node with the larger buffer), Tree/Simple up to 64 MB per node, and Ring/Simple on 48 channels from 64 to 128 MB per node. On 16 nodes it selects Tree/Simple from 8 to 64 MB per node. ncclReduceScatter keeps LL128 up to 7 MB per rank instead of 6 MB, and switches to Simple above that on 16 nodes too. Tree/LL128, and Ring/LL128 ncclBroadcast and ncclReduce below about nRanks / 2 chunks per channel, keep the default chunk size. The algorithm and protocol choices do not apply when NCCL_ALGO or NCCL_PROTO is set. Read once per process. All ranks must use the same value.	0: Disabled (default). 1: Enabled.
RCCL_IB_UDMA_POLICY Selects how queue pairs (QPs) are assigned to the two uDMA pipelines of an AINIC. Applies only to AINIC with the IB-CAST transport. With QP sharing enabled, the sharing groups choose the pipeline in place of the per-channel assignment of 0 and 1.	0: Pin each channel’s send QPs, and separately its receive QPs, to one pipeline per device, alternating pipelines as each is first used (default). 1: Round-robin P2P QPs across the pipelines in creation order. Collective QPs behave as with 0. 2: Leave the pipeline choice to the driver. 3: Split each connection across both pipelines. The default NCCL_IB_QPS_PER_CONNECTION of 2 is enough. Splitting is lost with NCCL_IB_QPS_PER_CONNECTION=1 or RCCL_IB_QPS_PER_P2P=1, and on RMA comms, which use one QP; every QP of a channel then uses the same pipeline. Other values log a warning and fall back to 0.
NCCL_NCHANNELS_PER_NET_PEER Sets the number of channels used per network (remote) peer. This overrides the value of the nChannelsPerNetPeer field in ncclConfig_t. When neither this variable nor the config field is set, RCCL auto-tunes the per-peer channel count based on the available NIC bandwidth and rank count.	Integer value, 1 to MAXCHANNELS (default: unset/auto-tuned) Values <= 0 are ignored and a warning is logged. Values > MAXCHANNELS set through ncclConfig_t are rejected with ncclInvalidArgument at communicator initialization.
NCCL_P2P_MAX_PEERS Sets the maximum number of peers a rank communicates with concurrently over P2P. This overrides the value of the maxP2pPeers field in ncclConfig_t. Where it applies, RCCL divides the P2P channel pool among this many peers instead of among all ranks, so a smaller value gives each peer more channels, affecting ncclSend/ncclRecv and the send/recv-based collectives (all-to-all, scatter, gather). It does not restrict which peers a rank is allowed to communicate with. It is read in two places only: the per-peer channel tiling enabled by RCCL_SATURATE_P2P_NCHANNELS (on by default for gfx1250 only), and the multi-node per-peer reduction, which requires more than one node and NCCL_NCHANNELS_PER_NET_PEER / nChannelsPerNetPeer unset. A single-node job on another architecture with default settings is therefore unaffected.	Integer value, 1 to the number of ranks (default: unset, which means the number of ranks in the communicator) Values <= 0 are ignored and a message is logged. Values greater than the communicator size are capped to it. Values <= 0 other than NCCL_CONFIG_UNDEF_INT set through ncclConfig_t are rejected with ncclInvalidArgument at communicator initialization.
RCCL_BCM_LINKS_PATH Fallback directory for Broadcom PEX inter-switch link topology. When the switch_discovery kernel module’s sysfs path (/sys/kernel/pci_switch_link/virtual_switch_links) does not exist, RCCL reads the same directory layout from this path instead. The userspace tool rccl_bcm_isl_discover.sh can populate it. Knowing which upstream ports share a physical PEX switch lets RCCL classify cross-partition PCIe paths as PXB and enable GDR across them.	String path to a directory with the same layout as the sysfs tree. Default: unset (falls back silently when sysfs is also absent).
NCCL_RINGS Defines custom ring topology.	Ring topology specification string Overrides automatic topology detection
RCCL_TREES Defines custom tree topology.	Tree topology specification string Alternative to ring topology
NCCL_RINGS_REMAP Controls ring remapping for specific topologies.	Remapping specification string Used with Rome 4P2H topology
RCCL_PCIE_ORDERED_RING This varilable applies only to gfx1250 scale-up single-host communicators. Pins ring graph search to the order GPUs appear in the topology array, so every channel uses the same ring. Without it the search picks a different permutation per channel, so a device exchanges with many different peers; pinning the order reduces fan-out to one peer per direction per device across all rings. On partitioned (DPX) GPUs the unpinned search also mixes scale-up and scale-in hops between channels, though ring throughput is bounded by its slowest hop either way. This constrains GPU visitation order only; path validation, bandwidth constraints, and backtracking are unchanged, and the ring still travels whatever paths the search validates. Applies to gfx1250 ring graphs on a single host where every rank is a GPU of that topology; trees, CollNet, other architectures, and multi-host communicators are unaffected. NCCL_GRAPH_FILE and NCCL_RINGS take precedence. Also accepted as NCCL_PCIE_ORDERED_RING.	0: Disabled, use the unconstrained permutation search. 1: Enabled (default). If no ordered ring passes validation, warn and fall back to the default ordering. 2: Enabled, strict. If no ordered ring passes validation, fail communicator initialization instead of falling back.
NCCL_GIN_PLUGIN Selects external GIN (GPU-Initiated Networking) plugins.	Comma-separated list of paths or short names A short name is resolved against the librccl-gin prefix, so example loads librccl-gin-example.so A plugin reporting the proxy device type is superseded by the built-in GIN proxy. See Using the RCCL GIN plugin API
NCCL_GIN_ENABLE Controls whether any GIN backend is registered. RCCL-specific.	1: Register GIN backends (default) 0: Register none, so GIN reports as unsupported
NCCL_GIN_TYPE Requires a specific GIN backend type, skipping all others.	-1: Disables the generic filter, but does not auto-enable the in-tree device backends 2: Proxy 3: GDAKI 4: GPI 5: EFA GDA 6: rocSHMEM GDA (required for that backend to initialize) 7: Anvil SDMA (also accepts unset)
NCCL_GIN_ALLREDUCE_GFX1250_ENABLE Opt in to GinScatter AllReduce on gfx1250.	0: gfx1250 keeps the pre-existing AllReduce selectors (default) 1: symmetric ncclSum fp32/fp16/bf16 messages from 64 MiB through 4 GiB use GinScatter when scratch can hold them Eligible sums below 64 MiB yield to DDA. Sums below 8 MiB already used DDA. From 8 MiB to 64 MiB, DDA LL/LL128 can take those sums
NCCL_GIN_ALLREDUCE_SCRATCH_BYTES GinScatter incoming scratch size, in bytes.	536870912 (512 MiB) by default, matching the CE AllReduce pool 0: do not allocate scratch, and do not select GinScatter
NCCL_RMA_PLUGIN Selects external one-sided RMA plugins, which are also the backend the built-in GIN proxy forwards to.	Comma-separated list of paths or short names A short name is resolved against the librccl-rma prefix, so example loads librccl-rma-example.so See Using the RCCL RMA plugin API

QP scheduling (CAST)

CAST (Congestion Aware Sprayed Traffic) adds a dynamic QP (Queue Pair) scheduler that balances RDMA traffic across multiple QPs per connection based on measured round-trip time (RTT). The following variables tune the scheduler and are accessible with either the RCCL_ or NCCL_ prefix; the RCCL_ form is shown below.

These variables only take effect when the IB-CAST net plugin is loaded (NCCL_NET=IB-CAST or ROCM-IB, or an AINIC NIC with NCCL_NET unset) and the CAST QP scheduler is enabled (see RCCL_IB_QP_SCHED_ENABLE below).

Environment variable	Values
RCCL_IB_QP_SCHED_ENABLE Enables the CAST QP scheduler.	-1: Auto (default). 0: Force off. 1: Force on. Auto resolves to enabled only when NCCL_NET=IB-CAST is set explicitly. Everywhere else — AINIC with NCCL_NET unset, or NCCL_NET=ROCM-IB — auto resolves to disabled; use RCCL_IB_QP_SCHED_ENABLE=1 to override. On AINIC, the scheduler only has an effect once ID-based receive matching (NCCL_IB_RECEIVER_SIDE_MATCHING_SCHEME=1) is enabled. Without it, forcing the scheduler on (1) reports “enabled”, but WRR and split-data are no-ops, so nothing is actually rebalanced. Always disabled when port failover/recovery (NCCL_IB_RESILIENCY_PORT_FAILOVER/_PORT_RECOVERY) or QP sharing (RCCL_IB_QP_SHARING_ENABLE) is active — load-balancer integration with those features is still pending.
RCCL_IB_QP_SCHED_RESET_INTERVAL Interval at which accumulated RTT statistics are reset, to prevent stale samples from permanently biasing the scheduler. Value is in milliseconds.	Integer milliseconds. Default: 60000 (60 seconds). 0: Never reset.
RCCL_IB_QP_SCHED_UPDATE_INTERVAL Minimum interval between scheduler weight updates. Value is in microseconds.	Integer microseconds. Default: 50. Values outside the range 1 µs to 60 s are ignored.
RCCL_IB_QP_SCHED_WEIGHT Exponential moving average (EMA) weight applied to new RTT samples.	Floating-point value in the range 0 to 1.0. Default: 0. 0: Simple average. Values closer to 1.0 react faster to recent samples.
RCCL_IB_QP_SCHED_SPLIT_DATA_MIN Minimum chunk size when splitting a message across QPs (split-data mode, enabled via NCCL_IB_SPLIT_DATA_ON_QPS). Value is in bytes.	Integer bytes. Default: 65536. Only positive values are applied.
RCCL_IB_QP_SCHED_LOG_PATH Directory for a per-process scheduler log file (one line per QP: device, min/current/max weight, and smoothed RTT). Log file is named cast_log_<hostname>_<pid>.	String directory path. Default: unset (logging disabled).
RCCL_IB_QP_SCHED_LOG_INTERVAL Interval at which scheduler statistics are written to the log file. Value is in microseconds. Only used when RCCL_IB_QP_SCHED_LOG_PATH is set.	Integer microseconds. Default: 1000000 (1 second). Values outside the range 1 µs to 60 s are ignored.

Development and testing (advanced)

The development and testing environment variables for RCCL are collected in the following table. These variables are primarily intended for debugging and development purposes.

Environment variable	Values
CUDA_LAUNCH_BLOCKING Controls CUDA kernel launch blocking behavior.	0: Non-blocking launches 1 or non-zero: Blocking launches
NCCL_COMM_ID Enables multi-process mode in test applications.	Any non-empty value enables multi-process mode Used with test executables for distributed testing
NCCL_DISABLE_MEM_MANAGER Disables the internal RCCL memory manager. This is an internal parameter intended for testing and debugging only. When the memory manager is disabled, ncclCommSuspend, ncclCommResume, and ncclCommMemStats return ncclInvalidUsage.	0: Memory manager enabled (default). 1: Memory manager disabled.
NCCL_NO_CACHE Disables caching for selected RCCL environment parameters so their values are re-read from the environment on each access. By default, RCCL caches parameter values after the first read for performance. This variable is intended for testing and debugging when parameters need to be changed without restarting the process. The value is parsed once on first use, so it must be set before RCCL reads any parameters. NCCL_NO_CACHE itself is always cached and cannot be listed.	Unset (default): all parameters are cached after first read. Comma-separated list of parameter names (for example, NCCL_DEBUG,NCCL_ALGO): disable caching for those keys only. ALL: disable caching for every parameter except NCCL_NO_CACHE.

Multi-communicator ordering

When an application uses multiple RCCL communicators on the same device, collective operations may execute in an unpredictable order unless the application adds explicit synchronization between streams.

Environment variable	Values
NCCL_LAUNCH_ORDER_IMPLICIT Serializes RCCL operations across different communicators on the same device according to their host-side launch sequence. This provides deterministic execution order for multi-communicator workloads such as chained collectives where one operation’s output feeds into the next.	0: Disabled (default). 1: Enabled. Operations execute in host launch order.

Inspector profiling

The RCCL Inspector is a profiler plugin that emits per-communicator, per-operation performance data (collectives and point-to-point) as JSON or Prometheus textfile metrics. For a full walkthrough, see Using the RCCL Inspector plugin. The Inspector environment variables are collected in the following table.

Environment variable	Values
NCCL_INSPECTOR_ENABLE Enables the Inspector profiler plugin. The plugin must also be loaded through NCCL_PROFILER_PLUGIN.	0: Disabled (default). 1: Enabled.
NCCL_INSPECTOR_ENABLE_P2P Enables tracking of point-to-point (Send/Recv) operations in addition to collectives. Required for the nccl_p2p_* Prometheus metrics and the P2P panels of the Grafana dashboard.	0: Disabled. 1: Enabled (default).
NCCL_INSPECTOR_PROM_DUMP Selects the Prometheus node-exporter textfile output format (nccl_inspector_metrics_<uuid>.prom) instead of the default JSON.	0: JSON output (default). 1: Prometheus textfile output.
NCCL_INSPECTOR_PROM_DUMP_STATS In Prometheus mode, also emits per-device ring-buffer counters: nccl_collectives_total, nccl_collectives_dropped_total, nccl_p2p_total and nccl_p2p_dropped_total. They are cumulative, so rate(dropped) / rate(total) gives the fraction of operations lost. JSON output always carries the same counts in its dump_stats record.	0: Disabled (default). 1: Enabled.
NCCL_INSPECTOR_CLUSTER Overrides the Prometheus cluster label. When unset, the Inspector uses SLURM_CLUSTER_NAME. Set this when that name is missing.	String. Default: unset (falls back to SLURM_CLUSTER_NAME, else unknown).
NCCL_INSPECTOR_DUMP_THREAD_ENABLE Enables the internal dump thread. When disabled, output is only written at communicator teardown, regardless of the configured dump interval.	0: Disabled. 1: Enabled (default).
NCCL_INSPECTOR_DUMP_THREAD_INTERVAL_MICROSECONDS Interval, in microseconds, at which the internal dump thread writes output. Output is always written at communicator teardown.	-1: Dump only at teardown (default). 0: Dump continuously. N: Dump every N microseconds. In Prometheus mode a minimum of 30000000 (30 s) is enforced to match node-exporter polling.
NCCL_INSPECTOR_DUMP_DIR Output directory for Inspector logs/metrics. For Prometheus mode, point this at the node-exporter textfile collector directory.	String path. Default: nccl-inspector-<jobid> from SLURM_JOB_ID, SLURM_JOBID, PBS_JOBID, or LSB_JOBID, else nccl-inspector-unknown-jobid.
NCCL_INSPECTOR_DUMP_VERBOSE Includes per-event trace information (sequence numbers and timestamps) in the JSON output.	0: Disabled (default). 1: Enabled.
NCCL_INSPECTOR_DUMP_MIN_SIZE_BYTES Minimum message size (in bytes) tracked by the Inspector.	Integer value in bytes (default: 8192).
NCCL_INSPECTOR_REQUIRE_KERNEL_TIMING Requires GPU-based kernel timing for an event to be recorded. When enabled, events that fall back to CPU-measured timing are discarded.	0: Record events regardless of timing source. 1: Record only GPU-timed events (default).
NCCL_INSPECTOR_DUMP_COLL_RING_SIZE Per-communicator capacity of the ring buffer holding completed collectives waiting to be dumped. When it fills, the oldest entries are overwritten and counted as dropped, with a single warning per process.	Integer number of entries (default: 1024).
NCCL_INSPECTOR_DUMP_P2P_RING_SIZE Per-communicator capacity of the ring buffer holding completed point-to-point operations waiting to be dumped. Overflow is handled as for NCCL_INSPECTOR_DUMP_COLL_RING_SIZE.	Integer number of entries (default: 1024).
NCCL_INSPECTOR_COLL_POOL_SIZE Initial size, and growth stride, of the collective event pool.	Integer number of entries (default: 256).
NCCL_INSPECTOR_P2P_POOL_SIZE Initial size, and growth stride, of the point-to-point event pool.	Integer number of entries (default: 256).
NCCL_INSPECTOR_COMM_POOL_SIZE Initial size, and growth stride, of the communicator event pool.	Integer number of entries (default: 256).
NCCL_INSPECTOR_POOL_GROW Allows the event pools above to grow beyond their initial size. When disabled, events are dropped once a pool is exhausted.	0: Fixed-size pools. 1: Pools grow on demand (default).
NCCL_INSPECTOR_OTEL_EXPORT Exports metrics over OTLP/HTTP instead of writing them to files. The Inspector emits one format only, and this setting takes precedence over NCCL_INSPECTOR_PROM_DUMP.	0: Disabled, so JSON or Prometheus files are written (default). 1: Enabled, posting OTLP/JSON to the endpoint below.
NCCL_INSPECTOR_OTEL_VERBOSE Selects per-collective metric points instead of aggregated ones. The per-collective form additionally reports nccl_collective_algobw_gbs and the per-device ring-buffer counters described under NCCL_INSPECTOR_PROM_DUMP_STATS.	0: Aggregated (default). 1: Per collective.
OTEL_EXPORTER_OTLP_METRICS_ENDPOINT Destination for the OTLP metric export. Also accepts OTEL_EXPORTER_OTLP_ENDPOINT. /v1/metrics is appended to whichever variable is used when its URL carries no path.	Plaintext http:// URL. https:// is unsupported and disables export, so use a local collector when the telemetry is sensitive. Default: http://localhost:4318/v1/metrics
OTEL_EXPORTER_OTLP_METRICS_HEADERS Extra headers attached to every OTLP request. Also accepts OTEL_EXPORTER_OTLP_HEADERS.	Comma-separated key=value list Sent verbatim over the plaintext connection above, so do not put a credential here unless the collector is local.

Profiler plugin

An external profiler plugin loaded through NCCL_PROFILER_PLUGIN receives the event types selected by NCCL_PROFILE_EVENT_MASK. RCCL 2.31 raises the plugin interface to v7, which adds two things a plugin can consume:

• Per-kernel barrier phases. Enabling ncclProfileKernelPhase reports an initial_sync, compute and final_sync span per kernel channel, each timed by the GPU globaltimer. A phase is a child of a kernel-channel event, so RCCL enables ncclProfileKernelCh implicitly whenever the phase bit is set. Only symmetric kernels emit phases, which requires buffers registered as symmetric windows. Regular collective and point-to-point kernels report kernel-channel events without phases.

• Symmetric-kernel variant metadata. Collective events carry the symmetric kernel variant that ran and a flag marking whether the collective was served symmetrically. The in-tree example surfaces these as the KernelVariant and IsSymColl arguments of its COLL trace events.

Both are v7-only: the v5 and v6 compatibility layers clear the phase bit, so a plugin must export ncclProfiler_v7 to receive them.

Environment variable	Values
NCCL_PROFILER_PLUGIN Selects an external profiler plugin.	Path or short name A short name is resolved against the librccl-profiler prefix, so example loads librccl-profiler-example.so
NCCL_PROFILE_EVENT_MASK Bitmask of the event types the plugin is offered.	1: Group 2: Collective 4: Point-to-point 8: Proxy op 16: Proxy step 32: Proxy control 64: Kernel channel 128: Net plugin 256: Group API 512: Collective API 1024: Point-to-point API 2048: Kernel launch 4096: CE collective 8192: CE synchronization 16384: CE batch 32768: Kernel phase (v7, symmetric kernels only; implies kernel channel) 65536: RCCL proxy diagnostics Combine by adding, so 32771 selects group, collective and kernel phase. Default: 0

Algorithm dispatch and tuning (gfx1250 / MI450)

The following variables control per-architecture algorithm and protocol dispatch, including the DDA fabric tiers, Copy Engine (CE) AllReduce, and the per-architecture threshold table introduced for gfx1250.

Environment variable	Values
RCCL_IGNORE_ARCH_TABLE Controls whether the per-architecture dispatch table (rcclArchThresholds) is used to set DDA, CE, and symmetric-kernel thresholds, or whether pre-table compile-time constants are used instead. When the table is active (0), each threshold variable below (RCCL_DDA_THRESHOLD, RCCL_DDA_LL_THRESHOLD, etc.) defaults to -1 (unset) and is resolved from the table at runtime; setting any of those variables explicitly still overrides the table.	0: Use the per-architecture table for threshold defaults (default). 1: Ignore the table; fall back to pre-table compile-time constants. This matches the behavior before the arch table was introduced.
RCCL_TUNING_PARTITION_MODE Selects which gfx1250 per-architecture table (SPX or DPX) is used for tuning thresholds. It only changes tuning, not results. A communicator uses the DPX table only when every rank selects DPX; otherwise every rank uses the SPX table. Other architectures have one table for every mode. Has no effect when RCCL_IGNORE_ARCH_TABLE=1.	auto (default): Follow the compute-partition mode the GPU reports. DPX selects the DPX table; SPX, CPX, or no reported mode selects the SPX table. Any other value is reported once and treated as auto. SPX: Use the SPX table. DPX: Use the DPX table.
RCCL_DDA_THRESHOLD Upper bound in bytes for the DDA VMM/Simple tier per collective. Messages above this size exit DDA and fall through to Ring, CE, or the symmetric kernel. When set to -1 (default), the value is resolved from the per-architecture table (ddaVmmMax field). Set RCCL_IGNORE_ARCH_TABLE=0 for the table to take effect.	-1: Resolved from the per-arch table at runtime (default). 0: Disable the DDA VMM tier for all collectives. N (bytes): Use N as the VMM tier ceiling for all collectives, overriding the table. Pre-table default: 134217728 (128 MiB).
RCCL_DDA_LL_THRESHOLD Upper bound in bytes for the DDA LL (low-latency) tier. Messages at or below this size use the LL protocol; larger messages move to LL128 or VMM. When -1 (default), resolved from ddaLLMax in the arch table.	-1: Resolved from the per-arch table at runtime (default). 0: Disable the DDA LL tier. N (bytes): Use N as the LL tier ceiling, overriding the table. Pre-table default: 65536 (64 KiB).
RCCL_DDA_LL128 Enables the DDA LL128 protocol tier. When -1 (auto), LL128 is enabled only for architectures whose per-arch table has a non-zero ddaLL128Max entry (currently gfx1250 only). 1 forces the tier on but still reads its ceiling from the arch table, so it yields the same effective cap as -1 in every current configuration; 0 disables the tier on all architectures.	-1: Auto-enabled only when the arch table has non-zero LL128 thresholds (default). 0: Disabled on all architectures. 1: Force-enabled; ceiling is still read from the arch table (same cap as -1 in every current configuration).
RCCL_DDA_LL128_THRESHOLD Upper bound in bytes for the DDA LL128 tier. Messages above this size move to VMM/Simple or Ring. When -1 (default), resolved from ddaLL128Max in the arch table. When set explicitly, it takes precedence over RCCL_DDA_LL128=0 (the env var is read before the flag check).	-1: Resolved from the per-arch table at runtime (default). 0: Disable the DDA LL128 tier. N (bytes): Use N as the LL128 tier ceiling, overriding the table. Pre-table default: 0 (LL128 tier was off before the arch table).
RCCL_CE_ALLREDUCE Enables the Copy Engine (CE) registered-window AllReduce path. When -1 (auto), CE AllReduce is on by default for gfx1250 communicators that meet all eligibility criteria, and off for all other architectures.	-1: Auto — enabled on gfx1250, disabled elsewhere (default). 0: Disabled on all architectures. 1: Force-enabled (subject to other eligibility checks such as buffer registration and message size).
RCCL_FORCE_CE_ALLREDUCE Enables the non-registered (2-shot/staged) CE AllReduce path without requiring NCCL_CTA_POLICY=2 (CTA_POLICY_ZERO). Does not affect the registered-window CE AllReduce path, which is controlled independently by RCCL_CE_ALLREDUCE. Does not override the ceNonRegMax[AR] staging buffer size cap.	-1: Auto — same default as RCCL_CE_ALLREDUCE (default). 0: Disabled. 1: Force-enabled (CTA_POLICY_ZERO requirement bypassed for non-registered path).
RCCL_CE_REDUCESCATTER Opt-in Copy Engine ReduceScatter. Off by default. When enabled, it shares the CE AllReduce staging buffer (ceARTmpBuf); user buffers do not need symmetric registration. Requires NCCL_CTA_POLICY=2 (CTA_POLICY_ZERO). Single-node symmetric communicators only. There is no message-size cap; large messages are pipelined through the staging slots.	0: Disabled (default). 1: Enabled, subject to the other eligibility checks.
RCCL_FORCE_CE_COLL Bypasses the NCCL_CTA_POLICY=2 (CTA_POLICY_ZERO) requirement for CE AllGather and CE Alltoall registered-window paths. Does not affect AllReduce (use RCCL_FORCE_CE_ALLREDUCE for that), and does not affect hierarchical CE paths. When -1 (auto), enabled by default on gfx1250 where CE is the preferred collective path; disabled on all other architectures.	-1: Auto — enabled on gfx1250, disabled elsewhere (default). 0: Disabled. 1: Force-enabled (CTA_POLICY check bypassed for AG and A2A registered CE).
RCCL_FORCE_SYMMETRIC Biases the ncclAllReduce backend selector to prefer the symmetric kernel path over all other candidates. Hard eligibility checks (buffer registration, op support) are still enforced; if no symmetric path is eligible the selector falls back normally. Off by default.	0: Disabled (default). 1: Prefer symmetric kernel for AllReduce.
RCCL_FORCE_DDA Biases the ncclAllReduce backend selector to prefer DDA (Direct Data Access) paths over all other candidates, including symmetric and Copy Engine. If no DDA path is eligible for the given message size and communicator the selector falls back normally. Off by default.	0: Disabled (default). 1: Prefer DDA for AllReduce.
RCCL_CE_AR_2SHOT_MAX_BYTES Overrides the CE 2-shot AllReduce message size cap. When -1 (default), the cap is read from ceNonRegMax[AllReduce] in the per-arch table. A table entry of 0 (e.g. gfx1250, where the value is a placeholder) resolves to 256 MiB when NCCL_CTA_POLICY=2 (ZERO CU policy) is active; without ZERO policy it stays 0 and 2-shot is disabled. Setting this variable to 0 explicitly disables the non-registered 2-shot path but does not suppress the staging buffer reservation, which the registered-window CE AllReduce path also requires.	-1: Resolved from the per-arch table (default). 0: Disable 2-shot (does not disable the registered-window CE path). N (bytes): Use N as the 2-shot AllReduce size cap.
RCCL_CE_AR_STAGING_BYTES Overrides the total allocation size of the CE AllReduce staging buffer (ceARTmpBuf). When -1 (default), the buffer is allocated at the compile-time constant NCCL_CE_AR_STAGING_BYTES (16 MiB). Increasing this reduces pipelining overhead for large messages but raises per-rank GPU memory usage. This variable sizes the buffer only; the selector cap is controlled separately by RCCL_CE_AR_MAX_MSG_BYTES.	-1: Use the compile-time default of 16 MiB (default). N (bytes): Set the per-slot payload capacity to N; ceARTmpBuf is NCCL_CE_NUM_SLOTS (2) times that.
RCCL_TDM_LL128_ENABLE Routes the LL128 user destination buffer through the gfx1250 tensor data mover: each warp’s slice is staged in LDS and pushed out as one bulk transfer instead of per-lane vector stores. Gains are concentrated in large messages under LL128 (AllGather is the best case); below roughly 32 MiB the staging overhead makes it a small loss. The library must also be built with --enable-tdm-prim-ll128, and the path only compiles on gfx1250 with an SDK providing the tensor data mover builtins. On any other architecture, or without the build flag, this variable has no effect.	0: Disabled (default). The LL128 store uses the per-lane vector path. 1: Enabled, where the build and architecture support it.
RCCL_CE_AR_2SHOT_MIN_BYTES Overrides the minimum message size for the CE 2-shot AllReduce path. When -1 (default), the value is resolved from ceNonRegMin[AllReduce] in the per-arch table. Messages smaller than this minimum skip the 2-shot CE path even when it is otherwise eligible.	-1: Resolved from the per-arch table (default). N (bytes): Use N as the 2-shot AllReduce minimum size.
RCCL_CE_COLL_MAX_BYTES Overrides the CE size cap for all non-AllReduce collectives (AllGather, AlltoAll, ReduceScatter, etc.) on both the registered-window and non-registered (scratch) paths. When -1 (default), the cap is read from ceRegMax[func] or ceNonRegMax[func] in the per-arch table. AllReduce uses RCCL_CE_AR_2SHOT_MAX_BYTES and RCCL_CE_AR_REG_MAX_MSG_BYTES instead.	-1: Resolved from the per-arch table (default). N (bytes): Use N as the CE size cap for all non-AllReduce collectives, overriding the table.
RCCL_CE_COLL_MIN_BYTES Overrides the minimum message size for non-AllReduce CE paths, on both the registered-window and non-registered (scratch) paths. When -1 (default), the value is resolved from ceNonRegMin[func] in the per-arch table. Messages below this size skip CE even when CTAPolicy_ZERO is active and the size is within ceRegMax or ceNonRegMax.	-1: Resolved from the per-arch table (default). N (bytes): Use N as the CE minimum size for all non-AllReduce paths.
RCCL_SYM_K_MAX_BYTES Overrides the symmetric-kernel upper bound for all collectives. When -1 (default), the bound is resolved from symMaxR2[func] (eager mode) or symMaxR2Graph[func] (graph-capture mode) in the per-arch table. A table entry of kThreshUnlimited (SIZE_MAX) means no suppression. Setting this variable to 0 disables the symmetric kernel for all collectives.	-1: Resolved from the per-arch table (default). 0: Disable the symmetric kernel for all collectives. N (bytes): Cap the symmetric kernel at N bytes for all collectives.
RCCL_SYM_K_MIN_BYTES Overrides the symmetric-kernel lower bound for all collectives. Below this threshold, DDA is preferred over the symmetric kernel for registered buffers. When -1 (default), the bound is resolved from symMinR2[func] in the per-arch table. 0 in the table means no lower-bound suppression.	-1: Resolved from the per-arch table (default). 0: No lower-bound suppression (symk eligible at all sizes). N (bytes): Below N, prefer DDA over the symmetric kernel.
NCCL_SYM_TMA_ENABLE Controls the Tensor Data Mover (TDM) symmetric kernels (AllGather_TmaST, AllReduce_RSxTmaLD_AGxTmaST and ReduceScatter_TmaLD), which stage data through the gfx1250 Tensor Data Mover instead of per-lane vector loads and stores. They run only on gfx1250 builds whose ROCm SDK ships the TDM descriptor header, and only for collectives whose send and receive buffers are both in symmetric registered windows and 16-byte aligned. On every other architecture this variable has no effect. When enabled, the symmetric cost model offers these kernels only at message sizes where their TDM deep loop engages, and launches them with the widest allowed block count. Set the same value on every rank: ranks that disagree can pick different kernels for the same collective and hang. See Tensor Data Mover symmetric kernels on gfx1250.	0: Disabled. The symmetric path uses only the vector-load kernels. 1: Offered to the symmetric cost model (default). 2: Forced. Wherever the collective has a TDM kernel, it is selected regardless of predicted time or message size. Intended for A/B measurement, not production.