Multi-node networking guide

Multi-node networking guide#

Multi-node training depends on high-bandwidth, low-latency communication between GPUs. On AMD systems, RCCL (ROCm Collective Communications Library) provides GPU collectives with an API aligned to NCCL, so most NCCL-prefixed environment variables apply to RCCL as well.

This guide summarizes how Primus configures networking, how InfiniBand, RoCE, and AINIC (AMD AI NIC) fit in, and how to validate and troubleshoot cluster connectivity.

Primary sources in this repository

Topic	File
Default NCCL/RCCL and socket setup	`runner/helpers/envs/base_env.sh`
IB HCA detection	`runner/helpers/envs/get_nccl_ib_hca.sh`
Socket / interface detection	`runner/helpers/envs/get_ip_interface.sh`
AINIC hook (container/CLI integration)	`runner/helpers/hooks/03_enable_ainic.sh`
AINIC CLI defaults	`runner/use_ainic.yaml`
ANP / `NCCL_NET_PLUGIN` example	`examples/run_pretrain.sh`

1. Overview#

Goal: Keep gradient and parameter exchanges from becoming the bottleneck when scaling across nodes.
Stack: PyTorch distributed uses the ROCm NCCL backend name in many configs; the implementation is RCCL on AMD GPUs.
Transports: Common fabrics include InfiniBand (IB), RoCE (RDMA over Converged Ethernet), and AINIC on supported AMD platforms. Primus scripts set or detect HCAs, socket interfaces, and optional AINIC tuning.

2. InfiniBand configuration#

These variables are standard in NCCL/RCCL deployments. Primus seeds several from runner/helpers/envs/base_env.sh when that script is sourced.

Variable	Role
`NCCL_IB_HCA`	Selects InfiniBand Host Channel Adapters (device:port list).
`NCCL_IB_GID_INDEX`	GID index for the active port (RoCE and IB differ; see vendor docs).
`NCCL_IB_TC`	Traffic class for InfiniBand.
`NCCL_IB_FIFO_TC`	Traffic class for FIFO traffic.
`NCCL_IB_RETRY_CNT`	Retry count for IB operations (tune with vendor guidance).
`NCCL_IB_TIMEOUT`	Timeout for IB operations.
`NCCL_IB_QPS_PER_CONNECTION`	Queue pairs per connection.
`NCCL_NET_GDR_LEVEL`	GPUDirect RDMA level for NIC/GPU transfers.
`NCCL_DMABUF_ENABLE`	Use DMA-BUF path where supported.

Auto-detection in Primus#

If NCCL_IB_HCA is unset, base_env.sh runs runner/helpers/envs/get_nccl_ib_hca.sh, which enumerates /sys/class/infiniband/, skips bonded/storage-style devices, and builds a comma-separated device:port list for NCCL_IB_HCA.

Default in base_env.sh:

export NCCL_IB_GID_INDEX=${NCCL_IB_GID_INDEX:-3}

AINIC-oriented configs often override NCCL_IB_GID_INDEX to 1 (see runner/use_ainic.yaml and 03_enable_ainic.sh).

3. RoCE (RDMA over converged ethernet)#

RoCE reuses much of the IB verb stack; the same NCCL_IB_* knobs apply.

Variable	Typical use
`NCCL_IB_ROCE_VERSION_NUM`	RoCE version (commonly 2 for RoCE v2).

GID selection (NCCL_IB_GID_INDEX) and traffic classes (NCCL_IB_TC, NCCL_IB_FIFO_TC) remain important on RoCE fabrics. Follow your network team’s mapping (often GID index 1 for RoCE v2 vs 3 for some IB fabrics—your site might differ).

4. AINIC (AMD AI NIC)#

AINIC refers to AMD’s AI-optimized NIC path (for example, the AMD Pensando™ Pollara 400 AI NIC) used in some clusters. Enabling it is a combination of environment, container image, and device pass-through.

Enable AINIC#

Set USING_AINIC=1. The hook runner/helpers/hooks/03_enable_ainic.sh runs when this is set and exports AINIC-related variables back to the caller (env.VAR=VALUE lines).
Use container images built for AINIC when required by your site. Examples in this repository use tags such as docker.io/tasimage/primus:<version>-ainic (see examples/customer_package/ and .github/workflows/ci.yaml). Match the image to your ROCm and ANP bundle.

`runner/use_ainic.yaml`#

Primus CLI system defaults for AINIC-oriented runs include:

Container device mounts: /dev/kfd, /dev/dri, /dev/infiniband (required for GPU and IB access in the container).
Environment entries such as USING_AINIC=1, NCCL_PXN_DISABLE=0, and NCCL_IB_GID_INDEX=1.

Adjust NCCL_IB_GID_INDEX and container.options.image to match your cluster; comments in runner/use_ainic.yaml call this out explicitly.

AINIC hook#

runner/helpers/hooks/03_enable_ainic.sh is the supported hook path: it sets ANP/RCCL/MPI home directories, IB QoS, RoCE version, P2P channel counts, GDR flush behavior, LD_LIBRARY_PATH (including libibverbs and RCCL/ANP/MPI build paths), and related flags. Default NCCL_IB_FIFO_TC in the hook is 192; align this value with your fabric.

RCCL network plugin (ANP)#

For ANP-based networking, clusters often set NCCL_NET_PLUGIN to librccl-anp.so when that library is present under ANP_HOME_DIR, falling back to librccl-net.so otherwise—see the logic in examples/run_pretrain.sh. This complements the library paths from 03_enable_ainic.sh.

Variables commonly set for AINIC#

From 03_enable_ainic.sh (non-exhaustive):

Variable	Purpose
`ANP_HOME_DIR`, `RCCL_HOME_DIR`, `MPI_HOME_DIR`	Install roots for ANP, RCCL, and Open MPI.
`NCCL_IB_TC`, `NCCL_IB_FIFO_TC`	Traffic classes for IB/RoCE.
`NCCL_IB_GID_INDEX`	Often 1 for AINIC-oriented configs in Primus examples.
`NCCL_IB_ROCE_VERSION_NUM`	RoCE v2.
`RCCL_GDR_FLUSH_GPU_MEM_NO_RELAXED_ORDERING`	Stricter GDR flush ordering (set to `0` in these scripts).
`LD_LIBRARY_PATH`	Prepends `libibverbs`, RCCL, ANP, and MPI library paths.

5. Socket configuration#

CPU-side and fallback socket traffic uses interface selection:

Variable	Role
`NCCL_SOCKET_IFNAME`	Interface name or pattern for NCCL socket transport (e.g. `eth0`, or `^docker0,lo` to exclude virtual interfaces).
`GLOO_SOCKET_IFNAME`	Interface for Gloo process groups (CPU barriers and related).

Primus behavior: base_env.sh sets IP_INTERFACE via runner/helpers/envs/get_ip_interface.sh (fallback: first address from hostname -I). Both NCCL_SOCKET_IFNAME and GLOO_SOCKET_IFNAME default to IP_INTERFACE when unset.

Requirement: All nodes must agree on a reachable address family and interface choice; mismatched bindings are a frequent source of hangs.

6. PCIe cross-NIC (PXN)#

Variable	Default in `base_env.sh`	Meaning
`NCCL_PXN_DISABLE`	`1`	PXN disabled by default (saves GPU memory per comment in `base_env.sh`).

When NCCL_PXN_DISABLE=0, PCIe cross-NIC is enabled: GPUs might use NICs attached to other PCIe switches, which can improve multi-rail bandwidth at the cost of higher GPU memory use. runner/use_ainic.yaml sets NCCL_PXN_DISABLE=0 for AINIC-oriented runs.

7. Network diagnostics#

Preflight#

primus-cli direct -- preflight --network

For multi-node (Slurm example):

primus-cli slurm srun -N 4 -- preflight --host --gpu --network

See docs/02-user-guide/preflight.md for flags, output locations (output/preflight by default), and interpretation.

Set PRIMUS_EXPECT_IB=1 when InfiniBand is required for validation; preflight uses this in primus/tools/preflight/network/network_standard.py.

RCCL benchmark#

primus-cli slurm srun -N 4 -- benchmark rccl --op all_reduce --min-bytes 1M --max-bytes 128M

This exercises collective bandwidth and latency across a message-size sweep. See docs/02-user-guide/benchmarking.md and primus/tools/benchmark/rccl_bench_args.py for options (dtypes, operations, output files).

Verbose RCCL logs#

export NCCL_DEBUG=INFO

Use for short, controlled runs; TRACE can be extremely verbose.

8. Multi-node setup checklist#

ROCm version matches across all nodes (driver and container image).
NCCL_SOCKET_IFNAME / GLOO_SOCKET_IFNAME (or auto-detected IP_INTERFACE) identify the same logical network on every node.
InfiniBand or RoCE is up (ibstat, /dev/infiniband, kernel modules such as ib_core / mlx5_core as appropriate).
Firewall allows ports required by your launcher and collective tests (MASTER_ADDR / MASTER_PORT reachable).
MASTER_ADDR resolves and is reachable from all nodes.
GPUS_PER_NODE matches physical GPUs per node.
Containers mount /dev/kfd, /dev/dri, and /dev/infiniband when using IB/RoCE/AINIC (see runner/use_ainic.yaml).

9. Troubleshooting network issues#

Symptom	What to check
Timeout or hang at init	`MASTER_ADDR` / `MASTER_PORT`, firewall, VPN, wrong `NCCL_SOCKET_IFNAME`, or inconsistent interface across nodes.
Slow collectives	IB vs Ethernet path, `NCCL_NET_GDR_LEVEL`, fabric errors, or contention; compare `benchmark rccl` to baseline.
IB not detected	`/dev/infiniband` missing, modules not loaded, or wrong container devices.
Wrong interface	Restrict with `NCCL_SOCKET_IFNAME=^docker0,lo` (exclude loopback and Docker bridges).
GID / RoCE issues	`NCCL_IB_GID_INDEX` vs site documentation; RoCE v2 settings (`NCCL_IB_ROCE_VERSION_NUM`).

For a consolidated list of NCCL_* / RCCL_* variables, see docs/03-configuration-reference/environment-variables.md and the upstream RCCL environment variables documentation.