Running RCCL using Docker

Running RCCL using Docker#

To use Docker to run RCCL, Docker must already be installed on the system. To build the Docker image and run the container, follow these steps.

Build the Docker image

The Dockerfile.ubuntu file is located in the docker directory of the RCCL project. Run the following commands from the projects/rccl directory of the rocm-systems repository.

By default, the Dockerfile uses docker.io/rocm/dev-ubuntu-22.04:latest as the base Docker image. It then fetches projects/rccl, projects/rccl-tests, and projects/rocshmem from the develop branch of the rocm-systems repository and builds RCCL and rccl-tests for gfx942 GPUs.

Use this command to build the Docker image:
```
docker build -t rccl-tests -f docker/Dockerfile.ubuntu --pull .
```
The base Docker image, rocm-systems repository (ROCM_SYSTEMS_REPO), branch, tag, or commit SHA (ROCM_SYSTEMS_REF), and GPU targets can be modified by using --build-arg in the docker build command above. GPU_TARGETS takes one target (for example, gfx950), several separated by ; (for example, "gfx942;gfx950"), or an empty value (GPU_TARGETS=) to build every target RCCL and rccl-tests support, which makes the build several times slower and the image larger. ROCM_SYSTEMS_REF must be a branch, a tag that contains projects/ (for example, therock-*), or a full commit SHA, because short SHAs fail the shallow fetch and rocm-7.2.x tags predate the monorepo layout. For example, to use a different base Docker image and target a specific GPU architecture, use this command:
```
docker build -t rccl-tests -f docker/Dockerfile.ubuntu --build-arg="ROCM_IMAGE_NAME=rocm/dev-ubuntu-24.04" --build-arg="ROCM_IMAGE_TAG=6.4.2" --build-arg="GPU_TARGETS=gfx942" --pull .
```
Extra RCCL install.sh flags can be passed with RCCL_BUILD_ARGS, which is empty by default. For example, to build RCCL with rocSHMEM, use this command:
```
docker build -t rccl-tests -f docker/Dockerfile.ubuntu --build-arg="RCCL_BUILD_ARGS=--rocshmem" --pull .
```

Launch an interactive Docker container on a system with AMD GPUs:

docker run --rm --device=/dev/kfd --device=/dev/dri --group-add video --ipc=host --network=host --cap-add=SYS_PTRACE --security-opt seccomp=unconfined -it rccl-tests /bin/bash

To run, for example, the all_reduce_perf test from rccl-tests on 8 AMD GPUs from inside the Docker container, use this command for ROCm 6.4.2 or later:

mpirun --allow-run-as-root -np 8 --mca pml ucx --mca btl ^openib -x NCCL_DEBUG=VERSION /workspace/rccl-tests/build/all_reduce_perf -b 1 -e 16G -f 2 -g 1

For ROCm 6.4.1 or earlier:

mpirun --allow-run-as-root -np 8 --mca pml ucx --mca btl ^openib -x NCCL_DEBUG=VERSION -x HSA_NO_SCRATCH_RECLAIM=1 /workspace/rccl-tests/build/all_reduce_perf -b 1 -e 16G -f 2 -g 1

For more information on the rccl-tests options, see the Usage guidelines in the GitHub repository.