Run CVS RCCL performance and regression tests

Run CVS RCCL performance and regression tests#

2026-09-24

3 min read time

Applies to Linux

RCCL tests validate collective communication bandwidth and correctness across AMD Instinct GPU nodes using all-reduce, all-gather, broadcast, and other collectives. Run RCCL after IB Perf passes to confirm that GPU-to-GPU communication meets performance thresholds.

Set up config#

  1. Copy the RCCL configuration file:

    cvs config copy rccl/rccl_config.json --output ~/cvs_workspace/rccl/rccl_config.json

  2. Edit directory paths and any <changeme> placeholders:

    • rccl_dir, rccl_tests_dir, mpi_dir

    • mpi_path_var, rccl_path_var, rocm_path_var

For the complete field reference, see ROCm Communication Collectives Library (RCCL) test configuration files.

Run tests#

You can list all available RCCL test cases using the CLI:

cvs list rccl_perf

Available tests in rccl_perf:
  - test_collect_hostinfo
  - test_collect_networkinfo
  - test_disable_firewall
  - test_gen_graph
  - test_print_env_once
  - test_rccl_perf[all_gather_perf]
  - test_rccl_perf[all_reduce_perf]
  - test_rccl_perf[alltoall_perf]
  - test_rccl_perf[alltoallv_perf]
  - test_rccl_perf[broadcast_perf]
  - test_rccl_perf[gather_perf]
  - test_rccl_perf[reduce_scatter_perf]
  - test_rccl_perf[scatter_perf]
  - test_rccl_perf[sendrecv_perf]

cvs list rccl_regression

Available tests in rccl_regression:
  - test_collect_hostinfo
  - test_collect_networkinfo
  - test_disable_firewall
  - test_gen_graph
  - test_print_env_once
  - test_rccl_perf

Prerequisites for environment script staging#

Before running RCCL tests, ensure the environment script specified in env_source_script is available on all cluster nodes:

With shared storage#

Place the environment script in a shared directory accessible from all nodes.

Without shared storage#

Use cvs scp to copy the environment script to all nodes. See Copy files to all cluster nodes using Cluster Validation Suite (CVS) parallel SCP for detailed instructions and examples.

  1. Run RCCL performance suite:

cvs run rccl_perf \
  --cluster_file ~/cvs_workspace/cluster.json \
  --config_file ~/cvs_workspace/rccl/rccl_config.json \
  --html=/var/www/html/cvs/rccl_perf.html --capture=tee-sys --self-contained-html \
  --log-file=/tmp/rccl_perf.log -vvv -s

  1. Run RCCL regression suite:

cvs run rccl_regression \
  --cluster_file ~/cvs_workspace/cluster.json \
  --config_file ~/cvs_workspace/rccl/rccl_regression.json \
  --html=/var/www/html/cvs/rccl_regression.html --capture=tee-sys --self-contained-html \
  --log-file=/tmp/rccl_regression.log -vvv -s

  1. Generate RCCL performance heatmap:

cvs generate heatmap --actual /tmp/rccl_perf_results.json --reference /path/to/golden_reference.json --output /var/www/html/cvs/rccl_heatmap.html --title "RCCL Performance Comparison"
Contents