Run CVS RCCL performance and regression tests

Run CVS RCCL performance and regression tests#

2026-10-19

4 min read time

Applies to Linux

RCCL tests validate collective communication bandwidth and correctness across AMD Instinct GPU nodes using all-reduce, all-gather, broadcast, and other collectives. Run RCCL after IB Perf passes to confirm that GPU-to-GPU communication meets performance thresholds.

  • rccl_perf runs configured collectives across rccl_test_params.data_types.

  • rccl_regression runs the Cartesian product of the regression environment axes and collectives.

  • rccl_pairwise runs Phase 0 reference sanity, Phase 1 reference/candidate pairs, and a Phase 2 incremental group.

Set up config#

  1. Copy the RCCL configuration file:

    cvs config copy rccl/rccl_config.json --output ~/cvs_workspace/rccl/rccl_config.json

  2. Edit directory paths and any <changeme> placeholders:

    • rccl_dir, rccl_tests_dir, mpi_dir

    • mpi_path_var, rccl_path_var, rocm_path_var

  3. All three suites read the same file. rccl_regression additionally requires a non-empty regression object; rccl_pairwise uses cvs_params.pairwise_min_bw and cvs_params.pairwise_results_file.

For the complete field reference, see ROCm Communication Collectives Library (RCCL) test configuration files.

Run tests#

You can list all available RCCL test cases using the CLI:

cvs list rccl_perf

Available tests in rccl_perf:
  - test_collect_hostinfo
  - test_collect_networkinfo
  - test_disable_firewall
  - test_gen_graph
  - test_print_env_once
  - test_rccl_perf[all_gather_perf]
  - test_rccl_perf[all_reduce_perf]
  - test_rccl_perf[alltoall_perf]
  - test_rccl_perf[alltoallv_perf]
  - test_rccl_perf[broadcast_perf]
  - test_rccl_perf[gather_perf]
  - test_rccl_perf[reduce_scatter_perf]
  - test_rccl_perf[scatter_perf]
  - test_rccl_perf[sendrecv_perf]

cvs list rccl_regression

Available tests in rccl_regression:
  - test_collect_hostinfo
  - test_collect_networkinfo
  - test_disable_firewall
  - test_gen_graph
  - test_print_env_once
  - test_rccl_perf

cvs list rccl_pairwise

Available tests in rccl_pairwise:
--------------------------------------------------------------------------------
  • test_collect_hostinfo
  • test_collect_networkinfo
  • test_gen_graph
  • test_rccl_incremental
  • test_rccl_pairwise

Total: 5 tests

Prerequisites for environment script staging#

Before running RCCL tests, ensure the environment script specified in env_source_script is available on all cluster nodes:

With shared storage#

Place the environment script in a shared directory accessible from all nodes.

Without shared storage#

Use cvs scp to copy the environment script to all nodes. See Copy files to all cluster nodes using Cluster Validation Suite (CVS) parallel SCP for detailed instructions and examples.

Run on bare metal or with the container backend#

  1. Run RCCL performance suite:

    cvs run rccl_perf \
  --cluster_file ~/cvs_workspace/cluster.json \
  --config_file ~/cvs_workspace/rccl/rccl_config.json \
  --capture=tee-sys -vvv -s

  2. Run RCCL regression suite:

    cvs run rccl_regression \
  --cluster_file ~/cvs_workspace/cluster.json \
  --config_file ~/cvs_workspace/rccl/rccl_config.json \
  --capture=tee-sys -vvv -s

  3. Run RCCL pairwise suite:

    cvs run rccl_pairwise \
  --cluster_file ~/cvs_workspace/cluster.json \
  --config_file ~/cvs_workspace/rccl/rccl_config.json \
  --capture=tee-sys -vvv -s

  4. Generate RCCL performance heatmap:

    cvs generate heatmap --actual /tmp/rccl_perf_results.json --reference /path/to/golden_reference.json --output /var/www/html/cvs/rccl_heatmap.html --title "RCCL Performance Comparison"

Each cvs run writes its pytest HTML report and log as described in Pytest HTML report and bundle. Pass --workspace <shared path> to choose where the run directory is created.

Run on managed compute#

Follow Run Cluster Validation Suite (CVS) on managed compute to run RCCL on managed compute.

On SPUR 0.11, pairwise and incremental RCCL tests are skipped. These tests run under Slurm.

Contents