Run Cluster Validation Suite (CVS) test suites on AMD Instinct GPU clusters

Run Cluster Validation Suite (CVS) test suites on AMD Instinct GPU clusters#

2026-10-27

8 min read time

Applies to Linux

To run a test suite over SSH you need two files: a cluster file (--cluster_file) that describes your nodes and SSH access, and a test suite config (--config_file) with suite-specific settings. Set those up first — see Configure the Cluster Validation Suite (CVS) cluster file and Configure Cluster Validation Suite (CVS) test suite configuration files.

To choose the right template, see Choose a Cluster Validation Suite (CVS) config template for your workload.

On Slurm or SPUR, launch cvs run as a job step instead and the cluster file becomes optional (see Run Cluster Validation Suite (CVS) on managed compute).

For SSH mode, use cvs run on the head node to execute tests across the cluster.

List available suites#

Run the following command to see all available test suites:

cvs list

You can also run cvs run with no arguments to see the same catalog.

Run a suite#

Pass the cluster file and test suite config from your workspace:

cvs run agfhc_cvs \
  --cluster_file ~/cvs_workspace/cluster.json \
  --config_file ~/cvs_workspace/health_config.json \
  --capture=tee-sys -vvv -s

cvs run writes the pytest HTML report, text log, and zip bundle under the run directory. See Pytest HTML report and bundle.

List test cases in a suite#

cvs list <suite> lists the test functions in that suite, including parameterized tests generated from your config. Pass the same --cluster_file and --config_file you use for cvs run:

cvs list agfhc_cvs \
  --cluster_file ~/cvs_workspace/cluster.json \
  --config_file ~/cvs_workspace/health_config.json

Run one test function#

Add the test function name after the suite name:

cvs run agfhc_cvs test_agfhc_hbm \
  --cluster_file ~/cvs_workspace/cluster.json \
  --config_file ~/cvs_workspace/health_config.json \
  --capture=tee-sys -vvv -s

Run with containers#

The cluster file has an orchestrator key. Suites differ: some follow it and run the workload inside the container, some start a container and keep their checks on the host, some force a container, and some stay on host SSH. The table is in Cluster Validation Suite (CVS) cluster file: configuration and backend selection.

Bare metal — commands run on the host filesystem. Use this when the suite follows orchestrator and the ROCm stack is installed on each host.
Container — commands for a suite that enters the container run with docker exec in one container per host. per_run (the default lifetime) removes that container at the end of the run.

See Run Cluster Validation Suite (CVS) test suites with a per-host Docker container backend for the cluster file, image, and a health-suite example.

Run on managed compute#

Launch one CVS task per node as a job step and give every task the same shared workspace. CVS generates the cluster file from the allocated node list; see Run Cluster Validation Suite (CVS) on managed compute for setup and a complete quickstart.

srun -A <account> -p <partition> -N 2 --ntasks-per-node 1 --exclusive --mpi=none \
  bash -lc 'source <cvs-checkout>/.cvs_venv/bin/activate &&
    cvs run host_configs_cvs test_check_os_release test_check_kernel_version \
      --config_file /shared/<user>/cvs/host_config.json \
      --workspace /shared/<user>/cvs'

Common `cvs run` options#

Required

--config_file: Per-suite test configuration under cvs/input/config_file/. See Cluster Validation Suite (CVS) test configuration files reference.

Conditionally required

--cluster_file: Cluster JSON with node list, SSH credentials, and execution backend. Read more at Cluster Validation Suite (CVS) cluster file: configuration and backend selection. Required unless cvs run is launched as a Slurm/SPUR job step (see Run Cluster Validation Suite (CVS) on managed compute).

Optional

--workspace: Shared-filesystem root for this run. The HTML report and text log default to <workspace>/cvs_runs/<run_id>/. Falls back to $CVS_WORKSPACE then the venv parent directory. Every scheduler-managed run must set this (or CVS_WORKSPACE) to storage shared by all nodes. The files written there are listed in Pytest HTML report and bundle.
--html: HTML report path override. Omit to use the run-directory default; --no-html skips the auto-derived report.
--self-contained-html: Embed CSS/JS in the HTML report for a portable single file. Implied when the HTML path is auto-derived; pass it with an explicit --html path if you want a portable override.
--log-file: Text log path override. Omit to use the run-directory default; --no-log-file skips the auto-derived file (console logging is unaffected).
--capture=tee-sys: Capture stdout/stderr from tests while still printing to the console.
-vvv: Increase pytest verbosity.
-s: Disable pytest output capturing (print statements appear in the console).

All pytest options can be passed through cvs run. For the full option list see Cluster Validation Suite (CVS) command-line interface (CLI) reference.

Test suites#

Run suites in the order shown: validate single-node health before exercising the network, and validate the network before distributed training or inference. Each suite’s page includes setup and run steps.

Category	What it validates	Suites
Burn-in and diagnostics	Single-node GPU and host health: OS config, BIOS/firmware, driver load, GPU burn-in, and device access. Run before any cluster-wide workload.	Platform, Health, Preflight
Network	Interconnect bandwidth, latency, and GPU collective communication across all nodes. Run after burn-in passes and before distributed workloads.	IB Perf, RCCL, MORI
Training	Multi-node distributed training throughput, scaling efficiency, and model convergence. Run after network validation.	Aorta, JAX MaxText, Megatron, TorchTitan
Inference	LLM serving throughput, latency, accuracy, and diffusion model performance on AMD Instinct GPUs.	vLLM, ATOM, SGLang, xDiT

Scalability#

For clusters with 32 or more nodes, CVS automatically shards work across parallel worker processes. See Cluster Validation Suite (CVS) scalability and parallel SSH performance for tuning CVS_HOSTS_PER_SHARD and CVS_WORKERS_PER_CPU.

Test results#

CVS writes a pytest HTML report after each run. The report includes per-node pass/fail status, captured output, and links to any custom suite-specific reports.

Pytest HTML report and bundle#

Each run produces the following artifacts in the run directory.

Artifact	Default location	Notes
Pytest HTML report	`<run_dir>/<test-file-stem>.html`	For example, `<workspace>/cvs_runs/<run_id>/rccl_perf.html`. The auto-derived report is self-contained.
Text log	`<run_dir>/<test-file-stem>.log`	Pytest run log.
Attachments	`<run_dir>/<test-file-stem>_html/`	Per-test “Full Log” pages and a copy of the run log. When the suite supports a Run Deck, this directory also holds the Run Deck HTML and JSON, and might include a summary page or an interactive viewer. The pytest report’s Reports section links the Run Deck HTML. The other deck files stay in this directory and in the zip. Recreated at the start of each run.
Zip bundle	`<run_dir>/<test-file-stem>_<YYYY-mm-ddTHHMMSS>.zip`	Contains the report, the attachments directory, and the cluster and config files. Links still work after extraction.

<run_dir> is <workspace>/cvs_runs/<run_id>. <run_id> is local-<YYYYmmdd-HHMMSS> outside a scheduler, or the job ID under SPUR or Slurm.

A Run Deck summarizes results collected during the run. It is included when the selected suite supports it. Run Deck generation does not affect the suite result.

See Cluster Validation Suite (CVS) command-line interface (CLI) reference for the report and log options.

Test output examples#

Tip

If errors prevent tests from starting, CVS prints a message with a suggested fix in the console.

Install suites run the tool installation commands on each node and confirm the binary is on the expected path.

All tests passed. The console shows per-step timing and ends with the pytest summary line and an HTML report path.

One or more tests failed. The short test summary lists each failing test with the parametrization that triggered it and the assertion that failed.

The run was interrupted with Ctrl + C. CVS prints the KeyboardInterrupt traceback and generates a partial HTML report for the tests that completed.

Sample reports for each category are on that category’s page, under Test results:

Burn-in and diagnostics — pytest HTML report and Run Deck
Network — RCCL pytest HTML report, Run Deck, and interactive viewer
Training — pytest HTML report, Run Deck, and interactive viewer
Inference — pytest HTML report, Run Deck, and interactive viewer