Cluster Validation Suite (CVS) command-line interface (CLI) reference

2026-10-26

7 min read time

Applies to Linux

Complete option reference for all cvs commands. For step-by-step usage, see the how-to guides linked at the end of each section.

Top-level commands

CVS provides the following top-level commands.

Command	Description
cvs generate	Generate configuration files or templates.
cvs config	Browse and copy bundled configuration templates.
cvs run	Run a test suite (wrapper over pytest).
cvs list	List available tests.
cvs monitor	Cluster health monitoring.
cvs exec	Execute a command on all cluster nodes.
cvs scp	Copy files to cluster nodes in parallel.

cvs generate

Run cvs generate with no arguments to list available generators.

Subcommands: cluster_json, heatmap.

cluster_json

Generate a cluster JSON file from a host list.

Required options (one host source plus credentials):

Option	Description
--input_hosts_file PATH or --hosts HOSTS	Host list: file with one IP/hostname per line, or comma-separated list. Supports ranges such as 192.168.1.10-20 and hostname[1-10].
--output_json_file PATH	Output cluster JSON path.
--username USER	SSH username for cluster nodes.
--key_file PATH	SSH private key file.

Optional:

Option	Description
--head_node IP	Head node IP (defaults to the first host)

See Configure the Cluster Validation Suite (CVS) cluster file.

heatmap

Generate an RCCL performance heatmap HTML report from actual vs reference JSON.

Required options:

Option	Description
-a PATH, --actual PATH	Actual results JSON (RCCL graph format).
-r PATH, --reference PATH	Golden reference JSON.

Optional options:

Option	Description
-o PATH, --output PATH	Output HTML path (default: /tmp/rccl_heatmap_<timestamp>.html).
-t TITLE, --title TITLE	Chart title (default: RCCL Performance Heatmap).
--metadata	Include metadata table (actual JSON must have a metadata key).
--no-data-table	Omit the data table from the HTML report.

See Run CVS RCCL performance and regression tests.

cvs config

Browse and copy bundled configuration templates for test suites and cluster files. Subcommands: list-dirs, list, copy.

list-dirs [path]

List config directories grouped by category (config_file_dirs:, cluster_file_dirs:, env_file_dirs:). Optional path scopes to a subtree (for example training).

list [path]

List template files grouped by parent directory. Optional path scopes to a subdirectory or file prefix (for example platform, rccl).

copy [path] --output PATH

Copy bundled templates into your workspace.

Option	Description
--output PATH	Destination file or directory (required).
--all	Copy all templates, preserving directory structure.
--force	Overwrite existing destination files.

Positional path is optional: omit for bulk operations with --all; specify a template path for a single-file copy (for example platform/host_config.json).

See Configure Cluster Validation Suite (CVS) test suite configuration files and Configure the Cluster Validation Suite (CVS) cluster file.

cvs run

Positional arguments: test [function ...] — suite name and optional test function names.

Required options:

Option	Description
--config_file PATH	Suite-specific test configuration JSON.

Conditionally required options:

Option

Description

--cluster_file PATH

Cluster JSON: nodes, SSH credentials, and the orchestrator key. Suites differ on whether they follow that key; see Cluster Validation Suite (CVS) cluster file: configuration and backend selection. Required unless cvs run is launched as a Slurm/SPUR job step; then CVS writes <run_dir>/cluster_agents.json and an optional file supplies extra metadata. See Run Cluster Validation Suite (CVS) on managed compute.

Optional options:

Option	Description
--workspace PATH	Shared-filesystem root; run dir is <workspace>/cvs_runs/<run_id>. Falls back to $CVS_WORKSPACE, then the venv parent directory. Every scheduler-managed run must set this or CVS_WORKSPACE to storage shared by all nodes. HTML and log reports default to this run directory.
--html PATH	Pytest HTML report path. Defaults to <run_dir>/<test-file-stem>.html (self-contained). Parent directories are created automatically.
--no-html	Do not pass --html to pytest (overrides auto-derive).
--self-contained-html	Embed CSS and images in the HTML report. Implied when the HTML path is auto-derived.
--log-file PATH	Text log file. Defaults to <run_dir>/<test-file-stem>.log. Parent directories are created automatically. Console logging is unaffected.
--no-log-file	Do not write a pytest log file.
--log-level LEVEL	DEBUG, INFO, WARNING, ERROR, CRITICAL.
--capture MODE	no, tee-sys, tee-merged, fd, sys.

Omit --html and --log-file to use the run-directory defaults. Pass either flag with a path to override. --no-html and --no-log-file suppress generation.

All other pytest flags pass through. Run pytest --help for the full list.

CVS_SCHEDULER overrides scheduler detection with spur, slurm or bare_metal. CVS_WORKSPACE supplies the workspace when --workspace is omitted. See Run Cluster Validation Suite (CVS) on managed compute for managed runs.

See Run Cluster Validation Suite (CVS) test suites on AMD Instinct GPU clusters.

cvs list

Positional argument: [test] — optional suite name.

Optional options:

Option	Description
--cluster_file PATH	Cluster file for parameterized test collection.
--config_file PATH	Config file for parameterized test collection.

With no arguments, lists all suites (same catalog as cvs run with no arguments). With a suite name, lists test functions in that suite.

See Run Cluster Validation Suite (CVS) test suites on AMD Instinct GPU clusters.

cvs monitor

Run cvs monitor with no arguments to list available monitors.

Subcommand: check_cluster_health

check_cluster_health accepts the following options.

Option	Description
--cluster_file PATH	Cluster JSON (recommended; takes precedence over CLUSTER_FILE).
--iterations N	Number of check iterations.
--time_between_iters SECONDS	Sleep between iterations.
--report_file PATH	Output HTML report path (default: cluster_report.html in the current directory).

Deprecated (use --cluster_file instead): --hosts_file, --username, --password, --key_file.

See Health reports and Live dashboards.

cvs exec

Run a shell command on all cluster nodes (or a selected subset) in parallel over SSH and print the output grouped by node.

Required options:

Option	Description
--cmd COMMAND	Shell command to run on selected nodes.

Optional options:

Option	Description
--cluster_file PATH	Cluster JSON (or set CLUSTER_FILE).
--target {computes,switches,all}	computes (default), switches, or both.
--timeout SECONDS	Per-node command output timeout (default: 30).
--connect-timeout SECONDS	Per-node SSH connect timeout (default: 15).
--json	Emit structured JSON on stdout.
--verbose, -v	Show SSH connection diagnostics.

See Run ad-hoc commands across all Cluster Validation Suite (CVS) cluster nodes over SSH.

cvs scp

Copy a local file or directory to the same path on every cluster node in parallel.

Required options:

Option	Description
--file PATH	Local file or directory to copy.

Optional options:

Option	Description
--dest PATH	Remote destination (defaults to the same path as the source).
--recurse	Copy directories recursively.
--cluster_file PATH	Cluster JSON (or set CLUSTER_FILE).
--parallel N	Parallel SCP operations (default: 20).

See Copy files to all cluster nodes using Cluster Validation Suite (CVS) parallel SCP for more information.