Run Cluster Validation Suite (CVS) test suites with a per-host Docker container backend

Run Cluster Validation Suite (CVS) test suites with a per-host Docker container backend#

2026-10-27

10 min read time

Applies to Linux

Some CVS suites run workload commands inside a per-host container. Use that path when you want the test to see the image you ship, or when the host should keep only Docker, the GPU driver, and SSH. container.lifetime defaults to per_run: CVS starts the container at setup and removes it at teardown.

--cluster_file carries orchestrator (baremetal or container). Write those values in lowercase. The factory accepts any case, and the fixtures that start a container compare against the lowercase string container.

Suites do not all follow the key.

What the suite does

Suites

Honors the cluster key. With container, the fixture starts the container and the workload runs inside it (docker exec). On more than one node that fixture also starts sshd on port 2224. RCCL host probes (ROCm, RDMA, firewall, dmesg) stay on the host.

  • rvs_cvs

  • install_rvs

  • agfhc_cvs

  • install_agfhc

  • transferbench_cvs

  • install_transferbench

  • csp_qual_agfhc

  • rccl_perf

  • rccl_regression

  • rccl_pairwise

Honors the key and starts that same container (and sshd on more than one node), then runs its own checks on the host. install_ompi builds on the host, and its UCX ROCm-path probe also runs on the host using orch.head or orch.all.

  • preflight_checks

  • host_configs_cvs

  • install_ompi

Honors the key and skips the suite when the value is container. Baremetal commands run on the host.

  • anc_installation

  • anc_test_* suites

Honors the cluster key unless the MORI config sets orchestrator. A test starts the container; the benchmark then runs inside it. sshd is not started. Dmesg stays on the host.

mori_benchmark_test

Honors the cluster key. A first test step starts the container; GPU and NIC discovery, the perftest build and perftest then run inside it. sshd is not started. Dmesg and AFM vPOD discovery stay on the host.

ib_perf_bw_test, install_ibperf_tools

Forces container and ignores the cluster key. The workload runs inside the container. vLLM does not start sshd. ATOM does. Aorta, SGLang, xDiT, Megatron (megatron_single / megatron_distributed), TorchTitan, and JAX MaxText launch the container and do not call sshd setup.

  • aorta_single

  • aorta_distributed

  • vllm_single

  • vllm_distributed

  • sglang_single

  • sglang_distributed

  • sglang_disagg_distributed

  • xdit_* suites

  • atom

  • megatron_single

  • megatron_distributed

  • torchtitan_single

  • torchtitan_distributed

  • jaxmaxtext_single

  • jaxmaxtext_distributed

Ignores orchestrator. The command runs on the host over SSH.

  • cvs exec

  • cvs scp

Prerequisites#

On every cluster node:

  • Docker installed. The SSH user needs passwordless sudo docker or direct Docker access (for example membership in the docker group). CVS probes sudo -n true once per orchestrator and caches the first node_dict key’s answer. See Cluster Validation Suite (CVS) cluster file: configuration and backend selection.

  • Host driver loaded so /dev/kfd, /dev/dri/*, and /dev/infiniband/* (when RDMA is in scope) are present for passthrough.

  • Launch-user SSH directory at /home/<user running cvs>/.ssh when the suite starts in-container sshd. CVS mounts that path at /host_ssh.

  • Container image already loaded, given as image_tar on each node, or pullable. A suite that starts sshd on more than one node needs /usr/sbin/sshd. The image also needs the binaries that suite runs.

On the machine where you launch cvs run:

That machine is the launch host. It can be outside node_dict. It is not head_node_dict.mgmt_ip, and it is not automatically the orchestrator head. The orchestrator head is the first node_dict key.

Step 1: Copy the cluster template#

CVS ships a cluster_container.json template alongside the baremetal cluster.json. Copy it to a working location:

cvs config copy cluster_container.json --output ~/cvs_workspace/cluster_container.json

You can browse every available template directory with:

cvs config list-dirs

Step 2: Edit the placeholders#

Replace every <changeme> placeholder in the copied file. A suite that loads the cluster file exits if any remain. Key fields to set:

  • username and priv_key_file. {user-id} expands to the user running cvs run. cvs exec and cvs scp leave that text literal, so use a real username in a file you pass to those commands.

  • node_dict keys: real IPs or hostnames. The first key is the orchestrator head.

  • container.image: required. If any host lacks the image, CVS loads image_tar (or logs in with runtime.args.registry) and pulls image across all hosts.

  • container.name: optional. The default is <user running cvs>_<image>, with characters outside [A-Za-z0-9_] replaced by _. Pin it for lifetime: persistent and when two runs share a host.

  • container.lifetime: no_launch, per_run, or persistent. See the lifecycle note below.

  • container.env: variables inside the container (docker -e). Cluster env_vars is a host-shell export prefix, including on the docker commands themselves, and is not copied into the container.

For the full schema and runtime argument reference, see Cluster Validation Suite (CVS) cluster file: configuration and backend selection.

Step 3: Run a suite that follows the cluster key#

rvs_cvs follows orchestrator in the cluster file. Run it the same way you run any CVS test suite, with the container cluster file:

cvs run rvs_cvs \
    --cluster_file ~/cvs_workspace/cluster_container.json \
    --config_file ~/cvs_workspace/health/health_config.json \
    --capture=tee-sys -vvv -s

The same cluster file applies to the other health suites on Run AGFHC, TransferBench, and RVS health burn-in tests in Cluster Validation Suite (CVS) (AGFHC, TransferBench, and their installers). Suites that force a container, or that stay on the host, are listed in the scope table above.

What happens during an rvs_cvs run with orchestrator: container:

  • The shared fixture reads the cluster file and builds a container orchestrator.

  • With lifetime: per_run, CVS removes any container with the same name on each host and starts the image with the merged runtime.args. The container’s command is sleep infinity. Workload commands are docker exec into it.

  • With lifetime: persistent, CVS attaches to a container already running on every host, or starts one if none is running.

  • With lifetime: no_launch, CVS checks that a container with the resolved name is already running on every host and reuses it.

  • On more than one node, that fixture also starts sshd on port 2224. A single node skips sshd. rvs itself is docker exec, not an SSH connection to port 2224.

Step 4: Verify container execution#

Confirm that the workload ran inside the container. From any node, list running containers with the configured name:

ssh <user>@<node> sudo docker ps --filter name=^cvs_container$

You should see one running container per node with the configured name and image while the run is in per_run setup, or after the run when lifetime is persistent or no_launch. You can fan this check out from the launch host:

cvs exec --cluster_file ~/cvs_workspace/cluster_container.json \
    --cmd "sudo docker ps --filter name=^cvs_container$ --format '{{.Names}} {{.Image}}'"

cvs exec SSHes to each host and ignores orchestrator, so this shows the host’s docker ps. A per_run container is already removed when the suite has finished teardown. Check during the run, or use persistent or no_launch if you need the container afterwards.

docker exec and port 2224#

Ordinary container commands are docker exec <name> bash -c <cmd> over the host connection. Port 2224 is a second channel. CVS starts /usr/sbin/sshd -p2224 inside the container so mpirun can reach peer ranks. Setup copies /host_ssh to /root/.ssh first.

CVS skips that setup when the cluster has one node, and on a Slurm or SPUR job step. The health suites, RCCL, host_configs_cvs, install_ompi, preflight_checks, and ATOM call it. vLLM skips it. Aorta, SGLang, xDiT, megatron_single, megatron_distributed, TorchTitan, JAX MaxText, and MORI do not call it. With network: host, port 2224 is in the host network namespace on the suites that do start sshd.

Lifecycle and teardown#

container.lifetime decides who owns the container:

  • per_run (default) — CVS starts a fresh container at setup and force-removes it at teardown. Writes to the container overlay are gone when the run ends.

  • persistent — CVS attaches when the container is already running on every host, or starts it when it is running on none. Teardown leaves it in place, so the overlay survives the run.

    • You can run cvs run install_rvs and then cvs run rvs_cvs as separate invocations and keep the install.

    • If the container is running on some hosts and missing on others, CVS stops. A relaunch would remove the containers that are still up. Remove container.name on every host and rerun, or start it on the missing hosts.

    • Pin container.name. The default name changes with the image tag.

  • no_launch — CVS starts nothing. It checks that the resolved name is already running on every host. Teardown leaves it in place.

The full setup/teardown matrix is in Cluster Validation Suite (CVS) cluster file: configuration and backend selection.

To remove a container CVS left running (persistent or no_launch):

cvs exec --cluster_file ~/cvs_workspace/cluster_container.json \
    --cmd "sudo docker rm -f cvs_container"

Replace cvs_container with your container.name. cvs exec runs that on the host.

Common pitfalls#

  • Running a multi-node suite that starts sshd against an image without /usr/sbin/sshd fails at the sshd setup stage. Single-node runs skip that step, and suites that never call sshd setup do not require it in the image.

  • An image that is missing the workload binary fails the test. For example, rvs_cvs runs rvs inside the container, so /opt/rocm/bin/rvs must be present.

  • Using lifetime: persistent without a pinned name causes the container name to follow the image tag. When the tag changes, the previous container stays running and a new one is started alongside it.

  • If port 2224 is already in use on the host, suites that start sshd with network: host will fail to bind it. Stop the conflicting listener before running the suite.

  • Setting orchestrator: baremetal in the cluster file for a suite that honors that key causes the suite to run on the host instead of in a container. Suites that force container ignore the key and always start a container; see the scope table.

  • Passing {user-id} to cvs exec uses the literal string {user-id} as the SSH username rather than substituting the current user.

Contents