Cluster Validation Suite (CVS) cluster file: configuration and backend selection

2026-09-24

9 min read time

Applies to Linux

Each cvs run invocation is pointed at a cluster file via --cluster_file <path>. The cluster file declares the SSH credentials, the node list, and the execution backend that runs the workload. CVS ships two starter templates:

Template	Backend	Use when
cluster.json	baremetal (default)	ROCm, RVS, RCCL, and other workload binaries are installed on the host filesystem.
cluster_container.json	container	The workload runs inside a long-lived per-host container; the host only needs Docker, GPUs, and SSH.

Copy the template that matches your cluster shape, edit the placeholders, and pass it to cvs run and cvs exec. The choice of backend is made entirely in the cluster file. The cvs run invocation does not change.

Ensure Set up passwordless SSH for Cluster Validation Suite (CVS) cluster nodes is configured before running CVS against the cluster.

Backends

Note

Current scope: Of the test suites shipped with CVS, only rvs_cvs and install_rvs consume the orchestrator and honor the orchestrator key in the cluster file. All other cvs run test suites and the cvs exec CLI run on the host regardless of the orchestrator value. Migrating additional suites to the orchestrator is tracked separately. Custom Python scripts can use the OrchestratorFactory API directly as an escape hatch.

Baremetal

Baremetal is the default. Workload commands are sent over SSH to each node and executed directly on the host filesystem. This is the path that has always been used by CVS and is consumed by every existing test suite.

Container

Container mode routes workload commands through a container runtime on each node. CVS uses host SSH to the node, then docker exec into a long-lived per-host container. Inside the container, an SSH daemon listens on port 2224 so that MPI-style workloads can fan out using the in-container SSH transport.

The container backend is currently consumed by rvs_cvs and install_rvs. The container lifecycle (start, verify, sshd setup, teardown) runs from the test fixture in cvs/tests/health/rvs_cvs.py.

Cluster file shape

Both templates share the same top-level shape. The container block and the orchestrator key are only meaningful in container mode.

Note

In the cluster file, {user-id} resolves to the current login user at runtime. Replace it with a literal username if you want to pin it. The placeholders {xx.xx.xx.xx|hostname-N} are example placeholders for real IPs or hostnames.

cluster_container.json

{
    "orchestrator": "container",
    "username": "{user-id}",
    "priv_key_file": "/home/{user-id}/.ssh/id_rsa",

    "head_node_dict": {
        "mgmt_ip": "{xx.xx.xx.xx|hostname-1}"
    },

    "env_vars": {},

    "node_dict": {
        "{xx.xx.xx.xx|hostname-1}": {
            "bmc_ip": "NA",
            "vpc_ip": "{xx.xx.xx.xx|hostname-1}"
        },
        "{xx.xx.xx.xx|hostname-2}": {
            "bmc_ip": "NA",
            "vpc_ip": "{xx.xx.xx.xx|hostname-2}"
        }
    },

    "container": {
        "lifetime": "per_run",
        "image": "rocm/cvs:latest",
        "name": "cvs_container",
        "runtime": {
            "name": "docker",
            "args": {
                "network": "host",
                "ipc": "host",
                "privileged": true
            }
        }
    }
}

Top-level parameters

The following table describes every key accepted at the top level of the cluster file.

Configuration parameter	Default value	Description
orchestrator	baremetal	Execution backend. Set to container to route workload commands through the container runtime. Honored today only by rvs_cvs and install_rvs; see the Backends section above.
username	{user-id}	SSH username for all hosts. {user-id} resolves to the current login user at runtime.
priv_key_file	/home/{user-id}/.ssh/id_rsa	Absolute path to the SSH private key used for every host.
head_node_dict.mgmt_ip	(required)	Head node management IP or hostname (the host where you run the CVS CLI). This can be one of the keys in node_dict (usually the first node) or a completely separate host that is not in node_dict. See Install Cluster Validation Suite (CVS) on ROCm.
env_vars	{}	Custom environment variables exported on every host before each command. Honored by the legacy parallel-SSH path; cvs exec does not export this block.
node_dict	(required)	Cluster member nodes keyed by public IP or hostname. Each value is {"bmc_ip": "NA", "vpc_ip": "..."}. vpc_ip is the address reachable from other nodes; set it equal to the public IP if there is no separate VPC.
container	{}	Container backend configuration. Required when orchestrator is container. See the next section.

CVS’s own internal commands – the Docker CLI calls made by DockerRuntime (docker run/exec/rm/ps/load) and the MPI hostfile cleanup in BaremetalOrchestrator – automatically detect whether sudo is needed. Once per run, CVS probes each host with sudo -n true and caches whether passwordless sudo is available; every subsequent privileged command is then prefixed with ``sudo -n `` or left unprefixed accordingly, for the lifetime of that run. No cluster-file configuration is required.

Container block

The container block configures the container backend. It is consumed by the ContainerOrchestrator defined in cvs/core/orchestrators/container.py.

The following table describes each key in the container block.

Configuration parameter	Default value	Description
lifetime	per_run	Container lifecycle policy: no_launch, per_run, or persistent. See the truth table below.
image	(required)	Image with the test dependencies (for example rvs) pre-installed and an sshd you can start on port 2224. Must be present locally on each node or pullable from a reachable registry.
name	(required)	Container name on each host. For parallel runs, make this per-iteration unique (for example cvs_iter_<run_id>).
runtime.name	docker	Container runtime. Today only docker is implemented. enroot is registered as a stub and is not yet functional.
runtime.args	{}	Backend-specific runtime arguments (see the next section).

Docker runtime.args reference

When runtime.name is docker, the keys below configure the underlying docker run command. Defaults are merged from DEFAULT_CONTAINER_ARGS in cvs/core/orchestrators/container.py:

• List arguments (volumes, devices, cap_add, security_opt, group_add, ulimit) append to the baked-in defaults.

• Scalar arguments (network, ipc, privileged) override the default when set, otherwise inherit it.

• An empty args: {} already yields a working RDMA-ready container. The keys below are only needed to extend or override.

The following table lists the supported runtime.args keys and their defaults.

Argument	Default value	Description
network	host	--network mode. host for clusters sharing the host network stack.
ipc	host	--ipc mode. host enables cross-process IPC required for RDMA.
privileged	true	--privileged. Required for device passthrough and RDMA.
volumes	Appended	List of host:container[:ro] mounts. The container always also receives /home/$user/.ssh:/host_ssh injected by the orchestrator.
devices	["/dev/kfd", "/dev/dri", "/dev/infiniband"] (appended)	Device passthroughs. Per-host /dev/infiniband/* is also discovered at runtime.
cap_add	["SYS_PTRACE", "IPC_LOCK", "SYS_ADMIN"] (appended)	Linux capabilities.
security_opt	["seccomp=unconfined", "apparmor=unconfined"] (appended)	Security profile relaxations needed for RDMA and ptrace.
group_add	["video"] (appended)	Supplementary groups inside the container.
ulimit	["memlock=-1"] (appended)	Per-process resource limits. memlock=-1 is required for RDMA.

lifetime truth table

setup_containers and teardown_containers branch on container.lifetime. The behavior below is pinned by the per-lifetime unit tests in cvs/core/orchestrators/unittests/test_container.py.

The following table shows what each lifetime value does during setup and teardown.

lifetime	setup_containers	teardown_containers
no_launch	Verify a container with the configured name is already running on every host; set container_id. Never starts anything.	No-op. CVS does not own a container it did not launch.
per_run (default)	Start a fresh container on every host (force-removing any stale same-named container first).	Force-remove the container CVS started.
persistent	Attach if the container is already running on every host. Start fresh only if it is running on no host. Running on some hosts but not all is a hard error (CVS will not force-remove the still-running hosts and destroy their overlay). Idempotent across runs.	No-op. The container is left running for the next run; remove it yourself when done.

Note

With lifetime: persistent, pin container.name explicitly. The default <user>_<sanitized_image> name shifts when you bump the image tag, silently abandoning the previous container’s overlay.

Prerequisites on each cluster node

The following prerequisites must be satisfied on every cluster node before using the container backend.

To use the container backend, every cluster node must have:

• Docker installed. The SSH user needs either passwordless sudo docker or direct Docker access (for example membership in the docker group) – CVS probes once per run (sudo -n true) and caches which applies, prefixing every subsequent Docker command accordingly.

• Host driver loaded so /dev/kfd, /dev/dri/*, and /dev/infiniband/* (when RDMA is in scope) are present for passthrough.

• SSH user home directory accessible. The orchestrator mounts ~/.ssh as /host_ssh and copies keys into /root/.ssh inside the container so that the in-container sshd on port 2224 can authenticate.

• Container image either pre-loaded on every node (docker load) or pullable from a reachable registry. The image must contain openssh-server (for the in-container sshd) and the workload binaries the suite invokes (for example /opt/rocm/bin/rvs).

Related resources

• Run Cluster Validation Suite (CVS) test suites with a per-host Docker container backend - task-oriented walkthrough for running rvs_cvs in container mode.

• cvs/input/cluster_file/README.md - in-tree reference next to the templates.