.. meta:: :description: Reference for the CVS cluster file JSON format, including SSH credentials, node topology, baremetal and container backend selection, and Docker runtime options. :keywords: CVS, cluster file, JSON, baremetal, container, Docker, orchestrator, SSH, ROCm, RDMA, node topology *************************************************************************** Cluster Validation Suite (CVS) cluster file: configuration and backend selection *************************************************************************** Each ``cvs run`` invocation is pointed at a cluster file via ``--cluster_file ``. The cluster file declares the SSH credentials, the node list, and the **execution backend** that runs the workload. CVS ships two starter templates: .. list-table:: :widths: 3 2 6 :header-rows: 1 * - Template - Backend - Use when * - ``cluster.json`` - ``baremetal`` (default) - ROCm, RVS, RCCL, and other workload binaries are installed on the host filesystem. * - ``cluster_container.json`` - ``container`` - The workload runs inside a long-lived per-host container; the host only needs Docker, GPUs, and SSH. Copy the template that matches your cluster shape, edit the placeholders, and pass it to ``cvs run`` and ``cvs exec``. The choice of backend is made entirely in the cluster file. The ``cvs run`` invocation does not change. Ensure :doc:`/reference/cluster/passwordless-ssh` is configured before running CVS against the cluster. Backends ======== .. include:: /_includes/orchestrator-scope.rst Baremetal --------- Baremetal is the default. Workload commands are sent over SSH to each node and executed directly on the host filesystem. This is the path that has always been used by CVS and is consumed by every existing test suite. Container --------- Container mode routes workload commands through a container runtime on each node. CVS uses host SSH to the node, then ``docker exec`` into a long-lived per-host container. Inside the container, an SSH daemon listens on port ``2224`` so that MPI-style workloads can fan out using the in-container SSH transport. The container backend is currently consumed by ``rvs_cvs`` and ``install_rvs``. The container lifecycle (start, verify, sshd setup, teardown) runs from the test fixture in `cvs/tests/health/rvs_cvs.py `_. Cluster file shape ================== Both templates share the same top-level shape. The ``container`` block and the ``orchestrator`` key are only meaningful in container mode. .. note:: In the cluster file, ``{user-id}`` resolves to the current login user at runtime. Replace it with a literal username if you want to pin it. The placeholders ``{xx.xx.xx.xx|hostname-N}`` are example placeholders for real IPs or hostnames. .. dropdown:: ``cluster_container.json`` .. code:: json { "orchestrator": "container", "username": "{user-id}", "priv_key_file": "/home/{user-id}/.ssh/id_rsa", "head_node_dict": { "mgmt_ip": "{xx.xx.xx.xx|hostname-1}" }, "env_vars": {}, "node_dict": { "{xx.xx.xx.xx|hostname-1}": { "bmc_ip": "NA", "vpc_ip": "{xx.xx.xx.xx|hostname-1}" }, "{xx.xx.xx.xx|hostname-2}": { "bmc_ip": "NA", "vpc_ip": "{xx.xx.xx.xx|hostname-2}" } }, "container": { "lifetime": "per_run", "image": "rocm/cvs:latest", "name": "cvs_container", "runtime": { "name": "docker", "args": { "network": "host", "ipc": "host", "privileged": true } } } } Top-level parameters -------------------- The following table describes every key accepted at the top level of the cluster file. .. list-table:: :widths: 3 3 5 :header-rows: 1 * - Configuration parameter - Default value - Description * - ``orchestrator`` - ``baremetal`` - Execution backend. Set to ``container`` to route workload commands through the container runtime. Honored today only by ``rvs_cvs`` and ``install_rvs``; see the Backends section above. * - ``username`` - ``{user-id}`` - SSH username for all hosts. ``{user-id}`` resolves to the current login user at runtime. * - ``priv_key_file`` - ``/home/{user-id}/.ssh/id_rsa`` - Absolute path to the SSH private key used for every host. * - ``head_node_dict.mgmt_ip`` - (required) - Head node management IP or hostname (the host where you run the CVS CLI). This can be one of the keys in ``node_dict`` (usually the first node) or a completely separate host that is not in ``node_dict``. See :doc:`/install/install`. * - ``env_vars`` - ``{}`` - Custom environment variables exported on every host before each command. Honored by the legacy parallel-SSH path; ``cvs exec`` does not export this block. * - ``node_dict`` - (required) - Cluster member nodes keyed by public IP or hostname. Each value is ``{"bmc_ip": "NA", "vpc_ip": "..."}``. ``vpc_ip`` is the address reachable from other nodes; set it equal to the public IP if there is no separate VPC. * - ``container`` - ``{}`` - Container backend configuration. Required when ``orchestrator`` is ``container``. See the next section. CVS's own internal commands -- the Docker CLI calls made by ``DockerRuntime`` (``docker run``/``exec``/``rm``/``ps``/``load``) and the MPI hostfile cleanup in ``BaremetalOrchestrator`` -- automatically detect whether ``sudo`` is needed. Once per run, CVS probes each host with ``sudo -n true`` and caches whether passwordless sudo is available; every subsequent privileged command is then prefixed with ``sudo -n `` or left unprefixed accordingly, for the lifetime of that run. No cluster-file configuration is required. Container block =============== The ``container`` block configures the container backend. It is consumed by the ``ContainerOrchestrator`` defined in `cvs/core/orchestrators/container.py `_. The following table describes each key in the ``container`` block. .. list-table:: :widths: 3 3 5 :header-rows: 1 * - Configuration parameter - Default value - Description * - ``lifetime`` - ``per_run`` - Container lifecycle policy: ``no_launch``, ``per_run``, or ``persistent``. See the truth table below. * - ``image`` - (required) - Image with the test dependencies (for example ``rvs``) pre-installed and an ``sshd`` you can start on port ``2224``. Must be present locally on each node or pullable from a reachable registry. * - ``name`` - (required) - Container name on each host. For parallel runs, make this per-iteration unique (for example ``cvs_iter_``). * - ``runtime.name`` - ``docker`` - Container runtime. Today only ``docker`` is implemented. ``enroot`` is registered as a stub and is not yet functional. * - ``runtime.args`` - ``{}`` - Backend-specific runtime arguments (see the next section). Docker ``runtime.args`` reference ================================= When ``runtime.name`` is ``docker``, the keys below configure the underlying ``docker run`` command. Defaults are merged from ``DEFAULT_CONTAINER_ARGS`` in `cvs/core/orchestrators/container.py `_: - List arguments (``volumes``, ``devices``, ``cap_add``, ``security_opt``, ``group_add``, ``ulimit``) **append to** the baked-in defaults. - Scalar arguments (``network``, ``ipc``, ``privileged``) **override** the default when set, otherwise inherit it. - An empty ``args: {}`` already yields a working RDMA-ready container. The keys below are only needed to extend or override. The following table lists the supported ``runtime.args`` keys and their defaults. .. list-table:: :widths: 3 3 5 :header-rows: 1 * - Argument - Default value - Description * - ``network`` - ``host`` - ``--network`` mode. ``host`` for clusters sharing the host network stack. * - ``ipc`` - ``host`` - ``--ipc`` mode. ``host`` enables cross-process IPC required for RDMA. * - ``privileged`` - ``true`` - ``--privileged``. Required for device passthrough and RDMA. * - ``volumes`` - Appended - List of ``host:container[:ro]`` mounts. The container always also receives ``/home/$user/.ssh:/host_ssh`` injected by the orchestrator. * - ``devices`` - ``["/dev/kfd", "/dev/dri", "/dev/infiniband"]`` (appended) - Device passthroughs. Per-host ``/dev/infiniband/*`` is also discovered at runtime. * - ``cap_add`` - ``["SYS_PTRACE", "IPC_LOCK", "SYS_ADMIN"]`` (appended) - Linux capabilities. * - ``security_opt`` - ``["seccomp=unconfined", "apparmor=unconfined"]`` (appended) - Security profile relaxations needed for RDMA and ptrace. * - ``group_add`` - ``["video"]`` (appended) - Supplementary groups inside the container. * - ``ulimit`` - ``["memlock=-1"]`` (appended) - Per-process resource limits. ``memlock=-1`` is required for RDMA. ``lifetime`` truth table ======================== ``setup_containers`` and ``teardown_containers`` branch on ``container.lifetime``. The behavior below is pinned by the per-lifetime unit tests in `cvs/core/orchestrators/unittests/test_container.py `_. The following table shows what each ``lifetime`` value does during setup and teardown. .. list-table:: :widths: 2 4 4 :header-rows: 1 * - ``lifetime`` - ``setup_containers`` - ``teardown_containers`` * - ``no_launch`` - Verify a container with the configured name is already running on every host; set ``container_id``. Never starts anything. - No-op. CVS does not own a container it did not launch. * - ``per_run`` (default) - Start a fresh container on every host (force-removing any stale same-named container first). - Force-remove the container CVS started. * - ``persistent`` - Attach if the container is already running on every host. Start fresh only if it is running on no host. Running on some hosts but not all is a hard error (CVS will not force-remove the still-running hosts and destroy their overlay). Idempotent across runs. - No-op. The container is left running for the next run; remove it yourself when done. .. note:: With ``lifetime: persistent``, pin ``container.name`` explicitly. The default ``_`` name shifts when you bump the image tag, silently abandoning the previous container's overlay. Prerequisites on each cluster node ================================== The following prerequisites must be satisfied on every cluster node before using the container backend. To use the container backend, every cluster node must have: - **Docker installed**. The SSH user needs either passwordless ``sudo docker`` or direct Docker access (for example membership in the ``docker`` group) -- CVS probes once per run (``sudo -n true``) and caches which applies, prefixing every subsequent Docker command accordingly. - **Host driver loaded** so ``/dev/kfd``, ``/dev/dri/*``, and ``/dev/infiniband/*`` (when RDMA is in scope) are present for passthrough. - **SSH user home directory accessible**. The orchestrator mounts ``~/.ssh`` as ``/host_ssh`` and copies keys into ``/root/.ssh`` inside the container so that the in-container ``sshd`` on port ``2224`` can authenticate. - **Container image** either pre-loaded on every node (``docker load``) or pullable from a reachable registry. The image must contain ``openssh-server`` (for the in-container ``sshd``) and the workload binaries the suite invokes (for example ``/opt/rocm/bin/rvs``). Related resources ================= - :doc:`/how-to/run-with-containers` - task-oriented walkthrough for running ``rvs_cvs`` in container mode. - `cvs/input/cluster_file/README.md `_ - in-tree reference next to the templates.