Preflight checks configuration file reference for Cluster Validation Suite (CVS)

Preflight checks configuration file reference for Cluster Validation Suite (CVS)#

2026-10-27

5 min read time

Applies to Linux

The preflight checks validate cluster health and configuration consistency before running performance tests, RCCL training, or inference workloads. Checks include GPU node health, optional MI4XX scale-up fabric admission, IFoE L2 and TransferBench gates, RDMA inventory and connectivity, and Node Smoke Tier 1, Tier 2, and Tier 3 Primus checks. The Node Smoke tiers run by default. Set connectivity_mode to "skip" (and tier2_perf to false for Tier 2) to disable one.

Configuration file location: cvs/input/config_file/preflight/preflight_config.json

For the full parameter reference, examples, and troubleshooting, see README_preflight_config.md in the repository.

Run preflight checks#

Use the following commands to run the full preflight suite or individual Node Smoke tiers.

cvs run preflight_checks \
  --cluster_file cluster.json \
  --config_file cvs/input/config_file/preflight/preflight_config.json

Run individual Node Smoke tiers:

cvs run preflight_checks test_node_smoke_tier1 \
  --cluster_file cluster.json \
  --config_file preflight_config.json

cvs run preflight_checks test_node_smoke_tier3 \
  --cluster_file cluster.json \
  --config_file preflight_config.json

Configuration structure#

preflight/
├── node_check/              # GPU visibility, ROCm, optional MI4XX fabric
├── connectivity_check/
│   ├── rdma/                # GID, interfaces, ibv_rc_pingpong connectivity
│   └── ifoe/                # L2 ping and TransferBench smoketest
├── node_smoke_tier1/        # Node Smoke Tier 1/2 (Primus node_smoke)
├── node_smoke_tier3/        # Node Smoke Tier 3 (Primus preflight --host --gpu --network)
├── reporting/               # HTML report and artifact paths
└── debug/                   # ScriptLet and troubleshooting options

Legacy keys node_smoke and tier3_info are normalized to node_smoke_tier1 and node_smoke_tier3 at load time.

Node Smoke tiers#

Node Smoke checks run by default. Set a tier’s connectivity_mode to "skip" to disable it, or set node_smoke_tier1.tier2_perf to false to disable Tier 2 only. Preflight reports each tier separately in the console summary and HTML report. The console count is the number of checks Primus reported, not a fixed catalog size.

Tier

Config

Primus command

Count (8 GPU)

Tier 1

node_smoke_tier1.connectivity_mode: "run" (default)

node_smoke (per node)

one row per GPU plus 7 collectors

Tier 2

node_smoke_tier1.tier2_perf: true (default)

node_smoke --tier2-perf (per node)

GEMM and HBM per GPU, plus local RCCL

Tier 3

node_smoke_tier3.connectivity_mode: "run" (default)

preflight --host --gpu --network (cluster)

host, GPU, and network groups

Tier 1 and Tier 2 counts are per node (the summary does not multiply by node count). Tier 3 is cluster-wide. A row is passed only when Primus reported that check. A node-level pass does not mark unreported checks as passed.

Example console output:

✅ Node Smoke Tier 1: PASS - 2/2 nodes passed Node Smoke Tier 1; 15 tests run per node
✅ Node Smoke Tier 2: PASS - 2/2 nodes passed Node Smoke Tier 2; 17 tests run per node
✅ Node Smoke Tier 3: PASS - 2/2 nodes passed Node Smoke Tier 3; 3 tests run cluster-wide

Sample configuration#

Note

In this configuration file, {user-id} is resolved to the current username at runtime.

Minimal Node Smoke Tier 1 + Tier 3
{
  "preflight": {
    "node_smoke_tier1": {
      "connectivity_mode": "run",
      "auto_setup": true,
      "primus_dir": "/home/{user-id}/INSTALL/Primus",
      "venv_activate": "/home/{user-id}/envs/preflight/.venv/bin/activate",
      "gpus_per_node": 8
    },
    "node_smoke_tier3": {
      "connectivity_mode": "run",
      "ssh_timeout": 600
    },
    "reporting": {
      "artifacts_root_dir": "/home/{user-id}/preflight"
    }
  }
}

Key parameters#

Node check (node_check)#

  • enabled — GPU visibility, AMDGPU/KFD, kernel health, ROCm validation

  • gpus_per_node — Expected GPU count on every node

  • expected_rocm_version — ROCm version string (must match amd-smi version)

RDMA connectivity (connectivity_check.rdma)#

  • connectivity_mode — "basic", "full_mesh", or "skip"

  • interfaces — Expected RDMA device names on all nodes

  • gid_index — GID index validated on those interfaces

  • gid_type — Expected sysfs GID type (default "RoCE v2"); "any" disables type checking, and InfiniBand link layers are exempt

  • mtu_check — Enable RDMA active MTU validation, plus jumbo-frame netdev MTU validation on RoCE ports (default true)

  • min_netdev_mtu / min_active_mtu — Minimum netdev MTU on RoCE ports (default 4200; InfiniBand ports are exempt) and active MTU on all ports (default 4096); 0 disables either threshold

  • ibv_test_timeout / ibv_test_port_range — ibv_rc_pingpong test tuning

Node Smoke Tier 1 (node_smoke_tier1)#

  • connectivity_mode — "run" or "skip" (default "run")

  • auto_setup — Clone Primus and create venv before running

  • primus_dir / venv_activate — Required when connectivity_mode is "run"

  • tier2_perf — Node Smoke Tier 2 perf sanity checks (default true)

  • gemm_tflops_min / hbm_gbs_min / rccl_gbs_min — Tier 2 thresholds

Node Smoke Tier 3 (node_smoke_tier3)#

  • connectivity_mode — "run" (default) or "skip" (independent of Tier 1)

  • primus_dir / venv_activate — Optional; empty inherits from Tier 1

  • dist_timeout_sec — torch.distributed init timeout

  • report_file_name — Base name for Primus markdown report (default node_smoke_tier3)

Contents