Generate AMD Instinct GPU cluster health reports with Cluster Validation Suite (CVS)

Generate AMD Instinct GPU cluster health reports with Cluster Validation Suite (CVS)#

2026-10-23

3 min read time

Applies to Linux

Health reports run cvs monitor check_cluster_health from the head node. CVS SSHs to each compute node, collects GPU/NIC counters and logs, and writes a self-contained HTML report. No agents or exporters are installed on the cluster.

The monitor identifies hardware degradation (RAS, PCIe/XGMI, RDMA counters through AMD SMI) and software failures (dmesg and journalctl signatures). Use it before a test campaign, after a change, or to snapshot counters while workloads run and compare deltas across nodes.

Generate a health report#

  1. Complete Install CVS and Set up a cluster file.

  2. List available monitors:

    cvs monitor

  3. View help:

    cvs monitor check_cluster_health --help

  4. Run the monitor:

    Option

    Description

    --cluster_file PATH

    Cluster JSON (same file as cvs run or cvs exec). Takes precedence over CLUSTER_FILE.

    --iterations N

    Number of sampling iterations

    --time_between_iters SECONDS

    Sleep between iterations

    --report_file PATH

    Output HTML path (default: cluster_report.html in the current directory)

    Example:

    cvs monitor check_cluster_health \
  --cluster_file ~/cvs_workspace/cluster.json \
  --iterations 2

    Or set CLUSTER_FILE once:

    export CLUSTER_FILE=~/cvs_workspace/cluster.json
cvs monitor check_cluster_health --iterations 2

    Note

    Deprecated flags --hosts_file, --username, --key_file, and --password still work; prefer --cluster_file for consistency with other CVS commands.

  5. Open cluster_report.html in a browser.

Review the health report#

The report includes GPU and NIC snapshots, historic error logs, and per-iteration deltas for triage. Anomalies are highlighted in tables (PCIe, RDMA, GPU errors, cable issues, kernel logs).

RDMA counter table in the CVS cluster health HTML report, showing per-node RDMA statistics PCIe status table in the CVS cluster health HTML report, showing per-node PCIe link status journalctl kernel log entries in the CVS cluster health HTML report, highlighting error signatures

Metrics are collected with AMD SMI, ethtool, and rdma utilities on each node. See the AMD SMI CLI reference for field definitions.

For live dashboards instead of one-off HTML reports, see View live AMD Instinct GPU cluster dashboards with Cluster Validation Suite (CVS).

Contents