.. meta:: :description: Run AGFHC, TransferBench, and RVS health burn-in tests to validate GPU hardware, firmware, and memory bandwidth on AMD Instinct clusters with CVS. :keywords: CVS, health, burn-in, AGFHC, RVS, TransferBench, AMD Instinct, ROCm, AMD, GPU, HBM, PCIe **************************************************************************************** Run AGFHC, TransferBench, and RVS health burn-in tests in Cluster Validation Suite (CVS) **************************************************************************************** Health burn-in tests run AMD GPU Field Health Check (AGFHC), `TransferBench `_, and `ROCm Validation Suite (RVS) `_ on every node in the cluster. Use these tests to validate GPU hardware, memory bandwidth, and compute throughput before production workloads. .. _health-set-up-config: Set up config ============= 1. Copy the health configuration file: .. code:: bash cvs config copy health/health_config.json --output ~/cvs_workspace/health/health_config.json 2. Edit paths and any remaining ```` placeholders: .. list-table:: :header-rows: 1 :widths: 20 30 50 * - Section - Fields to set - Notes * - ``agfhc`` - ``path``, ``package_tar_ball``, ``install_dir``, ``env``, ``log_dir`` - A ``path`` other than ``/opt/amd/agfhc`` installs with ``./install --rocm-tar -o`` and no sudo; its parent and ``log_dir`` must be writable on the nodes. Update ``env`` when ``path`` changes. * - ``transferbench`` - ``git_install_path``, ``rocm_path`` - Leave ``rocm_path`` as ```` to auto-detect. For ``test_transfer_bench_a2a``, set ``results.gpu_to_gpu_a2a_per_link_min`` (default ``32.9``). For ``test_transfer_bench_a2asweep``, set ``a2asweep_timeout`` (default ``1500``) or omit it to derive the timeout from sweep size. See :doc:`/reference/configuration-files/burn-in-diag/health`. * - ``rvs`` - ``git_install_path``, ``path``, ``rocm_path`` - For the complete field reference including install paths and threshold values, see :doc:`/reference/configuration-files/burn-in-diag/health`. .. _health-run-tests: Run tests ========= Use these tests to validate single-node diagnostics for GPU cluster health. For the performance validation, use the reference bandwidth or latency numbers provided as part of the input ``config_file`` for the relevant test. Use these scripts to run each health test. These CVS test scripts have two parts: installing the functionality and running the tests. .. note:: These health suites honor ``orchestrator`` in the cluster file. With ``container``, the workload runs inside the per-host container (``docker exec``). See :doc:`/how-to/run-with-containers`. AGFHC ~~~~~ See the `AGFHC (AMD GPU Field Health Check) `_ docs for more information. You can list all available AGFHC test cases using the CLI: .. code:: bash cvs list agfhc_cvs .. code:: text Available tests in agfhc_cvs: - test_agfhc_hbm - test_agfhc_hbm1_lvl5 - test_agfhc_hbm2_lvl5 - test_agfhc_hbm3_lvl3 - test_agfhc_dma_all_lvl1 - test_agfhc_dma_lvl1 - test_agfhc_gfx_lvl1 - test_agfhc_pcie_lvl1 - test_agfhc_pcie_lvl3 - test_agfhc_xgmi_lvl1 - test_agfhc_all_perf - test_agfhc_all_lvl5 Run the AGFHC test: 1. Run the installation: .. code:: bash cvs run install_agfhc \ --cluster_file ~/cvs_workspace/cluster.json \ --config_file ~/cvs_workspace/health/health_config.json \ --capture=tee-sys -vvv -s 2. Run the AGFHC test: .. code:: bash cvs run agfhc_cvs \ --cluster_file ~/cvs_workspace/cluster.json \ --config_file ~/cvs_workspace/health/health_config.json \ --capture=tee-sys -vvv -s 3. Run the CSP qualification test: .. code:: bash cvs run csp_qual_agfhc \ --cluster_file ~/cvs_workspace/cluster.json \ --config_file ~/cvs_workspace/health/health_config.json \ --capture=tee-sys -vvv -s TransferBench ~~~~~~~~~~~~~ See the `TransferBench `_ docs for more information. You can list all available TransferBench test cases using the CLI: .. code:: bash cvs list transferbench_cvs .. code:: text Available tests in transferbench_cvs: - test_transfer_bench_a2a - test_transfer_bench_p2p - test_transfer_bench_healthcheck - test_transfer_bench_a2asweep - test_transfer_bench_scaling - test_transfer_bench_schmoo Run the TransferBench test: 1. Run the installation: .. code:: bash cvs run install_transferbench \ --cluster_file ~/cvs_workspace/cluster.json \ --config_file ~/cvs_workspace/health/health_config.json \ --capture=tee-sys -vvv -s 2. Start the TransferBench test: .. code:: bash cvs run transferbench_cvs \ --cluster_file ~/cvs_workspace/cluster.json \ --config_file ~/cvs_workspace/health/health_config.json \ --capture=tee-sys -vvv -s RVS ~~~ See the `ROCm Validation Suite (RVS) `_ docs for more information. You can list all available RVS test cases using the CLI: .. code:: bash cvs list rvs_cvs .. code:: text Available tests in rvs_cvs: - test_rvs_level_config - test_rvs_gpu_enumeration - test_rvs_gpup_single - test_rvs_mem_test - test_rvs_gst_single - test_rvs_iet_single - test_rvs_pebb_single - test_rvs_pbqt_single - test_rvs_peqt_single - test_rvs_rcqt_single - test_rvs_tst_single - test_rvs_babel_stream Run the RVS test: 1. Run the installation: .. code:: bash cvs run install_rvs \ --cluster_file ~/cvs_workspace/cluster.json \ --config_file ~/cvs_workspace/health/health_config.json \ --capture=tee-sys -vvv -s 2. Start the RVS test: .. code:: bash cvs run rvs_cvs \ --cluster_file ~/cvs_workspace/cluster.json \ --config_file ~/cvs_workspace/health/health_config.json \ --capture=tee-sys -vvv -s Run on managed compute ~~~~~~~~~~~~~~~~~~~~~~ See :doc:`/how-to/run-on-managed-compute` for guidance on running these health suites on managed compute.