Run TorchTitan pre-training benchmarks with CVS

Run TorchTitan pre-training benchmarks with CVS#

2026-10-20

4 min read time

Applies to Linux

TorchTitan validates single-node and multi-node pre-training on AMD Instinct GPUs. CVS drives training inside a container, parses logs, and gates on throughput, loss-curve, and optional checkpoint metrics.

There are two suites:

  • torchtitan_single — single-node configs (framework: torchtitan_single)

  • torchtitan_distributed — multi-node configs (framework: torchtitan_distributed); adds RDMA library setup

Both suites run test_backend_nic_speed after the container launches. It checks that every discovered backend NIC is linked at 400 Gbps before tokenizer download or training. The distributed suite still runs test_setup_rdma after that check to copy the Broadcom RDMA library into the container on thor2 nodes.

Prerequisites#

In addition to the common training prerequisites:

  • The container image must provide TorchTitan at torchtitan_root. The shipped default is /workspace/torchtitan.

  • A Hugging Face token file at hf_token_file for the tokenizer download. The nodes need network access to reach Hugging Face.

Set up config#

Follow these steps to set up the TorchTitan configuration.

  1. List available TorchTitan configuration files:

    cvs config list training/torchtitan

  2. Copy the configuration file you need, for example:

    cvs config copy training/torchtitan/mi355x_torchtitan_llama-3.1-8b_single.json --output ~/cvs_workspace/training/torchtitan/mi355x_torchtitan_llama-3.1-8b_single.json

  3. Replace every <changeme> with cluster-specific values (container image, HF token path, NCCL/RDMA fields on distributed configs).

  4. Edit the sibling *_threshold.json file if you gate on performance metrics.

  5. Set sweep.runs to the precision combos you want to execute.

Full parameter list: TorchTitan training configuration files for Cluster Validation Suite (CVS).

Run tests#

List available stages#

Run the following command to list available stages:

cvs list torchtitan_single

Available tests in torchtitan_single:
  - test_launch_container
  - test_backend_nic_speed
  - test_download_tokenizer
  - test_smoke
  - test_training[...]
  - test_checkpoint[...]
  - test_metric[...]
  - test_loss_curve[...]
  - test_teardown

cvs list torchtitan_distributed

Available tests in torchtitan_distributed:
  - test_launch_container
  - test_backend_nic_speed
  - test_setup_rdma
  - test_download_tokenizer
  - test_smoke
  - test_training[...]
  - test_checkpoint[...]
  - test_metric[...]
  - test_loss_curve[...]
  - test_teardown

Single-node example#

cvs run torchtitan_single \
  --cluster_file input/cluster_file/cluster.json \
  --config_file input/config_file/training/torchtitan/mi355x_torchtitan_llama-3.1-8b_single.json \
  -vvv -s

Distributed example#

cvs run torchtitan_distributed \
  --cluster_file input/cluster_file/cluster.json \
  --config_file input/config_file/training/torchtitan/mi355x_torchtitan_llama-3.3-70b_distributed.json \
  -vvv -s

Run a single stage#

cvs run torchtitan_single test_smoke \
  --cluster_file input/cluster_file/cluster.json \
  --config_file input/config_file/training/torchtitan/mi355x_torchtitan_llama-3.1-8b_single.json

Use a single-node config with torchtitan_single and a distributed config with torchtitan_distributed. The config framework field must match the suite name.

Run on managed compute#

Follow Run Cluster Validation Suite (CVS) on managed compute to run TorchTitan on managed compute.

Contents