Run TorchTitan pre-training benchmarks with CVS

Run TorchTitan pre-training benchmarks with CVS#

2026-09-24

3 min read time

Applies to Linux

TorchTitan validates single-node and multi-node pre-training on AMD Instinct GPUs. CVS drives training inside a container, parses logs, and gates on throughput, loss-curve, and optional checkpoint metrics.

There are two suites:

  • torchtitan_single — single-node configs (framework: torchtitan_single)

  • torchtitan_distributed — multi-node configs (framework: torchtitan_distributed); adds RDMA setup

Set up config#

Follow these steps to set up the TorchTitan configuration.

  1. List available TorchTitan configuration files:

    cvs config list training/torchtitan

  2. Copy the configuration file you need, for example:

    cvs config copy training/torchtitan/mi355x_torchtitan_llama-3.1-8b_single.json --output ~/cvs_workspace/training/torchtitan/mi355x_torchtitan_llama-3.1-8b_single.json

  3. Replace every <changeme> with cluster-specific values (container image, HF token path, NCCL/RDMA fields on distributed configs).

  4. Edit the sibling *_threshold.json file if you gate on performance metrics.

  5. Set sweep.runs to the precision combos you want to execute.

Full parameter list: TorchTitan training configuration files for Cluster Validation Suite (CVS).

Run tests#

List available stages#

Run the following command to list available stages:

cvs list torchtitan_single

Available tests in torchtitan_single:
  - test_launch_container
  - test_download_tokenizer
  - test_smoke
  - test_training[...]
  - test_checkpoint[...]
  - test_metric[...]
  - test_loss_curve[...]
  - test_teardown

cvs list torchtitan_distributed

Available tests in torchtitan_distributed:
  - test_launch_container
  - test_setup_rdma
  - test_download_tokenizer
  - test_smoke
  - test_training[...]
  - test_checkpoint[...]
  - test_metric[...]
  - test_loss_curve[...]
  - test_teardown

Single-node example#

cvs run torchtitan_single \
  --cluster_file input/cluster_file/cluster.json \
  --config_file input/config_file/training/torchtitan/mi355x_torchtitan_llama-3.1-8b_single.json \
  --html ./logs/torchtitan_single.html --self-contained-html -vvv -s

Distributed example#

cvs run torchtitan_distributed \
  --cluster_file input/cluster_file/cluster.json \
  --config_file input/config_file/training/torchtitan/mi355x_torchtitan_llama-3.3-70b_distributed.json \
  --html ./logs/torchtitan_distributed.html --self-contained-html -vvv -s

Run a single stage#

cvs run torchtitan_single test_smoke \
  --cluster_file input/cluster_file/cluster.json \
  --config_file input/config_file/training/torchtitan/mi355x_torchtitan_llama-3.1-8b_single.json

Use a single-node config with torchtitan_single and a distributed config with torchtitan_distributed. The config framework field must match the suite name.

Contents