Run xDiT diffusion inference tests with Cluster Validation Suite (CVS)

Run xDiT diffusion inference tests with Cluster Validation Suite (CVS)#

2026-10-27

8 min read time

Applies to Linux

CVS provides five xDiT suites under cvs/tests/inference/xdit/. Each suite is a separate pytest module; pick the one that matches your topology and launcher, then point --config_file at a template from cvs/input/config_file/inference/xdit/.

Single-node suites run one independent docker+torchrun job on every node in the cluster file (full model on each node).
Distributed suites run one coordinated torchrun job across server_params.nnodes (nnodes >= 2), using the first that many cluster hosts, or server_node_list when set.

FLUX.1-dev and FLUX.2-dev share the xdit_flux_dev_* suites; choose the matching flux1 or flux2 JSON. Stage model weights on every participating node before the run. Packaged configs keep Hugging Face downloads off.

mi3xx_* templates target MI300X, MI325X, and MI350X. mi35x_* templates target MI355X, including Ionic RDMA examples for distributed runs. Both families point threshold_json at the same sibling threshold file for that workload.

Config reference: xDiT inference benchmark configuration for Cluster Validation Suite (CVS).

Test suites#

The following suites are available. Every suite runs the same nine stages.

CVS suite name	Source module	What it runs
`xdit_flux_dev_single`	`xdit_flux_dev_single.py`	FLUX.1 (`run_usp.py`) or FLUX.2 (`flux2_example.py`); one job per cluster node.
`xdit_flux_dev_distributed`	`xdit_flux_dev_distributed.py`	Unified FLUX.1 and FLUX.2 torchrun across `nnodes`.
`xdit_wan22_14b_single`	`xdit_wan22_14b_single.py`	WAN 2.2 I2V native (`/app/Wan2.2/run.py`); one job per cluster node.
`xdit_wan22_14b_diffusers_single`	`xdit_wan22_14b_diffusers_single.py`	WAN Diffusers xFuser (`wan_i2v_example.py`); one job per cluster node.
`xdit_wan22_14b_diffusers_distributed`	`xdit_wan22_14b_diffusers_distributed.py`	Unified WAN Diffusers xFuser torchrun across `nnodes`.

Set up config#

Follow these steps to set up the xDiT configuration.

List available xDiT templates:
```
cvs config list inference/xdit
```

Copy the workload JSON and its sibling threshold file into the same directory. FLUX.1-dev and FLUX.2-dev share the same suites; copy the matching flux1 or flux2 JSON. Use mi35x_* on MI355X.

cvs config copy inference/xdit/mi3xx_xdit_flux1_dev_single.json \
  --output ~/cvs_workspace/inference/xdit/mi3xx_xdit_flux1_dev_single.json

cvs config copy inference/xdit/xdit_flux1_dev_single_threshold.json \
  --output ~/cvs_workspace/inference/xdit/xdit_flux1_dev_single_threshold.json

cvs config copy inference/xdit/mi35x_xdit_flux1_dev_distributed.json \
  --output ~/cvs_workspace/inference/xdit/mi35x_xdit_flux1_dev_distributed.json

cvs config copy inference/xdit/xdit_flux1_dev_distributed_threshold.json \
  --output ~/cvs_workspace/inference/xdit/xdit_flux1_dev_distributed_threshold.json

Copy a cluster file (GPU compute nodes only):

cvs config copy cluster_container.json --output ~/cvs_workspace/cluster.json

Edit the config:
- Set container.image (_image_example names a known image for that template).
- Set server_params.model to a Hugging Face repo id or an absolute host path, and stage those weights before the run.
- Set gpu_name by removing <changeme> (templates hint mi325 or mi355).
- On distributed templates, replace every <changeme> NCCL/socket field and confirm the RDMA library bind mounts match the host.
- Resolve {user-id} and {home} or leave them for CVS to expand.
- Keep threshold_json as the sibling filename unless you point it at your own file.

Shipped config templates and the suites they target:

Config file	Use with suite
`mi3xx_xdit_flux1_dev_single.json` or `mi35x_xdit_flux1_dev_single.json`	`xdit_flux_dev_single`
`mi3xx_xdit_flux1_dev_distributed.json` or `mi35x_xdit_flux1_dev_distributed.json`	`xdit_flux_dev_distributed`
`mi3xx_xdit_flux2_dev_single.json` or `mi35x_xdit_flux2_dev_single.json`	`xdit_flux_dev_single`
`mi3xx_xdit_flux2_dev_distributed.json` or `mi35x_xdit_flux2_dev_distributed.json`	`xdit_flux_dev_distributed`
`mi3xx_xdit_wan22_14b_single.json` or `mi35x_xdit_wan22_14b_single.json`	`xdit_wan22_14b_single`
`mi3xx_xdit_wan22_14b_diffusers_single.json` or `mi35x_xdit_wan22_14b_diffusers_single.json`	`xdit_wan22_14b_diffusers_single`
`mi3xx_xdit_wan22_14b_diffusers_distributed.json` or `mi35x_xdit_wan22_14b_diffusers_distributed.json`	`xdit_wan22_14b_diffusers_distributed`

Each workload has one shared threshold file (xdit_<workload>_threshold.json). mi3xx and mi35x configs for the same workload reference that same file. GPU keys inside it (mi300x, mi325, mi350, mi355, auto) select the limit.

Note

FLUX.2 configs bind-mount cvs/lib/inference/xdit/scripts/flux2_example.py when the image does not ship /benchmark/flux2_example.py. WAN Diffusers suites mount cvs/lib/inference/xdit/scripts/wan_i2v_example.py. Adjust the host side of those volume entries to your CVS checkout.

mi3xx distributed templates ship Broadcom bnxt RDMA mounts and rdma* HCA names. mi35x distributed templates ship Ionic library mounts and ionic_* HCA names. Replace the <changeme> interface, HCA, and GID values for your fabric.

On shared clusters, skip aggressive docker prune during container setup:

export CVS_SKIP_DOCKER_SYSTEM_PRUNE=1

Run tests#

List stages in a suite:

cvs list xdit_flux_dev_single

Every xDiT suite uses this order:

Available tests in xdit_flux_dev_single:
  - test_launch_container
  - test_verify_prerequisites
  - test_verify_model
  - test_verify_parallelism
  - test_run_benchmark
  - test_parse_thresholds
  - test_clip_score
  - test_print_results
  - test_teardown

test_verify_parallelism checks the parallel-degree product:

FLUX (single and distributed): ulysses_degree × ring_degree × pipefusion × tensor_parallel × data_parallel == nnodes × torchrun_nproc. Single-node uses nnodes = 1.
WAN distributed: ulysses_size × ring_size == nnodes × torchrun_nproc. Single-node WAN logs the layout and skips that product check.

Example run (FLUX.1-dev on an MI300-family node; use the flux2 JSON for FLUX.2-dev, or mi35x_* on MI355X):

cvs run xdit_flux_dev_single \
  --cluster_file ~/cvs_workspace/cluster.json \
  --config_file ~/cvs_workspace/inference/xdit/mi3xx_xdit_flux1_dev_single.json \
  -vvv

Distributed FLUX.1-dev on MI355X:

cvs run xdit_flux_dev_distributed \
  --cluster_file ~/cvs_workspace/cluster.json \
  --config_file ~/cvs_workspace/inference/xdit/mi35x_xdit_flux1_dev_distributed.json \
  -vvv

WAN 2.2 native:

cvs run xdit_wan22_14b_single \
  --cluster_file ~/cvs_workspace/cluster.json \
  --config_file ~/cvs_workspace/inference/xdit/mi3xx_xdit_wan22_14b_single.json \
  -vvv

WAN Diffusers:

cvs run xdit_wan22_14b_diffusers_single \
  --cluster_file ~/cvs_workspace/cluster.json \
  --config_file ~/cvs_workspace/inference/xdit/mi3xx_xdit_wan22_14b_diffusers_single.json \
  -vvv

cvs run xdit_wan22_14b_diffusers_distributed \
  --cluster_file ~/cvs_workspace/cluster.json \
  --config_file ~/cvs_workspace/inference/xdit/mi35x_xdit_wan22_14b_diffusers_distributed.json \
  -vvv

Direct pytest invocation#

Each module can also be run with pytest:

pytest cvs/tests/inference/xdit/xdit_flux_dev_single.py \
  --cluster_file ~/cvs_workspace/cluster.json \
  --config_file ~/cvs_workspace/inference/xdit/mi3xx_xdit_flux1_dev_single.json \
  --html ~/cvs_results/xdit_flux1_single.html

Run on managed compute#

Follow Run Cluster Validation Suite (CVS) on managed compute to run xDiT on managed compute.

Read the results#

cvs run writes the pytest HTML report described in Pytest HTML report and bundle. Benchmark pass/fail uses the docker exit code plus parsed artifacts and the GPU key in the sibling threshold file.

The suite reads PCI device IDs from /sys/class/drm/card*/device/device. 0x74a5 selects mi325, 0x75a0 selects mi350, and 0x75a3 selects mi355. Any other ID, including MI300X, selects mi300x. Lookup then uses that key, or auto when the threshold file has no entry for it. test_print_results prints PASS or FAIL for each host. With enforce_thresholds set to false, the same row is RECORDED and does not fail the stage.

Each stage corresponds to a test function in the HTML report.

Stage	Test function	Notes
Launch	`test_launch_container`	Starts the named container. Setup may run `docker system prune` unless `CVS_SKIP_DOCKER_SYSTEM_PRUNE=1`.
Prerequisites	`test_verify_prerequisites`	Requires `/dev/kfd` and `torchrun` inside the container on every participating node.
Model	`test_verify_model`	Checks staged weights and that the CLIP checkpoint is in the container cache. A missing cache or host path fails the stage. The suite does not download weights. `enforce_thresholds: false` does not skip a missing CLIP checkpoint.
Parallelism	`test_verify_parallelism`
Benchmark	`test_run_benchmark`
Parse	`test_parse_thresholds`	Compares average latency to the threshold file. FLUX.1 requires `num_repetitions` and that `timing.json` contains that many `pipe_time` samples. Native WAN requires `num_benchmark_steps` and that many `rank0_step*.json` files.
CLIP score	`test_clip_score`	Scores saved media inside the benchmark container. FLUX scores `flux_.png` and `flux2_.png` (`clip_score` against `min_clip_score`). WAN samples 8 frames from `video.mp4` (native) or `video_i2v.mp4` (Diffusers) and gates `mean_frame_clip_score` against `min_mean_frame_clip_score`. Uses `openai/clip-vit-base-patch32`; prompts longer than 77 tokens are truncated (`results/clipscore.json` sets `prompt_truncated`). WAN requires `imageio` with `imageio-ffmpeg`, OpenCV, or `torchvision` to decode the MP4. The shipped floor of 20 is an unvalidated placeholder. `enforce_thresholds: false` records a score that misses the floor without failing the stage, but does not skip a missing CLIP checkpoint.
Teardown	`test_teardown`	Stops the container.

The metrics, threshold keys, and output artifacts for each benchmark family are listed below.

Family	Metric	Threshold key	Artifacts
FLUX	average `pipe_time` from `results/timing.json`	`max_avg_pipe_time_s`	`timing.json`, `flux_*.png`
WAN native	average `total_time` from `rank0_step*.json`	`max_avg_total_time_s`	step JSONs, `video.mp4`
WAN Diffusers	average epoch and pipe time from `results/timing.json`	`max_avg_pipe_time_s`	`results/timing.json`, `results/video_i2v.mp4`
FLUX	mean CLIPScore of the saved PNGs	`min_clip_score`	`results/clipscore.json`
WAN native	mean CLIPScore of 8 frames from `video.mp4`	`min_mean_frame_clip_score`	`results/clipscore.json`
WAN Diffusers	mean CLIPScore of 8 frames from `video_i2v.mp4`	`min_mean_frame_clip_score`	`results/clipscore.json`

Single-node output dirs use the node hostname, for example ${paths.log_dir}/flux_<hostname>_outputs or wan_22_<hostname>_outputs. Distributed runs write to the rank-0 hostname directory.