DLRMv4: HSTU ranker on Yambda-5b (MLPerf Training dlrmv4)#
DLRMv4 is the MLPerf Training recommendation benchmark. It trains an HSTU sequential ranking model on the Yambda-5b music-listening dataset. The model trains on the data’s daily windows in time order. Every 0.1% of the training data, it is evaluated on a fixed held-out day that training never sees. The run stops when the eval AUC reaches 0.75.
In Primus it is the recommendation backend with model_family: dlrm_v4. The
recipe is examples/mlperf/dlrm_v4.
This page: requirements, running, configuration, results, multi-node, updating the image and troubleshooting.
Benchmark reference: the dataset, data order, model, loss, optimizer, quality target and eval schedule.
Requirements#
GPUs |
1, 2 or 4 nodes of 8× AMD Instinct MI355X ( |
Image |
|
Dataset |
227 GB, on node-local NVMe on every node (training reads it with random page faults) |
Host memory |
The data loaders pin about 180 GiB per node before |
Container |
Host network and IPC, |
At the first launch in a container, the recommendation install hook:
installs
polars-u64-idx,gin-configand the MLPerf logging package, all with--no-deps. The image’s torch, FBGEMM, torchrec, Triton and FlyDSL stay untouched, and the hook fails if any of them would change;installs FlyDSL 0.2.4 into a private directory,
/opt/primus-recommendation/site-packages, which only the trainer puts first on its path. The trainer’s kernels also run on the image’s FlyDSL 0.3.4.1 (setPRIMUS_RECOMMENDATION_SITE=to empty to use it), but 0.19% slower;links
/opt/rocmto the image’s ROCm. The trainer builds HIP kernels at run time with/opt/rocm/bin/hipcc.
Quick start#
On the host, from the Primus checkout:
# 1. Download the dataset (227 GB, ready to train) and verify it
export DLRM_DATA_PATH=/data/mlperf_dlrm_v4
bash <(curl -s https://raw.githubusercontent.com/mlcommons/r2-downloader/refs/heads/main/mlc-r2-downloader.sh) \
-d "$DLRM_DATA_PATH" https://training.mlcommons-storage.org/metadata/dlrmv4_dataset.uri
bash examples/mlperf/dlrm_v4/data/verify_dataset.sh
# 2. If it is on shared storage, copy it to node-local NVMe
DLRM_DATA_SRC=$DLRM_DATA_PATH bash examples/mlperf/dlrm_v4/data/stage_data_local.sh /mnt/nvme/dlrm_data 4086
# 3. One timed MLPerf run on one node
DATADIR=/mnt/nvme/dlrm_data LOGDIR=/mnt/nvme/dlrm_results \
bash examples/mlperf/dlrm_v4/run_with_docker.sh
run_with_docker.sh starts a rocm/primus:v26.9 container and runs
examples/mlperf/dlrm_v4/run_and_time.sh in it. Each trial prints
RESULT,DLRM_V4,<seed>,<seconds>,... and writes three files to LOGDIR: the
MLPerf log (mlperf_<date>_<n>.log), the train log (train_<date>_<n>.log)
and the console log. The recipe README covers the remaining options: number of
trials, fixed seeds, the compliance checker, short throughput runs, and running
inside an existing container. To build the dataset from the HuggingFace release
instead of downloading it, see
Benchmark reference: Dataset.
To run without the Docker launcher, from inside a container:
DLRM_DATA_PATH=/data ./primus-cli direct -- train pretrain \
--config examples/mlperf/dlrm_v4/configs/MI355X/dlrm_v4-yambda5b-mlperf.yaml
The trainer starts one process per GPU itself. primus-cli therefore runs one
Primus process per node, not a torchrun fan-out.
The first run in a container builds the trainer’s HIP extensions under
/tmp/dlrm_hstu_* before run_start, which makes initialization take 3–7
minutes. Later runs reuse them. At the end of a run, the train log prints
[Primus:dlrm_v4] MLPerf summary: with the initialization time (init_s), the
time to train (run_s), the status, the number of evals and the last AUC.
Configuration#
Settings come from three layers, each overriding the one before it:
The trainer’s gin file (
primus/backends/recommendation/dlrm_v4/src/generative_recommenders/dlrm_v4/train/gin/yambda_5b.gin). It sets the model, kernels and precision, and the defaults of the run parameters. Each run parameter is read from an environment variable when that variable is set.The Primus config. Each run parameter in the table below that is set (not
null) is written to its environment variable before the trainer starts, replacing any exported value.nullkeeps the gin value. All parameters are listed with their meaning inprimus/configs/modules/recommendation/pre_trainer.yaml. The MLPerf experiment,dlrm_v4-yambda5b-mlperf.yaml, fills them from environment variables of the same name, so an exportedBATCH_SIZEstill applies.gin_bindings: a list of gin bindings that the backend appends after the gin file. Later gin bindings win, so a binding overrides the file’s binding for that parameter, environment lookup included. Use it for settings that are not exposed as parameters.
Primus parameter |
Environment variable |
Default |
Meaning |
|---|---|---|---|
|
|
none ( |
Dataset root, holding |
|
|
1024 |
Batch per GPU; the global batch is |
|
|
1e-6 |
Learning rate after warmup (dense Adam, sparse row-wise Adagrad) |
|
|
24000 |
Linear warmup from 0 |
|
|
-1 |
-1: a new random seed per run |
|
|
0.75 |
Stop at the first eval that reaches it; 1.0 trains through every window |
|
|
0.001 |
Evaluate every 0.1% of the training data |
|
|
-1 |
-1: first eval by the MLPerf rule for the run’s global batch; 0: evaluate from the start |
|
|
0, 299 |
First daily window and number of training windows (0–298: all of them) |
|
|
4086 |
User history length; selects |
|
|
0 |
Steps between throughput and loss lines in the train log (0: none) |
|
|
none |
Directory for checkpoints; unset disables them (see below) |
|
|
Evict the dataset from the page cache before INIT and verify it (MLPerf rule) |
|
|
|
Stream each node’s dataset copy back into the page cache once that node’s ranks pass |
|
|
|
|
MLPerf log ( |
|
|
Generated gin file and default MLPerf log |
|
|
|
Train on a generated synthetic dataset instead of |
|
|
|
Where synthetic datasets are generated and kept |
In the MLPerf experiment, clear_caches, warm_cache_after_run_start and
output_dir are filled from CLEAR_CACHES, WARM_CACHE_AFTER_RUN_START and
DLRM_OUTPUT_DIR. Other environment variables reach the trainer through the
experiment’s env: block, or through EXTRA_ENV="K=V ..." with
run_with_docker.sh.
MLPerf rules. For a submission, only the global batch size and the post-warmup learning rate may be tuned. The reference points are 1e-6 at a global batch of 8192 (1 node), 2e-6 at 16384 and 4e-6 at 32768. Warmup (24000 steps from 0) and the eval schedule are fixed. See Benchmark reference: Quality.
Precision#
Training is bf16 mixed precision. On top of it, the recipe runs most HSTU GEMMs in FP8, and this is on by default:
Forward: the HSTU projections and attention use
e4m3operands. Their per-tensor scales are calibrated over the first 4 steps and then frozen.Backward: the attention backward GEMMs, the output projection’s input gradient, and the UVQK projection’s input and weight gradients. Gradients are
e5m2or block-scalede4m3; weights and activations aree4m3. Backward scales are recomputed during training, not frozen.Preprocessor: the second-layer GEMMs of the contextual preprocessor MLPs.
FP8 is folded into the kernels that produce each tensor (LayerNorm, projection, SiLU, residual add), never added as a separate cast. GEMMs accumulate in fp32. Embeddings, the optimizer, the loss, the metrics and the collectives do not use FP8.
The FP8 paths fail closed: if one cannot run, the run stops instead of
silently falling back to bf16. The train log shows [hstu-fp8] resolved: and
[hstu-fp8] engaged: lines for the forward paths in use, and armed OK: lines
for the other optional paths.
The Results are with FP8 on, and the recipe is validated only in
that configuration. Each FP8 path has its own off switch, documented with the
path in the gin file. The trainer refuses an FP8 backward with a bf16 forward:
setting only DLRM_HSTU_FP8_FWD=0 stops at startup, and the error lists the
other switches to set. They are DLRM_HSTU_FP8_BWD=0,
DLRM_HSTU_FP8_BWD_UVQK=0 and DLRM_HSTU_FP8_PREPROC_L2=0.
Throughput runs on synthetic data#
With synthetic_data: true, the backend generates a small dataset in the
layout of the staged cache and trains on it, so a throughput run needs no
download. The data loader, model, kernels and gin file are the ones the MLPerf
run uses. The generator,
synthetic_data.py,
draws its users from the statistics of a steady-state day of the real data:
sequence lengths, anchors per user (the trainer batches a window user-major),
the share of each event type, and items replayed from per-user artist
libraries. Tokens per batch, users per batch and unique items, artists and
albums per batch come within a few percent of the real day’s, and so does the
throughput: on one MI355X node with rocm/primus:v26.9, two 200-step runs gave
8397 samples/s per GPU (launched with primus-cli direct) and 8379 (through
tools/perf), against 8454 on the real day 200 (harmonic means over steps
31-200; TFLOP/s 1013 and 1011 against 1017).
The dataset has two daily windows, one to train on and one for the final eval,
sized from num_train_batches (required), batch_size and the world size. The
backend sets start_ts 0, num_train_ts 1 and the eval window itself.
Generating it takes about 20 s for 200 steps of one node (11 GB); later runs of
the same size reuse it, and a run of another size generates its own. It is for
a single node: synthetic_data refuses NNODES above 1, as every node would
generate the dataset into the same directory.
examples/recommendation/configs/MI355X/dlrm_v4-FP8-pretrain.yaml
is such a run, and the DLRMv4 entry of the Primus perf benchmarks
(tools/perf):
DATA_PATH=/mnt/nvme/primus-data ./primus-cli direct -- train pretrain \
--config examples/recommendation/configs/MI355X/dlrm_v4-FP8-pretrain.yaml
It trains 200 steps and logs the throughput every 10 (train - Step N perf:
lines, with global_sps and tflops_real/gpu); the first steps hold the
kernel builds and FP8 calibration.
Throughput depends on the data: the real windows’ sequences grow over the
dataset (tokens per rank-batch rise from about 2.2 M around window 20 to 2.8 M
from window 200), so the MLPerf recipe’s short check on the first windows
(NUM_TRAIN_TS=31, median global_sps over steps 20–200) runs faster than a
steady-state window. Compare synthetic runs with each other, or with a real
window of the same day range.
MI300X and MI325X (bf16)#
The recipe’s FlyDSL attention and FP8 / MXFP6 HSTU paths are gfx950-only, and
they stop a run on gfx942 instead of falling back, so MI300X and MI325X run
the model in bf16. examples/recommendation/configs/<GPU>/dlrm_v4-BF16-pretrain.yaml
sets, in its env: block:
HSTU_ATTN_FWD=triton,HSTU_ATTN_BWD=triton;DLRM_HSTU_FP8_FWD,DLRM_HSTU_FP8_BWD,DLRM_HSTU_FP8_BWD_UVQK,DLRM_HSTU_FP8_PREPROC_L2,DLRM_HSTU_FP8_FWD_MXFP6andDLRM_HSTU_FP8_BWD_OUTPROJ_MXFP6to 0 (the gin file arms each one separately, and refuses an FP8 backward without the FP8 forward);TRITON_HSTU_LAST_LAYER_TARGETS_ONLY=0: the targets-only last layer needs the separated-RNG LayerNorm dropout kernel, which the trainer selects on gfx950 only; without it the first step stops withindexed output dropout requires separated RNG masks.
They also lower hbm_cap_gb (the embedding placement cap, 260 GB in the gin
file) to 230 on MI325X and 170 on MI300X, and MI300X runs 512 samples per GPU:
at 1024 it runs out of its 192 GB. The other gfx950-only kernels turn
themselves off with a log line. dlrm_v4-BF16-pretrain.yaml for MI355X runs
the same settings, to compare the three GPUs on the same work.
On rocm/primus:v26.9, 200 steps on synthetic data: MI325X 1207 samples/s and
203 TFLOP/s per GPU, MI300X 1052 and 178 (batch 512), MI355X 2025 and 340 with
these settings (8379 and 1011 with the recipe’s FP8 kernels). The trainer’s pinned
Triton configurations are tuned on MI350X; TRITON_FULL_AUTOTUNE=1 gave MI325X
8.5% more throughput but took about 45 minutes of autotuning.
Checkpointing#
Checkpointing is off unless ckpt_path (CKPT_PATH) is set. A checkpoint
holds the full sharded model and is hundreds of GB. With the path set:
the run saves after every 100th window, after the last window, and every hour (
CKPT_TIME_INTERVAL_S, default 3600).CKPT_STEP_FREQandIN_WINDOW_CKPT_FREQadd step-based saves;it keeps the newest
KEEP_LAST_Ncheckpoints (default 1);on start, it resumes from the highest-numbered checkpoint under the path.
The MLPerf run state is saved too, so the MLPerf log continues across a resume. The MLPerf recipe leaves checkpointing off.
Results#
One MI355X node, rocm/primus:v26.9, the MLPerf recipe (global batch 8192, LR
1e-6), measured 2026-11-06:
Throughput, median |
84,506–84,533 samples/s |
Time to target, seed 731542219 |
913.3 s, 5 evals |
Time to target, seed 1234567 |
955.1 s, 6 evals |
Time to target, seed 20261103 |
1041.0 s, 8 evals |
MLPerf 6.1.0 compliance checker |
SUCCESS on every run |
Seed 20261103’s 7th eval lands within 0.0003 of the target, so this seed needs 7 or 8 evals from run to run (another run: 1000.2 s, 7 evals).
Time to target is run_start to run_stop, which excludes initialization.
Each eval takes about 13 s.
Multi-node#
The recipe has validated 2- and 4-node configurations:
examples/mlperf/dlrm_v4/config_MI355X_2x8x1.sh and config_MI355X_4x8x1.sh. Each
one sources the 1-node config and then changes what depends on the node count:
Nodes |
Config |
Global batch |
|
|
|---|---|---|---|---|
1 |
|
8192 |
1e-6 |
1 (Primus’ default) |
2 |
|
16384 |
2e-6 |
0 |
4 |
|
32768 |
4e-6 |
0 |
SKIP_EVAL_EPOCH_PCT stays unset, so the first eval follows the global batch.
Running. run_with_docker.sh launches one node. For 2 or 4 nodes, start one
task per node, each in a rocm/primus:v26.9 container with the Primus checkout
at /workspace/Primus and the dataset on the node’s local NVMe. In each task:
source examples/mlperf/dlrm_v4/config_MI355X_2x8x1.sh # or config_MI355X_4x8x1.sh
bash examples/mlperf/dlrm_v4/run_and_time.sh
Node layout. Under Slurm,
run_and_time.shandprimus-clitakeNNODESandNODE_RANKfromSLURM_*. Inside a container withoutscontrol, passMASTER_ADDRin as well. Without Slurm, setNODE_RANKandMASTER_ADDR(the first node) on each node before sourcing the config. Values set before sourcing are kept.CPUs. Give each task all of the node’s CPUs (
srun --ntasks-per-node=1 --cpus-per-task=$SLURM_CPUS_ON_NODE --cpu-bind=none). Otherwise a Slurm step can get one CPU per task, and RCCL runs far below line rate.Network. Multi-node runs over RDMA (RoCE or InfiniBand), with the usual RCCL settings: the routable network interface for the bootstrap (
NCCL_SOCKET_IFNAME), the RDMA devices (NCCL_IB_HCA), and your fabric’s GID index and traffic class. See Multi-node networking. The container needs/dev/infinibandand--ulimit memlock=-1:-1.
Why TORCH_NCCL_HIGH_PRIORITY=0 at 2+ nodes. Under the HIP in v26.9, all
streams share one pool of GPU_MAX_HW_QUEUES hardware queues per GPU. With 1,
the NCCL stream is pinned to the same queue as the train pipeline’s data-dist
stream. Every step, a small copy of the next batch’s all-to-all split sizes then
waits behind the backward collectives: the GPU idles 4–6 ms per step, and time to
target rises by about 3%. With 0, the streams land on separate queues. On one node,
1 shows no stall and stays. The full analysis is in the comment above the
setting in config_MI355X_2x8x1.sh.
Results (OLCF Lux, MI355X, 8 × 400 Gb/s RoCE per node, rocm/primus:v26.9,
2 runs per seed, run with DL_PIN_PREWARM_HEADROOM_GIB=128):
Nodes |
Seed 731542219 |
Seed 1234567 |
Seed 20261103 |
|---|---|---|---|
2 |
577.6–579.9 s, 6 evals |
626.2–626.9 s, 8 evals |
638.4–639.4 s, 9 evals |
4 |
376.3–376.7 s, 5 evals |
426.7–427.3 s, 9 evals |
436.8–437.1 s, 10 evals |
Every run passed the MLPerf 6.1.0 compliance checker. These times are 1.7% (2
nodes) and 2.1% (4 nodes) below the MLPerf submission version (training_dlrmv4
on rocm/primus:v26.3) on the same nodes.
Updating the image#
Some of the trainer’s startup work is driven by tables recorded for one exact software stack. On any other stack, the trainer ignores them:
Table |
Purpose |
Log line when ignored |
|---|---|---|
BLAS replay table |
Loads hipBLASLt’s kernel libraries before |
|
HIP preload table |
Loads deferred HIP code objects before |
|
FP8 GEMM solution pins |
Pin the FP8 GEMM kernels some fused paths need ( |
The Triton kernel replay table is keyed by Triton version instead and covers Triton 3.6 and 3.9.
Without these tables the run still trains correctly, but slower. Before the v26.9 tables were recorded, a full run had five 3–6 s stalls, about 19 s in total, and throughput without the FP8 pins was 0.36% lower.
After moving to a new image, re-record the tables inside the new image, on one node:
bash tools/recommendation/dlrm_v4/record_stack_tables.sh blas <label> # ~12 min on 8 GPUs
bash tools/recommendation/dlrm_v4/record_stack_tables.sh shim /tmp/shim
mkdir -p /tmp/coload
LD_PRELOAD=/tmp/shim/coload_shim.so COLOAD_LOG_DIR=/tmp/coload <one full training run>
bash tools/recommendation/dlrm_v4/record_stack_tables.sh hip <label> /tmp/coload <that run's MLPerf log>
Then register the new files and the new hipBLASLt build in the trainer package:
add the table files to
_BLAS_REPLAY_TABLESindlrm_v4/train/blas_warmup.pyand toSTACK_TABLESindlrm_v4/train/hip_preload.py;add the hipBLASLt build to the FP8 pins in
ops/blas_solution_pins.py. A pin applies only on the stacks it lists.
These are edits to the trainer package, so run its tests afterwards. See the
backend README.
To find windows that still stall, read the [m4] factors ... maxmed= lines of a
full run: maxmed is a window’s slowest step divided by its median step.
Use a fresh container after an image change. The HIP extension builds cached
in /tmp are reused whenever the sources match, even on a different stack.
Troubleshooting#
Symptom |
Cause and fix |
|---|---|
|
|
|
The run first reclaims page cache by itself; what remains is held by processes. Free that memory, or lower |
All ranks stall at random, worse with node uptime |
Transparent huge-page compaction. Keep |
Slow steps throughout, high I/O wait |
The data is on network storage, or the page cache is cold. Stage the data to local NVMe and keep |
A few windows have a step 16–28× the median |
The per-stack tables do not match the image. See Updating the image. |
|
No usable FlyDSL. The kernels need FlyDSL 0.2 or ≥ 0.3. The train log line |
One node: throughput about 5% below the results |
Primus’ default |
2+ nodes: every step slow, time to target about 3% high |
|
A CUDA or CPU torch replaced the image’s torch |
Something was installed with dependencies. Never reinstall torch, FBGEMM or torchrec; install extra packages with |
Triton cannot find |
Do not set |
For developers#
primus/backends/recommendation/README.md
describes the backend layout, the trainer’s origin (the MLPerf reference fork
training_dlrmv4) and the trainer’s test suite.