Primus documentation

Documentation for Primus, a large-scale foundation model training framework for AMD GPUs.

---

Choose your starting point

I am a…	Start here
New user	Getting started
User running training jobs	User guide
User writing YAML configurations	Configuration reference
Engineer tuning performance	Technical guides
Operator deploying to production	Operations
Contributor to the codebase	Developer guide

---

Documentation structure

Getting started

Start here if you are new to Primus.

• Project overview: what Primus does, who it is for, key capabilities

• Installation guide: prerequisites, Docker/bare-metal/Slurm setup

• Quickstart: first training run in 5 minutes

• Release notes: published training image tags and their full software stacks (single source of truth)

• Glossary: terms, acronyms, and domain concepts

User guide

Core workflows and day-to-day usage.

• CLI reference: primus-cli modes, flags, and subcommands

• Configuration system: YAML configuration model, presets, overrides, inheritance

• Environment and XLA flags: where to set env vars, how to override XLA_FLAGS safely, how to verify what took effect

• Pretraining: pretraining concepts: backends, YAML structure, parallelism, configuration inventory

• End-to-end training recipes: pretraining commands: copy-paste, GPU-arch-specific run commands

• SpecForge (offline): primus-cli entrypoint for SpecForge hidden states capture and train

• Megatron-LM training performance validation: reproduce the published Megatron backend benchmarks on the rocm/primus image

• TorchTitan training performance validation: reproduce the published TorchTitan backend benchmarks on the rocm/primus image

• JAX MaxText training performance validation: reproduce the AMD-published MaxText benchmarks via Primus, MAD, or the standalone scripts

• Post-training: SFT and LoRA fine-tuning via Megatron Bridge

• Micro-benchmarking: GEMM, RCCL, and dense-GEMM benchmark suites

• Preflight: cluster diagnostics and environment validation

• Projection: memory and performance projection tools

• Tuning agent: LLM-driven search for an optimal training configuration (uses projection as an oracle)

• Primus tools: catalog of all Primus tools and ecosystem projects with how-to starting points

Configuration reference

Parameter references for Primus presets, backend-facing keys, and commonly used environment variables.

• Megatron parameters: Megatron-LM backend YAML parameters and Primus overrides

• TorchTitan parameters: Primus TorchTitan preset keys and common JobConfig fields

• MaxText parameters: Primus MaxText overlay defaults and common fields

• Megatron Bridge parameters: Megatron Bridge recipe, SFT, and pretraining fields surfaced through Primus

• Environment variables: practical reference for commonly encountered environment variables

Technical guides

Deep technical topics for advanced users.

• Parallelism strategies: DP, TP, PP, SP, CP, EP, FSDP explained

• Parallelism configuration: per-backend parallelism setup and batch size relationships

• Collective operations: NCCL/RCCL operations and their role in each parallelism strategy

• Performance tuning: HipBLASLt, Primus-Turbo, FP8, MoE optimization

• MoE training deep-dive: bottlenecks and Primus-Turbo optimizations for Mixture-of-Experts models

• MegaMoE fused MoE layer: FlyDSL-based fused MoE layer for EP-only bf16 training, setup and reproduction

• Data preparation: tokenization, data formats, mock data

• Checkpoint management: formats, save/load, distributed checkpointing

• Multi-node networking: InfiniBand, RoCE, AINIC configuration

• AINIC bundle versions: rebuilding a training image against a different AINIC bundle

• Profiling and observability: Torch profiler, TraceLens, memory snapshots, projection, pp_vis

• Logging and experiment tracking: TensorBoard, WandB, MLflow setup per backend

• Fault tolerance and elastic training: graceful exit, auto-resume, in-process restart, torchft

• Determinism and reproducibility: deterministic mode, seeds, trade-offs

• Diffusion models: Flux diffusion architecture, data pipeline, and FP8 / MXFP4 training

• Native SFT and LoRA: Megatron-native SFT/LoRA runbook (BF16 / FP8 / FP4), no Megatron-Bridge dependency

• Native SFT LoRA on MI455X: 1-GPU gfx1250 native LoRA recipes and launch notes

Operations

Production deployment and operational guidance.

• Deployment: container, Slurm, and Kubernetes deployment

• Monitoring and logging: WandB, TensorBoard, MLflow, Primus logging

• Troubleshooting: common failures, diagnostics, and fixes

• Security: secrets handling, container security, dependencies

Developer guide

For contributors and maintainers.

• Architecture: system design, runtime, backends, patch system

• Contributing: development setup, code style, PR process

• Testing: test types, running tests, CI pipeline

• Extending backends: adding new training backends

• Adding models: adding model configurations per backend

• Model support matrix: supported models per backend and GPU

• CLI architecture: CLI internals: subcommand discovery, dispatch, and launch wrappers

• Backend patch notes: Primus-specific backend arguments and the files they patch

• Tooling: auxiliary analysis, benchmarking, visualization, and diagnostics tools

Technical blogs

Long-form articles on the reasoning behind Primus features, most of them published on ROCm Blogs.

• Blog index: every Primus article, published and in progress, with the documentation each one maps to

---

Common use cases

I want to…

Goal	Document
Understand what Primus is	Overview
Browse all Primus tools	Primus tools
Install Primus	Installation
Run my first training	Quickstart
Find out what is inside a training image	Release notes
Get an exact run command for my model/GPU	End-to-end training recipes
Train a SpecForge drafter on ROCm	SpecForge on Primus
Write a training YAML configuration	Configuration system
Look up a Megatron parameter	Megatron parameters
Look up a TorchTitan parameter	TorchTitan parameters
Look up an environment variable	Environment variables
Understand parallelism strategies	Parallelism strategies
Configure parallelism for my model	Parallelism configuration
Tune training performance	Performance tuning
Train a Mixture-of-Experts model	MoE training deep-dive
Use the fused MegaMoE layer	MegaMoE fused MoE layer
Train a diffusion (Flux) model	Diffusion models
Fine-tune with native SFT / LoRA	Native SFT and LoRA
Run native LoRA on MI455X (gfx1250)	Native SFT LoRA on MI455X
Auto-tune my training configuration	Tuning agent
Profile a training run	Profiling and observability
Track experiments (WandB/MLflow/TensorBoard)	Logging and experiment tracking
Survive node failures on long runs	Fault tolerance and elastic training
Reproduce results bit-for-bit	Determinism and reproducibility
Prepare training data	Data preparation
Deploy to a Slurm cluster	Deployment
Debug a training failure	Troubleshooting
Contribute to Primus	Contributing
Understand the code architecture	Architecture
Add a new training backend	Extending backends
Read the technical blogs behind a feature	Blog index
Cite Primus in a paper	Citation

---

External resources

• ROCm Blogs: where the Primus technical articles are published

• Primus-Turbo: high-performance operators and kernels

• Primus-SaFE: external stability/platform layer; this repository does not include a production integration guide

• AMD ROCm documentation

• Megatron-LM

• TorchTitan

• MaxText

---

Need help? Open an issue on GitHub.