Technical guides#
Deep technical topics for advanced users.
Parallelism strategies: DP, TP, PP, SP, CP, EP, FSDP explained
Parallelism configuration: per-backend parallelism setup and batch size relationships
Collective operations: NCCL/RCCL operations and their role in each parallelism strategy
Performance tuning: HipBLASLt, Primus-Turbo, FP8, MoE optimization
MoE training deep-dive: bottlenecks and Primus-Turbo optimizations for Mixture-of-Experts models
MegaMoE fused MoE layer: FlyDSL-based fused MoE layer for EP-only bf16 training, setup and reproduction
Data preparation: tokenization, data formats, mock data
Checkpoint management: formats, save/load, distributed checkpointing
Multi-node networking: InfiniBand, RoCE, AINIC configuration
RCCL SDMA AllGather for TorchTitan: FSDP2 AllGather on the copy engine: setup and verification, with Llama 3.1 as the example
AINIC bundle versions: rebuilding a training image against a different AINIC bundle
Profiling and observability: Torch profiler, TraceLens, memory snapshots, projection, pp_vis
Logging and experiment tracking: TensorBoard, WandB, MLflow setup per backend
Fault tolerance and elastic training: graceful exit, auto-resume, in-process restart, torchft
Determinism and reproducibility: deterministic mode, seeds, trade-offs
Diffusion models: Flux diffusion architecture, data pipeline, and FP8 / MXFP4 training
Recommendation models: DLRMv4 (HSTU on Yambda-5b) for MLPerf Training: running, configuration, results, troubleshooting and the benchmark reference
Hybrid models: Zebra-Llama hybrid recurrent-attention (Mamba/KDA/GDN + MLA) models, FLA-parity recipes, and checkpoint conversion
Native SFT and LoRA: Megatron-native SFT/LoRA runbook (BF16 / FP8 / FP4), no Megatron-Bridge dependency
SpecForge: speculative-decoding draft training — capture, offline train, and online train through
primus-cli