# Technical guides

Deep technical topics for advanced users.

- [Parallelism strategies](parallelism-strategies.md): DP, TP, PP, SP, CP, EP, FSDP explained
- [Parallelism configuration](parallelism-configuration.md): per-backend parallelism setup and batch size relationships
- [Collective operations](collective-operations.md): NCCL/RCCL operations and their role in each parallelism strategy
- [Performance tuning](performance-tuning.md): HipBLASLt, Primus-Turbo, FP8, MoE optimization
- [MoE training deep-dive](moe-training.md): bottlenecks and Primus-Turbo optimizations for Mixture-of-Experts models
- [MegaMoE fused MoE layer](mega-moe.md): FlyDSL-based fused MoE layer for EP-only bf16 training, setup and reproduction
- [Data preparation](data-preparation.md): tokenization, data formats, mock data
- [Checkpoint management](checkpoint-management.md): formats, save/load, distributed checkpointing
- [Multi-node networking](multi-node-networking.md): InfiniBand, RoCE, AINIC configuration
- [RCCL SDMA AllGather for TorchTitan](rccl-sdma-torchtitan.md): FSDP2 AllGather on the copy engine: setup and verification, with Llama 3.1 as the example
- [AINIC bundle versions](ainic-bundle-versions.md): rebuilding a training image against a different AINIC bundle
- [Profiling and observability](profiling-and-observability.md): Torch profiler, TraceLens, memory snapshots, projection, pp_vis
- [Logging and experiment tracking](logging-and-experiment-tracking.md): TensorBoard, WandB, MLflow setup per backend
- [Fault tolerance and elastic training](fault-tolerance-and-elastic-training.md): graceful exit, auto-resume, in-process restart, torchft
- [Determinism and reproducibility](determinism-and-reproducibility.md): deterministic mode, seeds, trade-offs
- [Diffusion models](diffusion-models/README.md): Flux diffusion architecture, data pipeline, and FP8 / MXFP4 training
- [Recommendation models](recommendation/README.md): DLRMv4 (HSTU on Yambda-5b) for MLPerf Training: running, configuration, results, troubleshooting and the benchmark reference
- [Hybrid models](hybrid-models/README.md): Zebra-Llama hybrid recurrent-attention (Mamba/KDA/GDN + MLA) models, FLA-parity recipes, and checkpoint conversion
- [Native SFT and LoRA](native-sft-lora.md): Megatron-native SFT/LoRA runbook (BF16 / FP8 / FP4), no Megatron-Bridge dependency
- [SpecForge](specforge/introduction.md): speculative-decoding draft training — capture, offline train, and online train through `primus-cli`

---

[← Documentation home](../README.md)
