Megatron backend configuration reference#
This page lists the flat configuration keys exposed by Primus when framework: megatron. Unless a section says otherwise, values are the defaults from primus/configs/modules/megatron/trainer_base.yaml and related model presets. The effective pretraining preset is pre_trainer.yaml, which extends trainer_base.yaml and overrides several high-impact training defaults.
Where parameters live. Set overrides under modules.pre_trainer.overrides: in your experiment YAML. Model architecture keys usually come from models.<role>.overrides: (or your chosen model preset), but the same names map to Megatron’s argparse namespace either way.
Presets.
Module presets:
primus/configs/modules/megatron/(the main pretraining bundle ispre_trainer.yaml, which extendstrainer_base.yamland Primus Megatron add-ons).Model presets:
primus/configs/models/megatron/(for examplelanguage_model.yaml).
Mapping to Megatron-LM. Keys are passed through 1:1 to Megatron’s training arguments (same names as argparse / Namespace). Primus builds that namespace with MegatronArgBuilder.
Upstream reference. Full flag semantics and newer options are defined in Megatron-LM: megatron/training/arguments.py.
Example (experiment YAML)#
framework: megatron
modules:
pre_trainer:
overrides:
global_batch_size: 256
train_iters: 50000
tensor_model_parallel_size: 2
models:
pre_train:
overrides:
hidden_size: 2048
num_layers: 32
1. Base module parameters#
Source: primus/configs/modules/module_base.yaml (merged into Megatron presets; trainer_base.yaml sets trainable: true).
Parameter |
Default |
Description |
|---|---|---|
|
|
When |
|
|
Log level for the structured sink (Primus module plumbing); |
|
|
Minimum level for file-backed logging. |
|
|
Minimum level for stderr logging. |
2. Training and batching#
Source: primus/configs/modules/megatron/trainer_base.yaml; effective pre_trainer.yaml overrides are noted where they differ.
Parameter |
Default |
Description |
|---|---|---|
|
|
Reserved; not supported as a Megatron override in this preset. |
|
|
Optional trainer spec hook (unused in defaults). |
|
|
Samples per microbatch per data-parallel rank (per forward/backward step before gradient accumulation). |
|
|
Deprecated; use |
|
|
Total batch size across the data-parallel world (before or after splitting, per Megatron semantics). |
|
|
Optional batch-size ramp schedule string / config. |
|
|
Allow shrinking batch if memory is insufficient. |
|
|
Abort on NaNs in loss or gradients. |
|
|
Detect abnormal loss spikes. |
|
|
Detect abnormally large gradients. |
|
|
Pads vocabulary size for efficient kernels / partitioning. |
|
|
Install handlers for graceful shutdown signals. |
|
|
Stop training after this many minutes. |
|
|
Exit after this many iterations (if set). |
|
|
ONNX export compatibility tweaks. |
|
|
Use BERT binary classification head when applicable. |
|
|
Prefer FlashAttention kernels when available. |
|
|
RNG seed for reproducibility. |
|
|
Random init that varies across data-parallel ranks. |
|
|
Use Xavier uniform for some weights. |
|
|
Lightweight test path (fewer steps / checks). |
|
|
Total training iterations (mutually exclusive with sample-based stopping in typical setups). |
|
|
Total training samples (when using sample-based training). |
|
|
Validation iterations per eval. |
|
|
Run validation every this many iterations. |
|
|
Run a full pass over validation data. |
|
|
Multiple validation datasets / passes. |
|
|
Only run eval / test, no training updates. |
|
|
Periodic distributed sync barrier for debugging. |
|
|
ADLR autoresume integration. |
|
|
Autoresume checkpoint interval. |
|
|
Force Python GC on a schedule. |
|
|
GC every N steps when |
|
|
Run manual GC during evaluation. |
|
|
Masking strategy for MLM / similar objectives. |
|
|
Masking strength multiplier. |
|
|
Iterations interpreted as one “epoch” for logging. |
3. Mixed precision#
Source: trainer_base.yaml.
Parameter |
Default |
Description |
|---|---|---|
|
|
Enable FP16 mixed precision training. |
|
|
Enable BF16 mixed precision training. |
|
|
All-reduce gradients in BF16 (saves bandwidth). |
|
|
Normalize loss per token instead of per sample. |
|
|
Static loss scale for FP16; |
|
|
Initial dynamic loss scale. |
|
|
Floor for dynamic loss scale. |
|
|
Window for dynamic loss scaling updates. |
|
|
Hysteresis steps for loss-scale decreases. |
|
|
Accumulate and reduce gradients in FP32. |
|
|
Compute LM cross-entropy in FP16. |
|
|
FP8 recipe selection ( |
|
|
FP8 scaling margin. |
|
|
FP8 recipe variant (e.g. delayed scaling). |
|
|
Deprecated FP8 interval (kept for compatibility). |
|
|
History length for FP8 amax statistics. |
|
|
How to combine amax history ( |
|
|
Run weight gradients in FP8 where supported. |
|
|
FP8 parameter gather for distributed optimizer paths. |
|
|
Transformer Engine RNG tracker for FP8. |
|
|
Separate RNG tracker for inference FP8. |
|
|
FP4 mode; |
|
|
FP4 recipe name. |
|
|
Store parameters in FP4. |
|
|
Keep first/last layers in BF16 for stability. |
|
|
Count of early layers forced to BF16 when enabled. |
|
|
Count of final layers forced to BF16 when enabled. |
|
|
Primus: disable FP8 weight transpose cache (see |
4. Optimizer and learning rate#
Source: trainer_base.yaml.
Parameter |
Default |
Description |
|---|---|---|
|
|
Optimizer family ( |
|
|
Peak learning rate. |
|
|
LR decay schedule ( |
|
|
Decay duration in iterations. |
|
|
Decay duration in samples. |
|
|
Warmup as a fraction of total train steps. |
|
|
Linear warmup steps. |
|
|
Warmup in samples. |
|
|
LR at the start of warmup. |
|
|
Minimum LR after decay. |
|
|
Weight-decay schedule style for WSD when used. |
|
|
WSD decay window in samples. |
|
|
WSD decay window in iterations. |
|
|
LR multiplier for attention/head modules when supported. |
|
|
AdamW / L2-style weight decay. |
|
|
Starting weight decay for schedules. |
|
|
Ending weight decay for schedules. |
|
|
How weight decay changes between start/end. |
|
|
Global gradient norm clip. |
|
|
Adam first moment decay. |
|
|
Adam second moment decay. |
|
|
Adam epsilon. |
|
|
SGD momentum when |
|
|
Override optimizer parameter groups’ schedulers. |
|
|
Load optimizer scheduler state strictly from checkpoint. |
|
|
Alternate warmup specification (legacy / schedule hooks). |
|
|
Decoupled LR for certain param groups. |
|
|
Minimum for decoupled LR. |
|
|
Muon optimizer scaling. |
|
|
Muon scaling mode. |
|
|
Muon matmul precision hint. |
|
|
Muon Newton–Schulz iterations. |
|
|
Muon tensor-parallel mode. |
|
|
Muon Nesterov momentum. |
|
|
Split QKV for Muon. |
|
|
Muon momentum. |
|
|
Muon-specific decay. |
|
|
How Muon applies decay. |
|
|
Offload optimizer state to CPU. |
|
|
Fraction of optimizer state offloaded. |
|
|
Use PyTorch optimizer for offload path. |
|
|
Overlap CPU optimizer device transfers. |
|
|
Pin memory for CPU gradients. |
|
|
Pin memory for CPU params in offload. |
|
|
Use precision-aware optimizer (main grads/params in lower precision). |
|
|
Dtype for main gradients ( |
|
|
Dtype for master params. |
|
|
Optimizer first moment dtype ( |
|
|
Optimizer second moment dtype. |
|
|
Build Adam from |
5. Parallelism and distribution#
Sources: trainer_base.yaml (distributed runtime) and primus/configs/models/megatron/language_model.yaml (model-parallel sizes and TP communication).
5.1 Data / distributed runtime (trainer)#
Parameter |
Default |
Description |
|---|---|---|
|
|
Overlap pipeline P2P with compute. |
|
|
Process-group backend ( |
|
|
Collective timeout. |
|
|
Defer embedding weight gradients. |
|
|
Max deferred embedding wgrad steps. |
|
|
Align gradient reductions for efficiency. |
|
|
Number of DDP buckets. |
|
|
DDP bucket size in elements. |
|
|
Pad buckets for NCCL bus bandwidth. |
|
|
Average inside collective vs outside. |
|
|
Overlap gradient all-reduce with backward. |
|
|
Overlap param all-gather (distributed optimizer). |
|
|
Overlap param gather with optimizer step. |
|
|
Align param gather for distributed optimizer. |
|
|
Ring-exchange P2P for PP. |
|
|
Local rank override (normally from launcher). |
|
|
Defer Megatron parallel state init. |
|
|
Account for embedding in PP partition. |
|
|
Account for loss partition in PP. |
|
|
Aggressiveness of |
|
|
Dedicated PP stage for embeddings. |
|
|
Shard optimizer state across data parallel. |
|
|
Use SHARP for collectives when available. |
|
|
Which group SHARP applies to ( |
|
|
Custom FSDP integration path. |
|
|
Megatron FSDP path. |
|
|
Build model on |
|
|
FSDP / ZeRO style sharding ( |
|
|
Fuse division into reduce-scatter. |
|
|
Suggested communication chunk size. |
|
|
Keep FP8 transpose cache with custom FSDP. |
|
|
Sharded optimizer instances per rank group. |
|
|
Use PyTorch FSDP2 integration. |
|
|
JSON config for NCCL communicators. |
|
|
Custom TP/PP/DP process mapping. |
|
|
Data replication mode for certain schedules. |
|
|
Stride between replicated ranks. |
|
|
Replication factor. |
|
|
Prefer deterministic algorithms (slower). |
|
|
Periodically hash weights across DP replicas for debugging. |
|
|
Overlap MoE expert-parallel communication. |
|
|
Primus: manual PP split points for decoder (list of ints). |
|
|
Primus: patch MoE compute/comm overlap. |
5.2 Model parallelism (model preset)#
Parameter |
Default |
Description |
|---|---|---|
|
|
Legacy combined MP size override. |
|
|
Tensor parallelism degree (intra-layer split). |
|
|
Encoder TP size when encoder/decoder differ. |
|
|
Pipeline parallelism stages. |
|
|
Optional explicit PP layout string. |
|
|
|
|
|
Encoder PP stages (encoder–decoder models). |
|
|
Rank where encoder/decoder split. |
|
|
Layers on first decoder PP stage. |
|
|
Layers on last decoder PP stage. |
|
|
Virtual PP (interleaved) depth. |
|
|
Layers per virtual stage. |
|
|
Virtual stages per physical PP rank. |
|
|
Microbatch grouping for interleaved PP. |
|
|
Sequence parallelism when TP > 1. |
|
|
Context (sequence) parallelism degree. |
|
|
Context-parallel comm pattern ( |
|
|
Hierarchical CP group sizes. |
|
|
Expert parallelism for MoE. |
|
|
Expert tensor-parallel degree. |
|
|
Named groups that get high-priority CUDA streams. |
5.3 Tensor-parallel communication overlap (model)#
Parameter |
Default |
Description |
|---|---|---|
|
|
Async TP all-reduces for column-parallel layers. |
|
|
Enable TP communication overlap planner. |
|
|
Extra JSON / path for overlap configuration. |
|
|
Overlap all-gather in TP backward. |
|
|
Overlap reduce-scatter in TP backward. |
|
|
Overlap RS for data-grad path. |
|
|
Split all-gather for overlap. |
|
|
Split reduce-scatter for overlap. |
|
|
Bulk weight-gradient path for TP comm. |
|
|
Bulk data-gradient path for TP comm. |
|
|
Barrier using L1 timing hooks for TP comm profiling. |
|
|
Backend used to bootstrap TP communicators. |
6. Checkpointing#
Source: trainer_base.yaml.
Parameter |
Default |
Description |
|---|---|---|
|
|
Path prefix / pattern for checkpoints to write. |
|
|
Save every N iterations. |
|
|
Retain checkpoints at this interval. |
|
|
Skip optimizer state in checkpoints when truthy. |
|
|
Skip RNG state in checkpoints when truthy. |
|
|
Checkpoint path to load. |
|
|
Load only main parameters. |
|
|
Skip loading optimizer state. |
|
|
Skip loading RNG state. |
|
|
Finetune mode (do not require full optimizer match). |
|
|
When |
|
|
Restore model-parallel args from checkpoint. |
|
|
Restore tokenizer path from checkpoint args. |
|
|
Fail if |
|
|
Ephemeral checkpoint interval. |
|
|
|
|
|
Directory for non-persistent global checkpoints. |
|
|
Directory for non-persistent local checkpoints. |
|
|
|
|
|
Load weights from a pretrained checkpoint path. |
|
|
Specific step to load within a distributed checkpoint. |
|
|
Use deprecated distributed checkpoint format. |
|
|
Background worker for checkpoint IO. |
|
|
Infer checkpoint format automatically. |
|
|
Legacy format hint. |
|
|
|
|
|
Target format for one-shot conversion. |
|
|
Output path for conversion. |
|
|
Update legacy distributed-optimizer layout when converting. |
|
|
Deprecated fully-parallel save toggle. |
|
|
Save shards in parallel across ranks. |
|
|
Async checkpoint save ( |
|
|
Load shards in parallel. |
|
|
Assume identical layer structure across ranks. |
|
|
How to handle unexpected keys in distributed ckpt. |
|
|
Compatibility flag for older Megatron-Core checkpoints. |
|
|
Optimizer state fully reshardable layout. |
|
|
Primus: resume from latest checkpoint in the save directory when enabled. |
|
|
Primus: skip writing the final checkpoint at shutdown. |
7. Data#
Source: trainer_base.yaml.
Parameter |
Default |
Description |
|---|---|---|
|
|
Single blended dataset path / list. |
|
|
Shard data across ranks. |
|
|
Train/valid/test split ratios as comma string. |
|
|
Training data blend. |
|
|
Validation data blend. |
|
|
Test data blend. |
|
|
External JSON/YAML of dataset arguments. |
|
|
Per-split dataset args file. |
|
|
On-disk cache for indexed datasets. |
|
|
Use synthetic data (no real files). |
|
|
Merge file for blended datasets. |
|
|
Training sequence length. |
|
|
Encoder sequence length (encoder–decoder). |
|
|
Decoder sequence length. |
|
|
Sequence length for retriever models. |
|
|
Sampling rate for dataset blending. |
|
|
MLM mask probability. |
|
|
Probability of shorter sequences in BERT-style data. |
|
|
DataLoader worker processes per rank. |
|
|
Reset position IDs at document boundaries. |
|
|
Reset attention mask at boundaries. |
|
|
Mask loss at end-of-document tokens. |
|
|
Dataloader implementation ( |
|
|
Memory-map |
|
|
Build attention masks in the dataloader. |
|
|
Threads to build dataset indices. |
8. Recomputation (activation checkpointing)#
Sources: trainer_base.yaml and primus_megatron_module.yaml.
Parameter |
Default |
Description |
|---|---|---|
|
|
Enable activation recomputation globally. |
|
|
|
|
|
|
|
|
Layers to recompute per block / schedule. |
|
|
Primus: explicit global layer indices to recompute. Decoder layers are |
|
|
Distribute saved activations across TP/PP for memory balance. |
|
|
Deprecated alias for activation checkpointing. |
|
|
Recompute MoE layer activations (model preset). |
9. Logging and profiling#
Sources: trainer_base.yaml and primus_megatron_module.yaml.
9.1 Logging#
Parameter |
Default |
Description |
|---|---|---|
|
|
Skip first N iterations for throughput averaging. |
|
|
Reset moving averages periodically. |
|
|
Log L2 norms of parameters. |
|
|
Log fraction of zero gradients. |
|
|
Log tokens/sec and timing. |
|
|
Verbose progress logging. |
|
|
Verbosity for timing logs. |
|
|
Aggregate style for timing ( |
|
|
Steps between TensorBoard scalars. |
|
|
TensorBoard event queue size. |
|
|
Write timer stats to TensorBoard. |
|
|
Log batch size. |
|
|
Log LR. |
|
|
Log validation perplexity. |
|
|
Log memory usage. |
|
|
Log distributed world size. |
|
|
Log FP16/FP8 loss scale. |
|
|
Weights & Biases project name. |
|
|
W&B run name. |
|
|
W&B local directory. |
|
|
W&B entity / team. |
|
|
Enable NVIDIA OneLogger integration. |
|
|
OneLogger project string. |
|
|
OneLogger run name. |
|
|
Console log interval in iterations. |
|
|
TensorBoard output directory. |
|
|
Python logging level override. |
|
|
Directory for dumped config logs. |
|
|
Async OneLogger flushing. |
|
|
Application tag for telemetry. |
|
|
Application tag version. |
|
|
Primus: disable TensorBoard integration in Primus-wrapped runs. |
|
|
Primus: disable W&B. |
|
|
Primus: disable MLflow. |
|
|
Primus: MLflow run name. |
|
|
Primus: MLflow experiment name. |
|
|
Primus: collect ROCm memory info via |
|
|
Primus: iterations at which to log memory if |
9.2 Profiling#
Parameter |
Default |
Description |
|---|---|---|
|
|
Enable lightweight Nsight / CUDA profiling hooks. |
|
|
Enable |
|
|
Ranks to profile. |
|
|
First step to profile. |
|
|
Last step to profile. |
|
|
Skip listed iterations in profiling. |
|
|
Log rejected samples to this file. |
|
|
Create auxiliary Gloo groups for CPU-side ops. |
|
|
Record CUDA memory history (debug). |
|
|
Path for memory snapshot dumps. |
|
|
Primus: omit CPU activities from profiler traces. |
|
|
Primus: record tensor shapes in PyTorch profiler. |
|
|
Primus: capture Python stacks in profiler. |
|
|
Primus: gzip profiler outputs. |
10. Model architecture#
Sources: primus/configs/models/megatron/language_model.yaml and primus/configs/models/megatron/primus_megatron_model.yaml.
10.1 Core architecture#
Parameter |
Default |
Description |
|---|---|---|
|
|
Use legacy Megatron model code paths. |
|
|
Deprecated flag for Megatron-Core models; prefer current |
|
|
|
|
|
Transformer layers (decoder or unified stack). |
|
|
Encoder depth (encoder–decoder). |
|
|
Decoder depth. |
|
|
Hidden / model width. |
|
|
Attention heads. |
|
|
Attention kernel backend selection. |
|
|
Enable grouped-query attention (GQA). |
|
|
LayerNorm on Q/K projections. |
|
|
L2-normalize Q/K vectors. |
|
|
Number of query groups for GQA; |
|
|
Add absolute position embeddings (non-RoPE stacks). |
|
|
Position embedding style. |
|
|
Maximum sequence positions (context length cap). |
|
|
Original pretrained length for interpolation / scaling. |
|
|
Separate input embedding and LM head weights. |
|
|
FFN hidden size; |
|
|
Per-head KV channels override. |
|
|
Dropout on residual / hidden states. |
|
|
Attention dropout. |
|
|
Accumulate residuals in FP32. |
|
|
Apply residual after (vs before) norm where supported. |
|
|
Biases in linear / column-parallel layers. |
|
|
Biases in QKV projections. |
|
|
SwiGLU activation in FFN. |
|
|
Faster GeGLU path. |
|
|
OpenAI GELU variant. |
|
|
Squared ReLU activation. |
|
|
RoPE base frequency. |
|
|
Fraction of head dim spanned by RoPE. |
|
|
Interleaved RoPE layout. |
|
|
Positional interpolation factor for long contexts. |
|
|
Force RoPE on/off; |
|
|
Enable LLaMA-style rope scaling. |
|
|
Scaling factor for extended contexts (LLaMA-3 style). |
|
|
Backend library ( |
|
|
|
|
|
LayerNorm / RMSNorm epsilon. |
|
|
Norm type ( |
|
|
LayerNorm with +1 offset trick. |
|
|
Clone embedding scatter for autograd safety. |
|
|
Run weight initialization. |
|
|
Initialize on CPU then move to GPU. |
|
|
Use Transformer Engine activation kernels. |
|
|
Fuse gradient accumulation kernels. |
|
|
Delay weight-gradient computation for scheduling. |
10.2 Tokenizer and vocabulary#
Parameter |
Default |
Description |
|---|---|---|
|
|
Tokenizer class name ( |
|
|
Path to tokenizer model / vocabulary file. |
|
|
Vocabulary size (often inferred from tokenizer). |
|
|
Vocabulary file path for BPE/WP tokenizers. |
|
|
Extra reserved token slots. |
|
|
Regex pattern for tiktoken. |
|
|
Special token count for tiktoken setup. |
|
|
Legacy tokenizer behavior. |
|
|
|
10.3 Initialization and attention numerics#
Parameter |
Default |
Description |
|---|---|---|
|
|
Standard deviation for weight init. |
|
|
Scale Q/K by layer index (deprecated GPT-3 trick). |
|
|
Force softmax in FP32. |
10.4 Kernel fusion flags#
Parameter |
Default |
Description |
|---|---|---|
|
|
Fuse bias + GELU. |
|
|
Fused cross-entropy + softmax. |
|
|
|
|
|
Fuse bias + SwiGLU. |
|
|
Fused masked softmax. |
|
|
Non-persistent LayerNorm mode in TE. |
|
|
Fuse bias + dropout. |
|
|
Fused RoPE kernels. |
10.5 Multi-latent attention (MLA)#
Parameter |
Default |
Description |
|---|---|---|
|
|
Enable MLA blocks instead of standard MHA. |
|
|
Low-rank query projection rank. |
|
|
Low-rank KV compression rank. |
|
|
Q/K head dimension for MLA. |
|
|
Positional head dimension for MLA. |
|
|
Value head dimension for MLA. |
|
|
RoPE scaling inside MLA (distinct from |
|
|
Yarn / scaling m-factor. |
|
|
Yarn scaling on all dims. |
10.6 Mixture-of-experts (MoE)#
Parameter |
Default |
Description |
|---|---|---|
|
|
Experts per MoE layer; |
|
|
Every Nth layer is MoE (1 = every layer). |
|
|
Expert FFN hidden size. |
|
|
Shared expert overlaps routing. |
|
|
Shared expert FFN size. |
|
|
Grouped GEMM for experts. |
|
|
Router balancing ( |
|
|
Router activation dtype ( |
|
|
|
|
|
Experts to select per token. |
|
|
Apply softmax before top-k. |
|
|
Group-limited routing: number of expert groups. |
|
|
Groups to pick before top-k inside groups. |
|
|
Scaling for routing logits. |
|
|
Learnable per-expert bias. |
|
|
Update rate for expert bias. |
|
|
Legacy grouped GEMM path. |
|
|
Auxiliary load-balancing loss weight. |
|
|
Router z-loss coefficient. |
|
|
Input jitter for router stability. |
|
|
Token dispatch algorithm ( |
|
|
DeepEP-style expert parallelism. |
|
|
Per-layer MoE statistics logging. |
|
|
Capacity factor for token dropping / padding. |
|
|
Pad expert batches to capacity. |
|
|
Token dropping policy when over capacity. |
|
|
Extended tensor-parallel for experts. |
|
|
Expert upcycling initialization. |
|
|
Fuse token permutation for MoE. |
|
|
Primus: disable Primus top-k router patch. |
|
|
Primus: force load-balanced routing. |
|
|
Primus: legacy MoE layer implementation. |
|
|
Primus: Control the force load balancing type for the MoE router. Choices: even, uniform. |
|
|
Primus: run an FP32 router’s gating GEMM and its backward on Triton kernels, as accurate as Megatron’s FP32 GEMMs (BF16 input and weight, no bias, 16-256 experts, hidden a multiple of 256; other calls keep Megatron’s path). Not together with |
10.7 Logit softcapping (Primus / Grok-style)#
Parameter |
Default |
Description |
|---|---|---|
|
|
Softcap value for final logits; |
|
|
Softcap for attention logits. |
|
|
Softcap for MoE router logits. |
11. Primus extensions#
11.1 Build and compile#
Parameter |
Default |
Description |
|---|---|---|
|
|
Primus: avoid compiling dependency stacks in the trainer wrapper. |
11.2 Primus-Turbo (primus_turbo.yaml)#
Parameter |
Default |
Description |
|---|---|---|
|
|
Master switch for Primus-Turbo integrations. Many sub-features require this plus specific kernels. |
|
|
Turbo attention implementation. |
|
|
GPT-OSS-style learned sink attention. |
|
|
Sliding-window size for sink attention (GPT-OSS uses |
|
|
Apply the sliding window only to even layers (GPT-OSS pattern). |
|
|
Active Turbo GEMM flag for Dense paths. |
|
(removed) |
Removed—use |
|
|
Active Turbo grouped GEMM flag for MoE paths. |
|
(removed) |
Removed—use |
|
|
Fused MoE router with auxiliary scores. |
|
|
FP8 path inside Turbo attention (spacing in YAML is normalized to this key). |
|
|
Turbo DeepEP expert communication. |
|
|
DeepEP compute units / channels. |
|
|
Use a dedicated communication stream for DeepEP. |
|
|
Stage selector for sync-free MoE. |
|
|
Fuse activation + probability tensors to remove redundant work. |
|
|
Turbo RMSNorm kernels. |
11.3 Zero-bubble pipeline (zero_bubble.yaml)#
Parameter |
Default |
Description |
|---|---|---|
|
|
Install Primus zero-bubble PP patches when |
|
|
Print PP scheduler tables (also in |
|
|
Unified runtime for zero-bubble and related schedules. |
|
|
Issue a tiny comm before real comm to tune overlap. |
|
|
Start iter for auto-scheduler timers. |
|
|
End iter for auto-scheduler timers. |
|
|
Max pending backward ops (ZB1p vs ZB2p style); |
|
|
GPU memory percentile cap for adaptive ZB. |
|
|
Post-optimizer validation step (needs FSDP path). |
|
|
Require bitwise match in post validation when enabled. |
|
|
Enable zero-bubble schedule features in the ZB runtime. |
|
|
Zero-bubble “V” schedule without extra memory vs some baselines. |
|
|
Memory setup variant: |
|
|
1F1B-V schedule variant. |
|
|
Allow PP layer padding for divisibility. |
|
|
Iteration to profile memory ( |
|
|
Interleaved PP group size. |
|
|
Activation offload chunk count. |
|
|
Time budget for offload (scheduler hint). |
|
|
Auto-tune offload timing. |
|
|
Overlap save/resume in offload path. |
|
|
Splits along sequence dimension for ZB. |
|
|
CPU offload of activations in ZB path. |
11.4 Primus pipeline (primus_pipeline.yaml)#
Parameter |
Default |
Description |
|---|---|---|
|
|
Enable Primus pipeline scheduling patches. |
|
|
Schedule name ( |
|
|
|
|
|
Generic PP activation offload toggle in Primus pipeline. |
|
|
Comma-separated offload targets ( |
|
|
|
|
|
|
|
|
|
|
|
|
pp_warmup and dump_pp_data are Primus helpers defined in primus_megatron_module.yaml (not primus_pipeline.yaml):
Parameter |
Default |
Description |
|---|---|---|
|
|
Primus: warm-up PP stages to reduce first-iteration latency. |
|
|
Primus: dump PP tensors for debugging. |
11.5 Muse Spark (models/megatron/muse_spark/muse_spark_base.yaml)#
These settings (YAML keys in muse_spark_base.yaml) apply only to a Muse Spark model (model_type: muse_spark). muse_spark is one of the Primus values of model_type (row model_type in §10.1, next to deepseek_v4, kimi_k3, glm5_next, zaya1 and inkling), beside the gpt / mamba of §10.1; muse_spark_base.yaml sets it, so a Muse Spark preset gets it by extends.
Vocabulary padding: a Muse Spark model does not read make_vocab_size_divisible_by (§2). Its embedding and output layers have vocab_size rounded up to a multiple of 64 rows at every TP (202048 for the released model), the same rule as the native converter. The muse_spark_vocab patch sets Megatron’s padded vocabulary size that way, so the converted checkpoint has the same embedding shape at every TP. The vocabulary-parallel layers also need the rows to divide by TP (202048 = 2^6 x 7 x 11 x 41). TP must also divide the 16 query groups, so the usable TPs are 1, 2, 4, 8 and 16 (both checks pass); Megatron’s query-group check refuses another TP (e.g. 3, 6 or 12) before the vocabulary is sized. A refusal from a Muse Spark patch stops the run with SystemExit (the ValueError is its cause). Training at TP 2 is tested; for the TP a converted checkpoint is tested to load at, see §11.6.
Parameter |
Default |
Description |
|---|---|---|
|
|
Primus: |
|
|
Primus: Turbo DeepEP ( |
|
|
Primus: run the per-expert |
|
|
Primus: run the weightless attention norms (the attention |
|
|
Primus: run the centered |
muse_spark_skip_memory_check, muse_spark_turbo_deepep_internode, muse_spark_fused_post_expert_norm, muse_spark_fused_attn_norm and muse_spark_fused_centered_norm are booleans. A string is true for 1 / true / yes / on and false for 0 / false / no / off (any case, surrounding whitespace ignored); an empty or unset value means the key’s default (so the fused-kernel keys stay on). Any other value (e.g. a ture typo, 2) stops the run with a ValueError.
The Muse Spark example configs set PRIMUS_TE_GEMM_WORKSPACE_MIB: "512" in their env: block. The base patch megatron.te.hipblaslt_gemm_workspace (te_patches/hipblaslt_workspace_patches.py) reads it and enlarges TE’s hipBLASLt GEMM workspace, which the large-K / small-N GEMMs of the latent MoE need on gfx950 (“HIPBLASLT Error: 6”). Muse Spark adds no workspace code of its own.
A Muse Spark patch that rejects a setting (the fused Turbo router, the memory check, an a2a+p2p CP layout, …) stops the run with SystemExit and the reason; the patch runner does not log it and continue.
11.6 Muse Spark checkpoint conversion (sft_trainer / post_trainer)#
The post-training hook runner/helpers/hooks/train/posttrain/megatron/01_convert_checkpoints.py finds a Muse Spark checkpoint by the architectures / model_type of <hf_path>/config.json and runs primus/backends/megatron/checkpoint/native_loader_muse_spark_text.py (a thin entry point; the code is in native_loader_muse_spark.py). For Muse Spark the converter writes a torch_dist checkpoint. Limitations of this path:
The HF source (
hf_path, elsetokenizer_model) must be a local HF directory. The hook does not download a Hub id; a Hub id falls back to Megatron-Bridge, which has no Muse Spark model.The converter runs in one process. It builds the model with TP=PP=EP=1 and keeps the weights in host memory (CPU), then saves a
torch_distcheckpoint, which Megatron reshards on load to the training layout (tested layout: a 4-layer cut of the released checkpoint, converted by the posttrain hook and fine-tuned withprimus-cli direct -- train posttrainfrom the MI355X SFT example on one node at TP 1 / EP 8; loading a converted checkpoint at TP > 1 is not yet tested). It still needs one visible GPU: Megatron’s argument module importsmamba_ssm, which reads the GPU at import, and the router allocates its small expert-bias buffers on the current device. It does not run on a GPU-less login node.The converter copies the weights into the built model one decoder layer at a time, so host RAM holds one copy of the model plus one layer (about 2 bytes per parameter in bf16 / fp16, about 4 in fp32), not a second full state dict. The torch_dist save writes each expert’s
linear_fc1as one shard (the same global tensor), so the save copies no weights. Before the build, the converter compares this need (plus 5 %) with the host’s available memory (MemAvailable, or the cgroup limit when lower) and stops with an error when it does not fit. The full model (822B parameters, the checkpoint’s count; 816B is the release name) needs about 1.5 TiB (1.64 TB) in bf16, plus the 5 % margin. The optionalsft_trainer/post_trainerparamnative_ckpt_max_params(a positive integer parameter count; the hook passes it to the converter) adds a cap on the estimated parameter count; it is unset by default.The Megatron checkpoint is finished when it has
latest_checkpointed_iteration.txt(Megatron writes it last). The hook skips the conversion only then; it converts again into a directory without it. While it converts, the converter keeps a marker.muse_spark_conversion_in_progress(host and process id) in the output directory, touches it every 60 s, and removes it after the save. The torch_dist checkpoint is written into<checkpoint>/release/, then the tracker namesrelease. On a rerun, it removes a tracker-lessrelease/only when that marker belongs to a killed conversion: a process of the same host that no longer runs, or another host’s marker whose heartbeat has not moved for 900 s on the file server’s clock (never when that clock cannot be read). It does so only after its argument and memory checks passed. It refuses a finished checkpoint (a tracker namingrelease), a directory with another checkpoint (a tracker naming an iteration, or aniter_*directory), a directory that another running conversion is writing, and any other tracker-lessrelease/(remove it by hand).In a multi-node job, the hook’s other nodes follow node 0’s Muse Spark conversion through the converter’s marker (
wait_for_hook_conversion), not the hook’s 600 s timeout of the other families. They wait as long as the marker keeps changing (node 0 touches it every 60 s; hours at full size). They measure that on their own monotonic clock, not against file times. They finish when the hook’s<checkpoint>.doneand the checkpoint’slatest_checkpointed_iteration.txtexist and node 0’s<checkpoint>.converting.lock(the hook’s lock file) is gone. They print a progress line every 10 minutes on their own node. They stop at once when node 0’s conversion writes its failure record<checkpoint>.failed(also for a refusal before the lock is seen, e.g. the memory check). A record already there when a node starts waiting counts at once only if it carries this launch’s id (the optionalPRIMUS_CONVERT_RUN_ID, one new value per launch on every node, set by you) and was written less than 1800 s ago. That age is the record’s mtime against the mtime of a file written now in the same directory (the file server’s clock); when that directory cannot be written, the record does not count (see environment variables). They also stop with an error when node 0’s lock is gone for 60 s without a finished checkpoint (the conversion failed), when node 0’s marker has not changed for 900 s (node 0 was killed), or when node 0 has not started (its.converting.lockand a new marker) within 1800 s. A killed run’s leftover marker or.donedoes not count. Converting first and settingpretrained_checkpointskips the conversion on every node.When the fine-tune’s
num_layersis below the HF checkpoint’s (for example--num_layers 4against the released 62-layer checkpoint), the family’s prepare step passes the depth to the converter (convert(num_layers=...)). The converter converts only the first layers into<hf dir name>_native_<format>_layers<N>and records the depth in<checkpoint>.muse_spark_layers. A checkpoint recorded at another depth is not reused. Anum_layersabove the checkpoint’s is refused before the conversion. A HF checkpoint that already has only those layers also works (no cut).MuseSparkForConditionalGenerationcheckpoints (andmodel_typemuse_spark/muse_spark_text) are routed to this converter, which converts the text decoder (the vision and audio tower weights are skipped, with a log line). The family is registered inprimus/backends/megatron/checkpoint/native_families/muse_spark.py(the native converter’s family registry);python -m primus.backends.megatron.checkpoint.native_registrylists it with its optionnative_ckpt_max_params.The conversion dtype is
native_ckpt_dtypewhen set, else the top-leveltorch_dtype, elsetext_config’sdtype/torch_dtype, else the top-leveldtype(transformers >= 4.56 writesdtype), else bf16. When the top-leveltorch_dtypeandtext_config’s dtype differ, the hook stops and asks fornative_ckpt_dtype.Node 0 writes
<checkpoint>.started(this launch’s id) when it takes the lock. A node that starts waiting after node 0 already failed and removed its lock (for example in the convert-phase patches, before the converter runs) reads it and stops within 60 s instead of 1800 s. Any failure inside the converter’sconvert()writes<checkpoint>.failed.The vision tower,
vision_adapter,vision_projectionand the audio tower are not ported. Their tensors (model.vision_tower.*,model.vision_adapter.*,model.vision_projection.*,model.audio_tower.*) are logged with their count and not converted. Any other HF tensor the converter does not map fails the conversion.
13. Additional specialized parameters#
Source: trainer_base.yaml (remaining domains).
13.1 Vision pretraining#
Parameter |
Default |
Description |
|---|---|---|
|
|
Enable vision backbone pretraining. |
|
|
Objective ( |
|
|
Vision backbone family. |
|
|
Swin variant size. |
|
|
Classification classes. |
|
|
Image height. |
|
|
Image width. |
|
|
Input channels. |
|
|
ViT patch size. |
|
|
Fraction of classes used. |
|
|
Fraction of data per class. |
13.2 RETRO#
Parameter |
Default |
Description |
|---|---|---|
|
|
RETRO project directory with indices. |
|
|
Add frozen retriever tower. |
|
|
Cyclic iterator length. |
|
|
Retriever encoder layers. |
|
|
Retriever dropout. |
|
|
Retriever attention dropout. |
|
|
Neighbors per query chunk. |
|
|
Chunks concatenated per neighbor set. |
|
|
Gating between retrieval and LM. |
|
|
Assert neighbor counts for debugging. |
13.3 DINO self-supervised#
Parameter |
Default |
Description |
|---|---|---|
|
|
Local crop size. |
|
|
Number of local crops. |
|
|
Projection head width. |
|
|
Bottleneck dimension. |
|
|
Freeze last layer epochs. |
|
|
Normalize last layer weights. |
|
|
Teacher temperature warmup start. |
|
|
Teacher temperature. |
|
|
Epochs to warm teacher temperature. |
13.4 Biencoder / ICT / retriever utilities#
Parameter |
Default |
Description |
|---|---|---|
|
|
ICT projection head width. |
|
|
Biencoder shared projection dimension. |
|
|
Share query/context encoders. |
|
|
ICT checkpoint path. |
|
|
BERT encoder checkpoint for biencoder. |
|
|
Titles file for ICT datasets. |
|
|
Probability of in-block queries. |
|
|
Single-sentence pseudo documents. |
|
|
Evidence passages for open-domain QA. |
|
|
k values for top-k accuracy logging. |
|
|
Scale retriever scores. |
|
|
Block JSON data for retrieval. |
|
|
Precomputed embeddings path. |
|
|
Batch size when building ANN index. |
|
|
Indexer progress log interval. |
13.5 Straggler detection#
Parameter |
Default |
Description |
|---|---|---|
|
|
Log straggler diagnostics. |
|
|
Skip straggler detection at startup. |
|
|
Controller port for straggler service. |
|
|
Min/max samples for straggler stats. |
13.6 Inference-oriented options#
Parameter |
Default |
Description |
|---|---|---|
|
|
Heuristic threshold tying batch and sequence length. |
|
|
Dynamic batching for inference server. |
|
|
GPU buffer budget (GB). |
|
|
Minimum reserved fraction of buffer. |
|
|
Overflow growth factor. |
|
|
Hard cap on concurrent requests. |
|
|
Hard cap on tokens in flight. |
|
|
Token limit guard before OOM abort. |
|
|
Return BERT pooled embeddings. |
|
|
|
|
|
Flash decode kernels for incremental generation. |
|
|
Capture CUDA graphs for inference. |
|
|
Warm-up steps before capturing graphs. |
|
|
External graph provider hooks. |
|
|
Graph scope ( |
|
|
Max concurrent requests. |
|
|
Max prefill + decode tokens per request. |
13.7 Fault tolerance package and tooling#
Parameter |
Default |
Description |
|---|---|---|
|
|
NVIDIA fault-tolerance package hooks. |
|
|
Auto-calculate FT timeouts. |
|
|
Run workload inspector sidecar. |
13.8 Heterogeneous layers and process resilience#
Parameter |
Default |
Description |
|---|---|---|
|
|
JSON describing variable layer widths/types per layer. |
|
|
Inline base64/JSON blob for heterogeneous layers. |
|
|
In-process restart for fault recovery experiments. |
13.9 Experimental and rerun controls#
Parameter |
Default |
Description |
|---|---|---|
|
|
Gate experimental Megatron features. |
|
|
Fraction of iterations with injected errors (testing). |
|
|
|
|
|
|