Troubleshooting guide

Troubleshooting guide#

This guide is the primary reference for diagnosing and resolving common failures when running Primus on AMD GPUs (ROCm, RCCL, Docker). It complements the CLI reference, Preflight, Micro-benchmarking, and Configuration system documentation.

1. Diagnostic tools#

Use these tools before scaling a job or when a failure is hard to localize.

Tool	Purpose
`primus-cli --debug`	Enables verbose logging in `primus-cli` for command construction, delegation, and runtime details.
`primus-cli --dry-run`	Prints the commands Primus would run without executing them—useful to verify wrappers, paths, and MPI/launcher wiring.
`--export_config <path>`	Parsed by the training config parser, but not written by the current default `PrimusRuntime` path. Treat it as legacy/future functionality unless your deployment has implemented it.
`NCCL_DEBUG=INFO`	Surfaces detailed RCCL/NCCL connection and collective logs (set in the job environment).
`primus-cli direct -- preflight --host --gpu --network`	Fast host, GPU, and network validation. See Preflight.
`PRIMUS_PATCHES=none`	Disables Primus patches to the selected backend to isolate whether a failure is Primus-specific or upstream.

Examples

# Verbose CLI + dry run (no training executed)
primus-cli --debug --dry-run direct -- train pretrain --config path/to/config.yaml

# Preflight: environment validation only
primus-cli direct -- preflight --host --gpu --network

# RCCL/NCCL verbose logs in the training process environment
export NCCL_DEBUG=INFO

# Isolate backend vs. Primus patch layer
export PRIMUS_PATCHES=none

For end-to-end checks including optional performance probes, see Preflight (preflight --perf-test).

2. Out of memory (OOM) errors#

Symptoms: HIP/CUDA OOM messages, worker processes killed by the OOM killer, or abrupt exits during forward/backward.

Primary levers and mitigations

Approach	What to change
Reduce per-device activation memory	Lower `micro_batch_size` (often the first knob).
Shard weights/activations across devices	Increase tensor parallelism (`tensor_model_parallel_size` or `parallelism.tensor_parallel_degree`, depending on backend). See Parallelism configuration.
Trade compute for memory	Activation recomputation: `recompute_granularity: full`, `recompute_method: uniform`, `recompute_num_layers: <N>`.
Shard optimizer state	`use_distributed_optimizer: true` (Megatron-style stacks).
FSDP / sharded data parallel	Megatron: `use_torch_fsdp2: true`. TorchTitan: increase `data_parallel_shard_degree`.
CPU offload	`optimizer_cpu_offload: true` where supported.
Sequence length	Reduce maximum sequence length if the workload allows.
MoE / scratch memory	Set `HSA_NO_SCRATCH_RECLAIM=1` to reduce scratch-memory conflicts on some MoE workloads.
Plan before you run	`primus-cli ... -- projection memory --config <yaml>`—see Projection.

3. Distributed communication failures#

Symptoms: Hangs at initialization, NCCL/RCCL timeouts, connection errors, or inconsistent ranks.

Cause	What to verify / fix
Wrong network interface	Set `NCCL_SOCKET_IFNAME` to the correct interface; exclude virtual interfaces, e.g. `^docker0,lo`.
`MASTER_ADDR` unreachable	From every node, resolve DNS/IP consistently; verify firewalls and routing.
Port already in use	Change `MASTER_PORT` to a free port on all nodes.
InfiniBand not used or missing	Confirm `/dev/infiniband` exists where expected; run `ibstat`; ensure IB kernel modules are loaded. See Multi-node networking.
Timeout too aggressive	Megatron: increase `distributed_timeout_minutes`. TorchTitan: increase `comm.init_timeout_seconds`.
Mismatched world size	Align `NNODES`, `GPUS_PER_NODE`, and launcher settings across all nodes.

Debugging

export NCCL_DEBUG=INFO

Validation

primus-cli direct -- preflight --network
primus-cli direct -- benchmark rccl --op all_reduce

Collective behavior and RCCL roles are summarized in Collective operations.

4. Container issues#

Symptoms: Permission denied on devices, GPUs not visible inside the container, immediate exit, or RCCL failures only under Docker.

Symptom	Typical cause	Mitigation
GPU not visible	Devices not passed through	Ensure `--device /dev/kfd` and `--device /dev/dri` (Primus container mode typically sets these).
Permission denied on GPU	Group membership / permissions	Add `--group-add video`; ensure the user has video/render access on the host.
InfiniBand missing in container	Device not mounted	Add `--device /dev/infiniband` (and related uverbs devices as required by your site).
Debugger/profiler failures	Missing capabilities	`--cap-add SYS_PTRACE` and `--cap-add CAP_SYS_ADMIN` are required for many ROCm tooling paths.
Driver/library mismatch	Image vs. host ROCm	Match container image ROCm to host ROCm driver version.
Data or code not found	Bind mounts	Use `--volume /host/path:/container/path` for datasets and workspace.
Env vars missing in container	Forwarding	Check `runner/.primus.yaml` `container.options.env` for auto-forwarded variables; add extras with `--env KEY=VALUE`.

Installation and container-oriented setup are covered in Installation.

5. Configuration errors#

Symptoms: YAML parse failures, “unknown key” or type errors, silent wrong behavior after edits.

Issue	Resolution
Unset `${VAR}`	`${VAR}` with no default fails if `VAR` is unset. Use `${VAR:default}` or export the variable before launch.
Broken `extends:` chain	Confirm every referenced file exists and paths resolve relative to the expected directory.
Wrong parameter name	Cross-check backend docs: Megatron, TorchTitan, MaxText, Megatron Bridge.
Override not applied	Review merge order: CLI > experiment overrides > module preset with model preset additions; duplicate top-level module keys win over model preset keys. See Configuration system.
Effective config unknown	Use `--dry-run` to inspect the launch command and manually trace the experiment YAML, module preset, model preset, and overrides. Resolved-config export is not currently written by the default core runtime.

6. Backend-specific issues#

Megatron#

Issue	Mitigation
Suspected Primus patch interaction	`export PRIMUS_PATCHES=none` and retry with vanilla Megatron behavior.
Custom kernel compile failures	`disable_compile_dependencies: true` skips custom kernel compilation where applicable.
Wrong third-party path	Set `BACKEND_PATH` to override third-party resolution.

TorchTitan#

Issue	Mitigation
Submodule drift	`git submodule update --recursive` so `third_party/torchtitan` matches the Primus revision you run.
`torch.compile` instability	`export TORCH_COMPILE_DISABLE=1` or set `compile.enable: false` in config.

MaxText (JAX)#

Issue	Mitigation
JAX / jaxlib vs ROCm	Verify JAX and jaxlib builds match your ROCm stack.
XLA memory pressure	Tune `XLA_PYTHON_CLIENT_MEM_FRACTION` to cap client-side allocator use.

7. Performance issues#

Symptoms: Low tokens/sec, long iteration time, or poor scaling versus expectations.

Diagnosis

Step	Command / action
Compute sanity	`primus-cli direct -- benchmark gemm`
Interconnect	`primus-cli direct -- benchmark rccl --op all_reduce`
Broader probe	`primus-cli direct -- preflight --perf-test` (see Preflight)

Common fixes

Area	Action
GEMM / kernels	Enable hipBLASLt tuning (multi-stage workflow—see Performance tuning).
Primus stack	Enable `enable_primus_turbo: true` where supported.
Communication overlap	`overlap_grad_reduce: true`, `overlap_param_gather: true` (when applicable to your backend).
MoE / scratch	Confirm `HSA_NO_SCRATCH_RECLAIM=1` when recommended for your model class.
FP8	Enable FP8 when hardware and backend support it.

8. Data issues#

Symptom	Checks
Mock data works; real data fails	`data_path` format: Megatron typically expects `.bin` / `.idx` pairs. See Data preparation.
Tokenizer errors	`tokenizer_type` and `tokenizer_model` must match (e.g., Hugging Face tokenizer ID for `HuggingFaceTokenizer`).
Hugging Face download failures	Set `HF_TOKEN` for gated models; verify outbound network and cache directories.

9. Known limitations#

Area	Note
MaxText	Parameter completeness depends on upstream MaxText `base.yml`; some keys are inherited from upstream defaults.
Megatron Bridge	Recipe parameters may be loaded dynamically; not every key appears in static reference tables.
HummingbirdXT	Less mature than other backends; expect sharper edges in configs and tooling.
Primus-Turbo	Requires a separate installation step; not always present by default.

For terminology, see the Glossary. For checkpoint-related failures, see Checkpoint management.