Troubleshooting guide

Troubleshooting guide#

This guide is the primary reference for diagnosing and resolving common failures when running Primus on AMD GPUs (ROCm, RCCL, Docker). It complements the CLI reference, Preflight, Benchmarking, and Configuration system documentation.

1. Diagnostic tools#

Use these tools before scaling a job or when a failure is hard to localize.

Tool	Purpose
`primus-cli --debug`	Enables verbose logging in `primus-cli` for command construction, delegation, and runtime details.
`primus-cli --dry-run`	Prints the commands Primus would run without executing them—useful to verify wrappers, paths, and MPI/launcher wiring.
`--export_config <path>`	Parsed by the training config parser, but not written by the current default `PrimusRuntime` path. Treat it as legacy/future functionality unless your deployment has implemented it.
`NCCL_DEBUG=INFO`	Surfaces detailed RCCL/NCCL connection and collective logs (set in the job environment).
`primus-cli direct -- preflight --host --gpu --network`	Fast host, GPU, and network validation. See Preflight.
`PRIMUS_PATCHES=none`	Disables Primus patches to the selected backend to isolate whether a failure is Primus-specific or upstream.

Examples

# Verbose CLI + dry run (no training executed)
primus-cli --debug --dry-run direct -- train pretrain --config path/to/config.yaml

# Preflight: environment validation only
primus-cli direct -- preflight --host --gpu --network

# RCCL/NCCL verbose logs in the training process environment
export NCCL_DEBUG=INFO

# Isolate backend vs. Primus patch layer
export PRIMUS_PATCHES=none

For end-to-end checks including optional performance probes, see Preflight (preflight --perf-test).

2. Out of memory (OOM) errors#

Symptoms: HIP/CUDA OOM messages, worker processes killed by the OOM killer, or abrupt exits during forward/backward.

Primary levers and mitigations

Approach	What to change
Reduce per-device activation memory	Lower `micro_batch_size` (often the first knob).
Shard weights/activations across devices	Increase tensor parallelism (`tensor_model_parallel_size` or `parallelism.tensor_parallel_degree`, depending on backend). See Parallelism configuration.
Trade compute for memory	Activation recomputation: `recompute_granularity: full`, `recompute_method: uniform`, `recompute_num_layers: <N>`.
Shard optimizer state	`use_distributed_optimizer: true` (Megatron-style stacks).
FSDP / sharded data parallel	Megatron: `use_torch_fsdp2: true`. TorchTitan: increase `data_parallel_shard_degree`.
CPU offload	`optimizer_cpu_offload: true` where supported.
Sequence length	Reduce maximum sequence length if the workload allows.
MoE / scratch memory	Set `HSA_NO_SCRATCH_RECLAIM=1` to reduce scratch-memory conflicts on some MoE workloads.
Plan before you run	`primus-cli ... -- projection memory --config <yaml>`—see Projection.

3. Distributed communication failures#

Symptoms: Hangs at initialization, NCCL/RCCL timeouts, connection errors, or inconsistent ranks.

Cause	What to verify / fix
Wrong network interface	Set `NCCL_SOCKET_IFNAME` to the correct interface; exclude virtual interfaces, e.g. `^docker0,lo`.
`MASTER_ADDR` unreachable	From every node, resolve DNS/IP consistently; verify firewalls and routing.
Port already in use	Change `MASTER_PORT` to a free port on all nodes.
InfiniBand not used or missing	Confirm `/dev/infiniband` exists where expected; run `ibstat`; ensure IB kernel modules are loaded. See Multi-node networking.
Timeout too aggressive	Megatron: increase `distributed_timeout_minutes`. TorchTitan: increase `comm.init_timeout_seconds`.
Mismatched world size	Align `NNODES`, `GPUS_PER_NODE`, and launcher settings across all nodes.

Debugging

export NCCL_DEBUG=INFO

Validation

primus-cli direct -- preflight --network
primus-cli direct -- benchmark rccl --op all_reduce

Collective behavior and RCCL roles are summarized in Collective operations.

4. Container issues#

Symptoms: Permission denied on devices, GPUs not visible inside the container, immediate exit, or RCCL failures only under Docker.

Symptom	Typical cause	Mitigation
GPU not visible	Devices not passed through	Ensure `--device /dev/kfd` and `--device /dev/dri` (Primus container mode typically sets these).
Permission denied on GPU	Group membership / permissions	Add `--group-add video`; ensure the user has video/render access on the host.
InfiniBand missing in container	Device not mounted	Add `--device /dev/infiniband` (and related uverbs devices as required by your site).
Debugger/profiler failures	Missing capabilities	`--cap-add SYS_PTRACE` and `--cap-add CAP_SYS_ADMIN` are required for many ROCm tooling paths.
Driver/library mismatch	Image vs. host ROCm	Match container image ROCm to host ROCm driver version.
Data or code not found	Bind mounts	Use `--volume /host/path:/container/path` for datasets and workspace.
Env vars missing in container	Forwarding	Check `runner/.primus.yaml` `container.options.env` for auto-forwarded variables; add extras with `--env KEY=VALUE`.

Installation and container-oriented setup are covered in Installation.

5. Configuration errors#

Symptoms: YAML parse failures, “unknown key” or type errors, silent wrong behavior after edits.

Issue	Resolution
Unset `${VAR}`	`${VAR}` with no default fails if `VAR` is unset. Use `${VAR:default}` or export the variable before launch.
Broken `extends:` chain	Confirm every referenced file exists and paths resolve relative to the expected directory.
Wrong parameter name	Cross-check backend docs: Megatron, TorchTitan, MaxText, Megatron Bridge.
Override not applied	Review merge order: CLI > experiment overrides > module preset with model preset additions; duplicate top-level module keys win over model preset keys. See Configuration system.
Effective config unknown	Use `--dry-run` to inspect the launch command and manually trace the experiment YAML, module preset, model preset, and overrides. Resolved-config export is not currently written by the default core runtime.

6. Backend-specific issues#

Megatron#

Issue	Mitigation
Suspected Primus patch interaction	`export PRIMUS_PATCHES=none` and retry with vanilla Megatron behavior.
Custom kernel compile failures	`disable_compile_dependencies: true` skips custom kernel compilation where applicable.
Wrong third-party path	Set `BACKEND_PATH` to override third-party resolution.

TorchTitan#

Issue	Mitigation
Submodule drift	`git submodule update --recursive` so `third_party/torchtitan` matches the Primus revision you run.
`torch.compile` instability	`export TORCH_COMPILE_DISABLE=1` or set `compile.enable: false` in config.

MaxText (JAX)#

Issue	Mitigation
JAX / jaxlib vs ROCm	Verify JAX and jaxlib builds match your ROCm stack.
XLA memory pressure	Tune `XLA_PYTHON_CLIENT_MEM_FRACTION` to cap client-side allocator use.

7. Performance issues#

Symptoms: Low tokens/sec, long iteration time, or poor scaling versus expectations.

Diagnosis

Step	Command / action
Compute sanity	`primus-cli direct -- benchmark gemm`
Interconnect	`primus-cli direct -- benchmark rccl --op all_reduce`
Broader probe	`primus-cli direct -- preflight --perf-test` (see Preflight)

Common fixes

Area	Action
GEMM / kernels	Enable hipBLASLt tuning (multi-stage workflow—see Performance tuning).
Primus stack	Enable `enable_primus_turbo: true` where supported.
Communication overlap	`overlap_grad_reduce: true`, `overlap_param_gather: true` (when applicable to your backend).
MoE / scratch	Confirm `HSA_NO_SCRATCH_RECLAIM=1` when recommended for your model class.
FP8	Enable FP8 when hardware and backend support it.

8. Data issues#

Symptom	Checks
Mock data works; real data fails	`data_path` format: Megatron typically expects `.bin` / `.idx` pairs. See Data preparation.
Tokenizer errors	`tokenizer_type` and `tokenizer_model` must match (e.g., Hugging Face tokenizer ID for `HuggingFaceTokenizer`).
Hugging Face download failures	Set `HF_TOKEN` for gated models; verify outbound network and cache directories.

9. Known limitations#

Area	Note
MaxText	Parameter completeness depends on upstream MaxText `base.yml`; some keys are inherited from upstream defaults.
Megatron Bridge	Recipe parameters may be loaded dynamically; not every key appears in static reference tables.
HummingbirdXT	Less mature than other backends; expect sharper edges in configs and tooling.
Primus-Turbo	Requires a separate installation step; not always present by default.

For terminology, see the Glossary. For checkpoint-related failures, see Checkpoint management.