Security considerations

Security considerations#

This document describes security-relevant properties of Primus as a YAML-driven training framework for AMD GPUs (ROCm, RCCL, containers). It is intended for operators, platform engineers, and security reviewers. It does not replace organizational policies, threat models, or vendor hardening guides.

Related documentation: Environment variables, Installation, CLI reference.

1. Overview#

Aspect	Description
Role	Primus orchestrates distributed training jobs; it is not a general user-facing network service.
Authentication / authorization	No built-in authentication, authorization, or multi-tenant isolation in Primus itself.
Responsibility	Security posture is determined by the scheduler, container runtime, network, storage, identity systems, and operational practices of the deployment environment.

Treat Primus like privileged infrastructure software: run it on appropriately isolated hosts and networks, and govern secrets and data the same way you would for large-scale ML training elsewhere.

2. Secrets management#

Secrets are commonly passed as environment variables consumed by Primus, launchers, or third-party libraries.

Variable (examples)	Typical use
`HF_TOKEN`	Hugging Face token for gated models and authenticated downloads.
`WANDB_API_KEY`	Weights & Biases API key for experiment logging.
`DATABRICKS_HOST` / `DATABRICKS_TOKEN`	Databricks or MLflow-related credentials when those integrations are used.

Practices

Practice	Detail
Do not hardcode secrets	Avoid putting tokens or passwords directly in YAML, shell history, or committed scripts.
Prefer indirection	Use `${VAR}` substitution in configs to reference environment-injected values rather than literals.
Slurm	Use `--export` deliberately; prefer site-specific secret injection or credential helpers where available.
Containers	Pass secrets with `--env` or via `runner/.primus.yaml` env forwarding—never bake them into images.
Rotation	Rotate API keys and tokens on a schedule and after personnel or scope changes.

A broader catalog of variables appears in Environment variables.

3. Container security#

Primus-oriented container runs often require elevated access so ROCm, profilers, and high-performance networking behave correctly.

Common high-privilege options

Option	Typical purpose
`--privileged true`	Broad device access (often required for ROCm workflows on some setups).
`--cap-add SYS_PTRACE`	Debugging and profiling tooling.
`--cap-add CAP_SYS_ADMIN`	Administrative operations expected by parts of the ROCm/tooling stack.
`--security-opt seccomp=unconfined`	Relaxes seccomp constraints for compatibility with drivers and tools.
`--ipc host`	Shared memory semantics for large tensors and collectives.
`--network host`	Host networking—frequently used for multi-node RCCL performance and simplicity.

Device access (examples)

Device	Role
`/dev/kfd`	ROCm kernel interface.
`/dev/dri`	GPU render nodes.
`/dev/infiniband`	InfiniBand character devices when using IB.

Risks

Risk	Why it matters
Privileged containers	Substantial host access; container escape or compromise has high impact.
Host networking	Exposes the container to the host’s network namespace; services may bind broadly.
Shared IPC	Potential for cross-process interference or information leakage if workloads share hosts improperly.

Mitigations

Mitigation	Detail
Dedicated training nodes	Run training on isolated machines rather than mixed with user-facing services.
Network controls	Apply firewall rules and segmentation so only required ports and peers are reachable.
Trusted images	Pull from trusted registries, pin digests, and verify image provenance.
Monitoring	Track CPU, memory, GPU, and network usage; alert on anomalous processes or egress.

4. Third-party dependencies#

Primus integrates third-party submodules and Python packages; each carries its own license and maintenance cadence.

Representative submodules

Component	Notes (non-exhaustive)
Megatron-LM	MIT License; NVIDIA upstream.
TorchTitan	Apache 2.0; PyTorch / Meta ecosystem.
MaxText	Apache 2.0; Google upstream.
Megatron-Bridge	NVIDIA NeMo ecosystem.
Emerging-Optimizers	NVIDIA NeMo ecosystem.
HummingbirdXT	AMD AGI ecosystem.

Python dependencies

Runtime tooling often includes packages such as loguru, wandb, nltk, matplotlib, mlflow, and others as declared in project requirements—verify the canonical list in the repository’s requirements.txt (or lockfile) for your revision.

Recommendations

Recommendation	Rationale
Pin versions	Reproducible builds and controlled upgrade paths.
Update submodules	Security and correctness fixes flow from upstream projects.
Monitor advisories	Subscribe to upstream security notices for frameworks you enable.

5. Network security#

Property	Detail
RCCL / NCCL traffic	Not encrypted at the application layer; assumes a trusted network path.
Coordination	`MASTER_ADDR` and `MASTER_PORT` should reside on a private or otherwise trusted segment.
InfiniBand	Often on a dedicated fabric; still treat adjacent compromised hosts as in-scope for lateral movement.
TLS / mTLS	Primus does not provide TLS or mTLS for inter-node training traffic by default.

For physical and logical networking topics, see Multi-node networking.

6. Data security#

Asset	Consideration
Training data	May include PII, licensed corpora, or export-controlled material—classify and restrict accordingly.
Checkpoints	Contain full model state; treat as sensitive intellectual property.
Storage permissions	Use least privilege on shared filesystems and object stores.
`HF_TOKEN`	Grants access to gated Hugging Face assets—protect like any other long-lived credential.

Checkpoint formats and operational practices are described in Checkpoint management.

7. What is not verified#

The following items reflect typical gaps in public-facing evidence for many research and infrastructure codebases; confirm against your organization’s audits and CI for your fork and deployment.

Topic	Status (evidence-based caveat)
Independent security audit	No comprehensive third-party audit of this codebase is asserted here.
CI secrets scanning	No guarantee of automated secret detection in CI unless your pipeline adds it.
Dependency vulnerability scanning	No guarantee of continuous SCA unless your pipeline adds it.
Container images	Images may contain unpatched OS or Python packages—scan and rebuild on a schedule.
RCCL / NCCL traffic	Not encrypted or mutually authenticated by default; rely on network trust boundaries.

Use this section as a checklist for your production controls: add scanning, signing, policy-as-code, and periodic reviews appropriate to your threat model.