# Monitoring and logging

This page summarizes how Primus configures application logging, experiment tracking (Weights & Biases, TensorBoard, MLflow), training metrics, profilers, ROCm memory probes, and how to capture a reproducible configuration snapshot.

---

## 1. Primus logging system

Primus uses **loguru** for structured logging. Initialization wires **file sinks** (per log level) and a **stderr** sink, binds experiment and distributed context (`team`, `user`, `exp`, `module_name`, `node_ip`, `rank`, `world_size`), and installs an **intercept handler** so legacy `logging` output from frameworks such as Megatron is forwarded to loguru with consistent formatting.

**Rank-aware behavior**

- Worker processes write under `{exp_root}/logs/{module_name}/rank-{rank}/` with separate rotated files for `debug`, `info`, `warning`, and `error` (subject to `file_sink_level`).
- The launcher **master** process can use `logs/master/` when the master logger is configured with `is_head=True`.

**Levels from module configuration** (`primus/configs/modules/module_base.yaml`)

| Parameter | Default | Role |
|-----------|---------|------|
| `sink_level` | `null` | If set, overrides both file and stderr sink levels. |
| `file_sink_level` | `DEBUG` | Minimum level for file sinks when `sink_level` is unset. |
| `stderr_sink_level` | `INFO` | Minimum level for stderr when `sink_level` is unset. |

`init_worker_logger` in `primus/core/runtime/logging.py` reads `sink_level`, `file_sink_level`, and `stderr_sink_level` from the merged module config. The Megatron trainer maps `stderr_sink_level` to Megatron’s numeric `logging_level` (the `logging_level` field in `trainer_base.yaml` is deprecated; this mapping supersedes it)..

**Shell / runner environment** (see `docs/03-configuration-reference/environment-variables.md`)

| Variable | Purpose |
|----------|---------|
| `PRIMUS_LOG_LEVEL` | Runner verbosity: `DEBUG`, `INFO`, `WARN`, `ERROR` (default `INFO`). |
| `PRIMUS_LOG_TIMESTAMP` | `1` enables timestamps on runner logs; `0` disables. |
| `PRIMUS_LOG_COLOR` | `1` enables ANSI colors when appropriate; often `0` in non-TTY contexts. |

**CLI**

- `primus-cli --debug` sets `PRIMUS_LOG_LEVEL=DEBUG` so launcher and shell logging are verbose (see `docs/02-user-guide/cli-reference.md`).

---

## 2. Weights & Biases

### Megatron

Configuration files: `primus/configs/modules/megatron/trainer_base.yaml` and `primus_megatron_module.yaml`.

Defaults in `primus_megatron_module.yaml` disable Weights & Biases; trainer fields in `trainer_base.yaml` supply names and paths when enabled.

| Parameter | Default (module / trainer) | Description |
|-----------|----------------------------|-------------|
| `disable_wandb` | `true` (`primus_megatron_module.yaml`) | Master switch; when `false`, Primus sets paths and default project/run names from experiment metadata. |
| `wandb_project` | `null` | If unset when Weights & Biases is enabled, defaults to `{work_group}_{user_name}`. |
| `wandb_exp_name` | `null` | If unset, defaults to `exp_name`. |
| `wandb_entity` | `null` | Optional WandB entity/team. |
| `wandb_save_dir` | `null` | Deprecated in favor of `{exp_root}`; artifacts use `{exp_root}/wandb`. |

**Environment**

- `WANDB_API_KEY` is **required** when Weights & Biases is enabled; Primus emits a warning if it is missing (`primus/backends/megatron/patches/args/wandb_config_patches.py`).

### TorchTitan

Configuration file: `primus/configs/modules/torchtitan/pre_trainer.yaml`.

| Parameter | Default | Description |
|-----------|---------|-------------|
| `metrics.enable_wandb` | `false` | Enables WandB in the TorchTitan metrics stack. |

When enabled, `primus/backends/torchtitan/patches/wandb_patches.py` can set `WANDB_PROJECT` and `WANDB_RUN_NAME` from Primus experiment metadata if unset. Use `WANDB_API_KEY` for authentication.

---

## 3. TensorBoard

### Megatron

**Module toggles**

Configuration file: `primus_megatron_module.yaml`

| Parameter | Default | Description |
|-----------|---------|-------------|
| `disable_tensorboard` | `true` | When `false`, TensorBoard output is placed under `{exp_root}/tensorboard` (Primus overrides deprecated `tensorboard_dir` with this path). |

**Trainer**

Configuration file: `trainer_base.yaml`

| Parameter | Default | Description |
|-----------|---------|-------------|
| `tensorboard_log_interval` | `1` | Steps between TensorBoard writes. |
| `tensorboard_queue_size` | `1000` | Event file queue size. |
| `log_timers_to_tensorboard` | `false` | Log timer stats. |
| `log_batch_size_to_tensorboard` | `false` | Log batch size. |
| `log_learning_rate_to_tensorboard` | `true` | Log learning rate. |
| `log_validation_ppl_to_tensorboard` | `false` | Log validation perplexity. |
| `log_memory_to_tensorboard` | `false` | Log memory stats. |
| `log_world_size_to_tensorboard` | `false` | Log world size. |
| `log_loss_scale_to_tensorboard` | `true` | Log loss scale. |
| `tensorboard_dir` | `null` | Deprecated; Primus sets the directory under `exp_root`. |

**Note:** Enabling Megatron **profiling** (`profile: true`) forces `disable_tensorboard` off in `update_primus_config` so TensorBoard is available for profile-related views.

### TorchTitan

Configuration file: `pre_trainer.yaml`.

| Parameter | Default | Description |
|-----------|---------|-------------|
| `metrics.enable_tensorboard` | `false` | Enables TensorBoard logging. |
| `metrics.save_tb_folder` | `tb` | Subfolder name (typically under the job dump directory in TorchTitan layouts). |

**Launch TensorBoard locally**

```bash
tensorboard --logdir <path-to-tensorboard-or-tb-folder>
```

Point `<path>` at the Megatron `tensorboard` directory under the experiment root, or at the TorchTitan metrics folder that contains the `save_tb_folder` subtree.

---

## 4. MLflow

MLflow integration is **Megatron-only** in the paths described here.

**Module**

Configuration file: `primus_megatron_module.yaml`

| Parameter | Default | Description |
|-----------|---------|-------------|
| `disable_mlflow` | `true` | When `false`, MLflow run setup runs on the **last** global rank (`world_size - 1`). |
| `mlflow_run_name` | `null` | If unset when enabled, defaults to `{work_group}_{user_name}`. |
| `mlflow_experiment_name` | `null` | Passed to `mlflow.set_experiment` when set. |

**Startup behavior**

Configuration file: `primus/backends/megatron/training/global_vars.py`

- Logs training `args` as parameters.
- Logs filtered environment variables with an `env__` prefix.
- Collects git metadata, sets MLflow source tags, and writes `system/git_metadata.json` as a run artifact.

**Environment** (typical Databricks / hosted tracking)

| Variable | Role |
|----------|------|
| `DATABRICKS_HOST` | Checked by the Megatron trainer when MLflow is enabled; a warning is printed if unset. |
| `DATABRICKS_TOKEN` | Authentication for Databricks-hosted tracking (see environment reference). |
| `MLFLOW_TRACKING_URI` | Tracking server URI; optional depending on deployment. |

---

## 5. Training metrics

### Megatron

Configuration file: `trainer_base.yaml`

| Parameter | Default | Description |
|-----------|---------|-------------|
| `log_interval` | `100` | Steps between standard training log lines. |
| `log_throughput` | `false` | Log throughput metrics. |
| `log_avg_skip_iterations` | `2` | Skip initial iterations when computing averages. |
| `log_avg_reset_interval` | `10` | Interval for resetting running averages. |
| `log_params_norm` | `false` | Log parameter norm. |
| `log_num_zeros_in_grad` | `false` | Log count of zero gradients. |
| `log_progress` | `false` | Progress-style logging. |
| `timing_log_level` | `0` | Timing log verbosity. |
| `timing_log_option` | `minmax` | Timing aggregation option. |

### TorchTitan

Configuration file: `pre_trainer.yaml`.

| Parameter | Default | Description |
|-----------|---------|-------------|
| `metrics.log_freq` | `10` | Metric logging frequency (steps). |
| `metrics.disable_color_printing` | `false` | Disable colored console metrics. |
| `metrics.save_for_all_ranks` | `false` | Save metrics from every rank vs. reduced ranks. |

---

## 6. Profiling

### Megatron

Configuration files: `trainer_base.yaml` and `primus_megatron_module.yaml`

| Parameter | Source | Default | Description |
|-----------|--------|---------|-------------|
| `profile` | `trainer_base.yaml` | `false` | Enables Megatron profiling path; also forces TensorBoard on when `true`. |
| `use_pytorch_profiler` | `trainer_base.yaml` | `false` | Use PyTorch profiler integration. |
| `profile_ranks` | `trainer_base.yaml` | `[0]` | Ranks to profile. |
| `profile_step_start` | `trainer_base.yaml` | `10` | First step to profile. |
| `profile_step_end` | `trainer_base.yaml` | `12` | Last step to profile. |
| `record_memory_history` | `trainer_base.yaml` | `false` | Record memory history. |
| `memory_snapshot_path` | `trainer_base.yaml` | `snapshot.pickle` | Memory snapshot file name. |
| `disable_profiler_activity_cpu` | `primus_megatron_module.yaml` | `false` | Disable CPU activities in the profiler. |
| `torch_profiler_record_shapes` | `primus_megatron_module.yaml` | `true` | Record tensor shapes. |
| `torch_profiler_with_stack` | `primus_megatron_module.yaml` | `true` | Capture Python stacks. |
| `torch_profiler_use_gzip` | `primus_megatron_module.yaml` | `false` | Gzip profiler traces. |

### TorchTitan

Configuration file: `pre_trainer.yaml`.

| Parameter | Default | Description |
|-----------|---------|-------------|
| `profiling.enable_profiling` | `false` | Master profiling toggle. |
| `profiling.profile_freq` | `10` | How often to capture traces. |
| `profiling.enable_memory_snapshot` | `false` | Enable memory snapshots. |
| `profiling.save_memory_snapshot_folder` | `memory_snapshot` | Output folder for snapshots. |
| `profiling.save_traces_folder` | `profile_traces` | Folder for profiler traces. |

---

## 7. ROCm memory monitoring

Configured in `primus/configs/modules/megatron/primus_megatron_module.yaml` and applied in the Megatron trainer when logging throughput.

| Parameter | Default | Description |
|-----------|---------|-------------|
| `use_rocm_mem_info` | `false` | When `true`, collect ROCm memory information via `rocm-smi` on **every** iteration that hits the throughput logging branch. |
| `use_rocm_mem_info_iters` | `[1, 2]` | When `use_rocm_mem_info` is `false`, `rocm-smi` runs only on these iteration numbers (same branch). |

Collection is evaluated where `log_throughput` drives the extended iteration log (see `primus/backends/megatron/patches/training_log/print_rank_last_patches.py`): enable `log_throughput` in `trainer_base.yaml` (or overrides) when you need ROCm memory lines in the training log.

---

## 8. Experiment snapshots

**On disk (every run)**

- **Experiment root**: `{workspace}/{work_group}/{user_name}/{exp_name}` is created at config load time (`PrimusConfig`).
- **Per-rank logs**: `{exp_root}/logs/{module_name}/rank-{rank}/` with rotated level-specific files.
- **Checkpoints**: Megatron uses `{exp_root}/checkpoints` (trainer sets `save` to this path).
- **TensorBoard / WandB**: Under `exp_root` as described above when those features are enabled.

**MLflow** (Megatron, when enabled): Parameters, environment snapshot, and git metadata artifact provide a structured record of the run configuration and repository state.

**Resolved configuration**

The launcher and parser accept `--export_config`, but the default core training path (`primus/cli/subcommands/train.py` into `PrimusRuntime`) does not currently write a resolved YAML file. Archive the submitted experiment YAML, any referenced presets, launcher config, and runtime logs with each run. Treat resolved-config export as a legacy or future capability unless your deployment has implemented it on the core runtime path.