# ATOM Environment Variables

This document describes the environment variables used in the ATOM project.

## OpenTelemetry tracing

| Variable | Type | Default | Description |
|----------|------|---------|-------------|
| **ATOM_ENABLE_OTEL_TRACE** | bool | false | Enable standalone request tracing with `1` or `true` (case insensitive). Requires the optional `tracing` dependencies and an explicit OTLP/HTTP traces endpoint. Read at process startup; restart to change. |
| **ATOM_TRACE_VERSION** | str | `unknown` | Exact build version attached to each tracing resource as `atom.version`. Set before startup. |

The standard `OTEL_SERVICE_NAME` and `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT`
variables configure service identity and delivery. See [tracing setup and
limitations](observability/README.md).

## Metadata H2D

| Variable | Type | Default | Description |
|----------|------|---------|-------------|
| **ATOM_H2D_BACKEND** | str | `packed` | `packed` combines forward metadata into one H2D and GPU scatter per consumer group. `direct` copies each member separately. Both preserve source reuse gates and full cudagraph padding. Set before starting the runner. See [metadata publication](h2d_publication.md). |

## Data parallelism

| Variable | Type | Default | Description |
|----------|------|---------|-------------|
| **ATOM_DP_RANK** | int | 0 | The rank ID for the current process in data parallelism. |
| **ATOM_DP_RANK_LOCAL** | int | 0 | The local rank ID for the current process (used in SPMD mode). |
| **ATOM_DP_SIZE** | int | 1 | Total number of data parallel ranks. |
| **ATOM_DP_MASTER_IP** | str | 127.0.0.1 | Master IP address for DP ranks coordination. |
| **ATOM_DP_MASTER_PORT** | int | 29500 | Master port for DP ranks coordination. |
| **ATOM_DP_LB_REQ_EQUIV** | int | 512 | Token-equivalent decode pressure assigned to each in-flight request by `least_tokens` routing. |
| **ATOM_DP_SESSION_AFFINITY** | bool | false | Load-place each new session, then keep later turns on the same prefix-cache owner. Reads `X-Dynamo-Session-ID`, falling back to `X-Correlation-ID`. |
| **ATOM_DP_PREFILL_BALANCE_TOKEN_WEIGHT** | int | 360000 | DP prefill cost balancing (always on under DP attention with more than one rank; needs the prefill delayer): each lockstep prefill step aims at min(costliest, mean x (1 + slack)) of the ranks' estimated chunk costs, and a costlier rank takes fewer tokens. This is the per-token term of the chunk cost, in attended-context tokens: cost = new tokens x (weight + prior context + new / 2). |
| **ATOM_DP_PREFILL_BALANCE_SLACK** | float | 0.1 | The target cost is at most mean x (1 + slack) over the ranks that prefill. |
| **ATOM_DP_PREFILL_BALANCE_MIN_TOKENS** | int | 4096 | Fewest tokens a shrunk rank still prefills in a step. |

## Prefill delayer (TP/DCP and DP attention)

Coalesces waiting prefills while decode continues. DP attention enables it by
default through `ATOM_ENABLE_PREFILL_DELAYER`. For a single scheduler (DP=1,
PP=1), including TP/DCP, it is opt-in: set `ATOM_PREFILL_DECODE_INTERVAL` above
zero and keep the master switch enabled. Interval 0 leaves TP scheduling
unchanged; setting only the master switch does not enable TP coalescing.
On TP, the interval and coalescer are enabled together.
This applies to the standard scheduler, including connector-based P/D roles.
RapidServe's dedicated `PrefillScheduler`/`DecodeScheduler` do not use the delayer.

After each executed prefill, the decode interval runs before all coalescing
bounds. Once it expires, fill, queue age, KV pressure, partial-prefill and stall
bounds decide when to release. `MAX_QUEUE_MS` stops extra coalescing after that
interval; it does not guarantee end-to-end TTFT. DP decisions reduce local
signals across ranks to keep their phases aligned.

The local fill signal discounts HBM cache hits and uses the admission path's
chunk limits. To bound CPU work, it stops probing fresh requests after their
total prompt length reaches one batch budget. It reports only work found so
far; it does not treat unseen requests as a full batch. A deep, cache-heavy
queue can therefore release through the stall or hold bounds before reaching
the fill target. If parked transfers exhaust the unreserved slots, a fresh
request may signal possible work with zero estimated tokens until admission
resolves its connector match. This is an estimate, not a reservation: checkpoint dependencies,
connector results and resource changes during admission can still reduce a batch.
Repeated probes reuse immutable prompt hashes while rechecking pool contents
and resource fit. During decode protection, only the existence of queued or
partial work is checked; partial-prefill hold bounds start after the interval.

Local hybrid models with state checkpointing can wait for an in-flight
producer's reusable prompt-end checkpoint. The expected prefix must exceed both
the HBM hit and any offload match by at least one prefill budget. P/D transfers
and already-started offload loads keep their own progress paths. Deferred
requests retain their relative order, while at most 16 later queue entries are
examined for independent work each pass. Each request's wait expires after
`TTFT_MAX_TICKS` scheduler passes from its first dependency wait; bypassing it
does not restart that deadline. `MAX_QUEUE_MS` bounds coalescing, not this
dependency wait: time spent queued before a producer becomes runnable does not
make duplicate prefill useful. Pure-attention models do not use checkpoint waits.

| Variable | Type | Default | Description |
|----------|------|---------|-------------|
| **ATOM_ENABLE_PREFILL_DELAYER** | bool | true | Master switch for the prefill coalescer. |
| **ATOM_PREFILL_DELAYER_TARGET_FILL** | float | 0.9 | Release once accumulated pending tokens reach `target_fill × max_num_batched_tokens` (averaged across prefillable ranks). In (0, 1]; higher = fewer, larger prefills at some TTFT cost. Clamped to (0, 1]. |
| **ATOM_PREFILL_DELAYER_TTFT_MAX_TICKS** | int | 200 | Max consecutive scheduler ticks a held prefill waits before force-release. Values `< 1` clamped to 1. |
| **ATOM_PREFILL_DELAYER_PARTIAL_MAX_TICKS** | int | 100 | Tighter bound for a held mid-chunked-prefill (it holds allocated KV). Values `< 1` clamped to 1. |
| **ATOM_PREFILL_DELAYER_STALL_TICKS** | int | 10 | After this many consecutive non-growing ticks, release (burst ended, more won't come), including while some DP rank has no prefill. Values `< 1` clamped to 1. |
| **ATOM_PREFILL_DELAYER_COST_GATE** | bool | true | With DP prefill cost balancing (DP attention, DP > 1): while some DP rank has no prefill, also release when a held decode step is not expected to pay off (request arrival rate on the ranks without prefill × the decode stall of one unaligned prefill step < waiting requests), or once the held requests have waited (in measured decode steps) as long as that stall. Step times and arrival rates are measured per rank and combined over the DP group; until every rank has timed both steps, only the stall bound releases. |
| **ATOM_PREFILL_DELAYER_KV_HIGH_WATERMARK** | float | 0.9 | At/above this KV usage a prefillable rank force-releases (can't accumulate a bigger batch anyway). |
| **ATOM_PREFILL_DELAYER_TOKEN_USAGE_LOW_WATERMARK** | float\|"" | "" (None) | If set, a prefillable rank below this KV usage force-releases (GPU starving). |
| **ATOM_PREFILL_DELAYER_MAX_QUEUE_MS** | float\|"" | "" (None) | After decode protection, release coalescing when the oldest schedulable waiting prefill reaches this age since arrival. Empty disables the age guard. Checkpoint dependency waits use `TTFT_MAX_TICKS`. This is not a hard TTFT limit. |
| **ATOM_PREFILL_DECODE_INTERVAL** | int | 0 | Protect this many scheduler passes after an executed prefill. On DP=1, PP=1, a positive value also enables local coalescing when the master switch is on; `0` leaves TP scheduling unchanged. On DP>1, `0` disables only the interval. |
| **ATOM_PREFILL_DELAYER_DEBUG** | bool | false | Per-tick FIRE/HOLD debug logging. |
| **ATOM_PREFILL_DELAYER_LOG_EVERY** | int | 1000 | Emit aggregate stats (per-exit fire counts + hold rate) every N decisions (0 disables). |

## Dynamic chunking

Interpolates pipeline-parallel chunk sizes between the configured batch budget
and the equal-latency size solved from the calibrated chunk model. Used when
neither `--dynamic-chunking-smooth-factor` nor `Config.dynamic_chunking_smooth_factor`
is given, and read only when dynamic chunking is enabled, so a malformed value
cannot affect a server that does not use the feature.

| Variable | Type | Default | Description |
|----------|------|---------|-------------|
| **ATOM_DYNAMIC_CHUNKING_SMOOTH_FACTOR** | float | 0.75 | Blend from the initial chunk (`0`) to the equal-latency chunk (`1`). Must be in `[0, 1]` when dynamic chunking is enabled. |

## Model loading

| Variable | Type | Default | Description |
|----------|------|---------|-------------|
| **ATOM_DISABLE_MMAP** | bool | false | If set to `true`, disable memory-mapped file loading for model weights. Useful in containerized environments where mmap may cause issues. |
| **ATOM_LOADER_NUM_THREADS** | int | 16 | Worker threads for weight loading. `>1` (default `16`) enables the batched parallel loader (routed expert weights staged in a CPU buffer, flushed with a single H2D copy when every routed expert of that parameter has arrived) with that many threads; set to `1` to fall back to the original sequential per-expert path. Raise on high-core hosts if loading is CPU-bound. |
| **ATOM_TOKENIZE_CACHE_MB** | int | 512 | Host memory, in MB, the API server keeps for reusing prompt tokenization across requests. A chat template separates messages with added (special) tokens, and the text between two of them tokenizes the same on its own. Each prompt is kept with its ids up to its last added token, so the next turn of the conversation reuses them up to the last added token both share; the rest is split at added tokens and each segment's ids are cached by content, the missing ones encoded in one parallel batch. Both are exact: they are used only when the tokenizer's added tokens, normalizer and pre-tokenizer allow it, and the first prompts are checked against the plain encoder (any difference turns the cache off). Prompts under 100K characters are encoded whole. One budget covers segment ids and stored prompts, counted by the bytes they retain (ids, text, per-entry overhead). The result still covers the whole prompt, so a request costs time linear in its length; what is saved is re-encoding text already encoded. Lowers TTFT on long multi-turn conversations; ITL and throughput are unaffected. `0` turns it off. |
| **ATOM_LOADER_STRICT_COVERAGE** | bool | `true` | Fail loading when a fused MoE parameter does not receive every routed expert from the checkpoint. Set to `false` to downgrade to a warning and load anyway, leaving those expert slots at their init values — useful when bringing up a checkpoint known to be partial, misleading otherwise (the symptom is an accuracy drop much later). |
| **ATOM_LOADER_PREFETCH** | bool | `true` | Warm the page cache by reading this rank's share of the checkpoint sequentially on a background thread, instead of leaving it to demand faults through the mmap. The fault pattern sustains ~3.2 GB/s on a local NVMe that a single sequential reader drives at 6.06 GB/s, so this is an access-pattern fix, not a queue-depth one. Measured on DeepSeek-R1 MXFP4 (350 GiB, TP=4): cold load 154s → 69s. Set to `false` to restore demand faulting. Also applies with `ATOM_DISABLE_MMAP=true`, whose whole-file reads go through the same page cache (DeepSeek-V4.1-Flash, TP=4, cold: 282–315s without it, 169–207s with it). |
| **ATOM_LOADER_PREFETCH_THREADS** | int | 4 | Concurrent sequential readers per rank used by the prefetcher. The device saturates at ~2 streams, so raising this mostly adds contention with the loader; `0` is clamped to `1` (use `ATOM_LOADER_PREFETCH=false` to switch prefetching off). |
| **ATOM_LOADER_PREFETCH_BLOCK_MB** | int | 16 | Read block size for the prefetcher, in MiB. |
| **ATOM_LOADER_FADVISE** | bool | `false` | Issue `posix_fadvise(SEQUENTIAL\|WILLNEED)` per shard before reading it. Off by default and ignored while `ATOM_LOADER_PREFETCH` is on: `WILLNEED` is a hint the kernel drops for most of a 350 GiB checkpoint, and running both makes the kernel read ahead over random-ish ranges while the prefetcher streams the same files, so the two compete for the device. Only useful with prefetching disabled. |
| **ATOM_ONLINE_QUANT_STREAMING** | bool | `false` | Opt in to quantizing eligible online-quant modules as soon as their checkpoint weights are complete, then release source storage to reduce load-time peak memory. Only active with a valid online quantization config. See the [streaming online quantization guide](./online_quantization_streaming_guide.md). |
| **ATOM_ONLINE_QUANT_STREAMING_HOST_STAGING** | bool | `true` | Assemble streamed module weights in CPU storage before one H2D transfer. Keeps the checkpoint walk parallel; disabling it buffers loader calls and forces the checkpoint walk to one thread. |
| **ATOM_ONLINE_QUANT_STREAMING_THREADS** | int | `4` | Tail workers for H2D, per-module quantization, and source release. More workers increase overlap and in-flight memory; `0` runs finalization inline. |

## Plugin mode

| Variable | Type | Default | Description |
|----------|------|---------|-------------|
| **ATOM_DISABLE_VLLM_PLUGIN** | bool | 0 (false) | If set to `1`, disable the vLLM plugin registration entirely. |
| **ATOM_V4_CG_INSTR_DIR** | str | — (off) | vLLM plugin, DeepSeek-V4 diagnostic: directory for per-process cudagraph dispatch counters (dispatched modes, steps whose uniform decode was overridden for a still-prefilling row, guard trips), one `cg_<pid>.json` per process, rewritten on every step. |

## Kernel / backend selection

| Variable | Type | Default | Description |
|----------|------|---------|-------------|
| **ATOM_USE_TRITON_GEMM** | bool | 0 (false) | If set to `1`, use AITER Triton FP4 weight preshuffled GEMM. Otherwise use AITER ASM FP4 weight preshuffled GEMM. |
| **ATOM_FP8_BLOCKSCALE_USE_E8M0_SCALE** | bool | unset (per checkpoint) | E8M0 rather than FP32 128x128 FP8 block scales, for the weight and the activation quantized for it; on gfx950 E8M0 operands take the AITER FlyDSL GEMM. Unset: E8M0 on every arch but gfx942 when the checkpoint declares `scale_fmt: ue8m0` (DeepSeek-V4), whose power-of-two scales E8M0 restates exactly; FP32 otherwise. `1`/`0` force it on/off; forcing it on for a checkpoint with non-power-of-two FP32 scales rounds them. |
| **ATOM_GROUP32_WEIGHT_PRESHUFFLE** | bool | 1 (true) | On gfx950, (16, 16)-shuffle native FP8 32x32 group32 weights (DeepSeek-V4.1) at load and after a weight sync, so AITER's preshuffled group32 GEMM (FlyDSL) reads them; that GEMM emits BF16 only. Set to `0` to keep checkpoint bytes and the row-major group32 GEMM. V4.1's grouped `wo_a` is shuffled either way. |
| **ATOM_USE_FP4_NON_SHUFFLE_TRITON_GEMM** | bool | 0 (false) | If set to `1`, use AITER Triton FP4 GEMM with non-shuffled weights. Takes precedence over the FP4 preshuffled GEMM path selected by `ATOM_USE_TRITON_GEMM`. |
| **ATOM_MHC_USE_BF16** | bool | 1 (true) | Use AITER BF16 hi/lo mHC computation for attention, FFN and head. After loading, replace FP32 fn storage with `mhc_shuffle_fn` output; no FP32 copy is retained. Set to `0` for FP32 mHC. Takes effect at model load; restart to change modes. On gfx1250, AITER enables shuffled residuals only while its runtime `mhc_fused_post_pre` policy remains fused (`M < 1024`); larger M uses ordinary residual layout and the standalone post/pre fallback. |
| **ATOM_DSV41_MHC_BACKEND** | str | `auto` | V4.1 delayed post/pre/RMSNorm backend, selected after checkpoint loading. `auto` packs each FP32 MHC weight once for HIP on gfx950 with `hc_mult=4`, H=4096/5120/6144/7168 and AITER `mega_mhc_shuffle_fn` support; otherwise it uses Triton. `hip` requires those conditions; `triton` forces Triton. Activations/norm weights and outputs are BF16. Each MHC fn keeps one packed copy shared across shapes and streams; AITER reuses weight-independent scratch per device/stream. Warm up each capture stream with sufficient scratch capacity; calls and graphs sharing that scratch must execute serially. Set the backend before loading. Reload repacks weights and requires graph recapture. Original FP32 weights remain available for mono decode's own kernels. |
| **ATOM_USE_TRITON_MXFP4_BMM** | bool | 0 (false) | If set to `1`, use FP4 BMM in MLA attention module. |
| **ATOM_USE_FLYDSL_GATHER_KV_B_PROJ** | bool | 1 (true) | Use the FlyDSL fused gather + `kv_b_proj` GEMM for MLA's cached-prefix path. Covers page_size-1 fp8 (e4m3) KV with an fp8 weight on gfx950 — i.e. Kimi-K3 / DeepSeek MLA under `--kv-cache-dtype fp8`. Unavailable imports or unsupported tensor configurations use Triton; kernel execution errors propagate. Set `0` to force Triton. |
| **ATOM_USE_FLYDSL_FP8_PREFILL_ATTN** | bool | 0 (false) | Enable FlyDSL FP8 MLA prefill on gfx950, including fused K concatenation with QKV quantization and direct FP8 output from FlyDSL gather where supported. AITER capability checks are cached per layer. Missing FP8 attention support or unsupported device/model configurations select BF16 attention before quantization and emit a warning once per process. FP16 activations and synthetic RoPE-padding configurations also use the normal attention path. Cached K/V reuse descales `max(new_token_descale, 1e-6) * 2`; larger outliers saturate. Unsupported FP8 gather configurations use BF16 gather followed by fused K/V dynamic quantization when FP8 attention is supported. Kernel execution errors propagate without retry. Added 2026-09-10. |
| **ATOM_PA_FLYDSL** | bool | 0 (false) | Route the MHA paged decode to aiter's FlyDSL kernel, with its work planner, instead of gluon. Shapes outside FlyDSL's domain, calls over 4096 sequences and calls whose context bound is below the query window fall back to gluon. Applies to the vLLM bridge too. **Needs aiter with #5809**; upgrade ATOM and aiter together. Plans use aiter's default workgroup budget (2×CU) and partition ceiling (CU count), except at head_dim 256 (Qwen3.5), where they use 64 partitions and a 256-workgroup budget to keep aiter's head-256 reducer on its fast path (until aiter #5949). In native ATOM, captured batch sizes use a plan built by the metadata builder (shared by target and EAGLE draft); other calls, and every vLLM bridge call, get a runtime plan, and a new row count compiles the planner once. Other native dense plans use aiter's default partition ceiling; MiniMax-M3 sparse and the vLLM bridge keep their static split count. MiniMax-M3 sparse decode refreshes its plan once per step. `ATOM_PA_FLYDSL_PLAN` was removed and is ignored with a warning. |
| **ATOM_V4_HCA_PERSIST** | bool | 1 (true) | DeepSeek-V4 HCA (`compress_ratio` 128) fp8 decode through aiter's persistent V4-NM kernel (`aiter.mla.mla_decode_fwd_v4_nm_ps`): one launch that splits the KV on the GPU and merges in-kernel, instead of the decode ASM plus a host-built split plan. Taken only with fp8 KV, 128 local heads, gfx950, row-dense KV pools and `ATOM_V4_HCA_PERSIST_MIN_ROWS <= rows <= 32768`; everything else stays on the ASM path. An aiter without the kernel keeps the ASM path (logged once). One workspace per device (about 64 MiB) is allocated at model load, before KV sizing and CUDA-graph capture. Set `0` to force the ASM path. |
| **ATOM_V4_HCA_PERSIST_MIN_ROWS** | int | 15 | Smallest decode call (q rows) that uses `ATOM_V4_HCA_PERSIST`; smaller calls stay on the ASM path. |
| **ATOM_DCP_SPARSE_PREFILL_NONPS** | bool | 1 (true) | DSA models under DCP: run sparse prefill attention on the non-persistent fp8 gqa64 MLA kernel, which needs no work metadata. Applies with fp8 q/KV, 64 gathered heads (tp4/dcp4, tp8/dcp8), gfx950, `ATOM_MLA_PAGE_SIZE=1` and `--max-num-batched-tokens` <= 32768 (each prefill token is one kernel row; a larger step budget logs a warning and keeps the default kernel). Needs an aiter that ships the kernel (ROCm/aiter#6132); with an older aiter set `0`, or the first sparse prefill past `index_topk` raises. |
| **ATOM_SPARSE_INDEXER_LOGITS_BUDGET_MB** | int | 2048 | Sparse (DSA) indexer dense fp32 logits budget, in MiB. Prefill scores query rows in bands within it. It also sizes the decode logits workspace (`LogitsWorkspace`, as wide as a request of `max_model_len` sees -- a pooled indexer's pools of it -- allocated once and counted in the KV budget) and so how many bands a decode step is scored in: lowering it saves memory but adds launches. `0` keeps only aiter's hard 2 GiB buffer-descriptor cap. |

### GLM-5.3

| Variable | Type | Default | Description |
|----------|------|---------|-------------|
| **ATOM_GLM5_KPOOL** | bool | 1 (true) | Enable the pooled sparse indexer. Setting `0` is an exact token-granular A/B only at or below `index_topk`; longer requests are refused. |
| **ATOM_GLM5_FORCE_DENSE_MLA** | bool | 0 (false) | Disable sparse MLA for short-context bring-up comparisons. |
| **ATOM_GLM5_DISABLE_FUSED_MHC** | bool | 0 (false) | Force the PyTorch mHC reference path instead of AITER's fused kernels. |

### Native model decode

| Variable | Type | Default | Description |
|----------|------|---------|-------------|
| **ATOM_MONO_ENABLE** | bool | 1 (true) | Enable fused decode for supported GLM-5.2, Kimi-K3, MiniMax-M3, and DeepSeek-V4.1 deployments and step geometries. Unsupported configurations and forwards keep the production path; `0` disables every Mono runner. |

### MiniMax-M3

| Variable | Type | Default | Description |
|----------|------|---------|-------------|
| **ATOM_MONO_CHECK** | bool | 0 (false) | Debug aid for `ATOM_MONO_ENABLE`. MiniMax-M3: the dense layers run mono and then the original modules, compared where they hand the first sparse layer its input; every sparse layer also runs the original decoder layer on the same input (after the mono layer, so mono reads only the cache entries it inserted) and rank 0 logs the per-layer, per-token difference of the residual and of the reduced output. DeepSeek-V4.1: K2a and K2b are launched apart and every layer's kernel outputs are compared with the original modules run on the same inputs (K1: seam, query, window row, indexer query / weights; the indexer's selection and candidate blocks against the original scorer; K2a: its inverse-RoPE quant, `wo_a`, residual and FFN seam; K2b: router logits and the reduced MoE output, the reference also run twice to show its own run-to-run change), logged on rank 0 as the fraction of differing elements, the largest difference and the relative error (`atom/mono/runtime/compare.py`; MiniMax-M3 logs the same three). Run it with `--enforce-eager`. |
| **ATOM_MONO_TRACE** | path | unset | Debug aid for `ATOM_MONO_ENABLE`: rank 0 appends every mono step's input tokens, positions, top-2 logits and greedy pick to this file (one JSON line per step), with a fingerprint of every stage (dense layers, each sparse layer's output, the cache history it reads). Two runs of the same prompt then align token by token. Run it with `--enforce-eager`. |
| **ATOM_MONO_DEBUG** | bool | 0 (false) | Debug aid for `ATOM_MONO_ENABLE`: the mono kernels are built with bounded mailbox waits. A wait still unmet after 10 s gives up and records its region, pair index, expected and seen tags and CTA; at the end of the step the runner raises with every record (a region recorded by its `region_id`, the CRC-32 of its name, and reported by name: the kernels' hand-off tables; DeepSeek-V4.1 adds its kernels' scratch regions, the peer regions and K2's readiness flags) instead of the GPU hanging on a peer or stage that never delivers; the step's peer fence (`atom/mono/runtime/step_begin.py`, after each zeroing of the peer buffer) likewise gives up and names the ranks that never arrived. Run it with `--enforce-eager`: the records are read after a synchronize, which a graph replay does not run. |
| **ATOM_MONO_TIMELINE** | path prefix | unset | Debug aid for `ATOM_MONO_ENABLE`: the mono layer kernels are built with their per-phase `s_memrealtime` stamps, each sparse layer into its own buffer, and after 20 warm-up steps of every decode token count S the next 5 steps are saved per rank as `<prefix>_r<rank>_s<S>_<i>.pt` (int64 [layer][CTA][stamp], 100 MHz ticks, 0 = not reached); DeepSeek-V4.1 saves its K1 and K2 stamps apart, as `<prefix>_k1_r...` and `<prefix>_k2_r...` (K2a's points, then K2b's; with `ATOM_MONO_CHECK` `<prefix>_k2a_r...` and `<prefix>_k2b_r...`). Run it with `--enforce-eager`: a graph replay runs no Python. |
| **ATOM_GLM52_FUSED_INDEXER** | bool | 0 (false) | GLM-5.2 only. Moves the sparse indexer *inside* the mono decode launch on the 18 "full" layers that actually select: index Q/K/W projection, the paged FP8 index-cache update, scoring and the exact top-2048 all run in one kernel instead of `_refresh_indexer`'s separate pass. Off leaves the external-indexer mono path untouched. Requires `--index_cache_dtype fp8` (the default when `--kv_cache_dtype fp8`), `--block-size 16` exactly (the index plane shares `block_tables` with the KV cache), and `--max-model-len` at or under `FUSED_INDEX_MAX_SEQ`, which is **1048576** tokens. 16384 is not a ceiling but the build granularity: `fused_index_capacity()` rounds each deployment UP to a whole 16384-token leaf and builds at that, because the scoring work map is enumerated from the compiled capacity rather than from the request — a 16 k deployment built against a 1 M capacity inflates that table 64x to do identical work (measured: +11.14 % against the external indexer at a 1 M build, -0.09 % at a 16 k build). Anything else is refused by name in `atom.models.glm52.mono.config.fused_indexer_refusal` and falls back to the external indexer unchanged; "shared" layers (57 of 78) never use this kernel either way. |
| **ATOM_GLM52_INDEXER_CP** | bool | 0 (false) | GLM-5.2 only, and strictly narrower than `ATOM_GLM52_FUSED_INDEXER` — on its own it does nothing. Shards index scoring and selection across the TP ranks inside the fused indexer: each rank scores a contiguous 64-token-aligned shard of the query's live context and the ranks exchange score shards through in-kernel peer mailboxes. Selection stays exact and the index caches stay replicated — CP distributes computation, not storage. Refusals in `atom.models.glm52.mono.config.indexer_cp_refusal`; a refusal leaves the fused indexer running exactly as it does with CP off. |

#### Mono decode routing

**DeepSeek-V4.1** (`atom/models/deepseek_v41/mono/`): two kernels a layer -- K1 (mHC seam, attention norm, q / kv projections, the indexer query) and K2, one launch of K2a (the indexer's scores and top-k at any context length, the sparse decode attention, the output projections and their all-reduce, the FFN seam) then K2b (the MoE and its all-reduce) -- with the compressor and Engram left to the original modules. It routes a decode step of up to 48 rows, however they split into requests (a DSpark verify of any length a request, confidence-scheduled or ragged included, or one token a request without a drafter; each row's window from its own batch id; one kernel build per width) and the DSpark draft's block backbone of up to 8 requests at the full block width (`draft_runner.py`; after a prefill as after a decode), the draft's LM head and Markov sampler unchanged. It needs TP2, TP4 or TP8 without EP (the routed experts sharded across TP; every per-rank width from `kernels/dims.py`; TP8's kernels are compile-checked only, since the original path does not start at TP8), DP / PP / CP 1, no TBO, no plugin mode, FULL cudagraph or eager, bf16 KV and an fp8 or fp4 index cache (fp4: K1 quantizes the indexer query to E2M1 and K2a scores with `index_score_fp4`), no speculative decoding, or DSpark with a request's verify (speculative tokens + 1) of at most 48 rows (another drafter's hooks on the target layers would not fire under mono), `ATOM_DSV41_SIDE_STREAMS=0`, routed experts in A8W4 (MXFP4 weights, MXFP8 activations, gate / up interleaved: checked when the weights bind), at most 1024 attention keys a token (sliding window + index top-k) and the indexer shape the kernels are written for (top-512, 2048 candidate blocks of 8 rows, 32 index heads of 128).

**MiniMax-M3** (`atom/models/minimax_m3/mono/`): decode steps of up to 16 tokens (requests x speculative query tokens; a verify's tokens one row each), dense and sparse layers, Eagle3 aux hidden states included. It needs TP4, ptpc-FP8 attention linears, fp8 KV and index cache, KV block size 128, `max_model_len` up to 1M, no TBO / DP / PP / plugin mode, and FULL cudagraph or eager. A layer that reuses an earlier layer's selection is served, as long as the first sparse layer selects. The same entry class serves both paths.

**GLM-5.2** (`atom/models/glm52/mono/`): the MoE suffix of the layers, at decode widths of 1 token (no speculation) or an MTP verify of 4, 5 or 6, batched 1 or 2. It needs TP4, ptpc-FP8 attention linears, fp8 KV, `ATOM_MLA_PAGE_SIZE=1`, and no DP attention / plugin mode / shuffled MLA cache. Unlike DeepSeek-V4.1 and MiniMax-M3 it does **not** refuse TR_HYBRID: it routes inside the *uncompiled* outer `GlmMoeDsaForCausalLM` and bypasses the compiled inner `DeepseekV2Model` entirely, so a token graph of that module never runs underneath mono and cannot conflict with it (`common_refusals(..., refuse_tr_hybrid=False)`). TR_HYBRID — which a config with no `cudagraph_mode` resolves to at level 3 (`Config.__post_init__`; the CLI itself defaults `--cudagraph-mode` to FULL) — is accepted, as is FULL and eager. By default the indexer runs outside the kernel (`_refresh_indexer`); `ATOM_GLM52_FUSED_INDEXER=1` moves it inside the launch on the 18 full layers, with the extra requirements listed in the table above, and `ATOM_GLM52_INDEXER_CP=1` additionally shards its scoring across the four ranks. Which one a width ended up on is named in the `GLM-5.2 MonoKernel on: S=.. chunk=.. indexer=..` line, not inferred.

## MoE all2all (MoRI) wire format

Both are opt-in and default to off; they only apply with DP attention + expert
parallelism. They are *not* symmetric — FP4 dispatch only moves a quantization
the MoE GEMM was going to perform anyway (it consumes FP4 activations either
way, and `per_1x32` is per-row, so it does not matter which rank runs it), while
FP8 combine adds a quantization that would not otherwise happen, since the
expert output is bf16. Treat the dispatch knob as format matching and the
combine knob as a quality/throughput tradeoff.

| Variable | Type | Default | Description |
|----------|------|---------|-------------|
| **ATOM_MORI_FP4_DISPATCH** | bool | 0 (false) | If set to `1`, quantize activations to packed FP4 (E2M1, `per_1x32`) before the MoE all2all instead of sending bf16 — a quarter of the bytes on the dispatch wire — which selects `EpDispatchIntraNodeKernel_fp4`. MoRI picks its dispatch kernel from the dtype of the tensor handed to `dispatch()` but sizes its staging buffers from the config built at init, so this also switches `scale_dim` to `hidden_dim/32` and the scale type to e8m0. All three are resolved together by `mori_prepare_finalize.resolve_mori_dispatch()`; never set one without the others, as a mismatch strides the staging scale buffer wrong and faults on the first real batch instead of erroring cleanly. |
| **ATOM_MEGA_STAGE2_FUSED** | bool | 1 (true) | Under `ATOM_MORI_V2=1`, run the MoE layer on aiter's `MegaMoEGfx1250` (MegaMoE): the gemm2 epilogue P2P-writes each weighted route into the peers' combine staging, so combine only barriers and sums. `0` falls back to mori's v2 op-layer with plain gather combine. Formerly `ATOM_MORI_V2_FUSED`. |
| **ATOM_MEGA_STAGE1_FUSED** | bool | 1 (true) | MegaMoE compact stage 1 (aiter `stage1_fused`): routing/layout planning fused with the flydsl TDM dispatch, which writes the grouped GEMM's per-expert rows directly. Only takes effect with `ATOM_MEGA_STAGE2_FUSED=1` and `MEGA_DISPATCH_WIRE=fp8`/`fp4`; a bf16 wire keeps the token-major path. Not supported together with the Triton experts. |
| **ATOM_MEGA_COMBINE_WIRE** | str | `bf16` | MegaMoE (`ATOM_MEGA_STAGE2_FUSED=1`) combine wire: `bf16`, `fp8` (mxfp8) or `fp4` (mxfp4). Prefill-only: decode steps always combine in bf16, and the choice is DP-agreed so every rank reduces in the same format. |
| **ATOM_MEGA_DECODE_MTPR** | int | 128 | Capacity (rows per rank) of the small MegaMoEV2 instance that DP-unified decode / verify / draft passes reuse (`ATOM_MEGA_DECODE_FAST_PATH=1`); one of 128, 256, 512, 1024. A pass with more rows falls back to the instance sized for `max_num_batched_tokens`. Size it to cover the DP-padded decode graphs (`running_bs * (num_speculative_tokens + 1)` rows); above 128 the instance stays on aiter's fixed-slot path only if `AITER_MEGA_FIXED_SLOT_MAX_MTPR` admits it (511 / 1023 / 2047), otherwise it runs compact. |
| **ATOM_MEGA_MASK_PAD_ROWS** | bool | 1 (true) | By default, the MegaMoE backend (`--moe-backend mega`) and the fused MoRI v2 transport (`ATOM_MORI_V2_FUSED=1`, FlyDSL experts) route this rank's DP pad rows (past `scheduled_tokens` on a step padded to the DP group's `running_tokens`) to expert `-1`: they are not dispatched or computed and come back as zeros, so identical pad rows also stop piling onto the same experts. Covers target and draft passes, eager and CUDA graph. With an aiter whose MegaMoEV2 masks `-1` slots in combine (`supports_combine_mask`) the op returns the zeros; otherwise ATOM zeroes only the padded rows with a small Triton kernel. Set to `0` to disable. Not used by plugin frontends. |
| **ATOM_MORI_COMBINE_QUANT** | str | `none` | Combine-side codec passed into the MoRI config. `none` returns bf16; `fp8_blockwise` selects `EpCombineIntraNodeKernel_*_fp8bwq_*`; MoRI also accepts `fp8_direct_cast`. |

## EPLB

Applies with `--enable-eplb`; the schedule itself is set by `--eplb-config`
(`load_window_size`, `rebalance_interval`, `rebalance_min_balancedness`, ...).

| Variable | Type | Default | Description |
|----------|------|---------|-------------|
| **ATOM_EPLB_MAX_REBALANCES** | int | 0 (no limit) | Stop periodic rebalancing once this many rebalances have run, and serve with that expert placement for the rest of the process: balance on early traffic, then pay no further migrations. Only rebalances that run count; an interval the balancedness gate skips does not. The limit is checked after a rebalance's last migration chunk commits, on every rank in lockstep; after it, the manager also stops reducing the per-step has-prefill flag. Load recording continues (its buffers are fixed for CUDA graphs) but is no longer read. `0` or a negative value keeps rebalancing for the life of the process. Read when the EPLB manager is built, so set it before the server starts. |

## Fusion passes

### RMSNorm

| Variable | Type | Default | Description |
|----------|------|---------|-------------|
| **ATOM_USE_MODEL_SENSITIVE_RMSNORM** | bool | 0 (false) | If set to `1`, use AITER's model-sensitive RMSNorm rounding mode. This can change numerical results and prefill performance, so it is opt-in. |

### TP AllReduce fusion

| Variable | Type | Default | Description |
|----------|------|---------|-------------|
| **ATOM_ENABLE_ALLREDUCE_RMSNORM_FUSION** | bool | 1 (true) | If set to `1`, fuse allreduce with RMSNorm in tensor parallel mode. |

### DeepSeek-style

| Variable | Type | Default | Description |
|----------|------|---------|-------------|
| **ATOM_ENABLE_DS_INPUT_RMSNORM_QUANT_FUSION** | bool | 1 (true) | If set to `1`, fuse RMSNorm with quantization. |
| **ATOM_ENABLE_DS_QKNORM_FUSION** | bool | 1 (true) | If set to `1`, use the fused Q/K RMSNorm path (`fused_qk_rmsnorm`) in the DeepSeek MLA attention module when Q-LoRA is enabled and QK norm+quant fusion is not used. If set to `0`, apply separate RMSNorm for the Q and KV branches instead. |
| **ATOM_ENABLE_DS_QKNORM_QUANT_FUSION** | bool | 1 (true) | If set to `1`, fuse QK norm with quantization in MLA attention module. |
| **ATOM_DUAL_STREAM_MOE_TOKEN_THRESHOLD** | int | 1024 | Upper bound on MoE token count (`num_tokens` in the MoE forward) for using the dual-stream path: shared experts on a secondary CUDA stream while routed experts run on the default stream. If `num_tokens` exceeds this value, that forward uses single-stream MoE instead. Set to `0` to disable dual-stream setup entirely (no alt stream, no `maybe_dual_stream_forward` registration). The fork is taken in eager runs, FULL capture and token-graph segments alike; only a plugin frontend's own PIECEWISE graphs stay single-stream. |

### DSpark block sampling

DSpark drafts a `num_speculative_tokens`-wide block in one backbone pass, then
samples it left-to-right with a low-rank first-order Markov head
(`logits_k = base_logits_k + W1[x_{k-1}] @ W2ᵀ`, `x_k = argmax(logits_k)`). The
unfused loop casts the whole `[V, r]` `W2` table to fp32 on every iteration and
materializes two `[B, V]` fp32 tensors that only an `argmax` reads. See
`atom/model_ops/dspark_markov_sample.py`.

| Variable | Type | Default | Description |
|----------|------|---------|-------------|
| **ATOM_DSPARK_FUSED_MARKOV_SAMPLE** | bool | 1 (true) | Sample the DSpark block with a fused Triton kernel that computes the rank-`r` bias GEMV, adds the base logits in the GEMM epilogue and reduces to token ids in registers — so `W2` stays bf16 and is read exactly once per block position, and no `[B, V]` intermediate exists. Covers both native DSpark block samplers, Kimi-K3 (`r=256`) and DeepSeek-V4 (`r=512`); the op is shape-generic and hands anything it cannot index back to the reference, but only K3 has been run on hardware. Tie-breaking matches `torch.argmax` (lowest index). The bias moves from an fp32 matmul to bf16 MFMA with an fp32 accumulator: every product is exact in fp32 either way, so the result is equal to the reference up to accumulation order. Measured on Kimi-K3 (MI355X, TP8, fp8 KV, full GSM8K 5-shot at 64 concurrency): acceptance 87.08% against 87.06% unfused with the accept-length distribution equal to within 0.1pp, and flexible-extract inside the run-to-run band. Saves 145 µs per drafting step at B=1 and ~235 µs at B=64. Set to `0` to force the reference spelling if an acceptance-rate regression is suspected — the two paths are not bit-identical by construction, so this is the fastest way to rule the kernel in or out. Read at Markov-head construction, so set it before the server starts. |

### Qwen3 style

| Variable | Type | Default | Description |
|----------|------|---------|-------------|
| **ATOM_ENABLE_QK_NORM_ROPE_CACHE_QUANT_FUSION** | bool | 0 (false) | If set to `1`, fuse QK norm, RoPE, and cache quantization into one kernel for Qwen3 dense and MoE models. |
| **ATOM_QK_NORM_ROPE_CACHE_FUSION_MAX_TOKENS** | int | 8192 | With the fusion above on, Gemma-norm models (Qwen3-Next, Qwen3.5) run the q/k norm, RoPE and cache write as separate kernels when a batch has more tokens than this; the fused kernel is slower at prefill sizes. |

### Llama-style

| Variable | Type | Default | Description |
|----------|------|---------|-------------|
| **ATOM_LLAMA_ENABLE_AITER_TRITON_FUSED_RMSNORM_QUANT** | bool | 1 (true) | If set to `1`, use Triton kernel to fuse RMSNorm with quantization. |
| **ATOM_LLAMA_ENABLE_AITER_TRITON_FUSED_SILU_MUL_QUANT** | bool | 1 (true) | If set to `1`, use Triton kernel to fuse SiLU and mul with quantization in MLP module. |

### Draft CUDAGraphs (all drafter flavors)

A drafter declares its forward passes as `DraftGraph`s (`atom/spec_decode/drafter.py`).
At the end of CUDAGraph capture the runner runs each one once per captured batch
size, so the per-shape JIT — aiter's flydsl builds an hgemm per tile config,
in-process — is paid at startup instead of stalling a serving step. At serve
time a pass runs at the batch the target just ran, which `ForwardMode.decide`
picks out of those same `capture_sizes` — that is what makes a warmed shape and a
reachable shape one set rather than two lists that drift. A pass must support capture; the switch below then decides whether its warmup also records. V4.1 DSpark currently warms and drafts eagerly because its request windows and expert dispatch are outside the whole-block capture contract.

| Variable | Type | Default | Description |
|----------|------|---------|-------------|
| **ATOM_DRAFT_CUDAGRAPH** | bool | 1 (true) | Capture each declared draft pass into a per-`capture_sizes` CUDAGraph as it is warmed, so a draft pass replays instead of relaunching every kernel. `0` keeps the warmup (and therefore the JIT saving) but drafts eagerly. Only declared passes with capture support are recorded; every drafter ATOM ships declares at least one pass, including the separate-draft Kimi-K3 path, whose block pass builds its paged metadata at warmup so nothing host-side is left inside the recording. EPLB no longer declines the padding: the target pads on every cudagraph decode step and its rows reach the same expert-load recorder, so declining on the draft protected nothing. A DP-sync dummy DOES replay, in lockstep with the ranks holding work — `is_dummy_run` is per-rank, so gating on it splits one DP group across two collectives. Measured on V4-Flash-DSpark tp1: GSM8K 0.9527 / acceptance 65.25% captured against 0.9497 / 65.21% eager, i.e. indistinguishable; on tp4 with the LM head inside the capture, draft kernel launches went 30 → 0 per pass and draft wall time 915.8 → 118.9 µs. Read per pass at warmup time, so set it before the server starts. Grep a trace for a trailing ` graph` in a `propose_*` label to confirm which passes replayed. |

### DSpark drafting

The Kimi-K3 DSpark draft writes the target's context rows into its own paged MLA
cache once per draft layer per drafting step; the switch below shortens that
path. The first write of each process logs which path it took, and logs again if
that ever changes, so a fusion left inert by an unrecognised layout says so.

| Variable | Type | Default | Description |
|----------|------|---------|-------------|
| **ATOM_DSPARK_FUSED_CTX_KV** | bool | 1 (true) | Write the context rows with one Triton kernel (RMSNorm + RoPE + concat + paged store) instead of four launches plus a throwaway `empty_like` for the RoPE's query side. Falls back per call when the cache layout or the RoPE is not the plain one the kernel understands (seg / shuffled-KV layouts keep their own write kernels), and until the RoPE's cos/sin cache has reached the device. Measured on Kimi-K3 (MI355X, TP8, fp8 KV): one 4.65 µs kernel replaces a 14 µs three-kernel chain, saving ~39 µs per drafting step at B=1 and ~36 µs at B=64. Set to `0` to force the per-op chain; that chain is the fallback above rather than debug code, so it stays reachable either way (it runs the first write of every layer). |
| **ATOM_DSPARK_DISABLE_COMPILE** | bool | 0 (false) | Run the DSpark draft eager while the target stays compiled. Prefer it over `--level 0`, which drops compilation for both models; `--enforce-eager` does not reach it, because `support_torch_compile` keys off `compilation_config.level` alone. Flips the decorator's own bypass rather than handing the draft a cloned config, so the shared `static_forward_context` registry stays one object. |

### Speculative acceptance

| Variable | Type | Default | Description |
|----------|------|---------|-------------|
| **ATOM_ENABLE_RELAXED_MTP** | bool | 0 (false) | Accept a draft token when it lands in the target's top 10 within 0.6 of the top logit, instead of requiring the argmax. Intended for quantized MTP heads, whose drafts are right about the region and wrong about the exact winner often enough that strict acceptance throws away usable tokens. Read once at `rejection_sampler` import, so it must be set before the server starts. |

## Engram (DeepSeek-V4.1)

The n-gram tables are per-layer and large enough that where they live, and
whether they are rebuilt, both show up at startup. Both switches below are
all-or-nothing on purpose: a half-registered set would keep the host path for
some layers and the device path for others, which is the confusing state.

| Variable | Type | Default | Description |
|----------|------|---------|-------------|
| **ATOM_ENGRAM_UVA** | bool | 1 (true) | Page-lock this rank's shard of the hash tables in place and let a device kernel read the rows it needs across the bus, dequantizing there. No copy and no HBM for the table. `0` falls back to gathering the rows on the host, which returns the same rows but costs ~50 ms of CPU per decode step with the GPU idle behind it. Anything that would make the device path unsafe — no CUDA, more TP ranks than hash heads, a registration that will not fit — falls back on its own, so the switch is for taking the host path deliberately. The fallback is the whole TP group's: the lookup ends in an all-gather, so one rank that cannot register turns every rank around rather than leaving the others in a collective it never enters. |
| **ATOM_ENGRAM_CACHE_DIR** | path | `~/.cache/atom/engram` | Where the compressed-vocab table is cached between runs. The table is reproducible from the tokenizer, so this only trades startup time for disk; point it at shared storage to let several servers build it once. A truncated or stale cache is rebuilt rather than raised. |

## Attention side streams (DeepSeek-V4.1)

A layer's compressor reads the hidden row and its own arena state, and its
indexer reads the normed query latent; neither reads what the query projection
and the fused rope/window launch produce, so the three are branches of one
dependency graph that a single stream serializes.

| Variable | Type | Default | Description |
|----------|------|---------|-------------|
| **ATOM_DSV41_SIDE_STREAMS** | int (0/1/2) | 0 | How many of a layer's branches leave the main stream. **0** — none. **1** — the compressor, on the MoE's `alt_stream`, waited at the scorer rather than at attention, because `visible` counts the index row a boundary crossed in this same forward writes, so the scorer is its first reader a whole top-k chain earlier. It borrows that stream because the MoE joins it inside its own forward, a sublayer after the compressor was joined, so the two are never live at once. **2** — the same, plus the indexer on a stream of its own; it is the only branch live beside both others, so it is the only one worth a queue. A value outside 0–2 raises rather than rounding. |
| **ATOM_TOKEN_GRAPH_SIZES** | str | "256,512,1024,2048,3072" | Prefill token buckets to record as TR graphs (`atom/utils/region_graph.py`), comma-separated. Each bucket's forward is recorded once at startup as graph segments broken at the model's eager regions (indices, sparse attention, window write, compressor, indexer, Engram), and a prefill padded to the smallest bucket that holds it (under DP attention, every prefilling rank's to the one holding the group's largest) replays them instead of launching every kernel; a longer prefill runs eagerly. Empty records nothing. Only a backend with `supports_token_graph` (DeepSeek-V4.1) uses it, and a disaggregated decode instance records none. The default stops at 3072 because the graph pool grows with the largest bucket: on V4.1 TP4 per GPU, FULL decode alone takes 1.15 GB, the five default buckets 3.1 GB, buckets up to 16384 10.8 GB. Below 3072 a prefill step is CPU-bound, so a graph cuts its TTFT by 30–40 ms and padding to the next bucket costs about nothing; longer prefills are GPU-bound. The KV budget's graph estimate does not include these buckets: they come out of the safety margin. Under DP attention a rank with no prefill of its own runs a dummy prefill while a peer prefills, instead of decoding, and replays the same bucket as its peers. |
| **ATOM_TOKEN_GRAPH_DP_MAX_TOKENS** | int | 512 | Under DP attention with a drafter, the largest decode token bucket recorded as a TR_HYBRID token graph; larger `bs × q` buckets are skipped at capture and left out of the graph memory estimate. Such buckets never run under DP, and recording them only grows the graph pool. The same skip on every DP rank keeps their capture loops in step. |
| **ATOM_DECODER_GRAPH_SIZES** | str | "1,2,3,4,8,16" | The `running_bs` at which DeepSeek-V4.1's decoder (layers 20..39) on a bounded-replay prefill's tail is recorded as a request graph (`models/deepseek_v41/decoder_graph.py`), comma-separated. The tail is laid out for the step's `running_bs` requests, a window each, at one set of addresses per `running_bs`, so a recording replays the whole decoder in one launch; any other `running_bs` runs the same layout eagerly, and empty records nothing. Not on an FP8 index plane. Under DP attention each size also records the tail of padding a rank finishing no prompt runs beside its peers. On a 16k-token prefill step the decoder in one graph cuts the step's CPU time from 84 to about 45 ms; the six default sizes add about 0.33 GB to the graph pool on V4.1 TP4. |

**Level 0 is the default because forking measured slower, not faster.** Median
steady-state layer period, MI355X TP4 bf16 KV DSpark-5, 1024/1024 at
concurrency 64, ~10k sampled layers per trace:

| level | layer period | vs 0 |
|-------|--------------|------|
| 0 | 636.40 µs (repeat: 636.72) | — |
| 1 | 645.76 µs | +1.47% |
| 2 | 640.32 µs | +0.62% |

The anchor is the gap between consecutive `topk_gating` launches, so a branch
that moves to another stream cannot drop out of the sample. The two level-0
traces were taken either side of the other two, which puts the floor — session
drift included — at 0.05%, making those deltas 12× and 29× the noise; the
unfiltered medians (604.0/605.9 against 618.8 and 607.6) order the levels the
same way. End-to-end throughput cannot resolve this: one wall-clock number per
run carries 2–6% spread, which is why an earlier A/B called the same
arrangement a wash. Whoever revisits it should start from these numbers and
from `GPU_MAX_HW_QUEUES`, not from where the forks sit.

## V4 attention backend (Migration)

Selects between the legacy per-seq Python dispatch path in `atom/models/deepseek_v4.py`
and the new batched `V4AttentionBackend` (`atom/model_ops/v4_attention_backend.py`).
The new backend removes ~256 GPU→CPU `.item()` syncs per forward and is required
to enable CUDAGraph capture for V4. Legacy stays available during PR-A migration
for byte-equal A/B verification via dump-bisect; it is removed once all phases
land. See `atom/model_ops/v4_backend_gate.py` for the selector.

| Variable | Type | Default | Description |
|----------|------|---------|-------------|
| **ATOM_V4_BACKEND** | str | `legacy` | `legacy` keeps the per-seq dispatch loop. `new` routes through `V4AttentionBackend`. Layer-restricted by `ATOM_V4_BACKEND_LAYERS` if set. |
| **ATOM_V4_BACKEND_LAYERS** | csv int | "" (= all) | Comma-separated layer ids that use the new backend (others stay legacy). Empty means: apply `ATOM_V4_BACKEND` uniformly. Used for layer-by-layer bisect during migration (e.g. `0,3,15,30`). |

## Prefix cache lookup

All three default on and move work off the scheduler thread.
`ATOM_FRONTEND_PREFIX_HASH=0` and `ATOM_PREFIX_WALK_MEMO=0` give the same hits.
`ATOM_PREFIX_FAST_WALK=0` does not always: the fast walk compares tokens only on the last hit block, so a
64-bit xxhash chained-hash collision on an interior block reuses another
prompt's KV with no error (wrong output). `0` compares every hit block's tokens
and catches it; set it when chasing a corruption that does not reproduce.

| Variable | Type | Default | Description |
|----------|------|---------|-------------|
| **ATOM_FRONTEND_PREFIX_HASH** | bool | true | Hash the prompt's block chain while the request is built (API server preprocessing executor) and ship it to the engine with a fingerprint of the prompt tokens, beside the request in the ADD message (never inside a pickled `Sequence`; with `DisaggCoreManager` only the decode engine, which allocates, gets it). The engine adopts it if block size, seed, length and fingerprint match and its first, middle and last links rehash equal, then uses it for the prefix walk, `allocate` and publishing the prompt blocks after prefill, so prefill hashes no prompt block on the scheduler thread (it hashes the prompt bytes once for the fingerprint, about 26 us at 186k tokens; the LMCache/offload and PD-transfer publish paths still hash their blocks). Any mismatch falls back to engine-side hashing. Read in the API server process. |
| **ATOM_PREFIX_FAST_WALK** | bool | true | Look the prompt's chain up with C-speed `map` rounds (the first block alone, then 16, 64, ... blocks, so a cold prompt costs one lookup), compare tokens on the last hit block only (all blocks if that one differs), and let `allocate` claim the blocks `can_allocate` found instead of walking the prompt again; any block that changed in between sends it back to the block-by-block walk. `0` walks block by block with a token compare per block. |
| **ATOM_PREFIX_WALK_MEMO** | bool | true | Reuse a waiting request's prefix walk across the probes that run every scheduler tick: the delayer's head check and probe, and the DP chunk-cost estimate (`BlockManager.can_allocate_memoized`). It walks again when a hash the walk read is evicted, remapped to another block or revoked by a republish of its block, the hash at its first miss is published, `clear_cache` runs, more than 8192 such changes happened since its last check, or the request's token count or seed changed. The fit, state gates and replay are re-asked on every tick, so the answer equals the exact probe; admission still probes exactly. `0` re-walks every tick. |

## State checkpoints

For models carrying per-request recurrent state (GDN: Qwen3-Next / Qwen3.5;
Kimi-K3's KDA; DeepSeek-V4's compressor ring), a checkpoint lets a later prefix
hit resume mid-prompt instead of recomputing from zero. *Where* they are placed
is a policy, set by `--state-checkpoint-interval-tokens` (three regimes carried
by the sign — see the [configuration guide](configuration_guide.md)) and the
flag below. Details in the state-checkpoint section of the
[scheduling & KV cache guide](scheduling_kv_cache_guide.md).

| Variable | Type | Default | Description |
|----------|------|---------|-------------|
| **ATOM_STATE_CHECKPOINT_DEMAND** | bool | 1 (true) | Set to `0` to stop a prefix hit that was refused for want of a checkpoint from placing a rung of its own, leaving the prompt-end anchor as the only placement. Overrides `--state-checkpoint-demand`, so the policy can be A/B'd without editing a launch script. The rung is most of the checkpoint write traffic and little of the read-back, and every write evicts something — `StateSlotPool.mark_speculative` carries the measurement. |
| **ATOM_STATE_CHECKPOINT_END_MARGIN** | int | 0 | Minimum number of prompt tokens left after the prompt-end anchor. The anchor goes on the last hash-block boundary at least `max(successor room, min final prefill rows, margin)` tokens before the prompt end: the largest of the three, not their sum. A chat template can re-render the tail of a prompt when the next user turn is appended: Qwen3.5's generation prompt ends in `<think>\n`, which the next turn drops, so the next prompt diverges at P-2 and an anchor whose last block covers P-2 (every prompt with P % 16 == 1) is never matched again. Set it to the template's generation-prompt length (`5` for Qwen3.5) to keep the anchor before that suffix. `0` keeps the old placement. Cost when the next turn does resend the prompt verbatim: the anchor moves back by up to `ceil((margin - room) / block)` hash blocks of recompute, where room is the larger of the other two terms: at most one block while the margin exceeds the room by no more than one block (Qwen3.5: margin 5, PAGE room 0, block 16), several for a larger margin. A prompt shorter than `margin + one block` keeps no anchor at all. |
| **ATOM_DSV4_STATE_REPLAY** | bool | 0 (false) | DeepSeek-V4: keep a prefix hit that reaches past the nearest state checkpoint by more than the window's receptive field (7,748 tokens on V4-Pro, 7,875 with MTP or DSpark) and recompute only that field below it, which rebuilds the state exactly, instead of resuming at the checkpoint (`Config.state_replay_tokens`). Off on anything but the DeepSeek-V4 trunk with compress ratios 4 / 128, and with context parallelism, a speculative method other than MTP or the model's own DSpark layers (not a separate `--draft-model`), prefix caching off, or a KV connector other than LMCache MP offload alone. With that one LMCache saves carry the PAGE KV alone, no state image (the native layout registers the PAGE group alone, under a namespace of its own, so a stock LMCache server's hit is how far that KV reaches), and a hit whose PAGE KV LMCache holds past the HBM prefix loads that KV alone and replays below it. Every other model, and DeepSeek-V4 without the replay, keeps storing images. |
| **ATOM_STATE_CHECKPOINT_INTERVAL_TOKENS** | int | — (the model's native interval) | vLLM plugin (DeepSeek-V4 state checkpoints): overrides the checkpoint interval in tokens, snapped down to a multiple of the block size. `>0` a rung every N tokens plus the prompt-end anchor and the demand rung; `-1` no rungs, anchor and demand only; `0` no state checkpoint is published, so there is no prefix hit at all (vLLM reports a hit only when every state checkpoint group is cached; the state checkpoint groups still allocate their working blocks). |
| **ATOM_STATE_CHECKPOINT_CHECK_BLOCKS** | bool | 0 (false) | vLLM plugin (state checkpoints) diagnostic: after every schedule, raise if a request holds more than two blocks in any state checkpoint group. |
| **ATOM_GDN_STATE_MIDSTEP** | bool | 0 (false) | Set to `1` to take Qwen3-Next / Qwen3.5 GDN state checkpoints inside a prefill forward, from the chunk kernel's per-64-token states, instead of cutting the prefill into several forwards that end on them. PAGE checkpoint path only. Checkpoint positions move back onto the forward's 64-token grid, so a resume recomputes up to 48 more tokens. |

### LMCache offload tier

ATOM's own offload knobs are defined in `atom/utils/envs.py`; their behavior is
documented in `atom/kv_transfer/offload/README.md`. Where a
`kv_connector_extra_config` key also exists, it takes precedence over the env
var. `LMCACHE_EC_PIN_TIMEOUT_SEC` belongs to LMCache and is read only to
derive a bound.

| Variable | Type | Default | Description |
|----------|------|---------|-------------|
| **ATOM_KV_OFFLOAD** | str | "" (off) | Enables KV offload without `--kv-transfer-config`, so a launcher that owns that flag for P/D can still add offload. `lmcache` selects the in-process `lmcache_offload` connector, `lmcache_mp` the standalone-server `lmcache_mp` connector (start `lmcache server` first), `mooncake_store` the `mooncake_store` connector, which puts and gets KV chunks straight to a Mooncake Store with no LMCache (start the Store's master and owners first; dense layout only). With a P/D connector in `--kv-transfer-config`, both run behind a `multi` connector. Setting it while `--kv-transfer-config` already names an offload connector is an error. |
| **ATOM_KV_OFFLOAD_EXTRA_CONFIG** | JSON object | "" | The offload connector's `kv_connector_extra_config`: `lmcache.<field>` overrides (e.g. `{"lmcache.chunk_size": 256}`), for `lmcache_mp` `lmcache.mp.*` options (e.g. `{"lmcache.mp.port": 5556}`), and for `mooncake_store` the `mooncake_store.*` settings (e.g. `{"mooncake_store.master": "10.0.0.1:26051", "mooncake_store.metadata": "http://10.0.0.1:26080/metadata"}`; all of them are listed in `atom/kv_transfer/offload/README.md`). Requires `ATOM_KV_OFFLOAD`. |
| **OFFLOAD_COPY_WORKERS** | int | 1 | Save executor threads per offload worker. Also scales the default `OFFLOAD_MAX_PENDING_SAVES`. |
| **OFFLOAD_LOAD_WORKERS** | int | 1 | Load executor threads per offload worker. DSV4's in-process path ignores it (its SLOT load path needs a serial load executor). |
| **OFFLOAD_MAX_PENDING_SAVES** | int | unset: `max(2, 2 × OFFLOAD_COPY_WORKERS)` for connectors, **2** for the scheduler's state tier | Bound on running-plus-queued saves. KV and state saves share it because both pin the same pool. A non-integer raises on the connector path and warns (using 2) on the state-tier path. Overridden by `max_pending_saves` in `kv_connector_extra_config`. While both are unset, `mooncake_store` leaves its saves unbounded (one per request in flight at most). |
| **OFFLOAD_MIN_LOAD_TOKENS** | int | 8192 | Smallest external-tier hit worth loading; shorter hits are recomputed. Negative values clamp to 0. |
| **OFFLOAD_MIN_SAVE_TOKENS** | int | 8192 | Shortest prefix native `lmcache_mp` stores, as an absolute boundary for normal and late saves. With the default equal to `OFFLOAD_MIN_LOAD_TOKENS`, a shorter prefix could never be loaded back. Other connectors ignore it. |
| **OFFLOAD_LOOKUP_MEMO_STEPS** | int | 32 | Scheduler steps a memoised tier-lookup answer is replayed before the tier is asked again. |
| **OFFLOAD_LOOKUP_RETRY_STEPS** | int | 32 | Scheduler steps a failed tier lookup suppresses the next attempt. |
| **OFFLOAD_ASYNC_LOOKUP_DEPTH** | int | 16 | `lmcache_mp` sends tier lookups for the head of the waiting queue before admission, as non-blocking requests, and admission consumes the answer; a request whose answer has not arrived is passed over (in arrival order) instead of blocking the scheduler. This is how many waiting requests per scheduling pass get a prefetched lookup. |
| **OFFLOAD_LOOKUP_DEFER_S** | float | 2.0 | How long `lmcache_mp` admission may pass over a request whose prefetched lookup is still in flight before it waits for the answer (finite, ≥ 0). |
| **OFFLOAD_SKIP_LOOKUP_ON_HBM_HIT** | bool | 1 (true) | On by default; set to `0` to always wait for the lookup. Admit a prefill without waiting for its external-tier lookup when the lookup cannot change what is loaded: the HBM prefix leaves fewer than `OFFLOAD_MIN_LOAD_TOKENS` tokens uncached, so every load path would refuse the tier hit. Requires an asynchronous lookup (LMCache MP) and is not applied with joint KV/state offload. The lookup is still sent, because it refreshes the prefix's L1 LRU recency and opens the MP session whose end touches the request's chunks; its answer is not used as a load hit. The save floor starts at the HBM prefix and is lowered to the tier's contiguous hit when the answer arrives (or to 0 if it never does), so chunks the tier evicted are stored again. Read locks are released when the answer arrives; the MP session ends after that, and an unanswered lookup is given up after 120 s, leaving its locks to the read TTL. |
| **OFFLOAD_PROFILE** | bool | 0 | Emit `[OFFLOAD-SAVE-PROF]` / `[OFFLOAD-LOAD-PROF]` per-transfer records, and once a minute an `[OFFLOAD-PERF]` line per scheduler with lookup/save/load-skip counters and per-step timings. An empty value reads as off. |
| **OFFLOAD_SINGLE_STREAM** | bool | 0 | Experimental: run the staging pack and copy legs on one stream. |
| **OFFLOAD_GPU_STAGING_CHUNKS** | int | derived from KV geometry | GPU staging buffer size in LMCache chunks (≥ 1). |
| **OFFLOAD_GPU_STAGING_MAX_BYTES** | int | unset | Upper bound on the GPU staging buffer in bytes; must hold at least one chunk. Unset, `mooncake_store` bounds it at 48 MiB, which it holds back from the KV budget per save and load thread. |
| **OFFLOAD_RELEASE_GPU_STAGING_AFTER_TRANSFER** | bool | 0 | Free the GPU staging buffer after each transfer instead of keeping it. |
| **OFFLOAD_SLOT_STAGING_SLOTS** | int | 1 | DSV4 in-process SLOT sidecar staging rows. Overridden by `slot_sidecar_staging_slots`. |
| **OFFLOAD_COMMITTED_SIDECAR_CAPACITY** | int | 65536 | DSV4 in-process committed SLOT sidecar index capacity. Overridden by `committed_sidecar_index_capacity`. |
| **OFFLOAD_PUBLICATION_TIMEOUT_S** | float | 5.0 | DSV4 in-process wait for a saved SLOT sidecar to become visible (finite, ≥ 0). |
| **OFFLOAD_PUBLICATION_POLL_INTERVAL_S** | float | 0.01 | Poll period of that wait (finite, > 0). |
| **LMCACHE_MP_TRANSFER_MODE** | str | auto | `lmcache_mp` transfer mode: `auto` or `lmcache_driven` (`engine_driven` is rejected). Overridden by `lmcache.mp.mp_transfer_mode`. |
| **LMCACHE_EC_PIN_TIMEOUT_SEC** | float | LMCache's own (300) | LMCache's source-pin timeout. ATOM reads it only to derive the engine's save-abandon window (`pin + 30s`), so the two stay ordered: a lost store report is reclaimed only after LMCache would already have force-unpinned its source. Non-positive disables ATOM's reclamation. ATOM sets no default of its own and, when unset, assumes LMCache's. |

## KV cache events

The scheduler can publish prefix-cache changes (`BlockStored`, `BlockRemoved`,
`AllBlocksCleared`, and `BlockStored(medium=REMOTE)` for KV received from a
PD producer) over ZMQ so external routers and cache managers can mirror what
each engine holds. Every batch carries a monotonic 8-byte sequence number;
a consumer that sees a gap can ask for the missed batches over the optional
replay socket. Events are advisory and never stall inference: the in-process
queue drops the oldest batch when full. These variables are the only way to
configure the feature today: they build `KVEventsConfig` (see `atom/config.py`)
and there is no CLI flag.

Endpoint rules: each publisher binds its PUB and replay endpoints offset by its
data-parallel rank (see `ATOM_KV_EVENTS_ENDPOINT`), so with `tcp://` the two
configured ports must be at least `data_parallel_size` apart or rank N's PUB
lands on rank N-1's replay port. Under pipeline parallelism only the head stage
publishes; downstream stages bind nothing. Prefill/decode disaggregation runs
separate engine processes, and only decode publishes; if other engines share
the host, give each its own endpoints.

| Variable | Type | Default | Description |
|----------|------|---------|-------------|
| **ATOM_KV_EVENTS_ENABLE** | bool | 0 (false) | Set to `1` to publish KV cache events. |
| **ATOM_KV_EVENTS_PUBLISHER** | str | `zmq` | `zmq` or `null` (accepts events and discards them). |
| **ATOM_KV_EVENTS_ENDPOINT** | str | `tcp://127.0.0.1:5557` | ZMQ PUB bind address. Under data parallelism every rank binds its own socket: `tcp://` endpoints get the DP rank added to the port, `ipc://`/`inproc://` endpoints get a `_dp<rank>` suffix. Rank 0 uses the configured value unchanged. |
| **ATOM_KV_EVENTS_TOPIC** | str | `""` | Subscription topic prefix sent as the first frame of every message. |
| **ATOM_KV_EVENTS_HWM** | int | 0 | ZMQ high-water mark on the PUB socket (0 = unlimited). |
| **ATOM_KV_EVENTS_BUFFER_STEPS** | int | 10000 | Depth of the in-process queue between the scheduler and the sender thread. When full, the oldest batch is dropped and counted in the publisher's `dropped` stat; the dropped batch still consumes a sequence number, so subscribers see the loss as a gap. |
| **ATOM_KV_EVENTS_REPLAY_ENDPOINT** | str | `""` | ZMQ ROUTER bind address for replay requests. Empty disables replay (PUB-only). A consumer sends an 8-byte big-endian start sequence and receives every retained batch with `seq >= start`, followed by a `REPLAY_DONE` terminal frame carrying the retained `[oldest, latest]` window. Offset per DP rank the same way as the PUB endpoint. Replay is serviced on the sender thread with non-blocking sends: a client that stops reading has its replay abandoned (counted in `replay_aborted`) rather than stalling live publication. |
| **ATOM_KV_EVENTS_REPLAY_BUFFER_STEPS** | int | 10000 | Number of most recently *sent* batches retained for replay. Independent of `ATOM_KV_EVENTS_BUFFER_STEPS`; each entry holds an encoded payload including token ids, so size it against the event rate and memory budget. Must be >= 1 when replay is enabled. |

## Shutdown

| Variable | Type | Default | Description |
|----------|------|---------|-------------|
| **ATOM_SHUTDOWN_TIMEOUT_S** | float | 5 | Seconds that shutdown waits for child processes to exit on their own before terminating them (SIGTERM, then SIGKILL): the server process waits this long for its EngineCores, and each EngineCore for its ModelRunner workers. All the children of one process share the same deadline. Must be finite and positive. Raise it when workers need longer to finish on exit, for example to write a profiler's output (`rocprofv3` writing a thread trace can take minutes). Set it before starting the server; every process reads it. |

## Profiling & debugging

| Variable | Type | Default | Description |
|----------|------|---------|-------------|
| **ATOM_METRICS_UPDATE_INTERVAL_S** | float | 1.0 | Shared interval in seconds for ordinary/DP/PP engine metrics pushes and API snapshot refresh. Must be finite and positive; read when each loop starts, so set it before starting every service process. Prometheus scraping is configured independently. Does not cache rendered `/metrics` responses or change when histogram observations are recorded. |
| **ATOM_ENABLE_METRICS_DEVICE_TIMER** | bool | 0 (false) | Set to `1` before starting the service to collect GPU forward duration and cumulative request prefill GPU time. Uses CUDA/HIP events, a reusable pool capped at 256 pending pairs, and FIFO polling that stops at the first incomplete event. Adds event recording and query overhead; disabled services emit no GPU timing samples. Agentic dashboard CI explicitly enables it. |
| **ATOM_TORCH_PROFILER_DIR** | str | — | When set, enables PyTorch profiler and writes traces to this directory. Create subdirectories per rank (e.g., `rank_0`, `dp0_tp0`). |
| **ATOM_PROFILER_MORE** | bool | 0 (false) | When `ATOM_TORCH_PROFILER_DIR` is set and this is `1`, enables detailed profiling: `record_shapes`, `with_stack`, and `profile_memory`. Applies to both the run-phase profiler and the CUDA-graph capture profiler. Each option can be overridden individually by the per-option flags below. |
| **ATOM_PROFILER_RECORD_SHAPES** | bool | `ATOM_PROFILER_MORE` | Set to `1`/`0` to enable/disable `record_shapes`. Takes precedence over `ATOM_PROFILER_MORE`; unset or empty falls back to it. |
| **ATOM_PROFILER_WITH_STACK** | bool | `ATOM_PROFILER_MORE` | Set to `1`/`0` to enable/disable `with_stack` (Python stack capture). Takes precedence over `ATOM_PROFILER_MORE`; unset or empty falls back to it. |
| **ATOM_PROFILER_PROFILE_MEMORY** | bool | `ATOM_PROFILER_MORE` | Set to `1`/`0` to enable/disable `profile_memory`. Takes precedence over `ATOM_PROFILER_MORE`; unset or empty falls back to it. |
| **ATOM_ENABLE_DETAILED_ANNOTATION** | bool | 0 (false) | When profiling is active, appends detailed attention aggregates to the `prefill[]`/`decode[]` trace labels: `sqsq` (Σ N_Q²), `sqsk` (Σ N_Q·N_KV), and `sk` (Σ N_KV), where N_Q is the scheduled query tokens and N_KV the KV length per request. Used to estimate attention FLOPs for downstream roofline analysis. |
| **ATOM_LOG_MORE** | bool | 0 (false) | If set to `1`, use verbose logging format (includes process name, PID, path, line number, function name). |

## Garbage collection

CPython's generation-2 pass is stop-the-world and walks every tracked
container, so its cost tracks the live heap — which in a serving process is
almost entirely startup state (model, compiled graph, tokenizer, KV block
pool) that is never garbage. Measured on DeepSeek-V4-Flash-DSpark tp1: 242.8 ms
in the EngineCore, up to 596 ms in a ModelRunner worker, while reclaiming zero
objects once startup was done. See `atom/utils/gc_utils.py`.

| Variable | Type | Default | Description |
|----------|------|---------|-------------|
| **ATOM_GC_FREEZE** | bool | 1 (true) | Move the startup heap into CPython's permanent generation once warmup is done, so collections stop scanning it. Applied in every process that outlives startup — the API server, the atomesh frontend, every EngineCore and every ModelRunner worker; undone on engine shutdown so an in-process teardown does not leak. Set `0` to keep the pre-freeze behaviour. |
| **ATOM_GC_DEBUG** | bool | 0 (false) | Log every collection: generation, duration, objects reclaimed, objects tracked. Costly — counting the tracked set on every pass added ~90s of startup on a V4-Flash tp1 — but the only way to see these pauses, since a stall in the EngineCore idles the workers with no event in their torch trace. |
| **ATOM_GC_THRESHOLD** | csv int | "" (= CPython default 700,10,10) | `t0,t1,t2` for `gc.set_threshold()`. Thresholds are per-interpreter, so each process reads it independently; anything that is not three integers is logged and ignored, applying nothing. Raising these does not make a pass cheaper, it makes passes rarer — the same total scan lands in fewer, longer stop-the-world pauses, which is a trade against tail latency and not measured here. It is also **not uniform across processes**: at concurrency 4096 the API server's collector ran 13,956 times in twenty minutes over a set that grew to 688,646 objects and reclaimed **zero**, while each ModelRunner worker reclaimed thousands per pass, where spacing collections out defers real work. Read `atom:gc_collected_total` for the process you mean to tune before setting this — and note that only the API server exports it, so a worker has to be read with `ATOM_GC_DEBUG=1`. |

Raising a process's thresholds is free only while its collector keeps finding
nothing to free, which is a property of that one process, so it is exported
rather than assumed. All three are wired into the API server only; the engine
and worker processes serve no `/metrics`, so `ATOM_GC_DEBUG=1` is what reads
them there.

- **`atom:gc_collected_total`** (`/metrics`, per generation — `prometheus_client`
  appends the `_total`) is the invariant as a series. Flat after startup is the
  expected shape; a rising line means the process has started building reference
  cycles, and spacing its collections out would defer real work into a growing
  heap. `atom:gc_collections_total`, `atom:gc_uncollectable_total` and
  `atom:gc_threshold` sit beside it for context. All are O(1) reads taken at
  scrape time, which is a bound and not a preference: `/metrics` renders on the
  loop that delivers every stream. The frozen count is deliberately **not**
  here — `gc.get_freeze_count()` walks the permanent generation (11.9 ms at
  430k frozen, the cost freezing exists to remove) for a number that changes
  twice in a process's life. The startup log has it, and so does the census.
- **`reclaim_watch`** logs one warning — once, not per check — if that line
  ever rises, because a counter nobody looks at is not a safeguard. It sees
  only what a collection reclaimed, so cyclic garbage that reaches gen-2 before
  it dies is invisible to it where gen-2 passes are rare; the gen-2 size in
  `/debug/gc_census` is what shows that.
- **`GET /debug/gc_census`** breaks the scanned set down by type and by owning
  library. Unlike the metrics it walks every tracked object (~1s at a million),
  so it is asked for, never scraped, and it runs in a worker thread rather than
  on the loop that delivers the streams. It reports counts only: naming what a
  container *holds* would mean serialising the keys of parsed request bodies
  into an unauthenticated response. `top` and `types_per_owner` bound the two
  breakdowns.

## Incremental detokenizer

Streaming decodes each delta from two `tokenizer.decode` calls that share a
window start, so subtracting one from the other isolates the new text without
emitting a half-formed UTF-8 character. These two settings govern the shortcut
that removes one of those calls. Nothing here is related to the KV prefix
cache, which is what "cache" means everywhere else in this repository.

| Variable | Type | Default | Description |
|----------|------|---------|-------------|
| **ATOM_DETOKENIZER_DELTA_REUSE** | auto \| on \| off | auto | Whether the incremental detokenizer may reuse the delta it last emitted in place of one of its two `tokenizer.decode` calls per update. The two decodes share a window start so that subtracting one from the other isolates the new text; the first one only ever yields a length, and its span is what the previous call already emitted. Measured on DeepSeek-V4-Pro: **1.74x** at one token per update, **1.40x** at sixty-four. Holds only where decoding a token span does not depend on where the window started — true for the byte-level BPE tokenizers measured (DeepSeek, Qwen3.5, GLM-5.2), not for a SentencePiece-style decoder that adds or strips a leading space by position. `auto` therefore verifies it at startup (~15 ms) instead of assuming it from a class name, and leaves it off on any failure. Case-insensitive; an unrecognised value is logged and leaves reuse off, since off is what someone spelling this setting is reaching for. Unrelated to the KV prefix cache. |
| **ATOM_DETOKENIZER_AUDIT_EVERY** | int | 1000 | How often a reused delta is checked against a real decode once reuse is on. A mismatch answers that call from the decode, then turns reuse off for the process and logs once. Only audited calls are checked, so at the default up to 999 deltas can ship between a tokenizer starting to disagree and the audit that notices it; `1` checks every update, which is what the startup probe runs at. An unusable value — 0, negative, or not a number — is logged and the default used; this is not how reuse is turned off, `ATOM_DETOKENIZER_DELTA_REUSE=off` is. |

### Debug dump (`atom.utils.debug_helper`)

Env-gated dump / compare / monkey-patch primitives for forward bisect &
batch invariance investigation. All entries are **no-op when their
controlling `*_DIR` is unset**, so they are safe to leave wired into
production paths. See `.claude/skills/dump-bisect-debug.md` for the
methodology and `atom/utils/debug_helper/` for the implementation.

| Variable | Type | Default | Description |
|----------|------|---------|-------------|
| **ATOM_FWD_DUMP_DIR** | str | — | Enables `install_block_forward_hooks`. Per-Block hidden state is saved to `{DIR}/layer{LL}_{Cls}_rank{R}[_call{NNN}].pt`. |
| **ATOM_FWD_DUMP_LAYERS** | csv int | "" (= all) | Comma-separated layer ids to dump (e.g. `0,5,15,30`). Empty string means dump every layer. |
| **ATOM_FWD_DUMP_BLOCK_CLASS** | csv str | `Block` | Module class names to hook. Multiple values supported (e.g. `Block,DeepseekV4Attention,MoE,Compressor,Indexer`) for sub-stage bisect. Override per model. |
| **ATOM_FWD_DUMP_LAYER_ATTR** | str | `layer_id` | Attribute name on the block carrying its index. Some non-DeepSeek models use `layer_idx`. |
| **ATOM_FWD_DUMP_ONE_SHOT** | bool | 1 (true) | When `1`, only the first call per layer is dumped (typical: warmup). Set to `0` to enumerate every call (`_call000.pt`, `_call001.pt`, …) — required when bisecting per-seq dispatch loops. |
| **ATOM_WEIGHT_DUMP_DIR** | str | — | Enables `maybe_dump_weights_and_exit`. Per-rank params + buffers for selected layers dumped to `{DIR}/weight_rank{R}_layer{L}.pt`. Skips `.experts.*` (FP4 packed). |
| **ATOM_WEIGHT_DUMP_LAYERS** | csv int | `0` | Comma-separated layer ids to dump weights for. |
| **ATOM_WEIGHT_DUMP_EXIT** | bool | 1 (true) | When `1` (default), call `sys.exit(0)` after dumping. Set to `0` to continue inference after dump. |
| **ATOM_DEBUG_TOPK** | int | 0 | Set to `K > 0` to log top-K logits per row from `Sampler.forward` via `maybe_log_topk()`. Only rank 0 writes. |
| **ATOM_DEBUG_TOPK_PATH** | str | — | Optional output file for top-K logs. Writes to stderr if unset. |

CLI for comparing dumps:

```bash
python -m atom.utils.debug_helper.compare slot-invariance --dir DIR --n-slots 4
python -m atom.utils.debug_helper.compare ref-vs-target  --dir DIR
python -m atom.utils.debug_helper.compare layer-bisect   --dir DIR --threshold 0.99
python -m atom.utils.debug_helper.compare schema --a A.pt --b B.pt
```

## Benchmarks (optional)

| Variable | Type | Default | Description |
|----------|------|---------|-------------|
| **OPENAI_API_KEY** | str | — | API key for OpenAI-compatible benchmark requests. |
| **VLLM_USE_MODELSCOPE** | bool | false | If set to `true`, use ModelScope for model downloads in benchmarks. |
| **SAVE_TO_PYTORCH_BENCHMARK_FORMAT** | bool | false | If set, save benchmark results in PyTorch benchmark format. |

## Internal / Set by ATOM

The following variables are set internally by ATOM; users typically do not need to configure them:

| Variable | Description |
|----------|-------------|
| **AITER_QUICK_REDUCE_QUANTIZATION** | Set to `INT4` for Llama models with bf16/fp16. |
| **TORCHINDUCTOR_CACHE_DIR** | Set by compiler interface for inductor cache. |
| **TRITON_CACHE_DIR** | Set by compiler interface for Triton cache. |

## Reference

Environment variables are defined and accessed via `atom.utils.envs`:

```python
from atom.utils import envs

# Example: check data parallel size
dp_size = envs.ATOM_DP_SIZE
```

See `atom/utils/envs.py` for the full list of lazy-evaluated environment variables.

## Mooncake PD matched rails

| Variable | Type | Default | Description |
|----------|------|---------|-------------|
| **ATOM_MOONCAKE_MATCHED_RAILS** | str | "" | Use `auto` to discover ACTIVE local HCAs in the primary HCA's numbered name family, or set a comma-separated allowlist. Enables a lazy single-HCA engine pool on P, selected by D's advertised HCA name. Requires RDMA, a single primary HCA, and corresponding same-name rails across hosts. Unset preserves existing behavior. |

See [Mooncake matched rails](mooncake_matched_rails.md) for independent P/D
rank configuration, deployment requirements, and registration lifetime.

## Mooncake PD MLA staging

| Variable | Type | Default | Description |
|----------|------|---------|-------------|
| **ATOM_PD_MLA_STAGING** | bool | 1 | Prefill (`kv_producer`, plain `mooncake` connector) only. When the decode side runs DCP, gather the MLA tokens each decode rank owns into a GPU staging slot laid out as its destination pages, then RDMA one descriptor per run of adjacent destination pages instead of one per 576-byte token. Destination bytes are identical. The pool holds one 8 MiB slot per send worker (`num_worker_threads`, default 16), capped at 256 MiB per producer GPU, and is held back from the KV cache budget. A prefill learns the decode's DCP size only per request, so every Mooncake producer of a plain MLA model (not DeepSeek-V4 or an MLA hybrid such as Kimi) without producer DCP reserves the pool while this is on; set 0 where no decode runs DCP. 0 restores the per-token path. |
| **ATOM_PD_FAILED_WRITE_HOLD_S** | float | 60 | Mooncake P/D. Mooncake gives up on a sync RDMA write at `MC_TRANSFER_TIMEOUT` without cancelling the work it posted, so a failed write may still land afterwards; a prefill therefore never retries one. A decode holds a failed receive's KV pages (and its staging row) this many seconds before reporting the failure; the scheduler then aborts the request and frees its pages, which no late write can reach any more. A prefill keeps a landing slot whose write failed out of use for as long. 60 s covers a write Mooncake keeps re-posting while the network is down (each of its 9 tries waits at most a 5 s connect); behind a decode whose handshake hangs it can take about 10 minutes. 0 releases at once. |

## Mooncake PD MLA landing

Set on the decode side; the prefill side follows what each write request offers and needs MLA staging enabled.

| Variable | Type | Default | Description |
|----------|------|---------|-------------|
| **ATOM_PD_MLA_LANDING** | bool | 1 | Decode (`kv_consumer`, plain `mooncake` connector, DCP > 1, a plain MLA model) only. Allocate a GPU landing pool per rank and give prefill stages partitions of it. A stage writes a rank's MLA rows packed in rank order into a landing slot with one RDMA descriptor per slot (instead of one per destination page); the decode rank scatters the slot into its paged KV cache on a side stream and returns the slot. A request completes only after its slots are scattered. Destination bytes are identical to the staged path. The pool is 256 MiB of 8 MiB slots per decode rank; it and the scatter's segment table (about 12 MiB at block size 1, less for larger blocks) are held back from the KV cache budget. Only a non-DCP MLA prefill with `ATOM_PD_MLA_STAGING=1` lands, so set 0 where none does (for example symmetric DCP P/D) to give the pool back to the KV cache. Partitions go first come, first served: a prefill stage endpoint gets `32 // its PP size` slots on first contact and keeps them while the decode runs, so with several prefill instances per decode rank a later one may get fewer slots, or none and keep the staged path. A stage that reports it cannot land returns its partition at its first write-done and is not offered slots again. A prefill sends a transfer needing fewer than 2 slots, or the rest of one after waiting 10 ms for a free slot, through the staged per-page path. 0 keeps the staged per-page writes. |