Environment variables

Environment variables#

Every environment variable Infera reads, grouped by subsystem. These are set on the process (engine, daemon, or server) — most have a CLI-flag equivalent where noted, and flags win over env vars. Each feature one-pager repeats the subset relevant to it; this page is the single place that lists them all.

Tip

The three that bite people most: PYTHONHASHSEED=0 (vLLM cross-restart cache hits), MC_GID_INDEX=1 (cross-node RDMA), and --ipc=host so the engine and kvd share the L2 arena.

kvd cache — connector (set on the engine process)#

Env	Default	What it does
`INFERA_KVD_SOCKET`	`/var/run/infera-kvd.sock`	Unix socket the engine uses to reach the kvd daemon.
`INFERA_KVD_HIPFILE_ROOTS`	(none)	L3 write roots, `retention=path` CSV (e.g. `long=/nvme/kvd-long`). Set this to enable a disk tier; also turns on the GPU-direct chunk-file path.
`INFERA_KVD_AIS`	`auto`	GPU-direct (hipFile) read/write: `auto` probes P2PDMA, `1`/`0` force. Use `1` on NFS/AIC, measure on local NVMe. Legacy name: `INFERA_KVD_GPU_DIRECT`.
`INFERA_KVD_CHUNK_TOKENS`	`auto`	Tokens per L3 chunk; `auto` targets `INFERA_KVD_CHUNK_TARGET_MIB` (default `128`).
`INFERA_KVD_LAYERWISE_LOAD`	`parallel`	L3 load mode: `off` / `stepped` / `prefetch` / `parallel`.
`INFERA_KVD_L3_BUDGET_BYTES`	`0` (uncapped)	Hard cap on the GPU-direct L3 file volume. These files are connector-owned, so the daemon’s `--long-bytes` does not bound them — this does.
`INFERA_KVD_L3_FREE_FLOOR`	`0.05`	Free-space fraction the connector’s L3 reaper keeps; evicts (retention-priority, LRU) when violated. Matters most on a shared filesystem.
`INFERA_KVD_L3_REAP_INTERVAL`	`30` (s)	L3 reaper tick period.
`INFERA_KVD_SAVE_WORKERS` / `INFERA_KVD_LOAD_WORKERS`	`auto`	L3 I/O worker counts (`auto` = mount `nconnect`; load drops to 1 without P2PDMA).
`INFERA_KVD_RETENTION_DEFAULT`	`long` (vLLM)	Default per-request retention when the request carries none. Required on SGLang to exercise the long tier (its SPI doesn’t propagate per-request retention).
`HIPFILE_UNSUPPORTED_FILE_SYSTEMS`	(unset)	libhipfile env (not Infera): set `1` to allow hipFile DMA reads on NFS.

Per-request (not env) cache hint goes through extra_body.kv_transfer_params: infera_retention — the vLLM connector honors none|short|long (default long). See KV-Cache Offload.

Advanced kvd knobs

Edge-case tuning (async-load depth, ring buffers, save pool, fsync policy, arena hugetlb/prefault, per-shard workers) exists as additional INFERA_KVD_* vars. A normal deployment never needs them; run python -m infera.kvd --help for the ones that matter.

PD disaggregation — RDMA transport (set on each engine)#

Env	Typical	What it does
`MC_GID_INDEX`	`1`	Mooncake RoCEv2 (routable) GID index — `1` on our ionic fleet; verify with `show_gids` (often 1 or 3). Index 0 is the link-local/RoCEv1 GID and times out cross-node.
`NCCL_IB_GID_INDEX`	`1`	Same idea for the collectives (RCCL) path.
`MC_DISABLE_HIP_TRANSPORT`	`1` (cross-node)	Force RDMA instead of the intra-node HIP/XGMI shortcut (which advertises an empty segment a remote peer rejects).
`MORI_IB_GID_INDEX`	`1`	RoCEv2 GID index for the MoRI transport.
`VLLM_HOST_IP`	(none)	Routable IP a vLLM worker advertises for KV transfer (cross-node). Pair with `--advertise-host`.
`RDMAV_FORK_SAFE`	`1`	libibverbs fork-safety; set for RDMA workers.

See PD disaggregation for the bring-up checklist and the RDMA self-check.

Routing & transport (set on the server and every worker)#

Env	Default	What it does
`NATS_SERVER` / `NATS_URL`	(injected by operator)	NATS broker address when `--request-transport nats`.
`INFERA_NATS_REQ_IDLE_TIMEOUT`	`900` (s)	Max wait for the next reply chunk; expiry → 504 + cancel. `0` = forever.
`INFERA_NATS_REQ_MAX_DURATION`	`0` (off)	Hard wall-clock cap on a whole request.
`INFERA_NATS_REQ_MAX_PENDING`	`0` (off)	Per-worker admission limit; backlog at the cap → 429.
`INFERA_REQUEST_MAX_RETRIES`	`1`	Retry on an alternate worker before the first token (never mid-stream). `0` = fail fast.
`INFERA_K8S_LABEL_SELECTOR`	(operator default)	Pod selector for `--discovery-backend kubernetes`.
`INFERA_DRAIN_TIMEOUT`	`30` (s)	Graceful-drain window on shutdown.

Each has a CLI flag (flag > env > default); see Routing & transport and the CLI reference.

Engine correctness#

Env	Value	What it does
`PYTHONHASHSEED`	`0`	Mandatory on vLLM — vLLM v1 salts block hashes per process, so without a fixed seed you get 0 cross-restart cache hits. SGLang doesn’t need it.
`VLLM_SERVER_DEV_MODE`	`1`	Mounts dev routes including `POST /reset_prefix_cache` (used to clear L1 + kvd L3 without a restart).