CLI reference

The flags you’ll reach for most, by entry point. This is a curated reference — run python -m infera.<thing> --help for the complete, authoritative list.

Match transport/discovery fleet-wide

--discovery-backend, --request-transport, and --kv-event-transport must be the same on the server and every worker. See Routing & transport.

Server — python -m infera.server

Flag	Default	Meaning
--host / --port	0.0.0.0 / 8000	OpenAI-compatible HTTP bind
--router-tokenizer-path	(required)	tokenizer for prefix hashing; must match workers
--router-mode	auto	auto (in-process policy) | direct (trust a GAIE EPP’s x-worker-instance-id pick)
--router-policy	kv-aware	kv-aware | round-robin (used when --router-mode auto)
--kv-overlap-weight	1.0	KV-aware: cache locality vs load balance
--kv-prefill-overlap-weight	(unset)	KV-aware PD: prefill-side weight (typical 20.0); overrides the global
--kv-decode-overlap-weight	(unset)	KV-aware PD: decode-side weight (typical 2.0); overrides the global
--request-max-retries	1	retry on an alternate worker on pre-response failure (never mid-stream); 0 disables
--breaker-failure-threshold	3	consecutive worker faults (5xx / unreachable) before a worker leaves rotation; 0 disables the breaker
--breaker-cooldown-s	5	how long a tripped worker is excluded before one probe request is admitted
--breaker-max-cooldown-s	60	ceiling for that cooldown, which doubles on each failed probe
--discovery-backend	kubernetes	kubernetes | etcd
--etcd-endpoint	—	required for --discovery-backend etcd
--etcd-prefix	/infera/workers/	etcd key prefix the fleet registers under
--k8s-label-selector	$INFERA_K8S_LABEL_SELECTOR	worker-Pod selector (k8s discovery)
--k8s-namespace	(Pod’s own namespace)	namespace to watch (k8s discovery)
--request-transport	nats	nats | http
--kv-event-transport	nats	nats | zmq
--nats-server	$NATS_SERVER / nats://127.0.0.1:4222	broker URL
--nats-req-idle-timeout	900s	per-chunk inactivity backstop that ends the request
--http-req-idle-timeout-s	0 (off)	same, for --request-transport http
--stream-admission-warn-s	240s	log a stream still awaiting its first byte (reports only)
--stream-stall-warn-s	60s	log a stream that went silent after producing bytes (reports only)
--nats-req-max-duration	0 (off)	hard wall-clock cap per request
--nats-req-max-pending	0 (off)	per-worker admission limit (→ 429)
--kvd-socket-path	—	kvd UDS; enables POST /v1/cache/prewarm (L3 prefetch), else 503
--enable-profiling	off	torch-profiler control plane

Worker — python -m infera.engine.{sglang,vllm,atom}

Infera flags vs forwarded engine flags

The worker launcher parses the Infera flags below and forwards everything else verbatim to the underlying engine. So --advertise-host, --discovery-backend, --request-transport, --kv-event-transport, --enable-kv-events are Infera’s; engine-native flags like --tp-size / --tensor-parallel-size, --page-size / --block-size, and the --disaggregation-* set are passed through to SGLang/vLLM/ATOM (and validated by them, not by Infera). Consult the engine’s own docs for those.

Common:

Flag	Default	Meaning
--model-path (sglang/atom: --model)	(required)	served model
--port (atom: --server-port)	—	engine HTTP port
--host	0.0.0.0	bind address
--etcd-endpoint	—	shared etcd (etcd discovery)
--advertise-host	= --host	routable host/IP for cross-node
--discovery-backend	kubernetes	kubernetes | etcd
--request-transport	nats	nats | http
--enable-kv-events / --no-enable-kv-events	on	publish KV events for KV-aware routing
--kv-event-transport	nats	nats | zmq

Parallelism:

Flag	Engine	Meaning
--tp-size N	SGLang	tensor-parallel across N GPUs
--tensor-parallel-size N	vLLM	tensor-parallel across N GPUs
-tp N	ATOM	tensor-parallel across N GPUs
--page-size 16 / --block-size 16	SGLang / vLLM	KV block size (match fleet-wide)

PD disaggregation:

Flag	Engine	Meaning
--disaggregation-mode prefill|decode|null	SGLang	role; null = colocated
--disaggregation-bootstrap-port	SGLang	prefill bootstrap port
--disaggregation-transfer-backend mooncake|mori	SGLang	KV transport
--kv-transfer-config '{...}'	vLLM / ATOM	connector + kv_role (producer/consumer)

KV-cache daemon — python -m infera.kvd

Flag	Default	Meaning
--socket	—	UDS path engines connect to
--max-bytes	—	host RAM (L2) budget
--shared-arena-bytes	auto	zero-copy arena size (auto = --max-bytes; 0 = off)
--long-path / --long-paths	—	L3 region: single device / striped across devices (mutually exclusive)
--long-bytes	—	L3 region size
--use-tablespace	(deprecated)	no-op, accepted for backward compatibility; the L3 tier is always the container-file tablespace layout now
--long-backend	tablespace	tablespace (L3) | mooncake | lmcache (L4) — one only
--io-mode	auto	auto | direct | buffered (auto picks O_DIRECT on NVMe, buffered on NFS/SATA)
--long-workers-per-shard	8	intra-shard IO parallelism for the striped (--long-paths) region

Per-request KV params (OpenAI extra_body.kv_transfer_params)

Key	Values	Meaning
infera_retention	none|short|long (default long)	cache class. The vLLM connector honors none/short/long; ephemeral exists as a daemon-side eviction class but is not selectable per-request via the vLLM connector (it maps to long).

Per-request TTL (infera_ttl_seconds) is supported by the daemon/wire protocol but is not currently propagated by the vLLM connector — TTL is a daemon-level concern today, not a per-request knob on the vLLM path.

Simulator — inferasim

InferaSim has its own entry points and its own (large) flag surface. They are documented by task rather than by flag, starting at Simulation overview.

Entry point	Purpose
inferasim inference	project or simulate one serving recipe
inferasim anchor	harvest a GPU-calibrated anchor
inferasim-tune	LLM-driven recipe search

infera-projection aliases inferasim and infera-tuning aliases inferasim-tune; python -m infera.projection.cli works if the console scripts are not installed. The simulator’s environment variables are all INFERASIM_* — see Boundaries and verification.

Important environment variables

Var	Where	Why
PYTHONHASHSEED=0	vLLM worker	mandatory for cross-restart cache hits (vLLM salts block hashes per process)
HIP_VISIBLE_DEVICES	worker	which GPU(s) the worker uses
MC_DISABLE_HIP_TRANSPORT=1	PD worker	force RDMA, not the hipIpc transport (already the image default)
MC_GID_INDEX=1 / NCCL_IB_GID_INDEX=1	cross-node PD	RoCEv2 GID (default 0 is link-local)
VLLM_HOST_IP	vLLM PD	routable host IP for the connector