Routing & transport

One-pager

Four knobs govern the fleet: how workers are discovered (--discovery-backend), how requests reach a worker (--request-transport), how KV events flow (--kv-event-transport), and how the router picks a worker (--router-mode + --router-policy). Set the discovery/transport knobs the same on the server and every worker. For the server as a component, see Server & router.

The knobs

Knob	Flag (server + worker unless noted)	Options	Default (this build)
Discovery	--discovery-backend	kubernetes | etcd	kubernetes
Request transport	--request-transport	nats | http	nats
KV-event transport	--kv-event-transport	nats | zmq	nats
Router mode (server)	--router-mode	auto | direct	auto
Router policy (server)	--router-policy	kv-aware | round-robin	kv-aware

Important

The defaults (NATS + Kubernetes) assume a reachable NATS broker and a k8s API — the production/operator setup, where NATS_SERVER is injected for you. The Quickstart uses the simpler no-broker path (etcd + http + zmq); see Running without a broker.

How the router picks — mode → policy → topology

1. Mode (--router-mode) — who decides:

• auto (default) — the server selects the worker in-process via the policy below and the PD-preferring AutoRouter.

• direct — trust an upstream GAIE Inference Gateway Endpoint Picker (EPP): dispatch to the worker named by the x-worker-instance-id request header (and x-prefill-instance-id for the PD prefill leg), skipping in-process selection. Falls back to the policy when the header is absent. This is the per-worker frontend-sidecar topology the operator wires up with spec.gaie.

2. Policy (--router-policy) — how auto mode scores workers:

• round-robin — stateless even spread. Best when requests are unique/short (no prefix to reuse).

• kv-aware — route to the worker that already caches the prompt’s prefix. See KV-aware routing for the cost function and the --kv-overlap-weight dial.

3. Topology — mixed vs disaggregated is automatic: when PD workers exist, the AutoRouter prefers a prefill+decode pair and the DisaggRouter shapes the per-connector body (concurrent push or serial pull); otherwise it routes to a single mixed worker. You don’t select PD per request. See PD disaggregation.

Discovery: how the worker list is built

• etcd — workers self-register with a lease + heartbeat; the server watches /infera/workers/. No Kubernetes needed — the external/standalone path (and what the Quickstart uses).

• kubernetes — workers annotate their own Pod; the server watches Pods by --k8s-label-selector. Zero etcd — the operator default.

Request transport

• http — the router forwards directly to the worker’s engine HTTP. Simple, no broker — good for dev/CI/single box.

• nats — the router publishes each request to the worker’s per-instance NATS subject and streams the reply back. Adds admission control and clean cancellation, at the cost of running a broker.

NATS request controls

When --request-transport nats (flag > env var > default):

Flag	Env	Default	Meaning
--nats-req-idle-timeout	INFERA_NATS_REQ_IDLE_TIMEOUT	900s	max wait for the next reply chunk (reset per chunk). Expiry → 504 + cancel. 0 = forever.
--nats-req-max-duration	INFERA_NATS_REQ_MAX_DURATION	0 (off)	hard wall-clock cap on the whole request. Expiry → 504 + cancel.
--nats-req-max-pending	INFERA_NATS_REQ_MAX_PENDING	0 (off)	per-worker admission limit; backlog at the cap → 429.

On timeout or client disconnect the router publishes to infera.cancel.<worker> so the worker aborts the in-flight generation instead of burning GPU.

Request failover

--request-max-retries (default 1) retries on an alternate mixed worker when a request fails before the first response — unreachable worker, NATS error, idle-timeout-before-first-token, or a 429 admission reject. It never retries mid-stream (once tokens flow, a failure surfaces to the client). Raise it for more resilience; set 0 to fail fast.

KV-event transport

Powers KV-aware routing. --kv-event-transport:

• nats — one broker subscription (infera.kv.events.>) for the whole fleet; workers run a relay to forward their engine events.

• zmq — the server opens a per-worker SUB socket directly. No broker.

Running without a broker

For no NATS (local dev, CI, a single box), override transport + discovery to the direct path on both the server and every worker:

# server
python -m infera.server --host 0.0.0.0 --port 8000 \
  --router-tokenizer-path <model> \
  --discovery-backend etcd --etcd-endpoint 127.0.0.1:2379 \
  --request-transport http --kv-event-transport zmq

# each worker
python -m infera.engine.sglang --model-path <model> --port 30000 --host 0.0.0.0 \
  --discovery-backend etcd --etcd-endpoint 127.0.0.1:2379 \
  --request-transport http --kv-event-transport zmq

This is exactly the shape the Quickstart and the PD bring-up recipes use.

Match the flags fleet-wide

Discovery and transport must agree across the server and all workers. A worker on http/etcd won’t be reached by a server expecting nats/kubernetes.

Environment variables

Each knob is also a flag (flag > env > default); set on the server and every worker.

Env	Default	What it does
NATS_SERVER / NATS_URL	(operator-injected)	NATS broker address when --request-transport nats.
INFERA_NATS_REQ_IDLE_TIMEOUT	900 (s)	Max wait for the next reply chunk; expiry → 504 + cancel.
INFERA_NATS_REQ_MAX_DURATION	0 (off)	Hard wall-clock cap on a whole request.
INFERA_NATS_REQ_MAX_PENDING	0 (off)	Per-worker admission limit; backlog at the cap → 429.
INFERA_REQUEST_MAX_RETRIES	1	Retry on an alternate worker before the first token. 0 = fail fast.
INFERA_ROUTER_MODE	auto	direct to trust a GAIE EPP’s per-request worker pick.
INFERA_K8S_LABEL_SELECTOR	(operator default)	Pod selector for --discovery-backend kubernetes.

Full list on the environment variables page. The complete flag table is in the CLI reference.