Routing & transport

One-pager

Four knobs govern the fleet: how workers are discovered (--discovery-backend), how requests reach a worker (--request-transport), how KV events flow (--kv-event-transport), and how the router picks a worker (--router-mode + --router-policy). Set the discovery/transport knobs the same on the server and every worker. For the server as a component, see Server & router.

The knobs

Knob	Flag (server + worker unless noted)	Options	Default (this build)
Discovery	--discovery-backend	kubernetes | etcd	kubernetes
Request transport	--request-transport	nats | http	nats
KV-event transport	--kv-event-transport	nats | zmq	nats
Router mode (server)	--router-mode	auto | direct	auto
Router policy (server)	--router-policy	kv-aware | round-robin	kv-aware

Important

The defaults (NATS + Kubernetes) assume a reachable NATS broker and a k8s API — the production/operator setup, where NATS_SERVER is injected for you. The Quickstart uses the simpler no-broker path (etcd + http + zmq); see Running without a broker.

How the router picks — mode → policy → topology

1. Mode (--router-mode) — who decides:

• auto (default) — the server selects the worker in-process via the policy below and the PD-preferring AutoRouter.

• direct — trust an upstream GAIE Inference Gateway Endpoint Picker (EPP): dispatch to the worker named by the x-worker-instance-id request header (and x-prefill-instance-id for the PD prefill leg), skipping in-process selection. Falls back to the policy when the header is absent. This is the per-worker frontend-sidecar topology the operator wires up with spec.gaie.

2. Policy (--router-policy) — how auto mode scores workers:

• round-robin — stateless even spread. Best when requests are unique/short (no prefix to reuse).

• kv-aware — route to the worker that already caches the prompt’s prefix. See KV-aware routing for the cost function and the --kv-overlap-weight dial.

3. Topology — mixed vs disaggregated is automatic: when PD workers exist, the AutoRouter prefers a prefill+decode pair and the DisaggRouter shapes the per-connector body (concurrent push or serial pull); otherwise it routes to a single mixed worker. You don’t select PD per request. See PD disaggregation.

Discovery: how the worker list is built

• etcd — workers self-register with a lease + heartbeat; the server watches /infera/workers/. No Kubernetes needed — the external/standalone path (and what the Quickstart uses).

• kubernetes — workers annotate their own Pod; the server watches Pods by --k8s-label-selector. Zero etcd — the operator default.

Request transport

• http — the router forwards directly to the worker’s engine HTTP. Simple, no broker — good for dev/CI/single box.

• nats — the router publishes each request to the worker’s per-instance NATS subject and streams the reply back. Adds admission control and clean cancellation, at the cost of running a broker.

NATS request controls

When --request-transport nats (flag > env var > default):

Flag	Env	Default	Meaning
--nats-req-idle-timeout	INFERA_NATS_REQ_IDLE_TIMEOUT	900s	max wait for the next reply chunk (reset per chunk). Expiry → 504 + cancel. 0 = forever.
--nats-req-max-duration	INFERA_NATS_REQ_MAX_DURATION	0 (off)	hard wall-clock cap on the whole request. Expiry → 504 + cancel.
--nats-req-max-pending	INFERA_NATS_REQ_MAX_PENDING	0 (off)	per-worker admission limit; backlog at the cap → 429.

On timeout or client disconnect the router publishes to infera.cancel.<worker> so the worker aborts the in-flight generation instead of burning GPU.

Request failover

--request-max-retries (default 1) retries on an alternate mixed worker when a request fails before the first response — unreachable worker, NATS error, idle-timeout-before-first-token, or a 429 admission reject. It never retries mid-stream (once tokens flow, a failure surfaces to the client). Raise it for more resilience; set 0 to fail fast.

Circuit breaker

Failover on its own has no memory. Its tried set lives for one request, so a worker that is broken for inference but healthy to discovery — it accepts the connection, answers /health, stays ACTIVE — gets re-picked by the next request, and every request after that pays the failover cost again.

The breaker is that missing memory. After --breaker-failure-threshold consecutive faults a worker is dropped from the candidate list for --breaker-cooldown-s, then one probe request is admitted: if it succeeds the worker is restored, if it fails the cooldown doubles, up to --breaker-max-cooldown-s.

Flag	Env	Default	Meaning
--breaker-failure-threshold	INFERA_BREAKER_FAILURE_THRESHOLD	3	consecutive faults before removal; 0 disables
--breaker-cooldown-s	INFERA_BREAKER_COOLDOWN_S	5	exclusion window before a probe
--breaker-max-cooldown-s	INFERA_BREAKER_MAX_COOLDOWN_S	60	cap on the doubling backoff

Two exclusions are deliberate. 4xx never counts — a malformed request returns 400 from every worker it reaches, so counting it would trip the entire healthy fleet on one bad client. 429 never counts either: it means “full right now”, which the policy’s load accounting already routes around, and a doubling cooldown is far too heavy a response to transient backpressure.

If every candidate is open the router dispatches anyway rather than returning 503 — a request served by a probably-bad worker beats turning a partial outage into a total one.

The breaker never writes WorkerStatus; that field belongs to discovery. This is the router’s private opinion, and it is visible as infera_router_worker_breaker_state (0 closed / 1 half-open / 2 open) and infera_router_worker_breaker_trips_total. A worker tripping repeatedly while discovery still reports it ACTIVE is the signal worth alerting on.

Both the Python and Rust routers implement this identically, with the same flags.

The breaker is the router’s view of a worker that is failing. For the orderly case — a worker being removed on purpose — see Scaling a fleet, which covers draining in-flight generations before shutdown.

KV-event transport

Powers KV-aware routing. --kv-event-transport:

• nats — one broker subscription (infera.kv.events.>) for the whole fleet; workers run a relay to forward their engine events.

• zmq — the server opens a per-worker SUB socket directly. No broker.

Running without a broker

For no NATS (local dev, CI, a single box), override transport + discovery to the direct path on both the server and every worker:

# server
python -m infera.server --host 0.0.0.0 --port 8000 \
  --router-tokenizer-path <model> \
  --discovery-backend etcd --etcd-endpoint 127.0.0.1:2379 \
  --request-transport http --kv-event-transport zmq

# each worker
python -m infera.engine.sglang --model-path <model> --port 30000 --host 0.0.0.0 \
  --discovery-backend etcd --etcd-endpoint 127.0.0.1:2379 \
  --request-transport http --kv-event-transport zmq

This is exactly the shape the Quickstart and the PD bring-up recipes use.

Match the flags fleet-wide

Discovery and transport must agree across the server and all workers. A worker on http/etcd won’t be reached by a server expecting nats/kubernetes.

Environment variables

Each knob is also a flag (flag > env > default); set on the server and every worker.

Env	Default	What it does
NATS_SERVER / NATS_URL	(operator-injected)	NATS broker address when --request-transport nats.
INFERA_NATS_REQ_IDLE_TIMEOUT	900 (s)	Max wait for the next reply chunk; expiry → 504 + cancel.
INFERA_NATS_REQ_MAX_DURATION	0 (off)	Hard wall-clock cap on a whole request.
INFERA_NATS_REQ_MAX_PENDING	0 (off)	Per-worker admission limit; backlog at the cap → 429.
INFERA_REQUEST_MAX_RETRIES	1	Retry on an alternate worker before the first token. 0 = fail fast.
INFERA_ROUTER_MODE	auto	direct to trust a GAIE EPP’s per-request worker pick.
INFERA_K8S_LABEL_SELECTOR	(operator default)	Pod selector for --discovery-backend kubernetes.

Full list on the environment variables page. The complete flag table is in the CLI reference.