Server & router

Server & router#

infera.server is a FastAPI process that speaks the OpenAI API and holds the router. It carries no model weights — just a live view of the worker fleet (from etcd) and the logic that picks a worker per request. Run any number of replicas behind a load balancer; they share one fleet view.

Endpoints#

Inference (OpenAI-compatible)#

Endpoint

Use

POST /v1/chat/completions

chat-style requests (messages array)

POST /v1/completions

text-completion requests (raw prompt)

GET /v1/models

list the served model(s) — used by OpenAI clients and readiness checks

These match the OpenAI schema, so any OpenAI client library works — point its base_url at http://<server>:8000/v1.

Anthropic Messages API#

Endpoint

Use

POST /v1/messages

Anthropic-compatible chat (system blocks, tool use, streaming)

The server also speaks the Anthropic Messages API via a translation layer — point an Anthropic client’s ANTHROPIC_BASE_URL at http://<server>:8000 and it runs against the same workers. Text and tool-use are supported; multimodal and extended-thinking content are not translated.

curl localhost:8000/v1/messages \
  -H 'Content-Type: application/json' \
  -d '{"model":"Qwen/Qwen3-0.6B","max_tokens":50,
       "messages":[{"role":"user","content":"1+1=?"}]}'

PD + ATOM uses /v1/completions

ATOM only threads the KV-transfer params on the text-completion path, so disaggregated ATOM requests must go to /v1/completions. The chat path is unaffected. See PD disaggregation.

Inspection & admin#

Endpoint

Use

GET /v1/workers

read-only dump of the worker registry

GET /v1/admin/cache-view/<worker_id>

the router’s mirrored KV-cache state for one worker (KV-aware mode)

POST /v1/cache/prewarm

one-way hint to async-pull KV blocks into the warm tier ahead of a request (needs --kvd-socket-path)

POST /v1/admin/profile/start · /stop

start/stop the torch profiler on the workers (requires --enable-profiling; 403 otherwise)

Operations (health & metrics)#

Endpoint

Use

GET /health

liveness/readiness — returns the active-worker count (wire to k8s probes)

GET /metrics

Prometheus exposition — worker-pool + cache-view gauges (scrape with Grafana)

The router#

Every request is routed by the server. The router is selected by two axes:

Mode (--router-mode, default auto)

  • auto — the server selects the worker in-process using --router-policy (below) and the PD-preferring AutoRouter.

  • direct — trust an upstream GAIE Inference Gateway Endpoint Picker (EPP): the server dispatches to the worker named by the x-worker-instance-id request header and skips in-process selection (falling back to the policy when the header is absent). This is the per-worker frontend-sidecar topology; see Deployment → operator (GAIE).

Policy (--router-policy, default kv-aware) — used when mode is auto:

  • kv-aware — route by prefix-cache locality using each worker’s KV events. See KV-aware routing.

  • round-robin — stateless spread.

Independently, the AutoRouter picks the topology: it prefers a prefill+decode pair when PD workers exist, and falls back to a single mixed worker otherwise — you don’t choose per request. The full policy + transport reference is in Routing & transport.

client → server → router.dispatch(request, live_workers) → worker(s) → tokens → client

Running multiple replicas#

Every replica watches the same etcd, so they share one fleet view. Scale the front door independently of the GPUs:

python -m infera.server --port 8000 --etcd-endpoint <etcd-host>:2379 \
  --router-tokenizer-path Qwen/Qwen3-0.6B \
  --discovery-backend etcd --request-transport http --kv-event-transport zmq
python -m infera.server --port 8001 --etcd-endpoint <etcd-host>:2379 \
  --router-tokenizer-path Qwen/Qwen3-0.6B \
  --discovery-backend etcd --request-transport http --kv-event-transport zmq
# ...put a load balancer in front of :8000 and :8001

(The three transport/discovery flags are the no-broker dev path; on the production NATS + Kubernetes plane they’re the defaults and can be dropped — see Routing & transport.)

Key server flags#

Flag

Default

What it does

--port

8000

listen port

--etcd-endpoint

—

host:port of the shared etcd

--router-mode

auto

auto (in-process policy) | direct (GAIE EPP header)

--router-policy

kv-aware

kv-aware | round-robin

--router-tokenizer-path

(required)

small tokenizer for prefix hashing (e.g. Qwen/Qwen3-0.6B)

--kv-overlap-weight

1.0

KV-aware: trade cache locality vs load balance

The full flag list is in the CLI reference.

Contents