Overview

This page is the conceptual tour — the mental model, no commands. If you’d rather get something running first, jump to the Quickstart and come back.

Is this for you?

Yes, if you’re running multiple GPUs and want smarter request routing across them — by cache locality, by splitting prefill and decode, or both. Infera sits in front of vLLM, SGLang, or ATOM and turns a collection of model workers into a routed fleet.

Probably not yet, if you have a single GPU and one model server. vllm serve or sglang launch is the right tool for that — come back when you’re scaling out.

What problem does Infera solve?

A single model server (vllm serve, sglang launch) is great until you need more than one GPU’s worth of serving. Then come the questions one server can’t answer on its own — each maps to a feature:

• I have 8 GPUs. How do requests spread across them without throwing away the prompt cache each time? → KV-aware routing

• My prompts share a long prefix (a system prompt, a document, a conversation). Can each request go to the GPU that already holds that prefix? → KV-aware routing

• Prefill (reading the prompt) and decode (generating tokens) want very different hardware. Can I run them on separate GPUs and move the KV between them? → PD disaggregation

• The KV cache only lives in GPU memory. Can I keep it warm in RAM / on disk so a restart — or the next user — doesn’t pay to recompute it? → KV-Cache Offload

Infera is the thin layer that answers these. It does not replace vLLM / SGLang / ATOM — it sits in front of them and orchestrates a fleet.

The three moving parts

1. The server (infera.server) — a FastAPI process exposing the OpenAI-compatible API (/v1/chat/completions, /v1/completions) on port 8000. It holds no model — it holds the router. Run as many replicas as you like behind a load balancer; they share one view of the fleet.

2. The workers (infera.engine.{vllm,sglang,atom}) — each is one model engine on one (or a few) GPUs. A worker registers itself into etcd at startup and heartbeats; when it dies, its lease expires and it drops out. You never hand the server a static worker list — the fleet is discovered.

3. etcd — a small key-value store: the shared source of truth for “which workers exist, and what each can do.” The server watches it; workers write to it. That’s the whole coordination story.

(Optional) kvd — KV-Cache Management. A per-host daemon that gives the workers a tiered KV cache: blocks spill from GPU HBM → host RAM → NVMe → network and stay warm across engine restarts, shared by every engine on the host (read back zero-copy via a shared-memory arena). Turn it on when you want prefixes to survive restarts or to reuse them across workers. See KV-Cache Management.

The one-sentence version

Workers announce themselves to etcd; the server watches etcd and routes each request to the best worker.

How a request flows

1. A client sends POST /v1/chat/completions to any server replica.

2. The router reads the live worker list (from etcd) and the request, then picks a worker — round-robin, KV-aware (prefix locality), or PD. See Routing & transport for the policies.

3. On a prefill/decode disaggregated path, the router dispatches to a prefill and a decode worker and coordinates the KV hand-off over the KV transport (Mooncake / MoRI).

4. The worker streams tokens back; the server relays them to the client.

The client never knows how many GPUs are involved — it’s just the OpenAI API.

Where it runs

Infera is ROCm-native, built and validated on AMD Instinct MI355X GPUs:

Capability	MI355X
Quantization	fp8 and mxfp4
RDMA NIC	AMD AINIC

The RDMA NIC carries the KV transfer in PD disaggregation; mxfp4 widens which quantized checkpoints you can load.

What to read next

• Get it running → Quickstart

• Pick an engine → Engines

• Deploy for real → Deployment

• The KV cache everyone keeps mentioning → KV-Cache Offload