ROCm Infera#
A lightweight inference-serving layer for AMD ROCm GPUs.
Infera (the infera package) puts an OpenAI-compatible endpoint in front
of one or many model workers, and routes each request to the best worker — by
cache locality, by load, or by splitting prefill and decode across GPUs. It runs
vLLM, SGLang, or ATOM underneath, and adds an AMD-native tiered KV cache
that spills from GPU memory to RAM, NVMe, and the network.
If you have ever run vllm serve, you already know 80% of this. Infera is the
thin orchestration layer that turns one model server into a routed fleet.
Start here#
Five minutes from pip install to a served model and a working curl.
What Infera is, the three moving parts, and when to reach for it.
The OpenAI-compatible endpoints and how requests flow through the router.
Components#
The moving parts, one page each — what they are and how to run them.
OpenAI / Anthropic endpoints + the router (modes, policies).
vLLM / SGLang / ATOM workers — launch, TP, DP-attention.
kvd’s tier design, offload/onboard mechanism, and on-disk format.
The Kubernetes CRD that reconciles server + prefill/decode pools.
Feature one-pagers#
Each of these is a single, self-contained page — read only the one you need.
Route to the worker that already holds your prompt’s prefix.
Run prefill and decode on separate GPUs (or nodes) and stitch them with a KV transfer.
Spill KV to RAM / NVMe / network — the usage modes and per-request control.
Work out how many prefill/decode replicas hold your TTFT and ITL targets.
Simulation and capacity planning#
Before you buy the GPUs: InferaSim projects TTFT, latency, throughput and KV footprint for a serving recipe, and simulates a fleet under load. Measure one anchor per regime, then project everything else from it — no GPU per question.
What InferaSim is, the components, how the two engines differ, and where to start.
One recipe in, steady-state means out — and how to read the report.
Arrival-driven discrete-event simulation for p90/p99 and queueing.