Quickstart

Quickstart#

The fastest path from nothing to a served model you can curl. Single host, two GPUs. About five minutes.

You’ll start four things: etcd, one server, two engine workers, then send a request.

Choose your path#

🐳 Docker — recommended

Pull a prebuilt engine image from Docker Hub (rocm/infera:<engine>-v0.1.1) and run the stack inside it. This page walks that path.

🛠️ Build from source

Build the engine image from deploy/docker/Dockerfile.<engine> — it applies the vLLM / Mooncake / hipFile patches Infera needs. See Installation.

Installation
☸️ Kubernetes

Multi-host, autoscale, and PD via the operator — the production path.

Kubernetes deployment

```{admonition} Use an image, not a bare pip install

class:

important The engine images bundle Infera and the engine patches (vLLM / Mooncake / hipFile) it depends on. A hand-rolled pip install skips those, so it is not a supported path — pull a prebuilt image, or build one from deploy/docker/.


Steps 2–3 run **inside** a prebuilt engine image; you'll start it in step 2.

## 1. Start etcd

etcd is the shared registry the server and workers coordinate through. For dev,
one container is enough:

```bash
docker run -d --name infera-etcd --net host quay.io/coreos/etcd:v3.5.14 \
  etcd --advertise-client-urls http://127.0.0.1:2379 \
       --listen-client-urls http://0.0.0.0:2379

2. Start a server#

Open an engine image with host networking (so it shares etcd) — use the image for the engine you’ll run in step 3 (rocm/infera:{vllm,sglang,atom}-v0.1.1):

docker run --rm -it --name infera --network host --ipc host --shm-size 32g \
  --device /dev/kfd --device /dev/dri --group-add video --group-add render \
  rocm/infera:sglang-v0.1.1 bash

Then, inside the container, start the server. It holds the router, not the model — it needs a small tokenizer for the router’s prefix hashing (any small model works):

python -m infera.server --port 8000 --etcd-endpoint 127.0.0.1:2379 \
  --router-tokenizer-path Qwen/Qwen3-0.6B \
  --discovery-backend etcd --request-transport http --kv-event-transport zmq

Why those last three flags?

By default Infera runs the production plane — --discovery-backend kubernetes (needs a k8s API + label selector) and --request-transport nats (needs a NATS broker). This quickstart uses the no-broker dev path instead: etcd discovery + http request transport + zmq KV events. Set the same three flags on every server and worker, or they won’t find each other. See Routing & transport.

One behaviour differs on this path: stopping a worker still lets its in-flight generations finish, but nothing takes it out of routing until it is signalled. Leaving rotation before the process is told to stop needs Kubernetes to report the Pod as going away — see Graceful shutdown.

Tip

Run the server on as many ports/hosts as you like, all pointing at the same --etcd-endpoint. They share one view of the fleet — put a load balancer in front and you have HA for free.

3. Start two engine workers#

Run these inside the same container — open another shell into it with docker exec -it infera bash. Pick one engine (match the image from step 2). Each worker takes a GPU and self-registers into etcd. The --discovery-backend etcd --request-transport http --kv-event-transport zmq flags match the server (the no-broker dev path from step 2).

HIP_VISIBLE_DEVICES=0 python -m infera.engine.sglang \
  --model-path Qwen/Qwen3-0.6B --port 30000 --host 0.0.0.0 \
  --etcd-endpoint 127.0.0.1:2379 \
  --discovery-backend etcd --request-transport http --kv-event-transport zmq

HIP_VISIBLE_DEVICES=1 python -m infera.engine.sglang \
  --model-path Qwen/Qwen3-0.6B --port 30001 --host 0.0.0.0 \
  --etcd-endpoint 127.0.0.1:2379 \
  --discovery-backend etcd --request-transport http --kv-event-transport zmq

HIP_VISIBLE_DEVICES=0 python -m infera.engine.vllm \
  --model Qwen/Qwen3-0.6B --port 30000 --host 0.0.0.0 \
  --etcd-endpoint 127.0.0.1:2379 \
  --discovery-backend etcd --request-transport http --kv-event-transport zmq

HIP_VISIBLE_DEVICES=1 python -m infera.engine.vllm \
  --model Qwen/Qwen3-0.6B --port 30001 --host 0.0.0.0 \
  --etcd-endpoint 127.0.0.1:2379 \
  --discovery-backend etcd --request-transport http --kv-event-transport zmq

# ATOM registers via etcd only — it has no --discovery-backend/--request-transport/
# --kv-event-transport knobs (those are vLLM/SGLang-only). Just point it at etcd.
HIP_VISIBLE_DEVICES=0 python -m infera.engine.atom \
  --model Qwen/Qwen3-0.6B --server-port 30000 --host 0.0.0.0 \
  -tp 1 --etcd-endpoint 127.0.0.1:2379

HIP_VISIBLE_DEVICES=1 python -m infera.engine.atom \
  --model Qwen/Qwen3-0.6B --server-port 30001 --host 0.0.0.0 \
  -tp 1 --etcd-endpoint 127.0.0.1:2379

Give them a few seconds to load (each engine pulls + loads the model).

4. Verify#

Check the server is up and both workers have registered:

curl -sf localhost:8000/health && echo OK
curl -s localhost:8000/v1/workers | python -m json.tool   # expect 2 workers

5. Send a request#

curl localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"Qwen/Qwen3-0.6B",
       "messages":[{"role":"user","content":"1+1=?"}],
       "max_tokens":50}'

You should get an OpenAI-style completion (abbreviated):

{
  "model": "Qwen/Qwen3-0.6B",
  "choices": [{
    "index": 0,
    "message": {"role": "assistant", "content": "1 + 1 = 2."},
    "finish_reason": "stop"
  }],
  "usage": {"prompt_tokens": 12, "completion_tokens": 8, "total_tokens": 20}
}

The router picked one of the two workers for you — that’s a working Infera stack.

Nothing back yet?

  • 503 or empty /v1/workers — the engines are still loading the model; wait a few seconds and retry (watch a worker’s log for “registered”).

  • Connection refused — the server takes a moment to bind :8000; retry.

  • Workers up but the request errors/hangs — the server and every worker must share the same --etcd-endpoint and the three dev flags (--discovery-backend etcd --request-transport http --kv-event-transport zmq).

Where to go next#

Contents