Quickstart

Quickstart#

The fastest path from nothing to a served model you can curl. Single host, two GPUs. About five minutes.

You’ll start four things: etcd, one server, two engine workers, then send a request.

Choose your path#

🐳 Docker — recommended

Pull a prebuilt engine image from Docker Hub (rocm/infera:<engine>-v0.1.1) and run the stack inside it. This page walks that path.

🛠️ Build from source

Build the engine image from deploy/docker/Dockerfile.<engine> — it applies the vLLM / Mooncake / hipFile patches Infera needs. See Installation.

Installation
☸️ Kubernetes

Multi-host, autoscale, and PD via the operator — the production path.

Kubernetes deployment

```{admonition} Use an image, not a bare pip install

class:

important The engine images bundle Infera and the engine patches (vLLM / Mooncake / hipFile) it depends on. A hand-rolled pip install skips those, so it is not a supported path — pull a prebuilt image, or build one from deploy/docker/.


Steps 2–3 run **inside** a prebuilt engine image; you'll start it in step 2.

## 1. Start etcd

etcd is the shared registry the server and workers coordinate through. For dev,
one container is enough:

```bash
docker run -d --name infera-etcd --net host quay.io/coreos/etcd:v3.5.14 \
  etcd --advertise-client-urls http://127.0.0.1:2379 \
       --listen-client-urls http://0.0.0.0:2379

2. Start a server#

Open an engine image with host networking (so it shares etcd) — use the image for the engine you’ll run in step 3 (rocm/infera:{vllm,sglang,atom}-v0.1.1):

docker run --rm -it --name infera --network host --ipc host --shm-size 32g \
  --device /dev/kfd --device /dev/dri --group-add video --group-add render \
  rocm/infera:sglang-v0.1.1 bash

Then, inside the container, start the server. It holds the router, not the model — it needs a small tokenizer for the router’s prefix hashing (any small model works):

python -m infera.server --port 8000 --etcd-endpoint 127.0.0.1:2379 \
  --router-tokenizer-path Qwen/Qwen3-0.6B \
  --discovery-backend etcd --request-transport http --kv-event-transport zmq

Why those last three flags?

By default Infera runs the production plane — --discovery-backend kubernetes (needs a k8s API + label selector) and --request-transport nats (needs a NATS broker). This quickstart uses the no-broker dev path instead: etcd discovery + http request transport + zmq KV events. Set the same three flags on every server and worker, or they won’t find each other. See Routing & transport.

Tip

Run the server on as many ports/hosts as you like, all pointing at the same --etcd-endpoint. They share one view of the fleet — put a load balancer in front and you have HA for free.

3. Start two engine workers#

Run these inside the same container — open another shell into it with docker exec -it infera bash. Pick one engine (match the image from step 2). Each worker takes a GPU and self-registers into etcd. The --discovery-backend etcd --request-transport http --kv-event-transport zmq flags match the server (the no-broker dev path from step 2).

HIP_VISIBLE_DEVICES=0 python -m infera.engine.sglang \
  --model-path Qwen/Qwen3-0.6B --port 30000 --host 0.0.0.0 \
  --etcd-endpoint 127.0.0.1:2379 \
  --discovery-backend etcd --request-transport http --kv-event-transport zmq

HIP_VISIBLE_DEVICES=1 python -m infera.engine.sglang \
  --model-path Qwen/Qwen3-0.6B --port 30001 --host 0.0.0.0 \
  --etcd-endpoint 127.0.0.1:2379 \
  --discovery-backend etcd --request-transport http --kv-event-transport zmq

HIP_VISIBLE_DEVICES=0 python -m infera.engine.vllm \
  --model Qwen/Qwen3-0.6B --port 30000 --host 0.0.0.0 \
  --etcd-endpoint 127.0.0.1:2379 \
  --discovery-backend etcd --request-transport http --kv-event-transport zmq

HIP_VISIBLE_DEVICES=1 python -m infera.engine.vllm \
  --model Qwen/Qwen3-0.6B --port 30001 --host 0.0.0.0 \
  --etcd-endpoint 127.0.0.1:2379 \
  --discovery-backend etcd --request-transport http --kv-event-transport zmq

# ATOM registers via etcd only — it has no --discovery-backend/--request-transport/
# --kv-event-transport knobs (those are vLLM/SGLang-only). Just point it at etcd.
HIP_VISIBLE_DEVICES=0 python -m infera.engine.atom \
  --model Qwen/Qwen3-0.6B --server-port 30000 --host 0.0.0.0 \
  -tp 1 --etcd-endpoint 127.0.0.1:2379

HIP_VISIBLE_DEVICES=1 python -m infera.engine.atom \
  --model Qwen/Qwen3-0.6B --server-port 30001 --host 0.0.0.0 \
  -tp 1 --etcd-endpoint 127.0.0.1:2379

Give them a few seconds to load (each engine pulls + loads the model).

4. Verify#

Check the server is up and both workers have registered:

curl -sf localhost:8000/health && echo OK
curl -s localhost:8000/v1/workers | python -m json.tool   # expect 2 workers

5. Send a request#

curl localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"Qwen/Qwen3-0.6B",
       "messages":[{"role":"user","content":"1+1=?"}],
       "max_tokens":50}'

You should get an OpenAI-style completion (abbreviated):

{
  "model": "Qwen/Qwen3-0.6B",
  "choices": [{
    "index": 0,
    "message": {"role": "assistant", "content": "1 + 1 = 2."},
    "finish_reason": "stop"
  }],
  "usage": {"prompt_tokens": 12, "completion_tokens": 8, "total_tokens": 20}
}

The router picked one of the two workers for you — that’s a working Infera stack.

Nothing back yet?

  • 503 or empty /v1/workers — the engines are still loading the model; wait a few seconds and retry (watch a worker’s log for “registered”).

  • Connection refused — the server takes a moment to bind :8000; retry.

  • Workers up but the request errors/hangs — the server and every worker must share the same --etcd-endpoint and the three dev flags (--discovery-backend etcd --request-transport http --kv-event-transport zmq).

Where to go next#

Contents