# Deployment

Two ways to run a fleet, smallest to largest:

| Target | How | Use when |
|---|---|---|
| **Manual** | `python -m infera.*` by hand | dev, benches, a small no-orchestrator fleet |
| **Kubernetes** | the operator + `InferaDeployment` CRD | multi-host, rolling upgrades, autoscale |

Both paths use the same images under `deploy/docker/`.

## Manual (bare-metal)

No images, no orchestrator — the `python -m infera.*` path (the same one the
[Quickstart](../getting_started/quickstart.md) uses), framed for a small standing
fleet. Start etcd, one or more servers, then the workers; every process points at
the **same** `--etcd-endpoint` and shares the no-broker dev plane
(`--discovery-backend etcd --request-transport http --kv-event-transport zmq`).

```bash
# 1. etcd — the shared registry
docker run -d --name infera-etcd --net host quay.io/coreos/etcd:v3.5.14 \
  etcd --advertise-client-urls http://<host>:2379 \
       --listen-client-urls http://0.0.0.0:2379

# 2. server(s) — the router/frontend on :8000; put a load balancer in front for HA
python -m infera.server --host 0.0.0.0 --port 8000 \
  --etcd-endpoint <host>:2379 --router-tokenizer-path <model> \
  --discovery-backend etcd --request-transport http --kv-event-transport zmq

# 3. workers — one per GPU, each self-registers (repeat on any host/GPU)
HIP_VISIBLE_DEVICES=0 python -m infera.engine.vllm \
  --model <model> --port 30000 --host 0.0.0.0 --advertise-host <host> \
  --etcd-endpoint <host>:2379 \
  --discovery-backend etcd --request-transport http --kv-event-transport zmq
```

Scale by launching more workers (on any host) against the same etcd; stop one and
its lease expires so it drops out of the fleet. On the production **NATS +
Kubernetes** plane those three dev flags are the defaults — drop them and see
[Routing & transport](../features/routing_and_transport.md).

## Kubernetes

The production path — install the platform with Helm, submit an
`InferaDeployment` CRD (aggregated, PD, or multi-node via the operator),
monitor, and send a request — has its own guide:
**[Kubernetes deployment](kubernetes.md)**. Ready-to-fill CR templates live in
`examples/k8s-deployments/`, and the CRD field reference is on the
[Operator](../components/operator.md) page.

## Engine images

Engine images overlay the Infera connector + `sitecustomize` hook +
Mooncake/ionic RDMA shims on top of a vendor base. Pick by runtime:

| Dockerfile (under `deploy/docker/`) | Base | Use for |
|---|---|---|
| `Dockerfile.vllm` | `vllm/vllm-openai-rocm:nightly-cbe9c40f…` | vLLM on MI355X (incl. hipFile) |
| `Dockerfile.sglang` | `lmsysorg/sglang:v0.5.18-rocm720-mi35x` | SGLang on MI355X |
| `Dockerfile.sglang.gfx942` | `lmsysorg/sglang:v0.5.16-rocm720-mi30x` | SGLang on MI325X |
| `Dockerfile.atom` | `rocm/atom:rocm7.2.4_…atom0.1.4` | ATOM on ROCm |
| `Dockerfile.atom`, `ATOM_NIGHTLY=1` | `rocm/atom-dev:nightly_…` | the [GLM-5.2 ATOM PD kit](../examples/atom_4p4d_glm5.2.md); released as `atom-nightly-<version>` |

Build from the repo root, e.g.:

```bash
docker build -f deploy/docker/Dockerfile.sglang \
  -t rocm/infera:sglang-dev .
```

```{admonition} Pin the SGLang base image
:class: important
The SGLang base is tied to a specific ROCm + build-date tag and is selected via
the `SGLANG_BASE_IMAGE` build-arg — override it to match the tag you've validated
rather than relying on the default. vLLM uses a single base image. ATOM has
two: its release base, and the ATOM nightly base of the `ATOM_NIGHTLY=1` row,
whose full build command is on the GLM-5.2 ATOM PD kit page.
```

```{admonition} hipFile on upstream rocm/sgl-dev
:class: note
The official `rocm/sgl-dev` tags strip the hipFile stack (no `libhipfile.so`).
The rocm720 Dockerfile rebuilds it from source so the async-read patch has
something to patch. Run `ais-check` inside the container on an MI355X
host to confirm the kernel exposes P2PDMA — otherwise hipFile falls back to a
CPU bounce (still works, just slower).
```

### Decode radix cache and HiCache with speculative decoding

This experimental option requires both Infera's forwarding changes and the
SGLang engine patch. Stock v0.5.18 rejects the MTP/radix combination even if the
environment variable is set; updating only the Infera package is insufficient.
Rebuild `deploy/docker/Dockerfile.sglang`, then set
`SGLANG_EXPERIMENTAL_DECODE_RADIX_SPEC=1` in the Decode worker environment.

With EAGLE/NEXTN top-k 1 on a supported Mooncake Decode leg:

- KV events enable automatic Decode radix flag forwarding.
- Adding `--enable-hierarchical-cache` and the desired HiCache options enables
  the radix prerequisite before SGLang's first argument validation, so an extra
  explicit radix flag is unnecessary. This also works with KV events disabled.
- Unsupported models still fail the radix compatibility check; an explicit
  `--disable-radix-cache` is incompatible with an automatic HiCache request.

Without the runtime opt-in, speculative Decode keeps its old behavior. Neither
HiCache nor session affinity is enabled by default. HiSparse, DCP, backend and
cache-builder restrictions remain. Hybrid SWA/SSM and EAGLE3 are outside scope.
The gfx942 image is not patched by this change. Custom compatible bases can use
`APPLY_SGLANG_DECODE_RADIX_SPEC_PATCH=0` to skip the build patch.

#### Launch examples

Build from this PR's checkout:

```bash
docker build -f deploy/docker/Dockerfile.sglang \
  --build-arg APPLY_SGLANG_DECODE_RADIX_SPEC_PATCH=1 \
  -t infera-sglang:mtp-decode-cache .
```

Run the following **inside that rebuilt mi35x container**, with model files and
ROCm/RDMA devices available. These are Decode launch commands; a matching
Prefill worker, etcd and a Router using HTTP requests/ZMQ KV events must already
be running. Replace the example model path and addresses with your deployment's
values. The Prefill worker must advertise the same served model name.

Common Decode arguments (Bash):

```bash
MODEL_PATH=/models/GLM-5.2-MXFP4
DECODE_IP=10.0.0.12
ETCD_ENDPOINT=10.0.0.10:2379

export SGLANG_EXPERIMENTAL_DECODE_RADIX_SPEC=1

DECODE_ARGS=(
  --model-path "$MODEL_PATH"
  --served-model-name glm5.2-mxfp4
  --trust-remote-code
  --host 0.0.0.0 --port 30000
  --advertise-host "$DECODE_IP"
  --discovery-backend etcd --etcd-endpoint "$ETCD_ENDPOINT"
  --request-transport http
  --enable-kv-events --kv-events on --kv-event-transport zmq
  --disaggregation-mode decode
  --disaggregation-transfer-backend mooncake
  --tp-size 8 --dp-size 8 --enable-dp-attention
  --speculative-algorithm EAGLE
  --speculative-eagle-topk 1
  --speculative-num-steps 5 --speculative-num-draft-tokens 6
  --mem-fraction-static 0.85
  --enable-metrics
  --json-model-override-args '{"index_share_for_mtp_iteration":false}'
)
```

Choose one of the two modes below.

**Decode radix + MTP, without HiCache:**

```bash
python -m infera.engine.sglang "${DECODE_ARGS[@]}"
```

KV events cause Infera to append the Decode radix flag. The patched SGLang hook
accepts the explicit MTP opt-in for EAGLE/NEXTN top-k 1.

**Decode radix + HiCache + MTP** (requires SGLang v0.5.19 or newer; see
[SGLang version for Decode HiCache](#sglang-version-for-decode-hicache)):

```bash
python -m infera.engine.sglang "${DECODE_ARGS[@]}" \
  --enable-hierarchical-cache \
  --hicache-ratio 1.5 \
  --hicache-write-policy write_through \
  --hicache-io-backend kernel \
  --hicache-mem-layout page_first
```

HiCache's required radix flag is set **before** `ServerArgs.from_cli_args`
performs cache compatibility checks, and is also forwarded to the engine.
Users do not need to add `--disaggregation-decode-enable-radix-cache` manually.
An explicit HiCache request implies this prerequisite even with KV events off;
this does not turn KV-event publication back on. Explicit radix flags remain
supported and are not duplicated.

The HiCache settings above reproduce the earlier configuration; size the host
pool for the node's memory. They are not a performance recommendation: the
historical C40 `write_through` test reduced total-token throughput by about 4.5%
and increased mean ITL by about 6.2% compared with Decode radix alone.

An old benchmark harness may still reject `DECODE_MTP=1` plus `DECODE_HICACHE=1`
before invoking Infera. These direct engine commands do not pass through that
harness guard; updating such scripts is separate from engine support.

#### SGLang version for Decode HiCache

Decode HiCache needs an SGLang build that includes
[sgl-project/sglang#35081](https://github.com/sgl-project/sglang/pull/35081):
v0.5.19 or newer. The default `Dockerfile.sglang` base, v0.5.18, does not
include it, so the HiCache mode above fails on the default image. Decode radix +
MTP without HiCache works on v0.5.18, and Prefill HiCache is not affected.

On v0.5.18 the Decode worker initializes normally (every rank logs
`impl=UnifiedRadixCache ... hicache_attached=True`), then the scheduler crashes
on the first request:

```text
AttributeError: 'UnifiedRadixCache' object has no attribute 'query_storage_hit_length'
```

The cause:

- For DSA models such as GLM-5.x, SGLang builds HiCache on `UnifiedRadixCache`
  ([#30468](https://github.com/sgl-project/sglang/pull/30468), in v0.5.18).
- Decode-side HiCache ([#26227](https://github.com/sgl-project/sglang/pull/26227))
  calls `tree_cache.query_storage_hit_length()` while preallocating each request,
  and `tree_cache.is_load_back_event_done()` when loading KV back from host
  memory.
- v0.5.18's `UnifiedRadixCache` implements neither method. #35081 added both
  three days after the v0.5.18 branch point and was not backported; v0.5.19 is
  the first release with it.

To use Decode HiCache, build on a base that includes #35081, for example with
`--build-arg SGLANG_BASE_IMAGE=<v0.5.19 or newer mi35x image>`. The full
`Dockerfile.sglang` build on a newer base has not been validated here.

Radix + MTP was GPU-validated with GLM-5.2 on the rebuilt default v0.5.18 image:
Decode reused a shared prefix and real MTP acceptance was observed. That run
did not include an accuracy benchmark. Earlier Decode HiCache runs used a patched
SGLang build newer than v0.5.19.

### Prefill load accounting in the Rust router

By default a disaggregated request holds both Prefill and Decode routing load
until Decode finishes. Pass `--pd-prefill-guard-release completion` (with
`--router-backend rust`) to release Prefill's load after its complete response
has been consumed; `decode` retains the default behavior. Unset, the Rust router
reads `INFERA_PD_PREFILL_GUARD_RELEASE`.

This changes router load accounting, not engine KV-cache ownership or eviction.
It applies to HTTP streaming/unary and NATS delivery. Response headers alone do
not release Prefill. Decode retains its own load guard, and the existing
client-disconnect, failed-stream and Prefill-drain timeout handling still aborts
unfinished engine work. A failed or cancelled Prefill drain also releases its
accounting entry.

The option is most useful for KV-aware routing when long Decode requests would
otherwise keep completed Prefill work in the load estimate. It does not change
the scoring formula or enable session affinity. It is a Rust-backend option;
the Python router is unchanged.

### Optional session affinity

Pass `--session-affinity` to `infera.server`; with `--router-backend rust` the
launcher forwards it to `infera-router`, which takes the same flag. It sits on
top of `--router-policy`, which still picks a new session's target:

| Value | Behavior |
|---|---|
| `off` (default) | Ignore session affinity and use the configured routing policy. |
| `prefill` | Keep each session on its selected Prefill worker/rank; choose Decode normally. |
| `both` | Keep independent Prefill and Decode targets; also pin aggregated workers. |

Clients opt in per request with `X-Session-ID`; AIPerf sends it, with its stable
per-conversation `X-Correlation-ID` value, when
`AIPERF_HTTP_X_SESSION_ID_FROM_CORRELATION_ID=true`. The ID is scoped by model
and role; it is not added to model input. Requests without the header keep
normal routing. Enabled modes reject duplicate, empty or over-1024-byte IDs.
Prefill and Decode rank numbers need not match.

`--session-affinity-ttl-seconds` is the idle lifetime (default 3600, allowed
1–86400 seconds). Active request leases prevent idle expiry. A changed or
unavailable target is reselected; failed requests and failed upstream streams
invalidate the affected binding. An old failure callback cannot delete a newer
binding. Bound requests still perform cache lookup and normal load accounting.

Bindings live in one Router process, with a limit of 65,536 model/session/role
entries. At capacity, new entries fall back to normal routing; active bindings
are not evicted. Restarting the router loses bindings. Multiple router replicas
need their own ingress affinity if cross-request consistency is required.

This option changes placement, not cache allocation. Decode affinity is most
useful when the engine can retain and reuse Decode prefixes, but does not itself
enable radix cache or guarantee better throughput. Session active-lease, hit and
selection counters are exposed through `infera_router_session_*` metrics, with
role labels and no session-ID labels.

A client that omits `X-Session-ID` gets normal routing without any error, so
check that sessions arrive. For `role="prefill"` (`role="mixed"` on aggregated
workers), `infera_router_session_hits_total` plus
`infera_router_session_selected_total` approximates the picks that carried a
session, and `infera_router_picks_total` summed over workers counts all picks. If
nearly all picks lack a session, the client is not sending the header (for
AIPerf, set `AIPERF_HTTP_X_SESSION_ID_FROM_CORRELATION_ID=true`). The comparison
is approximate: sessions that fall back to normal routing at capacity count as
neither hits nor selections, `prefill` mode never counts Decode, and mixed
retries add picks. These metrics come from the Rust router.

The Python router keeps the same modes, scoping, TTL, capacity and ID checks,
and a failed request also drops its bindings, with less bookkeeping: idle time
counts from a session's last routed request, a failed PD request drops both
roles' bindings rather than only the failed leg's, and there are no session
metrics.