Example: SGLang 1P2D (PD-disaggregated) — Kimi-K2.6-MXFP4 on MI355X

Example: SGLang 1P2D (PD-disaggregated) — Kimi-K2.6-MXFP4 on MI355X#

A PD-disaggregated SGLang deployment for amd/Kimi-K2.6-MXFP4 on AMD MI355X: one prefill worker feeding two decode workers (1P2D), fronted by the Infera router, with the KV cache moved over Mooncake RDMA, then throughput-swept end to end.

The runnable scripts for this example live in the repo under examples/sglang_1p2d_kimi2.6/. This page is the short overview; the README.md in that directory is the authoritative, copy-paste step-by-step guide. Every script is driven by environment variables and needs no editing.

The package covers two tests — run the single-node one first:

  1. SGLang single-node — sanity-check that Kimi-K2.6-MXFP4 starts and serves on one 4-GPU SGLang instance.

  2. Infera router + SGLang PD (1P2D) — the real multi-node target.

For the concepts behind PD, read PD disaggregation first.

PD only pays off over real RDMA

Cross-node PD moves the KV cache over the fabric on every request. It’s a win only on an RDMA fabric (ionic RoCEv2 here) — over TCP it is slower than a single node. The container’s ionic provider must match the host driver, or KV transfer silently falls back to TCP. So run preflight_rdma.sh on every PD node before the bring-up and confirm the visible port count equals the node’s active ports (not 0) — see §3 of the README.md.

Topology (1P2D)#

1P2D = 1 prefill + 2 decodes, spread over three nodes (8× MI355X each):

Node

Role

node-0

etcd + Infera router + prefill + verify + benchmark entrypoint

node-1

decode-0

node-2

decode-1

The tuned recipe#

Both PD legs share the aiter attention backend, fp8 KV, DP-attention, and Mooncake KV transfer. They differ mainly by GPU count and --disaggregation-mode:

leg

GPUs

parallelism

mem-fraction

key flags

prefill

4 (TP4)

--dp-size 4 --enable-dp-attention

0.85

--disaggregation-mode prefill --disaggregation-bootstrap-port 8998

decode

8 (TP8)

--dp-size 8 --enable-dp-attention (EP1)

0.90

--disaggregation-mode decode

Shared by both: --attention-backend aiter --kv-cache-dtype fp8_e4m3 --trust-remote-code --no-enable-kv-events --disaggregation-transfer-backend mooncake, plus env MC_GID_INDEX=1 SGLANG_USE_AITER=1.

Prerequisites#

  • Hardware: 3 nodes, 8× MI355X each (ROCm 7.2.0), Docker with GPU access. node-0 prefill uses 4 GPUs; each decode node uses all 8.

  • Model: amd/Kimi-K2.6-MXFP4 (~550B, license modified-mit, NOT gated).

  • Image: rocm/infera:sglang-v0.1.1.

Download the model to a local directory (it is bind-mounted read-only) and point MODEL at it. Benchmarking needs a local clone of InferenceX (IX). Full details, including the model download and the “adapt to your cluster” env vars, are in §1 Prerequisites / §2 Adapt to your cluster of the README.md.

Scripts#

All scripts are in examples/sglang_1p2d_kimi2.6/:

Script

Purpose

sglang_naive_engine.sh

Single-node SGLang server, default 4 GPUs / TP4.

preflight_rdma.sh

RDMA preflight: container port visibility + (optional) cross-node fabric check.

infera_0_etcd.sh

Start etcd (PD shared registry).

infera_1_server.sh

Start the Infera router/server (PD).

infera_2_sglang_prefill.sh

SGLang prefill leg, default 4 GPUs / TP4 / DP4.

infera_3_sglang_decode.sh

SGLang decode leg, default 8 GPUs / TP8 / DP8 / EP1.

curl.sh

Smoke-test the router (workers + chat).

bench_inferencex.sh

Benchmark via InferenceX.

Run#

Export the shared vars (MODEL, IX, DATA_NET, IMAGE) on every node, then run in this order:

single-node: engine -> verify -> benchmark
1P2D:        preflight -> etcd -> server -> prefill -> decode -> verify -> benchmark

Single-node (any 4-GPU MI355X):

IMAGE=rocm/infera:sglang-v0.1.1 \
MODEL=/your/path/Kimi-K2.6-MXFP4 bash sglang_naive_engine.sh

1P2D — first run the RDMA preflight on each PD node (the reported port count must equal the node’s active ports, not 0):

bash preflight_rdma.sh

Then, on node-0 (example data-net IP 10.0.0.1):

bash infera_0_etcd.sh
ETCD_ENDPOINT=10.0.0.1:2379 bash infera_1_server.sh
ETCD_ENDPOINT=10.0.0.1:2379 bash infera_2_sglang_prefill.sh

On node-1 and node-2 (ETCD_ENDPOINT must point at node-0):

ETCD_ENDPOINT=10.0.0.1:2379 bash infera_3_sglang_decode.sh

Verify through the router (bash curl.sh) — expect 1 prefill + 2 decodes plus a coherent completion — then sweep concurrency with bench_inferencex.sh (we swept 32→2048 for 1P2D). See §4 Run in the README.md for the exact commands, verify/benchmark details, and how to stop the services.

Notes & gotchas#

  1. Infera router auto-pairs; no static worker list. infera.server watches etcd and shapes each request to the 1P2D group — one server, --router-policy round-robin.

  2. Advertise the data-network IP. --advertise-host (and SGLANG_HOST_IP / HOST_IP) must be each node’s data-net IP the peers can reach, not the public NIC.

  3. RDMA passthrough on the PD legs. --device=/dev/infiniband, --cap-add=IPC_LOCK, MC_GID_INDEX=1, and the host libionic.so bind-mount keep the container’s RDMA provider aligned with the host driver — otherwise KV transfer silently degrades to TCP. Run the preflight checks first.

  4. Attention backend is aiter for both single-node and PD on this image.

The full gotchas list (cold-start behaviour, --no-enable-kv-events rationale, the decode-leg MoRI env) is in Notes & gotchas of the README.md.

Contents