Example: SGLang 1P1D (PD-disaggregated) — DeepSeek-V4-Pro on MI355X

Example: SGLang 1P1D (PD-disaggregated) — DeepSeek-V4-Pro on MI355X#

A concrete, copy-paste walk-through of a PD-disaggregated SGLang pair — one prefill worker + one decode worker on separate nodes — served through the standard Infera stack, with KV transferred over Mooncake RDMA on the ionic fabric, and throughput-swept end to end. Same tuned DeepSeek-V4-Pro (fp4: FP8 attention + MXFP4 MoE) recipe as the aggregated example, split across two 8×MI355X nodes.

For the single-node aggregated counterpart see SGLang mixed — DeepSeek-V4-Pro. For the concepts, read PD disaggregation first.

PD only pays off over real RDMA

Cross-node PD moves the KV cache over the fabric every request. It’s a win only on an RDMA fabric (ionic RoCEv2 here) — over TCP it’s slower. Infera’s launch preflight refuses a config prone to silent TCP fallback. Bring the fabric up first (cross-node prereqs).

The tuned recipe in one line#

The PD engine reuses the aggregated recipe’s two levers (--attention-backend dsv4 + the full fused-compress env set) and adds DP-attention always on plus a large prefill chunk. Both legs share these; they differ only by --disaggregation-mode:

flag group

value

base

--attention-backend dsv4 --disable-radix-cache --page-size 256 --swa-full-tokens-ratio 0.15 --disable-shared-experts-fusion --cuda-graph-max-bs 512 --context-length 9472

DP-attn

--tp-size 8 --dp 8 --enable-dp-attention --ep-size 8 --enable-prefill-delayer --prefill-delayer-max-delay-ms 5000

prefill chunk

--chunked-prefill-size 163840 --max-prefill-tokens 163840 (÷8 = 20480/rank)

PD

--disaggregation-mode prefill|decode --disaggregation-transfer-backend mooncake (prefill also --disaggregation-bootstrap-port 8998)

mem-fraction

prefill 0.85 (DP-attn prefill OOMs high), decode 0.90

0. Shared variables (both nodes)#

HF_CACHE=/path/to/hf-cache                     # TODO: HuggingFace cache / model dir (host path)
MODEL=$HF_CACHE/DeepSeek-V4-Pro                 # fp4 DSv4-Pro checkpoint (real weights, not stubs)
IMG=rocm/infera:sglang-v0.1.1                   # published SGLang engine image
P_IP=10.0.0.1                                   # TODO: prefill node data-plane (RDMA-rail) IP
D_IP=10.0.0.2                                   # TODO: decode  node data-plane (RDMA-rail) IP
ETCD=$P_IP:2379                                 # etcd lives on the prefill node

1. etcd — shared registry (on the prefill node host)#

docker run -d --name infera-etcd --network host quay.io/coreos/etcd:v3.5.14 \
  etcd --advertise-client-urls http://$P_IP:2379 --listen-client-urls http://0.0.0.0:2379

2. Worker container + check RDMA (on each node host)#

Start one persistent container per node (host net + /dev/infiniband = RDMA passthrough):

docker run -d --name pd_bench --network=host --ipc=host --shm-size=32G \
  --device=/dev/kfd --device=/dev/dri --device=/dev/infiniband \
  --group-add video --group-add render --cap-add=SYS_PTRACE --cap-add=IPC_LOCK \
  --security-opt seccomp=unconfined -v /mnt/vast:/mnt/vast --entrypoint "" "$IMG" sleep infinity

Now check RDMA is visible inside the container — the count must equal the node’s active RDMA ports (e.g. 8):

docker exec pd_bench bash -lc "ibv_devinfo | grep -c PORT_ACTIVE"

If that prints 0 (or ibv_devinfo errors), the container’s bundled ionic provider doesn’t match the host kernel driver — RDMA would silently degrade to TCP. Fix it by overlaying the host’s libionic.so onto both the soname and the libibverbs provider symlink, then re-check:

HL=$(readlink -f /usr/lib/x86_64-linux-gnu/libionic.so.1); B=$(basename "$HL"); docker cp "$HL" pd_bench:/usr/lib/x86_64-linux-gnu/$B
docker exec pd_bench bash -lc "cd /usr/lib/x86_64-linux-gnu && ln -sf $B libionic.so.1 && ln -sf libionic.so.1 libionic.so && cd libibverbs && ln -sf ../$B libionic-rdmav34.so && ldconfig; ibv_devinfo | grep -c PORT_ACTIVE"

For a fuller cross-node check (RoCE bandwidth + Mooncake KV transfer over the fabric, not just device visibility), run the preflight suite before launching — see Verify the RDMA fabric below.

Run the rest inside pd_bench on each node; re-export the shared vars. Export your token first if the model is gated:

export HF_TOKEN=hf_...                          # TODO: set if the model/tokenizer is gated

3. Infera router — infera.server (inside pd_bench on the prefill node, :8000)#

One router fronts the whole pair. It watches etcd and auto-detects the prefill+decode pair — no static prefill/decode list, no sglang_router:

python -m infera.server --host 0.0.0.0 --port 8000 --etcd-endpoint $ETCD \
  --router-tokenizer-path $MODEL --router-policy round-robin \
  --discovery-backend etcd --request-transport http --kv-event-transport zmq --router-backend rust

4. PD engines — one command per leg (inside pd_bench)#

Export the fused-compress lever on both nodes (paste as-is):

export SGLANG_USE_AITER=1 AITER_BF16_FP8_MOE_BOUND=0 SGLANG_OPT_FP8_WO_A_GEMM=0 SGLANG_OPT_DEEPGEMM_HC_PRENORM=0 SGLANG_OPT_USE_AITER_INDEXER=1 SGLANG_OPT_USE_TOPK_V2=0 SGLANG_FP8_PAGED_MQA_LOGITS_TORCH=1 SGLANG_OPT_USE_FUSED_PAGED_COMPRESS=1 SGLANG_HACK_FLASHMLA_BACKEND=unified_kv_triton SGLANG_OPT_USE_MULTI_STREAM_OVERLAP=false SGLANG_ROCM_USE_MULTI_STREAM=false SGLANG_OPT_USE_FUSED_COMPRESS=true SGLANG_OPT_USE_FUSED_COMPRESS_TRITON=true SGLANG_EAGER_INPUT_NO_COPY=true SGLANG_USE_ROCM700A=0 SGLANG_OPT_USE_JIT_INDEXER_METADATA=false SGLANG_OPT_USE_TILELANG_INDEXER=false SGLANG_OPT_USE_TILELANG_MHC_PRE=false SGLANG_OPT_USE_TILELANG_MHC_POST=false SGLANG_DP_USE_GATHERV=1

Prefill leg (prefill node):

HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 python -m infera.engine.sglang --model-path $MODEL --tp-size 8 --trust-remote-code --host 0.0.0.0 --port 30000 --advertise-host $P_IP --etcd-endpoint $ETCD --discovery-backend etcd --request-transport http --kv-event-transport zmq --attention-backend dsv4 --disable-radix-cache --page-size 256 --cuda-graph-max-bs 512 --swa-full-tokens-ratio 0.15 --disable-shared-experts-fusion --context-length 9472 --dp 8 --enable-dp-attention --ep-size 8 --enable-prefill-delayer --prefill-delayer-max-delay-ms 5000 --chunked-prefill-size 163840 --max-prefill-tokens 163840 --mem-fraction-static 0.85 --disaggregation-mode prefill --disaggregation-transfer-backend mooncake --disaggregation-bootstrap-port 8998

Decode leg (decode node) — same recipe, --disaggregation-mode decode, mem-fraction 0.90, no bootstrap port, plus --no-enable-kv-events (see the note below):

HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 python -m infera.engine.sglang --model-path $MODEL --tp-size 8 --trust-remote-code --host 0.0.0.0 --port 30000 --advertise-host $D_IP --etcd-endpoint $ETCD --discovery-backend etcd --request-transport http --kv-event-transport zmq --no-enable-kv-events --attention-backend dsv4 --disable-radix-cache --page-size 256 --cuda-graph-max-bs 512 --swa-full-tokens-ratio 0.15 --disable-shared-experts-fusion --context-length 9472 --dp 8 --enable-dp-attention --ep-size 8 --enable-prefill-delayer --prefill-delayer-max-delay-ms 5000 --chunked-prefill-size 163840 --max-prefill-tokens 163840 --mem-fraction-static 0.90 --disaggregation-mode decode --disaggregation-transfer-backend mooncake

Why –no-enable-kv-events on the decode leg (SWA models)

With KV events on, Infera auto-appends --disaggregation-decode-enable-radix-cache to a mooncake decode worker (so the router can steer prefix repeats). But DeepSeek-V4-Pro is a sliding-window-attention (SWA) model, and SGLang rejects that flag on SWA — the decode scheduler dies at startup with --disaggregation-decode-enable-radix-cache is incompatible with sliding window attention (SWA) models. --no-enable-kv-events skips the auto-append. PD here uses round-robin, not KV-aware routing, so nothing is lost. Non-SWA models can leave KV events on.

What Infera fills in for you

Infera auto-sets the RDMA env on ROCm (set-if-unset): MC_GID_INDEX=1, MC_DISABLE_HIP_TRANSPORT=1, and pins the KV-transfer host IP to the RDMA rail — so you don’t hand-set them (infera/engine/rocm_rdma_env.py). --advertise-host must be each node’s data-plane (RDMA-rail) IP so the peer can reach it. --kv-cache-dtype fp8_e4m3 is the default. For cold cross-node bootstrap, bump SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=1800.

5. Verify (curl through the router)#

curl -sf http://$P_IP:8000/health && echo OK
curl -s http://$P_IP:8000/v1/workers | python3 -m json.tool          # expect 2: one prefill, one decode
curl -s http://$P_IP:8000/v1/chat/completions -H 'Content-Type: application/json' \
  -d "{\"model\":\"$MODEL\",\"messages\":[{\"role\":\"user\",\"content\":\"1+1=?\"}],\"max_tokens\":32,\"temperature\":0}"

Two workers with disagg_mode prefill and decode, plus a coherent completion, means the router paired them and the KV hand-off works.

6. Sweep (InferenceX-aligned, through the router)#

Point sglang.bench_serving at the router on :8000 with sglang-oai. Same aligned knobs as the aggregated example:

for C in 64 128 256 384 512; do
  python3 -m sglang.bench_serving --backend sglang-oai --base-url http://$P_IP:8000 \
    --model $MODEL --tokenizer $MODEL \
    --dataset-name random --random-input-len 8192 --random-output-len 1024 --random-range-ratio 1.0 \
    --max-concurrency $C --num-prompts $((C*10)) --warmup-requests $((C*2)) \
    --request-rate inf --output-file pd_c${C}.jsonl 2>&1 | tail -20
done

Read total_token_throughput per jsonl for each concurrency point.

Verify the RDMA fabric (optional)#

The ibv_devinfo check in §2 confirms the container sees the NICs; it doesn’t confirm the two nodes can actually move KV over RDMA. Infera ships a preflight suite (infera-preflight) that measures cross-node RoCE bandwidth and Mooncake KV-transfer over the fabric — run it before a real bring-up to catch a degraded/TCP-fallback path early. Run it inside the container on both nodes against a shared output dir:

# both nodes, shared --dump-path; SLURM srun coordinates the two tasks
srun --nodelist=<prefill-host>,<decode-host> -N2 --ntasks-per-node=1 \
  python -m infera.tools.preflight --dump-path <shared-dir> --netperf --mooncake

It writes a per-host JSON + a combined HTML report; healthy same-rail RoCE is ~325–345 Gb/s, and Mooncake should show an rdma (not just tcp) number. See infera/tools/preflight/README.md for the full check list.

Notes & gotchas#

  1. Infera router auto-pairs; don’t use sglang_router/mini-lb. infera.server watches etcd and shapes each request to the prefill+decode pair. That’s the “use Infera’s own router” path — one server, no static --prefill/--decode list.

  2. Advertise the data-plane IP, not the public NIC. --advertise-host $P_IP/$D_IP must be the RDMA-rail IP the peer can reach; Infera also pins the KV host IP to the rail, but the etcd-advertised URL comes from --advertise-host.

  3. mem-fraction is role-asymmetric: prefill 0.85 (DP-attn prefill OOMs at 0.90 under high conc), decode 0.90. See DP prefill OOM.

  4. Mooncake vs MoRI: this image’s MoRI backend regressed; use --disaggregation-transfer-backend mooncake. The two are otherwise throughput-equivalent for DSv4 (small MLA KV). Both legs must match.

  5. Cold start ~30 min (weights + cuda-graph capture, same as the aggregated example) — don’t kill a slow launch; shrink --cuda-graph-max-bs to cut it.

  6. Switching topology/backend: kill both legs (pkill -9 -f infera.engine.sglang), wait for VRAM≈0 on both nodes, then relaunch — else the next run OOMs.

Contents