KV-aware routing

KV-aware routing#

One-pager

What: route each request to the worker that already has its prompt prefix cached. Why: skip recomputing the shared prefix → lower TTFT, higher throughput. Cost: the server subscribes to each worker’s KV-cache events.

The idea#

Many workloads share long prefixes — a fixed system prompt, a RAG document, a growing chat history. If two requests with the same prefix land on different workers, the second one recomputes a prefix the first already cached. KV-aware routing sends them to the same worker so the prefix is a cache hit.

How it picks#

Each worker streams KV-cache events to the server, which mirrors that worker’s cached-block set. For an incoming request the router scores every worker:

cost(w) = overlap_weight × (request_blocks − cache_hits(w)) + active_blocks(w)

request_blocks − cache_hits(w) — how many prompt blocks worker w would have to compute from scratch (lower = better cache locality).
active_blocks(w) — distinct in-flight prompt blocks on w (shared prefixes counted once); a load term so a hot worker isn’t overwhelmed.
overlap_weight (--kv-overlap-weight, default 1.0) — the dial between cache locality (raise it → lower TTFT) and load balance (lower it → steadier ITL under load). At 0 the cache term drops out entirely and routing is pure load balance.

The worker with the lowest cost wins.

Worked example

A 10-block prompt with --kv-overlap-weight 1.0:

Worker	cache hits	miss blocks (10 − hits)	active blocks	cost
W0 — caches the prefix	8	2	6	`1.0 × 2 + 6` = 8
W1 — cold	0	10	1	`1.0 × 10 + 1` = 11

W0 wins: skipping 8 blocks of prefill beats being a little busier. Raise the weight and W0 wins by more; set it to 0 and only active_blocks counts.

Tuning prefill vs decode separately (PD)#

Under PD disaggregation the two sides want opposite routing. Prefill benefits enormously from a prefix cache hit (it’s the whole cost of prefill), while decode should mostly balance load. So the router takes per-role overlap weights that override the global --kv-overlap-weight:

Flag	Typical	Effect
`--kv-prefill-overlap-weight`	`20.0`	strongly prefer the prefill worker that already caches the prefix
`--kv-decode-overlap-weight`	`2.0`	mild locality on decode; mostly route by load

Leave them unset to fall back to the single --kv-overlap-weight for both.

Turn it on#

Two halves: events on the workers, the policy on the server. The examples use the no-broker dev flags (--discovery-backend etcd --request-transport http --kv-event-transport zmq); see the Quickstart for why, and drop them to use the production NATS+k8s defaults.

SGLang

python -m infera.engine.sglang \
  --model-path <model> --port 30000 --host 0.0.0.0 \
  --page-size 16 --enable-kv-events \
  --etcd-endpoint <etcd>:2379 \
  --discovery-backend etcd --request-transport http --kv-event-transport zmq

vLLM

python -m infera.engine.vllm \
  --model <model> --port 30000 --host 0.0.0.0 \
  --block-size 16 --enable-kv-events \
  --etcd-endpoint <etcd>:2379 \
  --discovery-backend etcd --request-transport http --kv-event-transport zmq

# server
python -m infera.server --port 8000 --etcd-endpoint <etcd>:2379 \
  --router-policy kv-aware \
  --router-tokenizer-path Qwen/Qwen3-0.6B \
  --kv-overlap-weight 1.0 \
  --discovery-backend etcd --request-transport http --kv-event-transport zmq

Important

The block/page size must match across the fleet and the router (16 is the canonical value). The router tokenizes the prompt and hashes it into blocks with a chained XXH3-64 — the tokenizer it uses is --router-tokenizer-path.

Verify it’s working#

# the router's mirrored view of one worker's cache
curl localhost:8000/v1/admin/cache-view/0.0.0.0:30000
# → {"worker_id":"0.0.0.0:30000","block_count":47}

Send two requests with the same long prefix and watch the second one’s TTFT drop and block_count stay flat (it reused, didn’t re-add).

When to use it#

Yes: shared system prompts, RAG over repeated documents, multi-turn chat, agentic loops with a growing prefix.
Less so: every request is unique and short — there’s no prefix to hit, so plain round-robin balances load just as well at lower overhead.

Environment variables#

KV-aware routing is configured by flags (--router-policy, --kv-overlap-weight, --kv-event-transport), not env vars. The one env var that affects whether the prefix cache it relies on persists:

Env	Value	Why it matters here
`PYTHONHASHSEED`	`0`	On vLLM, required for stable block hashes — without it cross-restart prefix hits are zero, defeating the routing win.

Full list on the environment variables page.