Fleet and routing

Fleet and routing#

One-pager

What: several engine replicas behind a router, each with a real content-addressed KV block cache. Why: cache hit rate is a property of routing, not a constant — and the kv policy scores the same cost function the deployed router does. Cost: you have to supply content, either a prefix pool or a trace.

Simulate a fleet#

inferasim inference ... \
  --request-rate 6 --arrival-model poisson --des-num-requests 200 \
  --des-instances 4 --des-routing kv \
  --des-num-prefixes 8 --des-prefix-len 2048 --des-block-size 512

  Fleet: 4 instance(s), routing=kv | prefix pool: 8 prefixes
  KV block cache: block_size=512 tok, capacity/instance=unbounded, evictions=0, block-reuse=44.8%
  Prefix-cache hit rate: 89.5% of requests (avg 1833 cached tok/req; per-instance 88–90%)

Hit rate is emergent, not configured#

A prompt is an ordered sequence of block-hash ids, and a hit is the longest contiguous leading run of blocks already resident on the replica the request was routed to. The cache is finite and LRU-evicts under pressure, so hit rate is an emergent property of content, capacity and routing rather than a number you supply.

This is the structural difference from --prefix-cache-hit-rate on the analytical path, which asserts a rate. Here, changing the routing policy changes the hit rate, because that is what routing policies do.

Routing policies#

--des-routing

Behaviour

kv

The serving router’s own policy: minimise --des-overlap-weight × blocks missed + replica load. The weight dials reuse against balance (0 = pure load balance).

prefix_aware

Consistently hash the leading block to a home replica, so same-prefix requests co-locate. Misses ≈ number of prefixes, independent of fleet size.

round_robin / random

Ignore locality, so every replica re-warms every prefix. Misses ≈ prefixes × replicas.

The trade-off is real in both directions. Locality-seeking policies maximise reuse but can overconcentrate a hot prefix onto one replica, raising its decode pressure and lengthening the makespan. The report shows pooled latencies alongside the per-replica hit-rate spread so you can see both halves — a suspiciously good hit rate next to a wide per-instance spread is a hot-spotting warning, not a win.

Because kv scores the same cost function the deployed router does, sweeping --des-overlap-weight projects what retuning that weight in production would cost before you change it there.

Cache capacity and eviction#

Flag

Meaning

--des-block-size

tokens per cache block (the unit of matching and eviction)

--des-kv-blocks

cap per-replica capacity in blocks; unset means unbounded

--des-instances

number of engine replicas behind the router

--des-overlap-weight

kv policy: cache locality versus load balance

Leave --des-kv-blocks unset to find the hit rate an infinite cache would give you — the ceiling. Set it to study eviction pressure, which is where the evictions= and block-reuse= fields in the report start moving and where a policy that looked best on an unbounded cache can lose.

A single engine#

Set --des-instances 1 to study one engine’s automatic prefix caching as temporal reuse across a stream, with no routing in the picture. This isolates “does my traffic reuse anything at all, given this cache size?” from “is my router sending reuse to the right place?”, and it is worth answering in that order.

Next steps#

Contents