KV-Cache Management (kvd)

KV-Cache Management (kvd)#

One-pager

The design of infera.kvd — the per-host daemon that builds a tiered KV cache below the engine’s GPU cache and offloads / onboards blocks across GPU HBM → host RAM → NVMe → network. This page is the mechanism: the tiers, the zero-copy transport, the on-disk format, and how it’s tuned. For how to turn it on and the usage modes, see KV-Cache Offload.

Higher tiers are faster and smaller; a miss falls through to the next tier down. L1 is the engine’s own GPU cache; L2–L4 are kvd’s. The long tier is one --long-backend — local tablespace (L3) or a distributed backend (L4), never both.

The daemon owns L2 (the RAM arena) and, with GPU-direct off, also does the L3 disk I/O via its tablespace writer. With GPU-direct on (hipFile/AIS) the engine reads/writes the L3 chunk files directly (DMA straight into GPU memory); those files are connector-owned, so the daemon’s --long-bytes doesn’t bound them — the connector caps that tier via INFERA_KVD_L3_BUDGET_BYTES / INFERA_KVD_L3_FREE_FLOOR.

Why a separate daemon#

The engine’s own KV cache (L1) lives in GPU memory: it’s fast, but small, not shared between engines, and gone on restart. kvd adds the tiers below it:

Tier	Storage	Managed by	Restart-durable	Shared
L1	GPU HBM	the engine (vLLM/SGLang)	No	No
L2	host pinned RAM (memfd shared arena)	kvd	No	cross-engine, one host, zero-copy
L3	local NVMe / NFS / WekaFS	kvd	Yes	via the shared mount
L4	distributed (Mooncake / Redis / S3 / NIXL / WEKA)	kvd	Yes	cross-node

What fires when — a GET#

A SET flows the other way: new blocks land in L2; eviction from L2 writes back to the long tier (L3/L4), unless the request marked them ephemeral. These offloads (and the load-backs) are asynchronous — they run off the token-generation path, so spilling a block to disk never stalls decode, and kvd batches the writes for throughput.

The zero-copy win (L2)#

The RAM tier is a memfd-backed pinned shared arena. When a vLLM/SGLang worker opts in at the UDS handshake, kvd passes the arena’s file descriptor over SCM_RIGHTS and the engine mmaps the same region. A get() then returns just (slot_offset, length, version) — the engine reads bytes straight from its own mmap. No socket body, no bytes(...) copy. The gain vs the legacy inline-bytes path is largest on NFS-backed L3 (prefetch warms the arena; the bytes path can’t benefit).

L3 on-disk design — the tablespace#

The L3 disk tier never writes one OS file per cached block. Instead it pre-allocates a small, fixed set of large “container” files and packs blocks into fixed-size slots inside them — the same pattern databases use for a tablespace (and 3FS for its chunk engine). The file count is a constant (max_bytes / container_bytes), so a 64 GB region with 1 GB containers is 64 files no matter how many blocks it holds — which is what keeps filesystem metadata (inode lookup, dentry cache, the restart scan) from becoming the bottleneck the way one-file-per-block did past ~100K entries, especially on NFS.

Slots. Each container is carved into fixed --tablespace-slot-bytes slots (default 64 KB, sized to the typical packed-KV block). A value larger than one slot is rejected (the caller recomputes) — slots are not chained.
Allocator. A bitset in RAM tracks free slots; allocation and the key → (container, slot) index are in-memory. Disk durability is the journal, not the bitset.

Write path & crash safety#

Every PUT is slot-reserve → write+fsync data → append+fsync journal, so a crash can never surface a half-written block:

A crash between steps 2 and 3 leaves a slot written on disk but with no journal record — on restart it’s simply seen as free and overwritten later, never served as a phantom block. Steps 3 and 4 are atomic from a restart’s point of view (step 4 is RAM-only). On restart kvd loads index.snapshot.json (written atomically on graceful shutdown / periodic compaction), then replays any journal entries appended since the snapshot — so both a clean shutdown and a hard kill recover the full index. The journal is plain JSON on purpose: cat index.log shows the operation history, and a parse failure is obvious (vs. an opaque embedded-DB corruption).

Storage-aware auto-tuning#

kvd inspects the backing device at startup so you don’t hand-tune per storage class:

--io-mode auto (default) — O_DIRECT on NVMe, buffered on NFS/SATA.
IO concurrency is selected internally by device class (no CLI flag); the L3 writer is plain pwrite/O_DIRECT — no extra dependencies to build or ship. For the striped region you can cap intra-shard parallelism with --long-workers-per-shard (default 8).
Low NFS wsize → a startup WARN with the exact remount command.

Sizing principles#

L2 (--max-bytes) — as much host RAM as you can spare for KV; this is the hottest, zero-copy tier. On a big-RAM host (~3 TB/host on this fleet) L2 can be large.
L3 (--long-bytes) — the working-set’s durable footprint; sized to hold the prefixes you want to survive restarts.
Grow capacity as you go down — each tier should be ≥ the one above (L3 ≥ L2 ≥ the engine’s GPU L1). A lower tier smaller than the one above it just churns: blocks evict before they’re ever reused.

Verify it’s running#

After starting the daemon and at least one engine, send the same prompt twice (or any prompt that shares a long prefix), then check the daemon counters:

python -m infera.kvd.statctl --socket /var/run/infera-kvd.sock
# healthy output: sets_total > 0, gets_total > 0, hits_total > 0

hits_total == 0 after a repeated prompt usually means the engine isn’t connected to the daemon — confirm INFERA_KVD_SOCKET is set and points to the same socket path. Under GPU-direct the daemon counters stay 0 by design (the connector owns the files directly); use the engine’s External prefix cache hit rate log line and check for .kvcache files under INFERA_KVD_HIPFILE_ROOTS instead.