Simulation runs

Simulation runs#

One-pager

What: the discrete-event simulator — requests arrive on their own schedule, queue, get batched, and finish. Why: it is the only way to get a p99, a queue wait, or a saturation verdict. Cost: seconds instead of milliseconds, and you have to say what the load is.

The analytical projector gives means at a fixed concurrency. Add an arrival model and the same cost kernel is driven through an event loop instead, producing distributions.

Arrival-driven simulation#

inferasim inference ... \
  --request-rate 1.5 --arrival-model poisson --des-num-requests 120

[inferasim:Inference] Discrete-Event Simulation (arrival-driven)
  Arrivals: poisson @ 1.5 req/s offered  (achieved 1.20 req/s, utilization 99%)
  Simulated: 120 requests over 100.20 s  → system throughput 1226 tok/s
  metric                        mean         p50         p90         p99
  TTFT (from admit)          31.22 ms     31.49 ms     32.48 ms     35.72 ms
    queue wait                4.59 ms      4.42 ms      7.85 ms     21.02 ms
    TTFT (from arrival)      35.80 ms     35.31 ms     39.89 ms     53.52 ms
  TPOT (per token)            9.11 ms      9.42 ms     10.25 ms     10.40 ms
  ITL (inter-token)           9.14 ms      9.30 ms     10.29 ms     29.94 ms
  End-to-end latency       9360.36 ms   9669.08 ms  10526.00 ms  10673.39 ms
  Batch packing: avg batch 10.5 (max 19) | avg prefill/decode reqs 0.0/10.5 | ...

The DES report is printed in addition to the analytical one, not instead of it, so you can see both descriptions of the same configuration.

Reading the report#

There are two TTFT rows, and the difference matters. From admit is the engine’s behaviour once the request is running; from arrival adds the queue wait and is what a client actually experiences. Under load these diverge sharply, and quoting the first as if it were the second is the classic way to make a saturated system look healthy.

[SATURATED] on the arrivals line means offered load exceeds capacity. The latencies then reflect an unbounded queue and are not meaningful as an operating point. Lower --request-rate below the reported max sustainable rate, or add replicas with --des-instances.

Batch packing shows what the scheduler actually assembled, which is how you tell a latency problem caused by batching from one caused by queueing.

Arrival options#

Flag

Meaning

--arrival-model

poisson, deterministic, or closed (no arrival stream — see closed loop)

--request-rate

offered load in requests per second

--des-num-requests

how many requests to simulate

--des-burstiness

makes arrivals gamma-distributed: 1.0 is Poisson, lower is burstier

--des-range-ratio

spreads per-request lengths around the configured input/output lengths

--des-sweep

sweep offered load and emit a throughput-versus-latency curve rather than a single point

--des-sweep is the one to reach for when the question is “what rate can this configuration take?” rather than “how does it behave at this rate”. It walks the load axis and shows you where the knee is.

Reproduce a fixed-concurrency benchmark#

A harness run with --max-concurrency C has no arrival stream: C clients each submit one request, block until it completes, then submit the next. --des-closed-loop simulates exactly that, which makes TTFT the difference between two simulated timestamps rather than a closed-form prefill time:

inferasim inference ... --max-concurrency 256 \
  --max-num-batched-tokens 8192 --chunked-prefill-size 256 \
  --des-closed-loop --des-num-requests 1024

This matters because the analytical path prices TTFT as prefill service time — how long the forward pass over the prompt takes — while a harness reports response time. Under a colocated engine a prefill chunk rides a scheduler step that is simultaneously carrying the resident decodes, so the step it waits on dilates with the decode batch. Measured TTFT is close to a constant number of steps across three decades of concurrency; a standalone-prefill model holds it constant in milliseconds and therefore reads progressively early as load rises. The step loop already mixes prefill chunks with decodes, so driving it from a closed load puts that dilation in the answer without a separate queue term.

Set –chunked-prefill-size

It is the engine’s per-request per-step prefill allowance (vLLM’s long_prefill_token_threshold) and it decides how many steps a prompt takes, so it is the one input this needs. Left unset, a prompt prefills in a single step and TTFT reads early for that reason instead. It is a declared input: read it off the server’s launch flags rather than fitting it, or the simulation just relocates the error it was meant to remove.

Concurrency is the load axis here, so there is no offered rate, no saturation flag — the population is bounded by the client count — and no --des-sweep. All C clients start together, which is what the harness does and why its mean TTFT carries an opening-burst transient: mean/p50 reaches 3–5.6x at high concurrency on real runs. Match --des-num-requests and the warmup you score against to whatever the harness reports over.

Next steps#

Contents