Sweeps and tuning

Sweeps and tuning#

One-pager

What: two ways to search the configuration space instead of projecting one point at a time — a scripted grid, or an LLM-driven search. Why: the whole reason to make a projection cheap is to run a lot of them. Cost: zero GPUs on both paths.

Sweep the configuration space#

From Python, for scripted searches:

from infera.projection.core.projection.inference_projection.sweep import sweep

res = sweep(
    "gpt_oss_120B",
    tp=[1, 2, 4, 8], ep=[1, 2, 4, 8], pp=[1],
    concurrency=[1, 8, 32, 128],
    isl=1024, osl=1024,
    gpu_arch="mi355x", hbm_gb=288.0,
    valid=lambda tp, ep, pp: ep <= tp,      # your own legality rules
)

for p in res.points:
    if p.feasible:
        print(p.tp, p.ep, p.concurrency, round(p.ttft_ms, 1),
              round(p.tpot_ms, 2), round(p.decode_tps_per_gpu, 1))

Sweeps force --profiling-mode simulate, so they need zero GPUs — a search that measured every point would not be a search. This is the one place the uncalibrated path is the default rather than the opt-in, and it is why a sweep produces a ranking to confirm rather than numbers to quote. Rank in the sweep, then re-run the top few against an anchor.

Infeasible points are kept, annotated with p.reason, rather than dropped. This is deliberate: a missing point is ambiguous between “did not fit” and “was never tried”, and those call for different next actions. Pass workload= to project against your own experiment config instead of the packaged default.

Rank on decode_tps_per_gpu rather than aggregate throughput unless every point in the sweep has the same GPU count — otherwise the sweep just rediscovers that more GPUs are faster.

Search with the tuning agent#

When you would rather describe the space than enumerate it:

inferasim-tune --inference --workload <workload.yaml> --target-cluster <cluster.yaml>

Two stages: a deterministic seed sweep for a warm start, then an LLM-driven search that continues from the incumbent, proposing recipes and scoring them through the projector. It searches 36 serving levers against any of 24 projected metrics, and — the part that makes it worth using over a grid — it enforces latency SLOs as hard constraints, so max_throughput returns the fastest config that still keeps the promise instead of the largest batch that fits.

Note the --inference flag: the agent tunes training configurations by default.

Full documentation, including the objective list and the SLO block, is on Tuning agent.

Draw frontiers and pick a deployment mix#

inferasim pareto projects every combination of the engines, precisions, layouts, concurrencies and speculative settings you list, and writes the throughput-per-GPU versus interactivity frontier per model and scenario:

inferasim pareto --model deepseek_v3 --engine sglang --precision fp8 \
  --parallelism tp8 tp8ep8 tp8ep8dp8 --mode simulate \
  --spec none mtp:golden --concurrency 1 4 16 64 256 \
  --scenario 1k/1k 8k/1k --out-dir pareto_out

Open pareto_out/pareto.html for the charts. Every point is also in pareto_out/pareto_points.json.

A frontier still leaves the operator to choose a point. inferasim mix makes that choice per latency tier, one configuration per model and scenario:

inferasim mix pareto_out

Tier

Interactivity floor

TTFT limit

realtime

150 tok/s/user

1 s

interactive

70 tok/s/user

2.5 s

responsive

25 tok/s/user

10 s

bulk

none

none

A point qualifies if it meets the floor and its projected TTFT is at most half the limit (--ttft-margin 0.5); if nothing qualifies, the plain limit is used and the table says so. Among qualifying points within 5% of the best projected throughput per GPU (--tie 0.05), the one with the most SLO headroom wins. A configuration projected in both modes is taken from its anchored point, and simulate-mode DCP points are left out. Pass --tiers tiers.yaml for your own tiers (name, min_interactivity, max_ttft_ms). The picks go to mix.json and mix.csv next to the points.

Treat the mix as a shortlist: confirm each pick with one measured run before you deploy it. --top 5 lists the pick and the four next-best points per tier to confirm. Points after the pick need only meet the plain TTFT limit, and the table flags the ones that miss the margin.

inferasim mix pareto_out --top 5

Then spend the GPU hours#

A sweep’s output is a shortlist, not a decision. The intended loop is to search in simulation, confirm the top few candidates on hardware with the benchmarking path, and — if the confirmation disagrees — harvest an anchor in that regime so the next search starts from a calibrated kernel.

Next steps#

Contents