GEAK pipeline#
2026-07-17
3 min read time
The e2e_workflow takes a running LLM server and a workload (ISL/OSL/concurrency) and raises its serving
throughput. Control flow is deterministic; each phase is owned by a specialized agent, and every accepted
change is gated on a measured end-to-end delta before it is kept. The single-kernel kernel_workflow runs
the same closed loop on one kernel and is called recursively when a kernel needs to be authored or tuned.
Inputs#
Model + serving backend: a model on disk served by
sglangorvllm(selected per run).Workload: input/output sequence length and concurrency (ISL/OSL/conc). Profiling and benchmarking use the same workload so numbers are comparable.
Phases#
The following phases make up an end-to-end run.
Setup#
The Director builds an isolated evaluation directory, launches a warm server, and records the **true baseline throughput and its noise band. Model weights and installed packages stay read-only.
Profile#
The Profiler traces the warm server under the real workload and produces one canonical, Amdahl-ranked hot-kernel list, separating the prefill and decode regimes.
Strategize#
The System Architect routes each hot kernel to a track by Amdahl leverage
(pct_gpu_time × achievable_speedup): config, head kernel, editable kernel, or host overhead.
ConfigSweep#
The Config Tuner tries service-level switches one at a time — attention backend, cuda/HIP graph, scheduling and memory knobs, backend toggles, and (when enabled) lower precision — keeping only changes that measurably help. This is the cheapest, landscape-reshaping lever, so it runs first; the server is re-profiled afterward.
HeadKernel#
For the heaviest GEMM/attention kernels: the Kernel Extractor builds a standalone, immutable unit test
from the real workload; the Op Benchmarker bakes off backends and tunes them; and the recursive
kernel_workflow authors/optimizes a kernel against that test. The e2e Integrator overlays the winner
back and gates it end-to-end.
Milestone#
The remaining editable kernels above a threshold are swept the same way — extract, optimize using the kernel layer, integrate, re-profile — until the budget or the Amdahl stop rule ends the loop.
Finalize, Report, Validate#
The Integrator assembles the deliverable (reversible overlay + patch + final_launch.sh); the Architect
writes final_report.md; and the Director independently re-measures the combined result against the
true baseline and arbitrates the official number. final_launch.sh is a self-contained script that
launches the server with all accepted overlays and configuration changes applied — run it in place of
your original launch command to use the optimized configuration.
Integration and gating#
Each accepted kernel is folded back into the live server through a reversible overlay (never editing installed packages). A change is accepted only when it actually runs live (engagement proof), clears the noise band under a tight back-to-back A/B, and preserves output quality (greedy parity, or a task accuracy gate for reduced-precision kernels).
Outputs#
Everything lands under exp/e2e_<model>_<timestamp>/: final_report.md, architect_report.md, the
final/ bundle (overlay, final_patch.diff, final_launch.sh), and per-stage artifacts.