Benchmark frameworks with Magpie

Benchmark frameworks with Magpie#

2026-07-20

4 min read time

Applies to Linux

Magpie’s benchmark mode runs end-to-end performance tests against LLM inference frameworks—vLLM, SGLang, and Atom—and collects throughput and latency metrics in a structured JSON report. Benchmarks can run inside a Docker container, directly on the host, or on a remote Ray cluster, and optionally capture torch profiler traces for deeper analysis with TraceLens and gap analysis.

TraceLens is an AMD tool for visualizing profiler traces; it installs automatically on first use, but can also be installed manually—see TraceLens installation if the auto-install fails.

Magpie uses InferenceX as its benchmarking backend; InferenceX is a collection of benchmark scripts for LLM inference frameworks and is cloned automatically on first run. Use this mode to measure inference performance on AMD Instinct™ GPUs and identify the GPU kernels that dominate runtime.

Review these topics for more information:

Quick start#

Before you begin#

  • Magpie installed: see Install Magpie.

  • Docker installed and running: benchmark mode defaults to run_mode: docker. Verify with docker info.

  • ROCm-compatible GPU with sufficient VRAM: the example configs target AMD Instinct™ GPUs (MI300X/MI355X). DeepSeek-R1 requires 8 GPUs at fp8; smaller models need less. Magpie selects idle GPUs automatically.

  • HuggingFace token: required for gated models. Set HF_TOKEN in your environment before running.

  • InferenceX: cloned automatically on first run; no manual install needed.

Commands#

The following commands cover the most common benchmark invocations. Examples use python -m Magpie; the magpie CLI entrypoint is equivalent—see Install Magpie for details.

# Basic vLLM benchmark (paths are under examples/benchmarks/)
python -m Magpie benchmark --benchmark-config examples/benchmarks/benchmark_vllm_dsr1.yaml

# vLLM with TraceLens analysis
python -m Magpie benchmark --benchmark-config examples/benchmarks/benchmark_vllm_tracelens.yaml

# vLLM with gap analysis (kernel bottleneck report)
python -m Magpie benchmark --benchmark-config examples/benchmarks/benchmark_vllm_kimi_k2.yaml

# Standalone gap analysis on existing traces
python -m Magpie benchmark --trace-dir results/benchmark_vllm_<timestamp>/

# SGLang benchmark
python -m Magpie benchmark --benchmark-config examples/benchmarks/benchmark_sglang_dsr1.yaml

# Ad-hoc CLI without a YAML file (framework + model; optional torch profiler)
python -m Magpie benchmark vllm --model deepseek-ai/DeepSeek-R1-0528 --torch-profiler

Output structure#

A successful benchmark run creates a timestamped workspace directory with the following layout.

results/benchmark_vllm_<timestamp>/
├── benchmark_report.json      # Main benchmark results
├── summary.txt                # Human-readable summary
├── config.yaml                # Snapshot of benchmark configuration
├── container_stdout.log       # Container stdout
├── container_stderr.log       # Container stderr
├── inferencex_result.json   # Raw InferenceX output
├── torch_trace/               # Raw torch profiler traces
│   ├── *-rank-0.*.pt.trace.json.gz
│   ├── *-rank-1.*.pt.trace.json.gz
│   └── ...
├── gap_analysis/              # Gap analysis output (if enabled)
│   ├── gap_analysis.csv       # Merged kernel stats across all ranks
│   ├── gap_analysis_rank0.csv # Per-rank kernel stats
│   ├── gap_analysis_rank1.csv
│   └── ...
├── tracelens/                 # TraceLens inference reports (if enabled)
│   ├── prefilldecode/         # Full TraceLens CSVs for mixed prefill+decode
│   ├── decode_only/           # Full TraceLens CSVs for decode
│   ├── prefill_only/          # Full TraceLens CSVs for prefill, when available
│   ├── prefilldecode_ISL1024_OSL1024_CONC64_kernel_roofline_simple.csv
│   ├── decode_only_ISL1024_OSL1024_CONC64_kernel_roofline_simple.csv
│   └── prefill_only_ISL1024_OSL1024_CONC64_kernel_roofline_simple.csv
├── tracelens_rank0_csvs/      # Legacy/direct PyTorch single-rank report
└── tracelens_collective_csvs/ # Legacy/direct PyTorch multi-rank collective report

For TraceLens inference runs, the *_ISL*_OSL*_CONC*_kernel_roofline_simple.csv files are the fastest starting point for review. Each file covers one inference stage, encodes the benchmark ISL, OSL, and CONC values in the filename, keeps the most important roofline and timing columns, and includes both a compact param_signature and machine-readable params_json for matched TraceLens param:* metadata.

Benchmark report#

The primary summary file is benchmark_report.json, written to the run workspace directory. It aggregates throughput, latency, and optional gap_analysis and tracelens_analysis sections. A typical shape (abbreviated, with ... marking elided values):

{
  "success": true,
  "framework": "vllm",
  "model": "amd/Kimi-K2-Thinking-MXFP4",
  "throughput": {
    "request_throughput": 0.16,
    "output_throughput": 1.13,
    "total_token_throughput": 1192.76,
    "completed_requests": 40
  },
  "latency": {
    "ttft": { "mean_ms": 1185.44, "p99_ms": 1969.59 },
    "tpot": { "mean_ms": 131.09, "p99_ms": 282.21 }
  },
  "gap_analysis": {
    "config": { "trace_start_pct": 50, "trace_end_pct": 80, "categories": ["kernel", "gpu"] },
    "csv_path": "results/.../gap_analysis/gap_analysis.csv",
    "top_kernels": [
      { "name": "rcclGenericKernel<...>", "calls": 19620, "self_cuda_total_us": 28999961.95, "pct_total": 44.0 },
      { "name": "kernel_moe_mxgemm_2lds<...>", "calls": 9360, "self_cuda_total_us": 12495324.68, "pct_total": 18.9 }
    ]
  },
  "tracelens_analysis": { "output_files": [...] }
}
Contents