Docker images for AI inference

2025-09-18

2 min read time

Applies to Linux

Note

For the latest iteration of AI training and inference performance for ROCm 7.0, see Infinity Hub and the ROCm 7.0 AI training and inference performance documentation.

This page accompanies preview Docker images designed to reproduce inference performance on AMD Instinct™ MI355X, MI350X, and MI300X series accelerators. The images provide access to preview versions of the ROCm 7.0 software stack and are targeted at early-access users evaluating AI inference workloads using next-generation AMD accelerators.

Important

The following AI workload benchmarks use the ROCm 7.0 release candidate preview on AMD Instinct MI355X, MI350X, and MI300X series accelerators.

If you’re looking for production-level workloads for MI300X series accelerators, see Infinity Hub.

Inference

• Benchmark Llama 3.1 405B FP4 inference with vLLM

• Benchmark Llama 3.3 70B FP8 inference with vLLM

• Benchmark GPT OSS 120B inference with vLLM

• Benchmark DeepSeek R1 FP4 inference with SGLang

• Benchmark DeepSeek R1 FP8 inference with SGLang