SGLang inference and serving on ROCm

SGLang is an open-source library for fast, memory-efficient LLM inference and serving. This page describes how to set up and run SGLang on AMD GPUs using either a prebuilt Docker image (recommended) or pip. It applies to supported AMD GPUs and platforms.

Device family

AMD Instinct™

AMD Radeon™

Instinct GPU

AMD Instinct MI355X (gfx950)AMD Instinct MI350X (gfx950)AMD Instinct MI350P (gfx950)AMD Instinct MI325X (gfx942)AMD Instinct MI300X (gfx942)AMD Instinct MI300A (gfx942)

Radeon GPU

AMD Radeon AI PRO R9700 (gfx1201)AMD Radeon AI PRO R9600D (gfx1201)AMD Radeon RX 9070 XT (gfx1201)AMD Radeon RX 9070 GRE (gfx1201)AMD Radeon RX 9070 (gfx1201)AMD Radeon RX 9060 XT LP (gfx1200)AMD Radeon RX 9060 XT (gfx1200)AMD Radeon RX 9060 (gfx1200)AMD Radeon PRO W7900 Dual Slot (gfx1100)AMD Radeon PRO W7900 (gfx1100)AMD Radeon PRO W7800 48GB (gfx1100)AMD Radeon PRO W7800 (gfx1100)AMD Radeon RX 7900 XTX (gfx1100)AMD Radeon RX 7900 XT (gfx1100)AMD Radeon RX 7900 GRE (gfx1100)AMD Radeon PRO W7700 (gfx1101)AMD Radeon RX 7800 XT (gfx1101)AMD Radeon RX 7700 XT (gfx1101)AMD Radeon RX 7700 (gfx1101)AMD Radeon PRO V710 (gfx1101)AMD Radeon RX 7600 (gfx1102)

SGLang version

0.15.3post1

Installation method

Docker

Prerequisites

• For Instinct and Radeon devices, ensure your host system has the AMD GPU Driver (amdgpu) installed. See the ROCm compatibility matrix for driver support information. For installation instructions, see the AMD GPU Driver documentation.

• Ensure the host system has Docker Engine installed.

Get started

1. Pull the ROCm SGLang 0.15.3post1 Docker image.

   docker pull rocm/sgl-dev:v0.5.13.post1-ubuntu24.04-py3.14-rocm7.14
   

2. Start the Docker container.

   docker run -it --rm \
      --device /dev/kfd \
      --device /dev/dri \
      --network=host \
      --ipc=host \
      --group-add=video \
      --cap-add=SYS_PTRACE \
      --security-opt seccomp=unconfined \
      -v <path/to/your/models>:/app/models \
      -e HF_HOME="/app/models" \
      rocm/sgl-dev:v0.5.13.post1-ubuntu24.04-py3.14-rocm7.14 \
      bash
   

1. Pull the ROCm SGLang 0.15.3post1 Docker image.

   docker pull rocm/sgl-dev:v0.5.13.post1-ubuntu24.04-py3.14-rocm7.14
   

2. Start the Docker container.

   docker run -it --rm \
      --device /dev/kfd \
      --device /dev/dri \
      --network=host \
      --ipc=host \
      --group-add=video \
      --cap-add=SYS_PTRACE \
      --security-opt seccomp=unconfined \
      -v <path/to/your/models>:/app/models \
      -e HF_HOME="/app/models" \
      rocm/sgl-dev:v0.5.13.post1-ubuntu24.04-py3.14-rocm7.14 \
      bash
   

1. Pull the ROCm SGLang 0.15.3post1 Docker image.

   docker pull rocm/sgl-dev:v0.5.13.post1-ubuntu24.04-py3.14-rocm7.14
   

2. Start the Docker container. On Radeon GPUs, disable AITER by unsetting SGLANG_USE_AITER and SGLANG_ROCM_FUSED_DECODE_MLA. See the known issue for more information.

   docker run -it --rm \
      --device /dev/kfd \
      --device /dev/dri \
      --network=host \
      --ipc=host \
      --group-add=video \
      --cap-add=SYS_PTRACE \
      --security-opt seccomp=unconfined \
      -v <path/to/your/models>:/app/models \
      -e HF_HOME="/app/models" \
      -e SGLANG_USE_AITER=false \
      -e SGLANG_ROCM_FUSED_DECODE_MLA=false \
      rocm/sgl-dev:v0.5.13.post1-ubuntu24.04-py3.14-rocm7.14 \
      bash
   

See also

Using Docker (SGLang docs)

3. After setting up your environment, follow the SGLang 0.15.3post1 usage documentation to get started: Basic usage (SGLang docs).

Known issues

ROCm 7.14 introduces initial SGLang support for AMD Radeon GPUs. Radeon GPU users should disable AITER and unset SGLANG_ROCM_FUSED_DECODE_MLA, as both are enabled by default in the SGLang Docker image and may cause some workloads to fail. See the SGLang environment variables reference for more details.

export SGLANG_USE_AITER=false
export SGLANG_ROCM_FUSED_DECODE_MLA=false

Additionally, some models may not function correctly on Radeon GPUs, including certain Mixture-of-Experts (MoE) models (such as GPT-OSS-20B and MiniMax-M2.7) and Qwen3-ASR models. Users experiencing these issues are recommended to use the latest upstream SGLang versions, which will include the necessary fixes once they are merged.