SGLang inference and serving on ROCm

SGLang is an open-source library for fast, memory-efficient LLM inference and serving. This page describes how to set up and run SGLang on AMD GPUs using either a prebuilt Docker image (recommended) or pip. It applies to supported AMD GPUs and platforms.

ROCm version

10.0.0

7.14.1

7.14.0

Device family

AMD Instinct™

AMD Radeon™

Instinct GPU

AMD Instinct MI355X (gfx950)AMD Instinct MI350X (gfx950)AMD Instinct MI350P (gfx950)AMD Instinct MI325X (gfx942)AMD Instinct MI300X (gfx942)AMD Instinct MI300A (gfx942)

Radeon GPU

AMD Radeon RX 9070 XT (gfx1201)AMD Radeon RX 9070 GRE (gfx1201)AMD Radeon RX 9070 (gfx1201)AMD Radeon RX 9060 XT LP (gfx1200)AMD Radeon RX 9060 XT (gfx1200)AMD Radeon RX 9060 (gfx1200)AMD Radeon RX 9050 (gfx1200)AMD Radeon RX 9050 (4GB) (gfx1200)AMD Radeon RX 7900 XTX (gfx1100)AMD Radeon RX 7900 XT (gfx1100)AMD Radeon RX 7900 GRE (gfx1100)AMD Radeon RX 7800 XT (gfx1101)AMD Radeon RX 7700 XT (gfx1101)AMD Radeon RX 7700 (gfx1101)AMD Radeon RX 7600 (gfx1102)AMD Radeon PRO W7900 Dual Slot (gfx1100)AMD Radeon PRO W7900 (gfx1100)AMD Radeon PRO W7800 48GB (gfx1100)AMD Radeon PRO W7800 (gfx1100)AMD Radeon PRO W7700 (gfx1101)AMD Radeon PRO V710 (gfx1101)AMD Radeon AI PRO R9700 (gfx1201)AMD Radeon AI PRO R9600D (gfx1201)AMD Radeon AI PRO R9600 (gfx1201)

SGLang version

0.5.15

0.5.13

Installation method

Docker

Prerequisites

• For Instinct and Radeon devices, ensure your host system has the AMD GPU Driver (amdgpu) installed. See the ROCm compatibility matrix (ROCm 10.0.0) for driver support information. For installation instructions, see the AMD GPU Driver documentation.

• For Instinct and Radeon devices, ensure your host system has the AMD GPU Driver (amdgpu) installed. See the ROCm compatibility matrix (ROCm 7.14.1) for driver support information. For installation instructions, see the AMD GPU Driver documentation.

• For Instinct and Radeon devices, ensure your host system has the AMD GPU Driver (amdgpu) installed. See the ROCm compatibility matrix (ROCm 7.14.0) for driver support information. For installation instructions, see the AMD GPU Driver documentation.

• Ensure the host system has Docker Engine installed. For more guidance on running ROCm workloads in Docker containers, see Run ROCm Docker containers.

Get started

1. Pull the ROCm SGLang 0.5.15 Docker image.

   docker pull rocm/sgl-dev:v0.5.15.post1-ubuntu24.04-py3.14-rocm10.0.0
   

2. Start the Docker container.

   docker run -it --rm \
      --device /dev/kfd \
      --device /dev/dri \
      --network=host \
      --ipc=host \
      --group-add=video \
      --cap-add=SYS_PTRACE \
      --security-opt seccomp=unconfined \
      -v <path/to/your/models>:/app/models \
      -e HF_HOME="/app/models" \
      rocm/sgl-dev:v0.5.15.post1-ubuntu24.04-py3.14-rocm10.0.0 \
      bash
   

1. Pull the ROCm SGLang 0.5.15 Docker image.

   docker pull rocm/sgl-dev:v0.5.15.post1-ubuntu24.04-py3.14-rocm10.0.0
   

2. Start the Docker container.

   docker run -it --rm \
      --device /dev/kfd \
      --device /dev/dri \
      --network=host \
      --ipc=host \
      --group-add=video \
      --cap-add=SYS_PTRACE \
      --security-opt seccomp=unconfined \
      -v <path/to/your/models>:/app/models \
      -e HF_HOME="/app/models" \
      rocm/sgl-dev:v0.5.15.post1-ubuntu24.04-py3.14-rocm10.0.0 \
      bash
   

1. Pull the ROCm SGLang 0.5.15 Docker image.

   docker pull rocm/sgl-dev:v0.5.15.post1-ubuntu24.04-py3.14-rocm10.0.0
   

2. Start the Docker container. On Radeon GPUs, disable AITER by unsetting SGLANG_USE_AITER and SGLANG_ROCM_FUSED_DECODE_MLA. See the known issue for more information.

   docker run -it --rm \
      --device /dev/kfd \
      --device /dev/dri \
      --network=host \
      --ipc=host \
      --group-add=video \
      --cap-add=SYS_PTRACE \
      --security-opt seccomp=unconfined \
      -v <path/to/your/models>:/app/models \
      -e HF_HOME="/app/models" \
      -e SGLANG_USE_AITER=false \
      -e SGLANG_ROCM_FUSED_DECODE_MLA=false \
      rocm/sgl-dev:v0.5.15.post1-ubuntu24.04-py3.14-rocm10.0.0 \
      bash
   

See also

Using Docker (SGLang docs)

Get started

1. Pull the ROCm SGLang 0.5.13post1 Docker image.

   docker pull rocm/sgl-dev:v0.5.13.post1-ubuntu24.04-py3.14-rocm7.14.1
   

2. Start the Docker container.

   docker run -it --rm \
      --device /dev/kfd \
      --device /dev/dri \
      --network=host \
      --ipc=host \
      --group-add=video \
      --cap-add=SYS_PTRACE \
      --security-opt seccomp=unconfined \
      -v <path/to/your/models>:/app/models \
      -e HF_HOME="/app/models" \
      rocm/sgl-dev:v0.5.13.post1-ubuntu24.04-py3.14-rocm7.14.1 \
      bash
   

1. Pull the ROCm SGLang 0.5.13post1 Docker image.

   docker pull rocm/sgl-dev:v0.5.13.post1-ubuntu24.04-py3.14-rocm7.14.1
   

2. Start the Docker container.

   docker run -it --rm \
      --device /dev/kfd \
      --device /dev/dri \
      --network=host \
      --ipc=host \
      --group-add=video \
      --cap-add=SYS_PTRACE \
      --security-opt seccomp=unconfined \
      -v <path/to/your/models>:/app/models \
      -e HF_HOME="/app/models" \
      rocm/sgl-dev:v0.5.13.post1-ubuntu24.04-py3.14-rocm7.14.1 \
      bash
   

1. Pull the ROCm SGLang 0.5.13post1 Docker image.

   docker pull rocm/sgl-dev:v0.5.13.post1-ubuntu24.04-py3.14-rocm7.14.1
   

2. Start the Docker container. On Radeon GPUs, disable AITER by unsetting SGLANG_USE_AITER and SGLANG_ROCM_FUSED_DECODE_MLA. See the known issue for more information.

   docker run -it --rm \
      --device /dev/kfd \
      --device /dev/dri \
      --network=host \
      --ipc=host \
      --group-add=video \
      --cap-add=SYS_PTRACE \
      --security-opt seccomp=unconfined \
      -v <path/to/your/models>:/app/models \
      -e HF_HOME="/app/models" \
      -e SGLANG_USE_AITER=false \
      -e SGLANG_ROCM_FUSED_DECODE_MLA=false \
      rocm/sgl-dev:v0.5.13.post1-ubuntu24.04-py3.14-rocm7.14.1 \
      bash
   

See also

Using Docker (SGLang docs)

Get started

1. Pull the ROCm SGLang 0.5.13post1 Docker image.

   docker pull rocm/sgl-dev:v0.5.13.post1-ubuntu24.04-py3.14-rocm7.14
   

2. Start the Docker container.

   docker run -it --rm \
      --device /dev/kfd \
      --device /dev/dri \
      --network=host \
      --ipc=host \
      --group-add=video \
      --cap-add=SYS_PTRACE \
      --security-opt seccomp=unconfined \
      -v <path/to/your/models>:/app/models \
      -e HF_HOME="/app/models" \
      rocm/sgl-dev:v0.5.13.post1-ubuntu24.04-py3.14-rocm7.14 \
      bash
   

1. Pull the ROCm SGLang 0.5.13post1 Docker image.

   docker pull rocm/sgl-dev:v0.5.13.post1-ubuntu24.04-py3.14-rocm7.14
   

2. Start the Docker container.

   docker run -it --rm \
      --device /dev/kfd \
      --device /dev/dri \
      --network=host \
      --ipc=host \
      --group-add=video \
      --cap-add=SYS_PTRACE \
      --security-opt seccomp=unconfined \
      -v <path/to/your/models>:/app/models \
      -e HF_HOME="/app/models" \
      rocm/sgl-dev:v0.5.13.post1-ubuntu24.04-py3.14-rocm7.14 \
      bash
   

1. Pull the ROCm SGLang 0.5.13post1 Docker image.

   docker pull rocm/sgl-dev:v0.5.13.post1-ubuntu24.04-py3.14-rocm7.14
   

2. Start the Docker container. On Radeon GPUs, disable AITER by unsetting SGLANG_USE_AITER and SGLANG_ROCM_FUSED_DECODE_MLA. See the known issue for more information.

   docker run -it --rm \
      --device /dev/kfd \
      --device /dev/dri \
      --network=host \
      --ipc=host \
      --group-add=video \
      --cap-add=SYS_PTRACE \
      --security-opt seccomp=unconfined \
      -v <path/to/your/models>:/app/models \
      -e HF_HOME="/app/models" \
      -e SGLANG_USE_AITER=false \
      -e SGLANG_ROCM_FUSED_DECODE_MLA=false \
      rocm/sgl-dev:v0.5.13.post1-ubuntu24.04-py3.14-rocm7.14 \
      bash
   

See also

Using Docker (SGLang docs)

Known issues

ROCm 7.14 introduces initial SGLang support for AMD Radeon GPUs. Radeon GPU users should disable AITER and unset SGLANG_ROCM_FUSED_DECODE_MLA, as both are enabled by default in the SGLang Docker image and may cause some workloads to fail. See the SGLang environment variables reference for more details.

export SGLANG_USE_AITER=false
export SGLANG_ROCM_FUSED_DECODE_MLA=false

Additionally, some models may not function correctly on Radeon GPUs, including certain Mixture-of-Experts (MoE) models (such as GPT-OSS-20B and MiniMax-M2.7) and Qwen3-ASR models. Users experiencing these issues are recommended to use the latest upstream SGLang versions, which will include the necessary fixes once they are merged.

• SGLang inference workloads using the default AITER attention backend might fail on some AMD Radeon graphics products, such as the Radeon PRO W7900, Radeon AI PRO R9700, and Radeon RX 9070 XT. As a workaround, configure SGLang to use the Triton attention backend (--attention-backend triton) or disable AITER:

  export SGLANG_USE_AITER=0
  export SGLANG_USE_AITER_AR=0