vLLM inference and serving on ROCm#
vLLM is an open-source library for fast, memory-efficient LLM inference and serving. This page describes how to set up and run vLLM on AMD GPUs and APUs using either a prebuilt Docker image (recommended) or pip. It applies to supported AMD GPUs and platforms.
Prerequisites#
For Instinct and Radeon devices, ensure your host system has the AMD GPU Driver (amdgpu) installed. See the ROCm compatibility matrix for driver support information. For installation instructions, see the AMD GPU Driver documentation.
Ensure the host system has Docker Engine installed.
Ensure the host system has Docker Engine installed.
For Instinct and Radeon devices, ensure your system has the AMD GPU Driver (amdgpu) installed. See the ROCm compatibility matrix for driver support information. For installation instructions, see the AMD GPU Driver documentation.
Ensure your system has Python 3.14 installed and accessible.
Install uv.
Note
It’s recommended to use uv to install the vLLM wheel. vLLM has many transitive dependencies, and pip may silently pull incompatible versions from PyPI when installing from a direct wheel URL.
uv pipresolves dependencies more predictably, respecting the exact versions bundled with or required by the wheel.
Get started#
Pull the ROCm vLLM 0.23.0 Docker image.
docker pull docker pull rocm/vllm:rocm7.14.0_cdna_ubuntu24.04_py3.14_pytorch_2.11.0_vllm_0.23.0
Start the Docker container.
docker run -it --rm \ --device /dev/kfd \ --device /dev/dri \ --network=host \ --ipc=host \ --group-add=video \ --cap-add=SYS_PTRACE \ --security-opt seccomp=unconfined \ -v <path/to/your/models>:/app/models \ -e HF_HOME="/app/models" \ docker pull rocm/vllm:rocm7.14.0_cdna_ubuntu24.04_py3.14_pytorch_2.11.0_vllm_0.23.0 \ bash
Pull the ROCm vLLM 0.23.0 Docker image.
docker pull rocm/vllm:rocm7.14.0_rdna_ubuntu24.04_py3.14_pytorch_2.11.0_vllm_0.23.0
Start the Docker container.
docker run -it --rm \ --device /dev/kfd \ --device /dev/dri \ --network=host \ --ipc=host \ --group-add=video \ --cap-add=SYS_PTRACE \ --security-opt seccomp=unconfined \ -v <path/to/your/models>:/app/models \ -e HF_HOME="/app/models" \ rocm/vllm:rocm7.14.0_rdna_ubuntu24.04_py3.14_pytorch_2.11.0_vllm_0.23.0 \ bash
See also
After setting up your environment, follow the vLLM 0.23.0 usage documentation to get started: Using vLLM.
Install vLLM using pip#
Set up your Python virtual environment.
python3.14 -m venv .venv
Activate your Python virtual environment.
source .venv/bin/activate
Install PyTorch 2.11 in your virtual environment. This should also install the ROCm core libraries as a dependency.
python -m pip install --index-url https://repo.amd.com/rocm/whl-multi-arch/ \ "torch[device-gfx950]==2.11.0+rocm7.14.0" \ "torchvision[device-gfx950]==0.26.0+rocm7.14.0" \ "torchaudio==2.11.0+rocm7.14.0"
python -m pip install --index-url https://repo.amd.com/rocm/whl-multi-arch/ \ "torch[device-gfx942]==2.11.0+rocm7.14.0" \ "torchvision[device-gfx942]==0.26.0+rocm7.14.0" \ "torchaudio==2.11.0+rocm7.14.0"
python -m pip install --index-url https://repo.amd.com/rocm/whl-multi-arch/ \ "torch[device-gfx1200]==2.11.0+rocm7.14.0" \ "torchvision[device-gfx1200]==0.26.0+rocm7.14.0" \ "torchaudio==2.11.0+rocm7.14.0"
python -m pip install --index-url https://repo.amd.com/rocm/whl-multi-arch/ \ "torch[device-gfx1201]==2.11.0+rocm7.14.0" \ "torchvision[device-gfx1201]==0.26.0+rocm7.14.0" \ "torchaudio==2.11.0+rocm7.14.0"
python -m pip install --index-url https://repo.amd.com/rocm/whl-multi-arch/ \ "torch[device-gfx1100]==2.11.0+rocm7.14.0" \ "torchvision[device-gfx1100]==0.26.0+rocm7.14.0" \ "torchaudio==2.11.0+rocm7.14.0"
python -m pip install --index-url https://repo.amd.com/rocm/whl-multi-arch/ \ "torch[device-gfx1101]==2.11.0+rocm7.14.0" \ "torchvision[device-gfx1101]==0.26.0+rocm7.14.0" \ "torchaudio==2.11.0+rocm7.14.0"
python -m pip install --index-url https://repo.amd.com/rocm/whl-multi-arch/ \ "torch[device-gfx1102]==2.11.0+rocm7.14.0" \ "torchvision[device-gfx1102]==0.26.0+rocm7.14.0" \ "torchaudio==2.11.0+rocm7.14.0"
python -m pip install --index-url https://repo.amd.com/rocm/whl-multi-arch/ \ "torch[device-gfx1103]==2.11.0+rocm7.14.0" \ "torchvision[device-gfx1103]==0.26.0+rocm7.14.0" \ "torchaudio==2.11.0+rocm7.14.0"
python -m pip install --index-url https://repo.amd.com/rocm/whl-multi-arch/ \ "torch[device-gfx1151]==2.11.0+rocm7.14.0" \ "torchvision[device-gfx1151]==0.26.0+rocm7.14.0" \ "torchaudio==2.11.0+rocm7.14.0"
python -m pip install --index-url https://repo.amd.com/rocm/whl-multi-arch/ \ "torch[device-gfx1150]==2.11.0+rocm7.14.0" \ "torchvision[device-gfx1150]==0.26.0+rocm7.14.0" \ "torchaudio==2.11.0+rocm7.14.0"
python -m pip install --index-url https://repo.amd.com/rocm/whl-multi-arch/ \ "torch[device-gfx1152]==2.11.0+rocm7.14.0" \ "torchvision[device-gfx1152]==0.26.0+rocm7.14.0" \ "torchaudio==2.11.0+rocm7.14.0"
Install Flash Attention.
python -m pip install https://rocm.frameworks.amd.com/whl-multi-arch/vllm-cdna/flash-attn/flash_attn-2.8.3-cp314-cp314-linux_x86_64.whl
python -m pip install https://rocm.frameworks.amd.com/whl-multi-arch/vllm-rdna/flash-attn/flash_attn-2.8.3-py3-none-any.whl
Install AITER.
python -m pip install https://rocm.frameworks.amd.com/whl-multi-arch/vllm-cdna/amd-aiter/amd_aiter-0.1.13.post2.dev1%2Bgb32deb267-cp314-cp314-linux_x86_64.whl
Install the vLLM 0.23.0 wheel using
uv pip.uv pip install https://rocm.frameworks.amd.com/whl-multi-arch/vllm-cdna/vllm/vllm-0.23.1.dev1%2Brocm7.14.0.g9ddef7117.d20260715-cp314-cp314-linux_x86_64.whl
Set the following environment variables to prevent errors related to ROCm platform and Flash Attention availability when running vLLM.
export PYTHONPATH=$VIRTUAL_ENV/lib/python3.14/site-packages/_rocm_sdk_core/share/amd_smi export FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE
Check your installation.
python -c "import vllm; print('vLLM version:', vllm.__version__)" python -c "import torch; print('PyTorch:', torch.__version__); print('HIP available:', torch.cuda.is_available()); print('HIP built:', torch.backends.hip.is_built() if hasattr(torch.backends, 'hip') else 'N/A')" python -c "import flash_attn; print('flash-attn:', flash_attn.__version__)"
After setting up your environment, follow the vLLM 0.23.0 usage documentation to get started: Using vLLM.
Install the vLLM 0.23.0 wheel using
uv pip.uv pip install https://rocm.frameworks.amd.com/whl-multi-arch/vllm-rdna/vllm/vllm-0.23.1%2Brocm7.14.0-cp314-cp314-linux_x86_64.whl
Set the following environment variables to prevent errors related to ROCm platform and Flash Attention availability when running vLLM.
export PYTHONPATH=$VIRTUAL_ENV/lib/python3.14/site-packages/_rocm_sdk_core/share/amd_smi export FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE
Check your installation.
python -c "import vllm; print('vLLM version:', vllm.__version__)" python -c "import torch; print('PyTorch:', torch.__version__); print('HIP available:', torch.cuda.is_available()); print('HIP built:', torch.backends.hip.is_built() if hasattr(torch.backends, 'hip') else 'N/A')" python -c "import flash_attn; print('flash-attn:', flash_attn.__version__)"
After setting up your environment, follow the vLLM 0.23.0 usage documentation to get started: Using vLLM.
See also
Known issues#
Significantly longer warmup times might be observed in some large language model inference workloads on AMD Radeon GPUs using vLLM versions v0.21.0 through v0.25.0. As a workaround, use a vLLM release earlier than v0.21.0 or upgrade to vLLM v0.26.0 or later, which includes a fix for this issue.