vLLM inference and serving on ROCm#
vLLM is an open-source library for fast, memory-efficient LLM inference and serving. This page describes how to set up and run vLLM on AMD GPUs and APUs using either a prebuilt Docker image (recommended) or pip. It applies to supported AMD GPUs and platforms.
Prerequisites#
Ensure your host system has the AMD GPU Driver (amdgpu) installed. See the ROCm compatibility matrix (ROCm 10.2.0) for driver support information. For installation instructions, see the AMD GPU Driver documentation.
Ensure your host system has the AMD GPU Driver (amdgpu) installed. See the ROCm compatibility matrix (ROCm 10.1.0) for driver support information. For installation instructions, see the AMD GPU Driver documentation.
Ensure your host system has the AMD GPU Driver (amdgpu) installed. See the ROCm compatibility matrix (ROCm 10.0.0) for driver support information. For installation instructions, see the AMD GPU Driver documentation.
Ensure your host system has the AMD GPU Driver (amdgpu) installed. See the ROCm compatibility matrix (ROCm 7.14.1) for driver support information. For installation instructions, see the AMD GPU Driver documentation.
Ensure your host system has the AMD GPU Driver (amdgpu) installed. See the ROCm compatibility matrix (ROCm 7.14.0) for driver support information. For installation instructions, see the AMD GPU Driver documentation.
Ensure the host system has Docker Engine installed. For more guidance on running ROCm workloads in Docker containers, see Run ROCm Docker containers.
Ensure the host system has Docker Engine installed. For more guidance on running ROCm workloads in Docker containers, see Run ROCm Docker containers.
Ensure your host system has the AMD GPU Driver (amdgpu) installed. See the ROCm compatibility matrix (ROCm 10.2.0) for driver support information. For installation instructions, see the AMD GPU Driver documentation.
Ensure your host system has the AMD GPU Driver (amdgpu) installed. See the ROCm compatibility matrix (ROCm 10.1.0) for driver support information. For installation instructions, see the AMD GPU Driver documentation.
Ensure your host system has the AMD GPU Driver (amdgpu) installed. See the ROCm compatibility matrix (ROCm 10.0.0) for driver support information. For installation instructions, see the AMD GPU Driver documentation.
Ensure your host system has the AMD GPU Driver (amdgpu) installed. See the ROCm compatibility matrix (ROCm 7.14.1) for driver support information. For installation instructions, see the AMD GPU Driver documentation.
Ensure your host system has the AMD GPU Driver (amdgpu) installed. See the ROCm compatibility matrix (ROCm 7.14.0) for driver support information. For installation instructions, see the AMD GPU Driver documentation.
Ensure your system has Python 3.14 and its development package are installed and accessible, along with
libgomp.sudo apt update && sudo apt install python3.14-dev libgomp1
sudo dnf install python3.14-devel libgomp
sudo zypper install python3-devel libgomp1
If your system has a separate ROCm installation and you set the
ROCM_PATHorLD_LIBRARY_PATHenvironment variables, for example after installing ROCm via tarball, unset them before running vLLM as they can interfere with vLLM’s ROCm platform detection.unset ROCM_PATH LD_LIBRARY_PATH
If you don’t need the tarball installation, you can remove it instead. See Uninstalling ROCm, which removes the installation directory and its environment configuration. Step 2 of this guide installs ROCm core libraries using pip alongside PyTorch.
Get started#
Pull the ROCm vLLM 0.32 Docker image.
docker pull rocm/vllm:rocm10.2.0_ubuntu24.04_py3.14-pytorch_2.14.0_vllm-0.32.0
Start the Docker container.
docker run -it --rm \ --device /dev/kfd \ --device /dev/dri \ --network=host \ --ipc=host \ --group-add=video \ --cap-add=SYS_PTRACE \ --security-opt seccomp=unconfined \ -v <path/to/your/models>:/app/models \ -e HF_HOME="/app/models" \ rocm/vllm:rocm10.2.0_ubuntu24.04_py3.14-pytorch_2.14.0_vllm-0.32.0 \ bash
After setting up your environment, follow the vLLM 0.32 usage documentation to get started: Using vLLM.
Get started#
Pull the ROCm vLLM 0.29 Docker image.
docker pull rocm/vllm:rocm10.1.0_ubuntu24.04_py3.14-pytorch_2.13.0_vllm-0.29.0
Start the Docker container.
docker run -it --rm \ --device /dev/kfd \ --device /dev/dri \ --network=host \ --ipc=host \ --group-add=video \ --cap-add=SYS_PTRACE \ --security-opt seccomp=unconfined \ -v <path/to/your/models>:/app/models \ -e HF_HOME="/app/models" \ rocm/vllm:rocm10.1.0_ubuntu24.04_py3.14-pytorch_2.13.0_vllm-0.29.0 \ bash
After setting up your environment, follow the vLLM 0.29 usage documentation to get started: Using vLLM.
Get started#
Pull the ROCm vLLM 0.27 Docker image.
docker pull rocm/vllm:rocm10.0.0_ubuntu24.04_py3.14_pytorch_2.12.0_vllm_0.27.0
Start the Docker container.
docker run -it --rm \ --device /dev/kfd \ --device /dev/dri \ --network=host \ --ipc=host \ --group-add=video \ --cap-add=SYS_PTRACE \ --security-opt seccomp=unconfined \ -v <path/to/your/models>:/app/models \ -e HF_HOME="/app/models" \ rocm/vllm:rocm10.0.0_ubuntu24.04_py3.14_pytorch_2.12.0_vllm_0.27.0 \ bash
After setting up your environment, follow the vLLM 0.27 usage documentation to get started: Using vLLM.
Get started#
Pull the ROCm vLLM 0.23 Docker image.
docker pull rocm/vllm:rocm7.14.1_cdna_ubuntu24.04_py3.14_pytorch_2.11_vllm_0.23.0
Start the Docker container.
docker run -it --rm \ --device /dev/kfd \ --device /dev/dri \ --network=host \ --ipc=host \ --group-add=video \ --cap-add=SYS_PTRACE \ --security-opt seccomp=unconfined \ -v <path/to/your/models>:/app/models \ -e HF_HOME="/app/models" \ rocm/vllm:rocm7.14.1_cdna_ubuntu24.04_py3.14_pytorch_2.11_vllm_0.23.0 \ bash
Pull the ROCm vLLM 0.23 Docker image.
docker pull rocm/vllm:rocm7.14.1_rdna_ubuntu24.04_py3.14_pytorch_2.11_vllm_0.23.0
Start the Docker container.
docker run -it --rm \ --device /dev/kfd \ --device /dev/dri \ --network=host \ --ipc=host \ --group-add=video \ --cap-add=SYS_PTRACE \ --security-opt seccomp=unconfined \ -v <path/to/your/models>:/app/models \ -e HF_HOME="/app/models" \ rocm/vllm:rocm7.14.1_rdna_ubuntu24.04_py3.14_pytorch_2.11_vllm_0.23.0 \ bash
After setting up your environment, follow the vLLM 0.23 usage documentation to get started: Using vLLM.
Get started#
Pull the ROCm vLLM 0.23 Docker image.
docker pull rocm/vllm:rocm7.14.0_cdna_ubuntu24.04_py3.14_pytorch_2.11.0_vllm_0.23.0
Start the Docker container.
docker run -it --rm \ --device /dev/kfd \ --device /dev/dri \ --network=host \ --ipc=host \ --group-add=video \ --cap-add=SYS_PTRACE \ --security-opt seccomp=unconfined \ -v <path/to/your/models>:/app/models \ -e HF_HOME="/app/models" \ rocm/vllm:rocm7.14.0_cdna_ubuntu24.04_py3.14_pytorch_2.11.0_vllm_0.23.0 \ bash
Pull the ROCm vLLM 0.23 Docker image.
docker pull rocm/vllm:rocm7.14.0_rdna_ubuntu24.04_py3.14_pytorch_2.11.0_vllm_0.23.0
Start the Docker container.
docker run -it --rm \ --device /dev/kfd \ --device /dev/dri \ --network=host \ --ipc=host \ --group-add=video \ --cap-add=SYS_PTRACE \ --security-opt seccomp=unconfined \ -v <path/to/your/models>:/app/models \ -e HF_HOME="/app/models" \ rocm/vllm:rocm7.14.0_rdna_ubuntu24.04_py3.14_pytorch_2.11.0_vllm_0.23.0 \ bash
After setting up your environment, follow the vLLM 0.23 usage documentation to get started: Using vLLM.
Install vLLM using pip#
Set up and activate your Python virtual environment.
python3.14 -m venv .venv source .venv/bin/activate
Install PyTorch 2.14.0 in your virtual environment. This should also install the ROCm core libraries as a dependency. See Install PyTorch for ROCm for full instructions; make sure to select PyTorch version 2.14.0 on that page.
python -m pip install --index-url https://stable.repo.amd.com/rocm/whl-next/ \ "torch[device-gfx950]==2.14.0+rocm10.2.0" \ "torchvision[device-gfx950]==0.29.0+rocm10.2.0" \ "torchaudio==2.11.0.3+rocm10.2.0"
python -m pip install --index-url https://stable.repo.amd.com/rocm/whl-next/ \ "torch[device-gfx942]==2.14.0+rocm10.2.0" \ "torchvision[device-gfx942]==0.29.0+rocm10.2.0" \ "torchaudio==2.11.0.3+rocm10.2.0"
python -m pip install --index-url https://stable.repo.amd.com/rocm/whl-next/ \ "torch[device-gfx1200]==2.14.0+rocm10.2.0" \ "torchvision[device-gfx1200]==0.29.0+rocm10.2.0" \ "torchaudio==2.11.0.3+rocm10.2.0"
python -m pip install --index-url https://stable.repo.amd.com/rocm/whl-next/ \ "torch[device-gfx1201]==2.14.0+rocm10.2.0" \ "torchvision[device-gfx1201]==0.29.0+rocm10.2.0" \ "torchaudio==2.11.0.3+rocm10.2.0"
python -m pip install --index-url https://stable.repo.amd.com/rocm/whl-next/ \ "torch[device-gfx1100]==2.14.0+rocm10.2.0" \ "torchvision[device-gfx1100]==0.29.0+rocm10.2.0" \ "torchaudio==2.11.0.3+rocm10.2.0"
python -m pip install --index-url https://stable.repo.amd.com/rocm/whl-next/ \ "torch[device-gfx1101]==2.14.0+rocm10.2.0" \ "torchvision[device-gfx1101]==0.29.0+rocm10.2.0" \ "torchaudio==2.11.0.3+rocm10.2.0"
python -m pip install --index-url https://stable.repo.amd.com/rocm/whl-next/ \ "torch[device-gfx1102]==2.14.0+rocm10.2.0" \ "torchvision[device-gfx1102]==0.29.0+rocm10.2.0" \ "torchaudio==2.11.0.3+rocm10.2.0"
python -m pip install --index-url https://stable.repo.amd.com/rocm/whl-next/ \ "torch[device-gfx1103]==2.14.0+rocm10.2.0" \ "torchvision[device-gfx1103]==0.29.0+rocm10.2.0" \ "torchaudio==2.11.0.3+rocm10.2.0"
python -m pip install --index-url https://stable.repo.amd.com/rocm/whl-next/ \ "torch[device-gfx1151]==2.14.0+rocm10.2.0" \ "torchvision[device-gfx1151]==0.29.0+rocm10.2.0" \ "torchaudio==2.11.0.3+rocm10.2.0"
python -m pip install --index-url https://stable.repo.amd.com/rocm/whl-next/ \ "torch[device-gfx1150]==2.14.0+rocm10.2.0" \ "torchvision[device-gfx1150]==0.29.0+rocm10.2.0" \ "torchaudio==2.11.0.3+rocm10.2.0"
python -m pip install --index-url https://stable.repo.amd.com/rocm/whl-next/ \ "torch[device-gfx1152]==2.14.0+rocm10.2.0" \ "torchvision[device-gfx1152]==0.29.0+rocm10.2.0" \ "torchaudio==2.11.0.3+rocm10.2.0"
Install Flash Attention and AITER.
python -m pip install --extra-index-url https://rocm.frameworks.amd.com/whl-multi-arch/vllm/ \ "flash-attn==2.8.3" \ "amd-aiter==0.1.24.post1"
Install Triton 3.8.0.
python -m pip install --extra-index-url https://rocm.frameworks.amd.com/whl-multi-arch/vllm/ \ "triton==3.8.0+git669b31ac"
This replaces Triton 3.9.0 installed alongside PyTorch in step 2.
pipreports the following dependency conflict. The warning is expected; you can ignore it.ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. torch 2.14.0+rocm10.2.0 requires triton==3.9.0+git0a6e13e6.rocm10.2.0, but you have triton 3.8.0+git669b31ac which is incompatible. Successfully installed triton-3.8.0+git669b31ac
Install the vLLM 0.32 wheel using
uv pip.Note
We recommend using uv to install the vLLM wheel. It resolves vLLM’s many transitive dependencies more predictably than pip, which can silently pull incompatible versions from PyPI. If you don’t have
uvon your system, install it with:python -m pip install uv
cat > overrides.txt <<'EOF' triton==3.8.0+git669b31ac EOF uv pip install --override overrides.txt https://rocm.frameworks.amd.com/whl-multi-arch/vllm/vllm/vllm-0.32.1.dev0%2Bg5df63a720.d20261116.rocm102-cp314-cp314-linux_x86_64.whl
Set the following environment variables to prevent errors related to ROCm platform and Flash Attention availability when running vLLM.
export PYTHONPATH=$VIRTUAL_ENV/lib/python3.14/site-packages/_rocm_sdk_core/share/amd_smi export FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE
To make any of these settings permanent, add it to your shell startup file;
~/.bashrc, for instance.Check your installation.
python <<'EOF' import vllm import torch import flash_attn from vllm.platforms import current_platform print('vLLM version:', vllm.__version__) print('vLLM platform:', current_platform.device_name) print('Is ROCm:', current_platform.is_rocm()) print('PyTorch:', torch.__version__) print('HIP available:', torch.cuda.is_available()) print('HIP version:', torch.version.hip) print('flash-attn:', flash_attn.__version__) for i in range(current_platform.device_count()): print(f'Device {i}:', current_platform.get_device_name(i)) EOF
You should see output similar to the following with at least one GPU listed:
vLLM version: 0.32.1.dev0+g5df63a720.d20261116 vLLM platform: rocm Is ROCm: True PyTorch: 2.14.0+rocm10.2.0 HIP available: True HIP version: 7.17.26434 flash-attn: 2.8.3 Device 0: <AMD GPU name>
After setting up your environment, follow the vLLM 0.32 usage documentation to get started: Using vLLM.
Install PyTorch 2.13.0 in your virtual environment. This should also install the ROCm core libraries as a dependency. See Install PyTorch for ROCm for full instructions; make sure to select PyTorch version 2.13.0 on that page.
python -m pip install --index-url https://stable.repo.amd.com/rocm/whl-next/ \ "torch[device-gfx950]==2.13.0+rocm10.1.0" \ "torchvision[device-gfx950]==0.28.0+rocm10.1.0" \ "torchaudio==2.11.0.2+rocm10.1.0"
python -m pip install --index-url https://stable.repo.amd.com/rocm/whl-next/ \ "torch[device-gfx942]==2.13.0+rocm10.1.0" \ "torchvision[device-gfx942]==0.28.0+rocm10.1.0" \ "torchaudio==2.11.0.2+rocm10.1.0"
python -m pip install --index-url https://stable.repo.amd.com/rocm/whl-next/ \ "torch[device-gfx1200]==2.13.0+rocm10.1.0" \ "torchvision[device-gfx1200]==0.28.0+rocm10.1.0" \ "torchaudio==2.11.0.2+rocm10.1.0"
python -m pip install --index-url https://stable.repo.amd.com/rocm/whl-next/ \ "torch[device-gfx1201]==2.13.0+rocm10.1.0" \ "torchvision[device-gfx1201]==0.28.0+rocm10.1.0" \ "torchaudio==2.11.0.2+rocm10.1.0"
python -m pip install --index-url https://stable.repo.amd.com/rocm/whl-next/ \ "torch[device-gfx1100]==2.13.0+rocm10.1.0" \ "torchvision[device-gfx1100]==0.28.0+rocm10.1.0" \ "torchaudio==2.11.0.2+rocm10.1.0"
python -m pip install --index-url https://stable.repo.amd.com/rocm/whl-next/ \ "torch[device-gfx1101]==2.13.0+rocm10.1.0" \ "torchvision[device-gfx1101]==0.28.0+rocm10.1.0" \ "torchaudio==2.11.0.2+rocm10.1.0"
python -m pip install --index-url https://stable.repo.amd.com/rocm/whl-next/ \ "torch[device-gfx1102]==2.13.0+rocm10.1.0" \ "torchvision[device-gfx1102]==0.28.0+rocm10.1.0" \ "torchaudio==2.11.0.2+rocm10.1.0"
python -m pip install --index-url https://stable.repo.amd.com/rocm/whl-next/ \ "torch[device-gfx1103]==2.13.0+rocm10.1.0" \ "torchvision[device-gfx1103]==0.28.0+rocm10.1.0" \ "torchaudio==2.11.0.2+rocm10.1.0"
python -m pip install --index-url https://stable.repo.amd.com/rocm/whl-next/ \ "torch[device-gfx1151]==2.13.0+rocm10.1.0" \ "torchvision[device-gfx1151]==0.28.0+rocm10.1.0" \ "torchaudio==2.11.0.2+rocm10.1.0"
python -m pip install --index-url https://stable.repo.amd.com/rocm/whl-next/ \ "torch[device-gfx1150]==2.13.0+rocm10.1.0" \ "torchvision[device-gfx1150]==0.28.0+rocm10.1.0" \ "torchaudio==2.11.0.2+rocm10.1.0"
python -m pip install --index-url https://stable.repo.amd.com/rocm/whl-next/ \ "torch[device-gfx1152]==2.13.0+rocm10.1.0" \ "torchvision[device-gfx1152]==0.28.0+rocm10.1.0" \ "torchaudio==2.11.0.2+rocm10.1.0"
Install Flash Attention and AITER.
python -m pip install --extra-index-url https://rocm.frameworks.amd.com/whl-multi-arch/vllm/ \ "flash-attn==2.8.3" \ "amd-aiter==0.1.22.post2.dev0"
Install the vLLM 0.29 wheel using
uv pip.Note
We recommend using uv to install the vLLM wheel. It resolves vLLM’s many transitive dependencies more predictably than pip, which can silently pull incompatible versions from PyPI. If you don’t have
uvon your system, install it with:python -m pip install uv
uv pip install https://rocm.frameworks.amd.com/whl-multi-arch/vllm/vllm/vllm-0.29.1.dev0%2Brocm10.1.0.g98dff2a81.d20261002-cp314-cp314-linux_x86_64.whl
Upgrade vLLM’s
tensorizerdependency as a workaround for a compatibility issue.python -m pip install --upgrade "tensorizer==2.12.1"
Set the following environment variables to prevent errors related to ROCm platform and Flash Attention availability when running vLLM.
export PYTHONPATH=$VIRTUAL_ENV/lib/python3.14/site-packages/_rocm_sdk_core/share/amd_smi export FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE
To make any of these settings permanent, add it to your shell startup file;
~/.bashrc, for instance.Check your installation.
python <<'EOF' import vllm import torch import flash_attn from vllm.platforms import current_platform print('vLLM version:', vllm.__version__) print('vLLM platform:', current_platform.device_name) print('Is ROCm:', current_platform.is_rocm()) print('PyTorch:', torch.__version__) print('HIP available:', torch.cuda.is_available()) print('HIP version:', torch.version.hip) print('flash-attn:', flash_attn.__version__) for i in range(current_platform.device_count()): print(f'Device {i}:', current_platform.get_device_name(i)) EOF
After setting up your environment, follow the vLLM 0.29 usage documentation to get started: Using vLLM.
Install PyTorch 2.12 in your virtual environment. This should also install the ROCm core libraries as a dependency. See Install PyTorch for ROCm for full instructions; make sure to select PyTorch version 2.12.0 on that page.
python -m pip install --index-url https://stable.repo.amd.com/rocm/whl-next/ \ "torch[device-gfx950]==2.12.0+rocm10.0.0" \ "torchvision[device-gfx950]==0.27.0+rocm10.0.0" \ "torchaudio==2.11.0+rocm10.0.0"
python -m pip install --index-url https://stable.repo.amd.com/rocm/whl-next/ \ "torch[device-gfx942]==2.12.0+rocm10.0.0" \ "torchvision[device-gfx942]==0.27.0+rocm10.0.0" \ "torchaudio==2.11.0+rocm10.0.0"
python -m pip install --index-url https://stable.repo.amd.com/rocm/whl-next/ \ "torch[device-gfx1200]==2.12.0+rocm10.0.0" \ "torchvision[device-gfx1200]==0.27.0+rocm10.0.0" \ "torchaudio==2.11.0+rocm10.0.0"
python -m pip install --index-url https://stable.repo.amd.com/rocm/whl-next/ \ "torch[device-gfx1201]==2.12.0+rocm10.0.0" \ "torchvision[device-gfx1201]==0.27.0+rocm10.0.0" \ "torchaudio==2.11.0+rocm10.0.0"
python -m pip install --index-url https://stable.repo.amd.com/rocm/whl-next/ \ "torch[device-gfx1100]==2.12.0+rocm10.0.0" \ "torchvision[device-gfx1100]==0.27.0+rocm10.0.0" \ "torchaudio==2.11.0+rocm10.0.0"
python -m pip install --index-url https://stable.repo.amd.com/rocm/whl-next/ \ "torch[device-gfx1101]==2.12.0+rocm10.0.0" \ "torchvision[device-gfx1101]==0.27.0+rocm10.0.0" \ "torchaudio==2.11.0+rocm10.0.0"
python -m pip install --index-url https://stable.repo.amd.com/rocm/whl-next/ \ "torch[device-gfx1102]==2.12.0+rocm10.0.0" \ "torchvision[device-gfx1102]==0.27.0+rocm10.0.0" \ "torchaudio==2.11.0+rocm10.0.0"
python -m pip install --index-url https://stable.repo.amd.com/rocm/whl-next/ \ "torch[device-gfx1103]==2.12.0+rocm10.0.0" \ "torchvision[device-gfx1103]==0.27.0+rocm10.0.0" \ "torchaudio==2.11.0+rocm10.0.0"
python -m pip install --index-url https://stable.repo.amd.com/rocm/whl-next/ \ "torch[device-gfx1151]==2.12.0+rocm10.0.0" \ "torchvision[device-gfx1151]==0.27.0+rocm10.0.0" \ "torchaudio==2.11.0+rocm10.0.0"
python -m pip install --index-url https://stable.repo.amd.com/rocm/whl-next/ \ "torch[device-gfx1150]==2.12.0+rocm10.0.0" \ "torchvision[device-gfx1150]==0.27.0+rocm10.0.0" \ "torchaudio==2.11.0+rocm10.0.0"
python -m pip install --index-url https://stable.repo.amd.com/rocm/whl-next/ \ "torch[device-gfx1152]==2.12.0+rocm10.0.0" \ "torchvision[device-gfx1152]==0.27.0+rocm10.0.0" \ "torchaudio==2.11.0+rocm10.0.0"
Install Flash Attention and AITER.
python -m pip install --extra-index-url https://rocm.frameworks.amd.com/whl-multi-arch/vllm/ \ "flash-attn==2.8.3" \ "amd-aiter==0.1.20.post1"
Install the vLLM 0.27 wheel using
uv pip.Note
We recommend using uv to install the vLLM wheel. It resolves vLLM’s many transitive dependencies more predictably than pip, which can silently pull incompatible versions from PyPI. If you don’t have
uvon your system, install it with:python -m pip install uv
uv pip install https://rocm.frameworks.amd.com/whl-multi-arch/vllm/vllm/vllm-0.27.1.dev5%2Brocm10.0.0.gf46a9dfe2.d20260826-cp314-cp314-linux_x86_64.whl
Upgrade vLLM’s
tensorizerdependency as a workaround for a compatibility issue.python -m pip install --upgrade "tensorizer==2.12.1"
Set the following environment variables to prevent errors related to ROCm platform and Flash Attention availability when running vLLM.
export PYTHONPATH=$VIRTUAL_ENV/lib/python3.14/site-packages/_rocm_sdk_core/share/amd_smi export FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE
To make any of these settings permanent, add it to your shell startup file;
~/.bashrc, for instance.Check your installation.
python <<'EOF' import vllm import torch import flash_attn from vllm.platforms import current_platform print('vLLM version:', vllm.__version__) print('vLLM platform:', current_platform.device_name) print('Is ROCm:', current_platform.is_rocm()) print('PyTorch:', torch.__version__) print('HIP available:', torch.cuda.is_available()) print('HIP version:', torch.version.hip) print('flash-attn:', flash_attn.__version__) for i in range(current_platform.device_count()): print(f'Device {i}:', current_platform.get_device_name(i)) EOF
After setting up your environment, follow the vLLM 0.27 usage documentation to get started: Using vLLM.
Install PyTorch 2.11 in your virtual environment. This should also install the ROCm core libraries as a dependency.
python -m pip install --index-url https://repo.amd.com/rocm/whl-multi-arch/ \ "torch[device-gfx950]==2.11.0+rocm7.14.1" \ "torchvision[device-gfx950]==0.26.0+rocm7.14.1" \ "torchaudio==2.11.0+rocm7.14.1"
python -m pip install --index-url https://repo.amd.com/rocm/whl-multi-arch/ \ "torch[device-gfx942]==2.11.0+rocm7.14.1" \ "torchvision[device-gfx942]==0.26.0+rocm7.14.1" \ "torchaudio==2.11.0+rocm7.14.1"
python -m pip install --index-url https://repo.amd.com/rocm/whl-multi-arch/ \ "torch[device-gfx1200]==2.11.0+rocm7.14.1" \ "torchvision[device-gfx1200]==0.26.0+rocm7.14.1" \ "torchaudio==2.11.0+rocm7.14.1"
python -m pip install --index-url https://repo.amd.com/rocm/whl-multi-arch/ \ "torch[device-gfx1201]==2.11.0+rocm7.14.1" \ "torchvision[device-gfx1201]==0.26.0+rocm7.14.1" \ "torchaudio==2.11.0+rocm7.14.1"
python -m pip install --index-url https://repo.amd.com/rocm/whl-multi-arch/ \ "torch[device-gfx1100]==2.11.0+rocm7.14.1" \ "torchvision[device-gfx1100]==0.26.0+rocm7.14.1" \ "torchaudio==2.11.0+rocm7.14.1"
python -m pip install --index-url https://repo.amd.com/rocm/whl-multi-arch/ \ "torch[device-gfx1101]==2.11.0+rocm7.14.1" \ "torchvision[device-gfx1101]==0.26.0+rocm7.14.1" \ "torchaudio==2.11.0+rocm7.14.1"
python -m pip install --index-url https://repo.amd.com/rocm/whl-multi-arch/ \ "torch[device-gfx1102]==2.11.0+rocm7.14.1" \ "torchvision[device-gfx1102]==0.26.0+rocm7.14.1" \ "torchaudio==2.11.0+rocm7.14.1"
python -m pip install --index-url https://repo.amd.com/rocm/whl-multi-arch/ \ "torch[device-gfx1103]==2.11.0+rocm7.14.1" \ "torchvision[device-gfx1103]==0.26.0+rocm7.14.1" \ "torchaudio==2.11.0+rocm7.14.1"
python -m pip install --index-url https://repo.amd.com/rocm/whl-multi-arch/ \ "torch[device-gfx1151]==2.11.0+rocm7.14.1" \ "torchvision[device-gfx1151]==0.26.0+rocm7.14.1" \ "torchaudio==2.11.0+rocm7.14.1"
python -m pip install --index-url https://repo.amd.com/rocm/whl-multi-arch/ \ "torch[device-gfx1150]==2.11.0+rocm7.14.1" \ "torchvision[device-gfx1150]==0.26.0+rocm7.14.1" \ "torchaudio==2.11.0+rocm7.14.1"
python -m pip install --index-url https://repo.amd.com/rocm/whl-multi-arch/ \ "torch[device-gfx1152]==2.11.0+rocm7.14.1" \ "torchvision[device-gfx1152]==0.26.0+rocm7.14.1" \ "torchaudio==2.11.0+rocm7.14.1"
Install Flash Attention.
python -m pip install https://rocm.frameworks.amd.com/whl-multi-arch/vllm-cdna/flash-attn/flash_attn-2.8.3-cp314-cp314-linux_x86_64.whl
python -m pip install https://rocm.frameworks.amd.com/whl-multi-arch/vllm-rdna/flash-attn/flash_attn-2.8.3-py3-none-any.whl
Install AITER.
python -m pip install https://rocm.frameworks.amd.com/whl-multi-arch/vllm-cdna/amd-aiter/amd_aiter-0.1.13.post2.dev1%2Bgb32deb267.d20260901-cp314-cp314-linux_x86_64.whl
Install the vLLM 0.23.1 wheel using
uv pip.Note
We recommend using uv to install the vLLM wheel. It resolves vLLM’s many transitive dependencies more predictably than pip, which can silently pull incompatible versions from PyPI.
uv pip install https://rocm.frameworks.amd.com/whl-multi-arch/vllm-cdna/vllm/vllm-0.23.1.dev1%2Brocm7.14.1.g9ddef7117.d20260901-cp314-cp314-linux_x86_64.whl
Upgrade vLLM’s
tensorizerdependency as a workaround for a compatibility issue.python -m pip install --upgrade "tensorizer==2.12.1"
Set the following environment variables to prevent errors related to ROCm platform and Flash Attention availability when running vLLM.
export PYTHONPATH=$VIRTUAL_ENV/lib/python3.14/site-packages/_rocm_sdk_core/share/amd_smi export FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE
Check your installation.
python <<'EOF' import vllm import torch import flash_attn from vllm.platforms import current_platform print('vLLM version:', vllm.__version__) print('vLLM platform:', current_platform.device_name) print('Is ROCm:', current_platform.is_rocm()) print('PyTorch:', torch.__version__) print('HIP available:', torch.cuda.is_available()) print('HIP version:', torch.version.hip) print('flash-attn:', flash_attn.__version__) for i in range(current_platform.device_count()): print(f'Device {i}:', current_platform.get_device_name(i)) EOF
- After setting up your environment, follow the vLLM 0.23 usage
documentation to get started: Using vLLM.
Install the vLLM 0.23.1 wheel using
uv pip.Note
We recommend using uv to install the vLLM wheel. It resolves vLLM’s many transitive dependencies more predictably than pip, which can silently pull incompatible versions from PyPI.
uv pip install https://rocm.frameworks.amd.com/whl-multi-arch/vllm-rdna/vllm/vllm-0.23.1.dev1%2Brocm7.14.1.g9ddef7117.d20260831-cp314-cp314-linux_x86_64.whl
Upgrade vLLM’s
tensorizerdependency as a workaround for a compatibility issue.python -m pip install --upgrade "tensorizer==2.12.1"
Set the following environment variables to prevent errors related to ROCm platform and Flash Attention availability when running vLLM.
export PYTHONPATH=$VIRTUAL_ENV/lib/python3.14/site-packages/_rocm_sdk_core/share/amd_smi export FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE
Check your installation.
python <<'EOF' import vllm import torch import flash_attn from vllm.platforms import current_platform print('vLLM version:', vllm.__version__) print('vLLM platform:', current_platform.device_name) print('Is ROCm:', current_platform.is_rocm()) print('PyTorch:', torch.__version__) print('HIP available:', torch.cuda.is_available()) print('HIP version:', torch.version.hip) print('flash-attn:', flash_attn.__version__) for i in range(current_platform.device_count()): print(f'Device {i}:', current_platform.get_device_name(i)) EOF
After setting up your environment, follow the vLLM 0.23 usage documentation to get started: Using vLLM.
See also
Install PyTorch 2.11 in your virtual environment. This should also install the ROCm core libraries as a dependency.
python -m pip install --index-url https://repo.amd.com/rocm/whl-multi-arch/ \ "torch[device-gfx950]==2.11.0+rocm7.14.0" \ "torchvision[device-gfx950]==0.26.0+rocm7.14.0" \ "torchaudio==2.11.0+rocm7.14.0"
python -m pip install --index-url https://repo.amd.com/rocm/whl-multi-arch/ \ "torch[device-gfx942]==2.11.0+rocm7.14.0" \ "torchvision[device-gfx942]==0.26.0+rocm7.14.0" \ "torchaudio==2.11.0+rocm7.14.0"
python -m pip install --index-url https://repo.amd.com/rocm/whl-multi-arch/ \ "torch[device-gfx1200]==2.11.0+rocm7.14.0" \ "torchvision[device-gfx1200]==0.26.0+rocm7.14.0" \ "torchaudio==2.11.0+rocm7.14.0"
python -m pip install --index-url https://repo.amd.com/rocm/whl-multi-arch/ \ "torch[device-gfx1201]==2.11.0+rocm7.14.0" \ "torchvision[device-gfx1201]==0.26.0+rocm7.14.0" \ "torchaudio==2.11.0+rocm7.14.0"
python -m pip install --index-url https://repo.amd.com/rocm/whl-multi-arch/ \ "torch[device-gfx1100]==2.11.0+rocm7.14.0" \ "torchvision[device-gfx1100]==0.26.0+rocm7.14.0" \ "torchaudio==2.11.0+rocm7.14.0"
python -m pip install --index-url https://repo.amd.com/rocm/whl-multi-arch/ \ "torch[device-gfx1101]==2.11.0+rocm7.14.0" \ "torchvision[device-gfx1101]==0.26.0+rocm7.14.0" \ "torchaudio==2.11.0+rocm7.14.0"
python -m pip install --index-url https://repo.amd.com/rocm/whl-multi-arch/ \ "torch[device-gfx1102]==2.11.0+rocm7.14.0" \ "torchvision[device-gfx1102]==0.26.0+rocm7.14.0" \ "torchaudio==2.11.0+rocm7.14.0"
python -m pip install --index-url https://repo.amd.com/rocm/whl-multi-arch/ \ "torch[device-gfx1103]==2.11.0+rocm7.14.0" \ "torchvision[device-gfx1103]==0.26.0+rocm7.14.0" \ "torchaudio==2.11.0+rocm7.14.0"
python -m pip install --index-url https://repo.amd.com/rocm/whl-multi-arch/ \ "torch[device-gfx1151]==2.11.0+rocm7.14.0" \ "torchvision[device-gfx1151]==0.26.0+rocm7.14.0" \ "torchaudio==2.11.0+rocm7.14.0"
python -m pip install --index-url https://repo.amd.com/rocm/whl-multi-arch/ \ "torch[device-gfx1150]==2.11.0+rocm7.14.0" \ "torchvision[device-gfx1150]==0.26.0+rocm7.14.0" \ "torchaudio==2.11.0+rocm7.14.0"
python -m pip install --index-url https://repo.amd.com/rocm/whl-multi-arch/ \ "torch[device-gfx1152]==2.11.0+rocm7.14.0" \ "torchvision[device-gfx1152]==0.26.0+rocm7.14.0" \ "torchaudio==2.11.0+rocm7.14.0"
Install Flash Attention.
python -m pip install https://rocm.frameworks.amd.com/whl-multi-arch/vllm-cdna/flash-attn/flash_attn-2.8.3-cp314-cp314-linux_x86_64.whl
python -m pip install https://rocm.frameworks.amd.com/whl-multi-arch/vllm-rdna/flash-attn/flash_attn-2.8.3-py3-none-any.whl
Install AITER.
python -m pip install https://rocm.frameworks.amd.com/whl-multi-arch/vllm-cdna/amd-aiter/amd_aiter-0.1.13.post2.dev1%2Bgb32deb267-cp314-cp314-linux_x86_64.whl
Install the vLLM 0.23.1 wheel using
uv pip.Note
We recommend using uv to install the vLLM wheel. It resolves vLLM’s many transitive dependencies more predictably than pip, which can silently pull incompatible versions from PyPI.
uv pip install https://rocm.frameworks.amd.com/whl-multi-arch/vllm-cdna/vllm/vllm-0.23.1.dev1%2Brocm7.14.0.g9ddef7117.d20260715-cp314-cp314-linux_x86_64.whl
Upgrade vLLM’s
tensorizerdependency as a workaround for a compatibility issue.python -m pip install --upgrade "tensorizer==2.12.1"
Set the following environment variables to prevent errors related to ROCm platform and Flash Attention availability when running vLLM.
export PYTHONPATH=$VIRTUAL_ENV/lib/python3.14/site-packages/_rocm_sdk_core/share/amd_smi export FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE
Check your installation.
python <<'EOF' import vllm import torch import flash_attn from vllm.platforms import current_platform print('vLLM version:', vllm.__version__) print('vLLM platform:', current_platform.device_name) print('Is ROCm:', current_platform.is_rocm()) print('PyTorch:', torch.__version__) print('HIP available:', torch.cuda.is_available()) print('HIP version:', torch.version.hip) print('flash-attn:', flash_attn.__version__) for i in range(current_platform.device_count()): print(f'Device {i}:', current_platform.get_device_name(i)) EOF
- After setting up your environment, follow the vLLM 0.23 usage
documentation to get started: Using vLLM.
Install the vLLM 0.23.1 wheel using
uv pip.Note
We recommend using uv to install the vLLM wheel. It resolves vLLM’s many transitive dependencies more predictably than pip, which can silently pull incompatible versions from PyPI.
uv pip install https://rocm.frameworks.amd.com/whl-multi-arch/vllm-rdna/vllm/vllm-0.23.1.dev1%2Brocm7.14.0.g9ddef7117.d20260715-cp314-cp314-linux_x86_64.whl
Upgrade vLLM’s
tensorizerdependency as a workaround for a compatibility issue.python -m pip install --upgrade "tensorizer==2.12.1"
Set the following environment variables to prevent errors related to ROCm platform and Flash Attention availability when running vLLM.
export PYTHONPATH=$VIRTUAL_ENV/lib/python3.14/site-packages/_rocm_sdk_core/share/amd_smi export FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE
Check your installation.
python <<'EOF' import vllm import torch import flash_attn from vllm.platforms import current_platform print('vLLM version:', vllm.__version__) print('vLLM platform:', current_platform.device_name) print('Is ROCm:', current_platform.is_rocm()) print('PyTorch:', torch.__version__) print('HIP available:', torch.cuda.is_available()) print('HIP version:', torch.version.hip) print('flash-attn:', flash_attn.__version__) for i in range(current_platform.device_count()): print(f'Device {i}:', current_platform.get_device_name(i)) EOF
After setting up your environment, follow the vLLM 0.23 usage documentation to get started: Using vLLM.
See also
Known issues#
An incompatibility with vLLM’s
tensorizerdependency results in errors when running vLLM. As a workaround, manually bump thetensorizerversion in your virtual environment.python -m pip install --upgrade "tensorizer==2.12.1"
Known issues#
An incompatibility with vLLM’s
tensorizerdependency results in errors when running vLLM. As a workaround, manually bump thetensorizerversion in your virtual environment.python -m pip install --upgrade "tensorizer==2.12.1"
Significantly longer warmup times might be observed in some large language model inference workloads on AMD Radeon GPUs using vLLM versions v0.21.0 through v0.25.0. As a workaround, use a vLLM release earlier than v0.21.0 or upgrade to vLLM v0.26.0 or later, which includes a fix for this issue.