llama.cpp inference on ROCm

llama.cpp inference on ROCm#

llama.cpp is an open-source inference engine for running LLMs locally. It runs quantized models efficiently on everyday hardware and has become a pillar of local LLM inference. This page explains how to set up and run llama.cpp on AMD Radeon™ GPUs and Ryzen™ APUs.

Device family

AMD Radeon

AMD Ryzen

Instinct GPU

Radeon GPU

Ryzen APU

Operating system

Ubuntu 24.04

Windows 11

ROCm version

7.14.0

ROCm installation

apt

pip

Tarball

Prerequisites#

To run llama.cpp with ROCm on Linux, you need the following prerequisites:

GPU platform: AMD Instinct accelerators, and AMD Radeon discrete GPUs and Ryzen APUs supported by ROCm 7.14.0. For the list of supported devices, see the ROCm compatibility matrix.

Driver: the AMD GPU Driver (amdgpu). This is a separate installation step and is not provided by the ROCm packages or the tarball. Instinct and Radeon devices require the AMD GPU Driver; supported Ryzen APUs use the inbox kernel driver included with the distribution. See Install ROCm 7.14.0 for the driver installation step that applies to your distribution.
Group membership: your user must be a member of the video and render groups.
Additional packages: libgomp1 provides the OpenMP runtime that llama.cpp links against, and libcurl4 is required for model downloading. Install them if they are not already present on your system:
```
sudo apt install libgomp1 libcurl4
```

Install ROCm#

llama.cpp requires a ROCm installation on the host system.

Device family: AMD Radeon
Select your GPU – importantly, gfx architecture – from the dropdown list.
Use case: Compute
Operating system: Ubuntu
Installation method: apt (package manager)

Device family: AMD Radeon
Select your GPU – importantly, gfx architecture – from the dropdown list.
Use case: Compute
Operating system: Ubuntu
Installation method: pip

Device family: AMD Radeon
Select your GPU – importantly, gfx architecture – from the dropdown list.
Use case: Compute
Operating system: Ubuntu
Installation method: Tarball

Device family: AMD Radeon
Select your GPU – importantly, gfx architecture – from the dropdown list.
Use case: Compute
Operating system: Windows
Installation method: pip

Device family: AMD Radeon
Select your GPU – importantly, gfx architecture – from the dropdown list.
Use case: Compute
Operating system: Windows
Installation method: Tarball

Device family: AMD Ryzen
Select your GPU – importantly, gfx architecture – from the dropdown list.
Use case: Compute
Operating system: Ubuntu
Installation method: apt (package manager)

Device family: AMD Ryzen
Select your GPU – importantly, gfx architecture – from the dropdown list.
Use case: Compute
Operating system: Ubuntu
Installation method: pip

Device family: AMD Ryzen
Select your GPU – importantly, gfx architecture – from the dropdown list.
Use case: Compute
Operating system: Ubuntu
Installation method: Tarball

Device family: AMD Ryzen
Select your GPU – importantly, gfx architecture – from the dropdown list.
Use case: Compute
Operating system: Windows
Installation method: pip

Device family: AMD Ryzen
Select your GPU – importantly, gfx architecture – from the dropdown list.
Use case: Compute
Operating system: Windows
Installation method: Tarball

Important

Install llama.cpp#

AMD does not publish its own binaries for llama.cpp. Download a prebuilt ROCm-enabled release from the llama.cpp releases page. Release assets are named in the following format:

llama-<build>-bin-ubuntu-rocm-7.14-x64.tar.gz

For ROCm 7.14, that is any asset containing ubuntu-rocm-7.14, for example llama-b10628-bin-ubuntu-rocm-7.14-x64.tar.gz.

Verify the checksum of the downloaded file against the digest shown for the asset, then extract the archive:

sha256sum llama-b10628-bin-ubuntu-rocm-7.14-x64.tar.gz
tar -xf llama-b10628-bin-ubuntu-rocm-7.14-x64.tar.gz

The archive extracts into a directory named after the build, for example llama-b10628.

Note

The archive contains only the llama.cpp executables and their ggml backend libraries. It does not include the ROCm runtime, which must be installed separately as described in the previous section.

Configure your environment (Linux)#

llama.cpp loads the ROCm math libraries at run time and must be able to find them.

A pip installation places the ROCm runtime inside the virtual environment, in two directories:

<venv>/lib/python<version>/site-packages/_rocm_sdk_core/lib
<venv>/lib/python<version>/site-packages/_rocm_sdk_libraries/lib

Add both to LD_LIBRARY_PATH, replacing <venv> with the path of your virtual environment. The Python version in the path is the one you created the environment with; Ubuntu 24.04 provides Python 3.12.

SP=<venv>/lib/python3.12/site-packages
export LD_LIBRARY_PATH="$SP/_rocm_sdk_core/lib:$SP/_rocm_sdk_libraries/lib:$LD_LIBRARY_PATH"

Note

Activating the virtual environment is not sufficient. It sets PATH, which ROCm’s own tools use, but not LD_LIBRARY_PATH, which the llama.cpp executables need.

To make any of these settings permanent, add it to your shell startup file; ~/.bashrc, for instance.

Configure your environment (Windows)#

A pip installation places the ROCm runtime inside the virtual environment in two directories:

<venv>\Lib\site-packages\_rocm_sdk_core\bin
<venv>\Lib\site-packages\_rocm_sdk_libraries\bin

Add both to your PATH in the command prompt you use to launch llama.cpp, replacing <venv> with the actual path of your virtual environment, for example:

set "PATH=C:\llamacpp\.venv\Lib\site-packages\_rocm_sdk_core\bin;C:\llamacpp\.venv\Lib\site-packages\_rocm_sdk_libraries\bin;%PATH%"

This setting applies to the current command prompt only. To make it permanent, add both directories to your user or system PATH in your Windows settings.

If the ROCm libraries are not on your PATH, llama.cpp does not report an error. It silently falls back to the CPU backend and runs several times slower. Use Verify the installation to confirm the GPU is detected.

Copy the HIP runtime libraries#

Copy the following three files from your ROCm installation into the directory that contains llama-cli.exe: <venv>\Lib\site-packages\_rocm_sdk_core\bin

amdhip64_7.dll
rocm_kpack.dll
amd_comgr.dll

Copy all three files. Copying amdhip64_7.dll on its own prevents llama.cpp from using the GPU: the HIP runtime supplied by ROCm depends on the matching rocm_kpack.dll and amd_comgr.dll, and rocm_kpack.dll is not present in System32.

copy "<venv>\Lib\site-packages\_rocm_sdk_core\bin\amdhip64_7.dll" C:\llamacpp\
copy "<venv>\Lib\site-packages\_rocm_sdk_core\bin\rocm_kpack.dll" C:\llamacpp\
copy "<venv>\Lib\site-packages\_rocm_sdk_core\bin\amd_comgr.dll"  C:\llamacpp\

Copy the following three files from your ROCm installation into the directory that contains llama-cli.exe: C:\TheRock\build\bin

amdhip64_7.dll
rocm_kpack.dll
amd_comgr.dll

Important

Copy all three files. Copying amdhip64_7.dll on its own prevents llama.cpp from using the GPU: the HIP runtime supplied by ROCm depends on the matching rocm_kpack.dll and amd_comgr.dll, and rocm_kpack.dll is not present in System32.

copy "C:\TheRock\build\bin\amdhip64_7.dll" C:\llamacpp\
copy "C:\TheRock\build\bin\rocm_kpack.dll" C:\llamacpp\
copy "C:\TheRock\build\bin\amd_comgr.dll"  C:\llamacpp\

Note

The AMD display driver installs its own copy of amdhip64_7.dll in C:\Windows\System32. Windows searches System32 before PATH, so the driver’s version is loaded in preference to the one supplied by ROCm, even when ROCm is on PATH. Because Windows searches the directory containing the executable first, copying these libraries next to llama-cli.exe ensures the ROCm version is used.

Verify the installation#

List the devices visible to llama.cpp:

cd llama-b10628
./llama-cli --list-devices

The output shows the detected GPU and its available memory:

Available devices:
  ROCm0: AMD Radeon RX 9070 XT (16304 MiB, 15770 MiB free)

If Available devices is empty, the ROCm libraries were not found. Review Configure your environment.

If instead you see error while loading shared libraries: libgomp.so.1, install the additional packages listed in System requirements.

Listing the devices confirms that the ROCm libraries were found, but it does not confirm that computation runs on the GPU. To verify that, run a short benchmark with a model in GGUF format; see Run a llama.cpp example.

List the devices visible to llama.cpp. Make sure to cd into where the llama.cpp binaries are located and that you’re in the same shell in which you set your PATH.

The output shows the detected GPU and its available memory. For example:

Available devices:
  ROCm0: AMD Radeon RX 7900 XT (20464 MiB, 20315 MiB free)

If no devices are listed at all, the ROCm runtime cannot be found. Review Configure your environment.

Optionally, run the llama.cpp unit tests to validate the installation more thoroughly. The test suite takes a considerable amount of time to complete.

test-backend-ops.exe

Select a GPU#

Systems with both a discrete GPU and an integrated GPU report more than one ROCm device:

Available devices:
  ROCm0: AMD Radeon RX 9070 XT (16304 MiB, 15770 MiB free)
  ROCm1: AMD Radeon Graphics (63199 MiB, 63174 MiB free)

The integrated GPU reports a large amount of shared system memory and may not be a supported ROCm device. To restrict llama.cpp to a specific GPU, set HIP_VISIBLE_DEVICES to its index:

export HIP_VISIBLE_DEVICES=0

Run a llama.cpp example#

Once your llama.cpp environment is set up, experiment with the following steps to run a model and benchmark your installation. The prebuilt llama.cpp release includes the executables to exercise the functionality of your installation.

The two most popular use cases are:

llama-cli: The main executable to run the model interactively or get a response to a prompt.
llama-bench: Run a benchmark of your model with different configurations.

llama-cli#

Use the CLI tool to start the client, replacing <model>.gguf with the path to a model on your system:

A prompt appears when the client is ready, and you can start interacting with the model:

> Explain what a GPU kernel is in one sentence.
A GPU kernel is a specialized program within a GPU (Graphics Processing Unit) designed to execute
instructions for rendering graphical content, such as images or video, on a GPU's hardware architecture.

[ Prompt: 593.4 t/s | Generation: 286.4 t/s ]

To exit, enter /exit or press Ctrl+C.

To send a single prompt and exit instead of starting an interactive session, add the -p and -st flags:

Note

llama-bench#

Use the CLI tool to start the application, replacing <model>.gguf with the path to a model on your system:

The result of the command above should be similar to the following when running on an AMD Radeon RX 7900 XT system:

ggml_cuda_init: found 1 ROCm devices (Total VRAM: 20464 MiB):
  Device 0: AMD Radeon RX 7900 XT, gfx1100 (0x1100), VMM: no, Wave Size: 32, VRAM: 20464 MiB
load_backend: loaded ROCm backend from C:\llamacpp\ggml-hip.dll
load_backend: loaded RPC backend from C:\llamacpp\ggml-rpc.dll
load_backend: loaded CPU backend from C:\llamacpp\ggml-cpu-haswell.dll
| model                          |       size |     params | backend    | ngl |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --------------: | -------------------: |
| qwen2 1B Q4_K - Medium         | 462.96 MiB |   630.17 M | ROCm       | 999 |            pp16 |     2406.95 ± 704.75 |
| qwen2 1B Q4_K - Medium         | 462.96 MiB |   630.17 M | ROCm       | 999 |            pp32 |     3842.56 ± 500.15 |
| qwen2 1B Q4_K - Medium         | 462.96 MiB |   630.17 M | ROCm       | 999 |            pp64 |    7104.29 ± 1218.73 |
| qwen2 1B Q4_K - Medium         | 462.96 MiB |   630.17 M | ROCm       | 999 |           pp128 |     8628.19 ± 928.69 |
| qwen2 1B Q4_K - Medium         | 462.96 MiB |   630.17 M | ROCm       | 999 |           pp256 |   16444.29 ± 1662.23 |
| qwen2 1B Q4_K - Medium         | 462.96 MiB |   630.17 M | ROCm       | 999 |           pp512 |   24628.13 ± 2498.10 |
| qwen2 1B Q4_K - Medium         | 462.96 MiB |   630.17 M | ROCm       | 999 |          pp1024 |     26143.12 ± 73.58 |
| qwen2 1B Q4_K - Medium         | 462.96 MiB |   630.17 M | ROCm       | 999 |            tg64 |        373.29 ± 9.81 |
| qwen2 1B Q4_K - Medium         | 462.96 MiB |   630.17 M | ROCm       | 999 |           tg128 |        375.70 ± 3.49 |

build: 78ec4c378 (10539)

The backend column shows ROCm when the GPU is in use. If it shows CPU, the ROCm libraries were not found. Review the environment configuration steps to troubleshoot.

Flash Attention is enabled with -fa 1 and is supported on Radeon GPUs through rocWMMA:

llama.cpp inference on ROCm

Contents

llama.cpp inference on ROCm#

Prerequisites#

Install ROCm#

Install llama.cpp#

Verify the installation#

Run a llama.cpp example#

llama-cli#

llama-bench#