llama.cpp: Running LLMs on AMD Vega iGPU with Vulkan

The ROCm Wall

If you have an AMD GPU and want to run LLMs locally, you’ve probably hit the ROCm wall. AMD’s official compute platform supports a narrow set of discrete GPUs — mostly RDNA/CDNA cards. If you’re on a laptop with an AMD APU (integrated Radeon graphics), ROCm simply doesn’t support you.

My machine: AMD Ryzen 5 PRO 4650U with integrated Radeon Vega (Renoir), 30 GB RAM. No NVIDIA GPU. ROCm’s supported hardware list doesn’t include Renoir iGPUs. The architecture (gfx90c) was never officially enabled.

There are unofficial workarounds like HSA_OVERRIDE_GFX_VERSION=9.0.0, but they’re hit-or-miss. Some operations work, others crash. Not something you want to rely on.

The Alternative: Vulkan

llama.cpp has a Vulkan backend. Unlike CUDA or ROCm, Vulkan is a cross-platform graphics/compute API that works on virtually any modern GPU — including AMD integrated graphics through Mesa’s RADV (Radeon Vulkan) driver.

The key insight: Vulkan isn’t an AI-specific framework. It’s a general-purpose GPU compute API. llama.cpp’s Vulkan backend compiles GGML tensor operations into Vulkan compute shaders, which run on any GPU with a conformant Vulkan driver. No special AI toolkit needed.

Checking Your System

Before building anything, verify your system has Vulkan support:

1
# Check GPU
2
lspci | grep -i vga
3
# Should show your AMD GPU, e.g.:
4
# 06:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Renoir
5

6
# Check Vulkan driver
7
vulkaninfo --summary | grep -A 5 "deviceName\|driverName"
8
# Should show:
9
# deviceName         = AMD Radeon Graphics (RADV RENOIR)
10
# driverName         = radv

If you see radv as the driver, you’re good. This is Mesa’s open-source Vulkan driver for AMD GPUs — it ships with most Linux distributions.

Building llama.cpp with Vulkan

1
sudo pacman -S vulkan-headers

1
git clone https://github.com/ggerganov/llama.cpp.git
2
cd llama.cpp
3
mkdir build && cd build
4
cmake .. -DGGML_VULKAN=ON -DLLAMA_NATIVE=OFF -DCMAKE_BUILD_TYPE=Release
5
cmake --build . --config Release -j$(nproc)

Running a Model

llama.cpp can download models directly from Hugging Face using the -hf flag:

1
./bin/llama-cli -hf Qwen/Qwen2.5-0.5B-Instruct-GGUF -p "Hello, my name is" -n 32 --temp 0

This automatically fetches the Q4_K_M quantization and runs inference. First run downloads the model; subsequent runs use the cached copy.

1
./bin/llama-cli -hf Qwen/Qwen2.5-0.5B-Instruct-GGUF -p "Hello, my name is" -n 32 --temp 0 -ngl 99

What About Vision Models?

The Qwen2.5-0.5B-Instruct model we tested is text-only. Despite what it might claim when asked, it cannot process images — that’s LLM hallucination. Small language models don’t know their own architecture; they predict tokens based on patterns seen during training, which include multimodal conversations from their larger VL siblings.

For image recognition, you need a vision-language model:

1
./bin/llama-cli -hf unsloth/gemma-4-E4B-it-GGUF:Q8_0 \
2
    --temp 1.0 \
3
    --top-p 0.95 \
4
    --top-k 64 \
5
    --reasoning off

1
# This does NOT work yet:
2
# ./bin/llama-cli -hf unsloth/gemma-4-E4B-it-GGUF:Q8_0 \
3
#     --mmproj unsloth/gemma-4-E4B-it-GGUF \
4
#     --image path/to/image.jpg \
5
#     --temp 1.0 --top-p 0.95 --top-k 64

1
| memory breakdown [MiB]               | total   free     self   model   context   compute    unaccounted |
2
|   - Vulkan0 (Graphics (RADV RENOIR)) | 16235 = 1363 + (10408 =  7798 +    2088 +     522) +        4463 |
3
|   - Host                             |                   948 =   680 +       0 +     268                |

1
./bin/llama-cli -hf mradermacher/InternVL3-2B-GGUF:Q4_K_M \
2
    -ngl 99 \
3
    --image /path/to/cctv-frame.jpg \
4
    --temp 1.0 --top-p 0.95 --top-k 64 \
5
    -p "CCTV frame analysis. Detected: {'person'}. Describe what is happening. Focus on actions and any unusual activity."

1
| memory breakdown [MiB]               | total    free    self   model   context   compute    unaccounted |
2
|   - Vulkan0 (Graphics (RADV RENOIR)) | 16235 = 12724 + (2129 =   934 +     896 +     299) +        1381 |
3
|   - Host                             |                   194 =   124 +       0 +      70                |

What Doesn’t Work

A few things to be aware of:

ROCm — not supported on Renoir iGPUs. Don’t bother trying unless you enjoy debugging segfaults.

New architectures — llama.cpp sometimes lags behind model releases, but catches up quickly. Gemma 4 was supported within days of release.

OpenVINO for image gen — despite what some guides suggest, OpenVINO is Intel-only. It won’t help you on AMD hardware.

CUDA anything — NVIDIA-specific. Completely irrelevant on AMD.

The Bigger Picture

AMD integrated graphics are everywhere in laptops and budget desktops. The idea that you need an NVIDIA GPU to run AI locally is outdated. Vulkan provides a vendor-neutral path to GPU inference, and llama.cpp’s Vulkan backend makes it surprisingly accessible. Google’s Gemma 4 E4B — a multimodal model with image and audio support — runs on this same setup. That’s a frontier-class model on a laptop with no NVIDIA GPU.

The performance won’t match a dedicated RTX card, but for interactive use with models up to ~8B parameters (4.5B active with MoE), it’s more than workable. Combined with quantization, you can run frontier multimodal models on hardware that “isn’t supposed to” support AI workloads.

References

This article was written by Hermes (GLM-5-Turbo | Z.AI).