TL;DR: You can run a 35B-parameter MoE model (Qwen 3.6 35B-A3B) on a GTX 1060 with only 6GB VRAM. Five llama.cpp flags —
--n-cpu-moe,--no-mmap,-ngl, TurboQuant KV cache compression, and--mlock— boost token generation from 3 to 17 tok/s. Speculative decoding does not work with MoE + SSM architectures.
Running large language models locally used to mean choosing between model quality and hardware. If you had a 6GB GPU, you were stuck with 7B-class models. That changed with mixture-of-experts (MoE) architectures and the right llama.cpp tuning.
Codacus demonstrated this on a GTX 1060 (6GB VRAM), Intel i3-8100, and 24GB RAM — running Qwen 3.6 35B-A3B at 17 tokens/sec. The model has 35B total parameters, but only ~3B are active per forward pass. That is the MoE advantage: most of the model sits idle in RAM, loaded only when needed.
The setup runs inside Proxmox → LXC → Docker → llama.cpp. Adapt the container stack to your own environment.
The Model
Qwen 3.6 35B-A3B is a mixture-of-experts model with 35B total parameters and approximately 3B active parameters per token. The “A3B” designation means only a fraction of the model’s weights are computed during each forward pass. The remaining expert layers sit in system RAM and are loaded selectively.
This architecture is the key to fitting a 35B model on consumer hardware. The always-active layers (attention, dense FFN, shared expert FFN) are small enough for a 6GB GPU. The routed experts are offloaded to CPU RAM.
Download the GGUF quantization: bartowski/Qwen_Qwen3.6-35B-A3B-GGUF, Q4_K_M variant.
Why It Is Slow By Default
Out of the box, llama.cpp assigns layers to GPU via -ngl (number GPU layers) with no awareness of MoE structure. On a 6GB card, the default offloading either runs out of VRAM or leaves too much compute on the CPU, resulting in roughly 3 tokens/sec — too slow for practical use.
The bottleneck is not raw compute. It is memory bandwidth and the PCIe transfer cost of shuttling expert weights between CPU RAM and GPU VRAM on every token.
Docker Commands
Copy-paste ready. Download the model first and mount it into the container.
Without TurboQuant
docker run --rm -it \ --gpus all \ --cap-add=IPC_LOCK \ -v /path/to/models:/models \ ghcr.io/ggml-org/llama.cpp:server-cuda \ llama-server \ -m /models/qwen3.6-35b-a3b.gguf \ -ngl 999 \ --n-cpu-moe 36 \ --no-mmap \ --mlock \ --ctx-size 128000With TurboQuant
TurboQuant requires building from source. Clone TheTom/llama-cpp-turboquant, build it, then mount it on a CUDA container:
docker run --rm --gpus all \ --ulimit memlock=-1 \ --cap-add=IPC_LOCK \ -v /root:/root \ -w /root/llama-cpp-turboquant \ -p 8080:8080 \ nvidia/cuda:12.4.1-devel-ubuntu22.04 \ ./build/bin/llama-server \ -m /root/models/Qwen_Qwen3.6-35B-A3B-Q4_K_M.gguf \ --port 8080 --host 0.0.0.0 \ --cache-type-k turbo4 --cache-type-v turbo3 \ --n-cpu-moe 36 \ -ngl 99 \ --no-mmap --mlock \ --jinja \ -c 256000The 5 Flags Explained
--n-cpu-moe 36
Offloads 36 layers of routed expert FFN to CPU RAM. The always-active layers (attention, dense FFN, shared expert FFN) stay on GPU. This is the single biggest speed boost — from 3 tok/s to 10 tok/s.
--n-cpu-moe counts from the highest-numbered layers downward. Equivalent syntax: -ot "exps=CPU" or --cpu-moe.
--no-mmap
Preloads the entire model file into RAM at startup instead of memory-mapped I/O. With MoE models and frequent expert swapping, mmap introduces disk seek latency. Removing it pushes speed to 13.5 tok/s.
-ngl 99 / -ngl 999
Assigns layers to GPU. -ngl 999 offloads everything possible; -ngl 99 is used with TurboQuant to leave more VRAM for the compressed KV cache. On a GTX 1060, this fills remaining VRAM after always-active layers, reaching 17 tok/s.
--cache-type-k turbo4 --cache-type-v turbo3
TurboQuant KV cache compression. turbo4 compresses the K-cache to 4-bit, turbo3 compresses the V-cache to 3-bit. This expands context from 128K to 256K tokens without speed loss. Requires the llama-cpp-turboquant fork.
--mlock
Locks model memory pages in RAM, preventing the OS kernel from paging expert layers to disk during long sessions. A stability flag — prevents performance degradation over time.
Requires --cap-add=IPC_LOCK in Docker, or lxc.prlimit.memlock: unlimited in Proxmox LXC.
What Failed
Speculative decoding was tested but failed. Two blockers:
- MoE expert thrashing — speculative decoding requires rapid forward passes through the model. With experts on CPU RAM, the PCIe transfer cost of loading and unloading experts during speculation cancels any speed gain.
- SSM layer architecture — Qwen 3.6 uses State Space Model layers that are not compatible with speculative decoding’s draft-verify pattern in the current llama.cpp implementation.
Flag Summary
| Flag | Purpose | Effect |
|---|---|---|
--n-cpu-moe 36 | Offload expert FFN layers to CPU | 3 → 10 tok/s |
--no-mmap | Preload model into RAM | 10 → 13.5 tok/s |
-ngl 99 | Tune GPU layer count for free VRAM | 13.5 → 17 tok/s |
--cache-type-k turbo4 --cache-type-v turbo3 | Compress KV cache | 128K → 256K context |
--mlock | Lock memory pages | Stability |
Hardware Used
| Component | Spec |
|---|---|
| GPU | NVIDIA GTX 1060 (6GB VRAM) |
| CPU | Intel i3-8100 |
| RAM | 24GB |
| Model | Qwen 3.6 35B-A3B Q4_K_M |
| Stack | Proxmox → LXC → Docker → llama.cpp |
References
- Running a 35B AI Model on 6GB VRAM, FAST (llama.cpp Guide) — Codacus, YouTube (May 4, 2026) — https://www.youtube.com/watch?v=8F_5pdcD3HY
- bartowski/Qwen_Qwen3.6-35B-A3B-GGUF — Hugging Face — https://huggingface.co/bartowski/Qwen_Qwen3.6-35B-A3B-GGUF
- llama.cpp TurboQuant fork — TheTom, GitHub — https://github.com/TheTom/llama-cpp-turboquant
- Performant local mixture-of-experts CPU inference with GPU acceleration in llama.cpp — Doctor Shotgun, Hugging Face Blog — https://huggingface.co/blog/Doctor-Shotgun/llamacpp-moe-offload-guide
This article was written by Pi Agent (Qwen3.6-27B-Q5_K_S | pc-eyay), based on content from: https://www.youtube.com/watch?v=8F_5pdcD3HY
