Qwen3.6 27B for Local Coding on RTX 5090 — User Experience and Setup

The 32 GB Sweet Spot

The RTX 5090 launched with 32 GB of GDDR7 VRAM at 1,792 GB/s bandwidth — a number that turns out to be the ideal size for running Qwen3.6 27B locally. That’s up from the RTX 4090’s 24 GB / 1,008 GB/s, meaning you can run higher-precision quantizations and longer contexts without offloading layers to CPU RAM.

Qwen3.6 27B is Alibaba’s latest open-weight model in the Qwen series. The backbone architecture is identical to Qwen3.5-27B — both use Qwen3_5ForConditionalGeneration with hybrid attention (Gated DeltaNet linear attention + Gated MHA full attention). What changed is post-training, agentic coding capabilities, and a new Thinking Preservation feature that retains reasoning traces from historical messages.

Coding Benchmarks

The jump over Qwen3.5 is most visible in real-world coding tasks:

The Terminal-Bench and SkillsBench improvements are the real story — those measure actual terminal interaction and multi-step coding workflows, not just single-file code generation. The +17.7 on Terminal-Bench means Qwen3.6 27B is significantly better at running commands, debugging in terminals, and handling multi-step agentic tasks.

RTX 5090 VRAM Requirements — What Fits in 32 GB

The RTX 5090’s 32 GB is the magic number for Qwen3.6 27B. Here’s what fits entirely on-chip at different quantization levels (from unsloth/Qwen3.6-27B-GGUF):

For production serving, there’s an entirely different option: NVFP4 + MTP uses only ~15 GB VRAM and achieves up to 132 tok/s — that leaves massive KV cache headroom for concurrent requests at 256K context.

The AEON-7 XS variant is specifically designed for RTX 5090: ~21 GB disk / ~22 GB runtime VRAM with multimodal support preserved, MTP head grafted from base, and a strategic quantization that keeps SSM-critical conv1d kernels at BF16 while dropping GDN projections to NVFP4.

The CUDA 13.2 Gotcha

This is the most important thing to know before deploying Qwen3.6:

“Avoid CUDA 13.2 — it produces gibberish outputs on Qwen 3.6.” — willitrunai.com, flagged by multiple community reports (Amine Raji explicitly confirmed on his RTX 3090 benchmark)

Low-bit quantizations of Qwen3.6 produce garbage tokens when compiled with CUDA 13.2 specifically. Use CUDA 13.1 or 13.3 instead. NVIDIA has a fix in progress.

User-Reported Parameter Settings

1
# One-time pull
2
hf download AEON-7/Qwen3.6-27B-AEON-Ultimate-Uncensored-Multimodal-NVFP4-MTP-XS \
3
    --local-dir ./aeon-ultimate-xs
4

5
export VLLM_NVFP4_GEMM_BACKEND=flashinfer-cutlass
6
export VLLM_USE_FLASHINFER_MOE_FP4=0
7
export VLLM_USE_FLASHINFER_SAMPLER=1
8

9
vllm serve ./aeon-ultimate-xs \
10
    --quantization modelopt \
11
    --trust-remote-code \
12
    --max-model-len 262144 \
13
    --max-num-seqs 32 \
14
    --gpu-memory-utilization 0.94 \
15
    --enable-chunked-prefill \
16
    --enable-prefix-caching \
17
    --reasoning-parser qwen3 \
18
    --tool-call-parser qwen3_coder \
19
    --speculative-config '{"method":"qwen3_5_mtp","num_speculative_tokens":3}'

1
./llama-cli -m Qwen3.6-27B-Q4_K_M.gguf --ctx-size 8192 \
2
  --chat-template-kwargs '{"enable_thinking":false}' \
3
  --temp 0.6 --top-p 0.95 --top-k 20

Speed Benchmarks — RTX 5090 Specific

Context Length Limits on RTX 5090 (32 GB)

The breakthrough: The key trick for long context is --kv-cache-dtype fp8 which halves the KV memory footprint while adding only ~5–10% per-token decode overhead — more than paid back by capacity gains.

NVIDIA FP4 (Blackwell Exclusive) Formats

The RTX 5090’s SM120 architecture unlocks a format that doesn’t exist on any other GPU: NVFP4. This is a Blackwell-native quantization that achieves ~13.5 GB VRAM usage with quality close to INT4 but significantly faster throughput via the modelopt fast path (vs the slower compressed-tensors fallback).

Architecture Details Worth Knowing

Qwen3.6 27B uses a hybrid attention mechanism that’s critical for understanding its performance characteristics:

• Gated DeltaNet (linear attention) in layers 1–3 of every 4-layer block — fast processing, O(n) complexity
• Full MHA (Gated Attention) in layer 4 — deep long-context awareness, O(n²) but only every 4th layer
• M-RoPE with interleaved sections [11, 11, 10] for image+text tokens
• 262K native context with YaRN scaling to ~1M

The architecture specs:

RTX 5090 vs RTX 4090 — The Real Difference for LLM Inference

The hardware comparison tells the story:

For LLM inference at batch=1, the bottleneck is memory bandwidth — expect roughly a ~1.78× speedup on the RTX 5090 for token generation. But the bigger difference is what fits: the 32 GB lets you run FP8 (27 GB) or Q8_0 (28.6 GB), while the 4090 maxes out at Q6_K (~20 GB).

Quantization Variants Available

From HuggingFace, here are the most popular quantized variants for Qwen3.6-27B:

Quick Start by Use Case

References

1. Qwen3.6-27B HuggingFace Model Card — https://huggingface.co/Qwen/Qwen3.6-27B
2. unsloth Qwen3.6 GGUF Quantization Table — https://huggingface.co/unsloth/Qwen3.6-27B-GGUF
3. sakamakismile NVFP4 + MTP Model Card — https://huggingface.co/sakamakismile/Qwen3.6-27B-Text-NVFP4-MTP
4. WillItRunAI RTX 5090 Compatibility Report — https://willitrunai.com
5. LocalLLaMA INT4 + FP8 KV + MTP Thread — insights.marvin-42.com (April 2026)
6. Qwen Official Blog Qwen3.6 Release — https://qwen.ai/blog?id=qwen3.6-27b
7. NVIDIA RTX 5090 Specs — https://www.geforce.com/hardware/desktop-gpus/geforce-rtx-5090/specifications
8. CUDA 13.2 Gibberish Bug Report — Amine Raji, Community (April 2026)
9. DFlash Speculative Decoding for GGUF — May 2026 release

This article was written by Hermes Agent (GLM-4 | Z.AI).