Qwen3.6 27B: From 20 t/s to 184 t/s — Full Optimization Pipeline

Why quantization speeds things up

With FP8, weights occupy 2 bytes per parameter. At 4-bit, each weight is half a byte — 4× less memory traffic over the same bandwidth. Same compute, less waiting.

The host benchmarks two approaches:

Both double throughput compared to FP8 (from 20 → 48 t/s). Quality loss is minimal — mostly occasional errors requiring retry tokens, which still cost less than the time saved.

Quantization quality assessment

The host tested against three datasets derived from ToolCall15 (tool-calling success rate), data extraction, and instruction following — 45 scenarios total. Most users cannot perceive the difference between FP8 and 4-bit quantized output. The extra token cost of correcting rare errors is far less than the time penalty of running unquantized.

GGUF vs AWQ — when to use which

How MTP works

During single-token generation, the GPU sits idle waiting for memory reads. MTP keeps it busy — speculating on future tokens while the current one loads. It recovers wasted compute.

MTP benchmark results (vLLM)

FP8 also benefits dramatically from MTP:

The RTX 4090 has native FP8 hardware acceleration (Tensor Cores), making FP8 + MTP competitive even against quantized models without speculative decoding.

Task-type variance

Speculative decoding speed depends on predictability of the output domain:

The host’s key insight: multi-agent coding workflows are ideal for speculative decoding because code has structural regularity (indentation, brackets) while remaining deterministic enough that small models predict well.

Diffusion applied to text

The analogy: generating an image starts as noise and iteratively refines toward clarity. DFlash applies this to tokens:

1. The draft sees [last_target_token, MASK x 15] plus 5 captured target hidden states from specific layers
2. It denoises all 16 masks in a single forward pass — no chain dependency
3. DDTree builds a best-first tree of up to 22 nodes (or flat chain without DDTree)
4. One target forward verifies the entire tree via causal mask; committed tokens feed back, rejected branches discarded

Diffusion models are inherently parallel — they do not depend on sequential chain-of-thought like autoregressive generation. Qwen3.5-27B is a hybrid model (every 4th layer = full softmax attention, rest = Gated DeltaNet), which Lucebox handles via ggml_gated_delta_net — the only kernel needed beyond standard ggml.

DFlash benchmark

Lucebox ported DFlash to GGUF via a custom C++/CUDA engine on top of ggml (no libllama). Qwen3.5-27B Q4_K_M (~16 GB target + 3.46 GB BF16 draft) fits on one RTX 3090 with 24 GB VRAM:

The host achieves 184 tokens per second peak throughput on an RTX 4090 — nearly 10× the FP8 baseline on a single card. Lucebox’s demo hits 207 tok/s on a less powerful RTX 3090, proving DFlash scales beyond high-end hardware.

Why dual GPU is not 2× for single requests

The bottleneck is inter-card synchronization of model weights, not compute:

• RTX 4090 has no P2P (peer-to-peer) VRAM communication between cards
• Data must route through CPU → system RAM → PCIe → second card
• Each hop introduces latency that erodes the benefit for single-token generation

Where multi-GPU shines: concurrency and context length

Dual GPUs unlock two things:

1. More context — split model weights across two VRAM pools, enabling longer KV cache windows (e.g., TurboQuant enables 200K context on a single 24GB card; dual cards double that capacity)
2. Concurrent throughput — 10+ simultaneous requests fill both cards like moving from 2-lane to 4-lane highway

The host’s multi-agent orchestration system relies on this: high-IQ models handle planning, while quantized, high-concurrency instances execute repetitive tasks simultaneously. The concept of “extreme leverage” — same hardware, same power draw, but 10×–50× token throughput through optimization and concurrency.

Host’s Pinned vLLM Launch Command

1
VLLM_USE_MODELSCOPE=true \
2
uv run vllm serve qwen/Qwen3.6-27B-FP8 \
3
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}' \
4
  --max-num-seqs 10 \
5
  --max-model-len auto \
6
  --port 5000 \
7
  --enable-prefix-caching \
8
  --gpu-memory-utilization 0.92 \
9
  --reasoning-parser qwen3 \
10
  --enable-auto-tool-choice \
11
  --served-model-name qwen3.6 \
12
  --tool-call-parser qwen3_coder

Key flags:

RTX 5090 Alternative — GPTQ-MARLIN MoE at 211 t/s

A viewer reported that the host’s FP8+MTP config struggled on an RTX 5090 (stuck at ~25 t/s), but switching to GPTQ-Int4 with MARLIN quantization on the MoE variant Qwen3.5-35B-A3B delivered dramatically better results:

1
vllm serve Qwen/Qwen3.5-35B-A3B-GPTQ-Int4 \
2
  --quantization gptq_marlin \
3
  --dtype bfloat16 \
4
  --kv-cache-dtype fp8 \
5
  --gpu-memory-utilization 0.9 \
6
  --max-model-len 32768 \
7
  --max-num-batched-tokens 2096 \
8
  --max-num-seqs 2 \
9
  --enable-prefix-caching \
10
  --reasoning-parser qwen3 \
11
  --enable-auto-tool-choice \
12
  --tool-call-parser qwen3_coder \
13
  --trust-remote-code \
14
  --port 5000 \
15
  --served-model-name qwen3

Result: 211 t/s on a single RTX 5090 with the GPTQ-MARLIN + FP8 KV cache config — compared to only 25 t/s with the host’s original FP8+MTP setup. The MoE architecture (35B total, ~3B active per token) means far less memory bandwidth per step than dense models, and MARLIN is specifically optimized for the RTX 50-series compute path.