OpenRouter Fusion API: Fable-Level Intelligence from Model Ensembles — And How to Do It Locally

Two Tiers: Quality vs Budget

The budget tier is the real story — three cheap models plus one expensive judge produces within 1% of Fable quality at half the price².

The Analysis Layer Is Key

What makes Fusion genuinely useful isn’t just getting a “better” answer — it’s the analysis layer that shows you:

• Where models agree: High-confidence consensus across three independent reasoning paths
• Where they disagree: Divergent viewpoints reveal nuance and edge cases you’d miss from a single model
• Unique insights: Each model has different training data emphasis, so each surfaces something the others don’t
• Blind spots: Critical for complex topics — the models only answer what you ask them. The analysis layer identifies what they didn’t consider

This is especially valuable when making decisions based on AI output. A single model gives you one perspective. Three models with a judge give you consensus, disagreement, and hidden blind spots — all before you read the final answer.

Limitations: Long-Horizon Tasks

OpenRouter themselves acknowledge this: Fusion has only been evaluated on standard benchmarks (deep research) — not long-horizon tasks like multi-hour coding sessions, continuous browsing, or extended agentic workflows³. Fable 5’s standout ability was sustained focus across hours of work. Fusion doesn’t replicate that yet.

(a) Ensemble Before Inference

Route the query to the best-suited model before generating. Examples include RouteLLM, MetaLLM, and the “Blending Is All You Need” approach — which dynamically weights models based on predicted utility for each input.

(b) Ensemble During Inference

Combine models at token-level or span-level during generation. Token-level methods like GaC treat token generation as a classification problem across multiple models’ probability distributions. Span-level methods like SpecFuse have one model draft segments and another verify them — similar to speculative decoding but across different models.

(c) Ensemble After Inference

Generate complete responses from all models, then combine. This is where Fusion lives:

• Non-cascade: All models respond simultaneously; a judge synthesizes the result (LLM-Blender, FuseLLM, LLM-PeerReview)
• Cascade: Models are ordered by size/cost; cheaper ones attempt first, escalating to expensive models only if confidence is low (FrugalGPT, AutoMix)

OpenRouter Fusion is squarely in (c1) Non-cascade — all panel models respond simultaneously, then a judge model performs generative fusion.

Approach 1: Ollama + LLM-Blender (Easiest)

LLM-Blender is the foundational open-source tool for post-inference ensemble — exactly the category Fusion uses. It takes multiple model responses, ranks them pairwise using a judge model, then generates a fused output.

Setup:

1
# Start Ollama with multiple models running on different ports
2
ollama serve & sleep 3
3
ollama pull qwen3:8b
4
ollama pull gemma2:9b
5
ollama pull mistral-nemo:12b
6
ollama pull qwen3:32b  # judge model
7

8
# Clone and install LLM-Blender
9
git clone https://github.com/yuchenlin/LLM-Blender.git
10
cd llm-blender && pip install -e .

Run an ensemble:

1
from blender import Blender, ModelType
2

3
# Define panel models (budget tier)
4
panel = {
5
    "qwen3:8b": {"type": ModelType.Ollama},
6
    "gemma2:9b": {"type": ModelType.Ollama},
7
    "mistral-nemo:12b": {"type": ModelType.Ollama},
8
}
9

10
# Define judge model (quality tier)
11
judge = {
12
    "qwen3:32b": {"type": ModelType.Ollama},
13
}
14

15
blender = Blender(panel_models=panel, judge_model=judge)
16
response = blender.run("What is the best approach to building a RAG system for legal documents?")
17
print(response["fused"])

The key insight: use smaller models (8B–12B range) as your panel — they’re fast and cheap. Use a larger model (32B+) only as the judge, which runs once per query instead of generating full responses from scratch.

Approach 2: AutoMix + vLLM (Cascade with Confidence)

AutoMix takes a different approach — it dynamically mixes models based on their confidence signals. Instead of running all three and then judging, it uses model logits to determine which parts each model is confident about and blends at the token level.

This is more sophisticated but also more complex:

1
# Install AutoMix with vLLM backend
2
pip install automix-llm[vllm]
3

4
# Run multiple models via vLLM (requires GPU)
5
python -c "
6
from automix import AutoMix, ModelConfig
7

8
models = [
9
    ModelConfig('Qwen/Qwen3.6-8B', port=8001),
10
    ModelConfig('google/gemma-4-9b-it', port=8002),
11
]
12

13
automixer = AutoMix(models)
14
result = automixer.generate(
15
    prompt='Explain how speculative decoding works',
16
    max_tokens=512,
17
    blend_method='confidence_weighted'
18
)
19
print(result.text)
20
"

AutoMix is best when you want token-level blending rather than full-response fusion — it’s like the (b) category from the taxonomy. Each model contributes where it’s most confident, and the less-confident models yield to stronger ones.

Approach 3: Custom Pipeline with llama.cpp + Bash (Most Flexible)

For maximum control, build your own pipeline using llama.cpp instances communicating via HTTP servers:

1
#!/bin/bash
2
# local-fusion.sh — Self-hosted model ensemble
3

4
PROMPT=$1
5
JUDGE_MODEL="qwen3.6-27b-q5_k_m.gguf"
6

7
# Start three panel models on different ports (background)
8
llama-server -m qwen3.6-8b-q4_k_m.gguf --host 0.0.0.0 --port 8001 &
9
llama-server -m gemma-4-9b-it-q5_k_m.gguf --host 0.0.0.0 --port 8002 &
10
llama-server -m mistral-nemo-12b-instruct-q5_k_m.gguf --host 0.0.0.0 --port 8003 &
11

12
sleep 5
13

14
# Fan out prompt to all three models in parallel
15
curl -s http://localhost:8001/v1/chat/completions \
16
  -d "{\"model\":\"qwen\",\"messages\":[{\"role\":\"user\",\"content\":\"$PROMPT\"}]}" > /tmp/qwen.json &
17

18
curl -s http://localhost:8002/v1/chat/completions \
19
  -d "{\"model\":\"gemma\",\"messages\":[{\"role\":\"user\",\"content\":\"$PROMPT\"}]}" > /tmp/gemma.json &
20

21
curl -s http://localhost:8003/v1/chat/completions \
22
  -d "{\"model\":\"mistral\",\"messages\":[{\"role\":\"user\",\"content\":\"$PROMPT\"}]}" > /tmp/mistral.json &
23

24
wait
25

26
# Extract responses
27
QWEN=$(jq -r '.choices[0].message.content' /tmp/qwen.json)
28
GEMMA=$(jq -r '.choices[0].message.content' /tmp/gemma.json)
29
MISTRAL=$(jq -r '.choices[0].message.content' /tmp/mistral.json)
30

31
# Start judge model
32
llama-server -m "$JUDGE_MODEL" --host 0.0.0.0 --port 8004 &
33
sleep 5
34

35
# Send analysis prompt to judge
36
FUSION_PROMPT="Three models answered the same question. Analyze their responses for consensus, contradictions, unique insights, and blind spots. Then produce a fused final answer that incorporates all perspectives.
37

38
QUESTION: $PROMPT
39

40
MODEL 1 (Qwen3):
41
$QWEN
42

43
MODEL 2 (Gemma4):
44
$GEMMA
45

46
MODEL 3 (Mistral-Nemo):
47
$MISTRAL"
48

49
curl -s http://localhost:8004/v1/chat/completions \
50
  -d "{\"model\":\"qwen\",\"messages\":[{\"role\":\"user\",\"content\":\"$FUSION_PROMPT\"}]}" | jq -r '.choices[0].message.content'
51

52
# Cleanup
53
pkill -f llama-server

This is the self-hosted equivalent of OpenRouter Fusion — three panel models, one judge, same architecture. The advantage: zero API costs after the initial model downloads.

Hardware Requirements for Local Fusion