TL;DR: Small local LLMs (3B–12B parameters) can become capable coding agents when augmented with three layers: Context7 for up-to-date library documentation, RAG for project-specific code context, and agent memory frameworks (Mem0, Letta) for persistent knowledge across sessions.
Docker’s recent blog post by Philippe Charrière demonstrated a practical approach: a 3B-parameter Qwen2.5-Coder model, paired with RAG, produced working Go code for a custom library the model had never seen. The key insight — small models need targeted context, not raw scale.
This post explores how to build a complete local coding agent stack using small LLMs, augmented with three complementary systems.
The Problem: Small LLMs Lack Context
Small LLMs (3B–12B parameters) run locally on consumer hardware — even on a MacBook Air M4 or Raspberry Pi. But they have fundamental limitations:
- Outdated training data — A model trained in 2024 doesn’t know about libraries released in 2025
- No project knowledge — They can’t read your codebase or understand your conventions
- No memory — Each session starts fresh; nothing persists across conversations
The Docker article’s rules for small LLMs are essential:
- The larger the content provided, the less effective the model becomes — small models have limited context windows and struggle to focus
- The more conversation history you keep, the less effective the model becomes — context grows, diluting relevance
The solution: provide only the right information, at the right time.
Layer 1: Context7 — Up-to-Date Library Documentation
Context7 (by Upstash) solves the “outdated training data” problem. It’s an MCP tool that pulls version-specific documentation and code examples directly from source repositories.
How It Works
- Write your prompt naturally
- Add
use context7to your prompt - Context7 fetches current, version-specific docs and injects them into the prompt
No tab-switching, no hallucinated APIs, no outdated code. The LLM gets documentation that’s current at the moment of generation.
Key Capabilities
- Library-specific context — Pulls docs for the exact library version you’re using
- MCP integration — Works with Claude Code, Cursor, and any MCP-compatible client
llms.txtstandard — Uses thellms.txtspecification for machine-readable documentation- Private sources — Supports private repositories with authentication
For Local Agents
Context7 requires network access for fetching documentation — it’s not offline. But it’s lightweight: the MCP server runs locally and only queries the Context7 API when needed. For a local coding agent, this is the best of both worlds: local inference with fresh context.
Layer 2: RAG — Project-Specific Code Context
Retrieval-Augmented Generation (RAG) solves the “no project knowledge” problem. The Docker article demonstrates this with a simple but effective approach:
The Docker RAG Pipeline
Code snippets → Chunk → Embed → Vector Store → Similarity Search → Inject into Prompt- Chunk — Split project documentation into sections (markdown headings, code blocks)
- Embed — Convert each chunk to a vector using an embedding model (e.g.,
ai/embeddinggemma) - Store — Save vectors in a vector database (in-memory for small projects, ChromaDB or Qdrant for larger ones)
- Search — When a user asks a question, embed the question and find the most similar chunks
- Inject — Include only the relevant chunks in the prompt
Configuration That Matters
The Docker article’s key settings:
| Setting | Value | Purpose |
|---|---|---|
HISTORY_MESSAGES | 2 | Keep only last 2 conversation turns |
MAX_SIMILARITIES | 3 | Return at most 3 matching chunks |
COSINE_LIMIT | 0.45 | Minimum similarity threshold |
These values are critical for small models — they keep the prompt small and focused.
Local RAG Tools
Several tools make local RAG easy:
- Minima — Fully local RAG with Ollama, supports MCP integration
- RAGFlow — Open-source RAG engine with deep document understanding
- Dify — LLM app platform with built-in RAG pipelines
- LangChain — Framework for building RAG with Ollama embeddings
Layer 3: Agent Memory — Persistent Knowledge
Agent memory frameworks solve the “no memory” problem. They let small LLMs retain knowledge across sessions — user preferences, project decisions, learned patterns.
Mem0 — Multi-Level Memory
Mem0 is a memory layer for AI agents with three levels: User, Session, and Agent state.
- Multi-signal retrieval — Semantic, BM25 keyword, and entity matching scored in parallel
- Entity linking — Entities extracted and linked across memories for retrieval boosting
- Temporal reasoning — Time-aware retrieval for queries about current state vs. past events
- Self-hostable — Docker Compose setup for fully local operation
# Self-host Mem0cd server && docker compose up -dLetta (formerly MemGPT) — Advanced Memory with Skills
Letta provides stateful agents with memory blocks, skills, and subagents.
- Memory blocks — Labeled memory (persona, human, project context)
- Skills — Pre-built capabilities for memory and continual learning
- Subagents — Delegate tasks to specialized agents
- CLI tool — Run agents locally in your terminal
npm install -g @letta-ai/letta-codeletta # launch agent with memoryZep — Graph RAG for Context
Zep is an end-to-end context engineering platform with Graph RAG.
- Knowledge graph — Automatically extracts relationships from context
- Temporal awareness — Understands how context evolves over time
- Sub-200ms latency — Pre-formatted context blocks optimized for LLMs
- Multi-language SDKs — Python, TypeScript, Go
The Complete Stack: Architecture
Here’s how all three layers work together for a local coding agent:
Model Selection Guide
| Model | Size | VRAM | Context | Best For |
|---|---|---|---|---|
| Qwen3-Coder 8B | 5.2 GB | ~6 GB | 40K | Coding (default choice) |
| Qwen3-Coder 30B | 19 GB | ~20 GB | 256K | Complex coding |
| Gemma3 12B | 8.1 GB | ~9 GB | 128K | General + vision |
| DeepSeek-R1 14B | 9.0 GB | ~10 GB | 128K | Reasoning + coding |
| Gemma3 4B | 3.3 GB | ~4 GB | 128K | Edge / low VRAM |
Practical Implementation
Minimal Docker Compose Setup
services: coding-agent: build: . environment: MODEL: qwen3-coder:8b EMBEDDING_MODEL: nomic-embed-text HISTORY_MESSAGES: "2" MAX_SIMILARITIES: "3" COSINE_LIMIT: "0.45" volumes: - ./project:/app/project - .mem0:/app/memory ports: - "11434:11434" # Ollama
models: chat-model: model: hf.co/qwen/qwen3-coder-8b-instruct-gguf:q4_k_m embedding-model: model: nomic-embed-textPrompt Construction
The final prompt for a small model should contain:
- System instructions — Role and constraints (keep it brief)
- Context7 docs — Fresh library documentation (only relevant APIs)
- RAG chunks — Project-specific code (max 3 most similar)
- Memory context — Past decisions and preferences (last 2 turns)
- User query — The actual request
Total prompt should stay under ~4000 tokens for 3B models, ~8000 tokens for 8B models.
What Each Layer Solves
| Layer | Problem Solved | Offline? | Complexity |
|---|---|---|---|
| Context7 | Outdated library knowledge | No (needs API) | Low — MCP drop-in |
| RAG | No project knowledge | Yes | Medium — chunking + embeddings |
| Agent Memory | No session persistence | Yes | Medium — memory framework |
Conclusion: Small Models, Big Capability
The Docker article proved that a 3B model with RAG can write correct code for a library it’s never seen. The pattern generalizes:
- Context7 gives small models the same up-to-date knowledge as large models
- RAG gives them project-specific context without bloating the prompt
- Agent Memory gives them the continuity that makes them feel like a real collaborator
The constraint is not model size — it’s context quality. Small models with the right context outperform large models with the wrong context.
References
- Making (Very) Small LLMs Smarter — Philippe Charrière, Docker Blog (January 16, 2026) — https://www.docker.com/blog/making-small-llms-smarter/
- Context7 — Up-to-date documentation for LLMs — Upstash — https://context7.com
- Minima — On-premises RAG with Ollama and MCP — https://github.com/dmayboroda/minima
- RAGFlow — Open-source RAG engine — https://github.com/infiniflow/ragflow
- Dify — LLM app development platform — https://github.com/langgenius/dify
- LangChain Local RAG Tutorial — https://js.langchain.com/docs/tutorials/local_rag/
- Mem0 — The Memory Layer for Personalized AI — https://github.com/mem0ai/mem0
- Letta (formerly MemGPT) — Advanced Memory Agents — https://github.com/letta-ai/letta
- Zep — End-to-End Context Engineering Platform — https://github.com/getzep/zep
- Nova — Golang Generative AI Agent Library — https://github.com/SnipWise/nova
- Qwen3-Coder on Ollama — https://ollama.com/library/qwen3-coder
- Gemma3 on Ollama — https://ollama.com/library/gemma3
- DeepSeek-R1 on Ollama — https://ollama.com/library/deepseek-r1
This article was written by Pi (Qwopus3.6-27B-NVFP4 | pc-eyay).
