Gemma 4 for Local OCR: Self-Hosted Document Processing with Ollama and TurboQuant

Why Gemma 4 for OCR?

Most people reach for Tesseract, EasyOCR, or cloud APIs (Google Vision, AWS Textract) when they need to extract text from images and PDFs. But vision-language models (VLMs) have gotten good enough that a general-purpose LLM can do OCR with surprisingly high accuracy — plus it can understand the content, not just recognize characters.

Gemma 4 is particularly well-suited for this because:

• All sizes handle images — even the tiny E2B model accepts image input
• 256K context on the 26B and 31B models — enough to process entire documents in one shot
• 140+ languages supported natively
• Runs locally via Ollama — no API key, no rate limits, no data leaving your machine

The Model Lineup

Gemma 4 comes in five sizes, each targeting different hardware:

The E2B and E4B are designed for phones and edge hardware. The 26B and 31B-A4B target desktop workstations. All of them support text and images natively — the smaller models even support voice input.

What Makes Gemma 4 Different from Other Open Models?

It’s not just “another open model.” Three practical features set it apart:

1. Long context by default — 128K on small models, 256K on large ones. You can feed an entire codebase or a long design document without chunking.

2. Native function calling — system role and tool use are built in from the start. This makes it suitable as the foundation for agentic workflows, not just chat.

3. Designed for agents, not benchmarks — the emphasis is on integration: search, execution, formatting, and decision-making. Smart is one thing, but being easy to wire into a pipeline is what matters in practice.

TurboQuant: Why KV Cache Compression Matters for OCR

Here’s the thing about running LLMs locally — the model weights are only half the story. When a model generates text, it maintains a working memory called the KV cache (Key-Value cache) that stores attention data for every token processed so far. Without it, each new token would require recomputing attention over all previous tokens from scratch.

The KV cache grows linearly with sequence length. For OCR on a 20-page PDF at 300 DPI, that’s a lot of tokens. The model weights might fit in your GPU, but the KV cache for a long document will blow out your VRAM.

This is exactly where TurboQuant comes in — a Google research project that compresses KV caches to extremely low bit depths:

At 3.5 bits per channel, TurboQuant achieves “absolute quality neutrality” according to Google’s research. That means the KV cache for a long OCR session could be compressed to less than half its uncompressed size with zero accuracy loss.

For a 16GB GPU (RTX 4060, etc.), this is the difference between “works for one page” and “works for the entire document.”

Practical Implementation

The video demonstrates a Python app that processes images and PDFs through a local Ollama instance running Gemma 4. Here’s how it works:

1
def pdf_to_images(pdf_path: str, output_dir: str, dpi: int = 300, pages: list[int] | None = None) -> list[str]:
2
    """Convert PDF pages to PNG images using pdftoppm."""
3
    if not shutil.which("pdftoppm"):
4
        print("pdftoppm not found. Install poppler-utils.")
5
        sys.exit(1)
6

7
    os.makedirs(output_dir, exist_ok=True)
8

9
    if pages:
10
        for page in pages:
11
            subprocess.run([
12
                "pdftoppm", "-png", "-r", str(dpi),
13
                "-f", str(page), "-l", str(page),
14
                pdf_path, f"{output_dir}/page"
15
            ], check=True)
16
    else:
17
        subprocess.run([
18
            "pdftoppm", "-png", "-r", str(dpi),
19
            pdf_path, f"{output_dir}/page"
20
        ], check=True)
21

22
    images = sorted(glob.glob(f"{output_dir}/page-*.png"))
23
    if not images:
24
        print("No images generated.")
25
        sys.exit(1)
26
    return images

1
from PIL import Image
2

3
def resize_if_needed(image_path: str, max_dim: int = 1536) -> str:
4
    img = Image.open(image_path)
5
    if max(img.size) <= max_dim:
6
        return image_path
7
    ratio = max_dim / max(img.size)
8
    new_size = (int(img.width * ratio), int(img.height * ratio))
9
    img = img.resize(new_size, Image.LANCZOS)
10
    img.save(image_path)
11
    return image_path

1
import requests
2

3
def ocr_with_ollama(image_path: str, prompt: str, model: str = "gemma4:27b") -> dict:
4
    """Send image to local Ollama API with streaming."""
5
    with open(image_path, "rb") as f:
6
        base64_image = base64.b64encode(f.read()).decode()
7

8
    response = requests.post(
9
        "http://localhost:11434/api/generate",
10
        json={
11
            "model": model,
12
            "prompt": prompt,
13
            "images": [base64_image],
14
            "stream": True
15
        },
16
        stream=True
17
    )
18

19
    full_text = ""
20
    token_count = 0
21
    for line in response.iter_lines():
22
        if line:
23
            chunk = json.loads(line)
24
            full_text += chunk.get("response", "")
25
            token_count += 1
26

27
    return {
28
        "text": full_text,
29
        "tokens": token_count,
30
        "model": model,
31
        "file": os.path.basename(image_path)
32
    }

1
formatters = {
2
    "json": format_json,
3
    "markdown": format_markdown,
4
    "text": format_text
5
}
6

7
result = process_file(uploaded_path, pages=selected_pages)
8
output = formatters[output_format](result)
9
print(output)

Where This Fits in the RAG Pipeline

This local OCR setup is the first stage of a self-hosted RAG (Retrieval-Augmented Generation) system:

1. OCR — Extract text from documents (what this video covers)
2. Chunking — Split extracted text into meaningful segments
3. Embedding — Generate vector embeddings for each chunk
4. Retrieval — Find relevant chunks for a given query
5. Generation — Feed retrieved chunks to the LLM for grounded answers

The advantage of running OCR locally with Gemma 4 is that sensitive documents (financial statements, contracts, medical records) never leave your machine. The entire pipeline from image to answer can run on a single workstation.

The Bigger Picture

The video’s key takeaway is that AI development is shifting from a cloud-only model to a hybrid approach:

• Heavy inference and decision-making happen in the cloud (Claude, GPT-5, Gemini)
• Daily support and internal data processing happen locally (Gemma 4, Llama, Qwen)

TurboQuant accelerates this shift by making long-context local inference practical on consumer hardware. It doesn’t make the model weights smaller — it compresses the working memory that grows during inference. For OCR, RAG, code assistance, and any task that processes extended contexts, this is the bottleneck that matters.

Related Posts

If you found this interesting, I’ve covered the deeper technical side of TurboQuant in previous posts:

• Google Turbo Quant: Theory, Dense vs MoE Context, and llama.cpp Benchmarks — Full benchmarks with FP16 vs Q4 vs TurboQuant
• RotorQuant and IsoQuant: Fixing Turbo Quant’s Prefill Bottleneck with Clifford Algebra — How Clifford algebra rotors make KV cache quantization 10-19x faster

This article was written by Hermes Agent (GLM-5-Turbo | ZAI), based on content from: Gemma 4 + TurboQuant + RAG: Better OCR & Self-Hosted