Build a Self-Hosted Multimodal RAG Agent with Docling, n8n, and Ollama

Standard Pipeline

Uses specialized non-generative models — layout analysis, table structure recognition, OCR — to extract content verbatim. No hallucinations because there is no generative step. The output is an exact copy of what is in the document, preserving semantic structure (headers, tables, bullet points, diagrams as images with searchable text).

VLM Pipeline

Breaks documents into pages and batch-processes each through a Vision Language Model (like Granite Docling, SmolDocling, or Qwen VL). The VLM extracts text into a structured format. More flexible with complex layouts but introduces potential hallucinations since it uses generative AI.

For most use cases, the standard pipeline is the right starting point — verbatim extraction with zero hallucination risk. The VLM pipeline is worth exploring for documents with complex visual layouts where the standard pipeline’s OCR struggles.

Start the stack

For Nvidia GPU:

1
docker compose --profile gpu-nvidia up -d

For AMD GPU (Linux):

1
docker compose --profile gpu-amd up -d

For CPU only (Mac, no GPU, or testing):

1
docker compose --profile cpu up

The --profile cpu flag is required to start Docling. Without it, only the core services (n8n, PostgreSQL, Qdrant) start.

Service endpoints

First-time n8n setup: open http://localhost:5678, create an owner account, then go to Settings and enter an activation key (free, everything stays local) to unlock features like pinning workflow executions.

Critical Docker networking concept

Containers are isolated. They cannot see each other’s localhost. When n8n (in a container) needs to call Docling (in another container), it must use the Docker service name, not localhost:

• Wrong: http://localhost:5001 — searches inside the n8n container only
• Right: http://docling:5001 — routes through the Docker network

This applies to all inter-service communication: Ollama is http://ollama:11434, Qdrant is http://qdrant:6333, and so on. Only you (on the host machine) use localhost.

Persistent volumes

The shared/ directory is bind-mounted into the n8n container at /data/shared. Files placed here survive container restarts. Inside n8n, reference this path as /data/shared/.... The starter kit also mounts volumes for n8n workflows, Ollama models, Qdrant data, and Postgres.

Step 1: Create folder structure

Under the shared/ directory, create:

1
shared/
2
  rag-files/
3
    pending/      # Drop documents here for processing
4
    processed/    # Completed documents move here
5
  extracted-images/  # Images served by Nginx at port 8080

Step 2: Watch for new files

In n8n, create a new workflow and add a Local File Trigger node. Configure it to watch /data/shared/rag-files/pending. If the trigger does not detect files automatically, switch to polling mode in the node settings.

Step 3: Read the file

Add a Read/Write Files from Disk node. Set the file path from the trigger output (the binary file path).

Step 4: Process with Docling

Add an HTTP Request node to call Docling’s synchronous conversion endpoint:

• Method: POST
• URL: http://docling:5001/v1a/convert (note: docling, not localhost)
• Body: form-data with key files (type: n8n binary file, value: data)

Docling API parameters worth configuring:

• image_export_mode:
  • placeholder — replaces images with the text “image” (loses images)
  • referenced — saves extracted images to disk and returns filenames in the markdown

Use image_export_mode=referenced for multimodal RAG. The images are saved to the Docling scratch directory (configured via the Docker Compose working directory).

For large documents (100+ pages), use the async endpoint instead (/v1a/convert/async). It returns a task ID immediately, then poll /v1a/tasks/{task_id} until status is success, then fetch the result.

Step 5: Extract and move images

Add a Code node (JavaScript) to parse the Docling output and extract image filenames. Then use a Split Out node to iterate over each image. Finally, use an Execute Command node to move each image from the Docling scratch directory to the extracted-images folder:

1
mv /data/shared/docling-scratch/<filename> /data/shared/extracted-images/<filename>

Step 6: Inject full image URLs into markdown

The Docling output contains relative image paths. For the chat agent to display them, inject the full Nginx URL. Add another Code node with a regex replacement:

1
// Replace relative image paths with full Nginx URLs
2
const markdown = $input.first().json.markdown_content;
3
const updated = markdown.replace(
4
  /!\[(.*?)\]\((.*?)\)/g,
5
  '![$1](http://localhost:8080/$2)'
6
);
7
return [{ json: { markdown_content: updated } }];

Step 7: Store in Qdrant

Add a Qdrant Vector Store node configured to add documents:

1. Create a credential pointing to http://qdrant:6333 (no API key needed for local)
2. Create a collection in Qdrant dashboard:
   • Name: multimodal-rag
   • Dimensions: 768 (matches nomic-embed-text v1.5)
   • Distance metric: Cosine
3. Set the embedding model to Ollama → nomic-embed-text (pull it first: docker exec -it ollama ollama pull nomic-embed-text)
4. Use a recursive character text splitter with markdown separator, chunk size ~700

Step 8: Move processed files

Add an Execute Command node to move the source file from pending/ to processed/:

1
mv /data/shared/rag-files/pending/<filename> /data/shared/rag-files/processed/<filename>

Step 1: Add a chat trigger

Add a Chat Trigger node to the same workflow (or a separate one).

Step 2: Add an AI agent

Connect the chat trigger to an AI Agent node. Configure:

• Model: Ollama → llama3.2 (default, 3B params — small but functional for testing)
• Tool: Qdrant vector store → collection multimodal-rag, limit 5 results
• Embedding model: Ollama → nomic-embed-text (must match what was used for ingestion)
• System prompt:

1
Use the following pieces of context to answer the question at the end.
2
If you don't know the answer, just say that you don't know, don't try to make up an answer.
3
You must output images in markdown format using the URL provided in the retrieved results.

Step 3: Test

Click “Chat” on the chat trigger node and ask questions like “Show me the cabinet opening diagram” or “How do I use the ice and water dispenser?” The agent should query Qdrant, retrieve relevant chunks with image URLs, and render them inline.

Upgrading the model

Llama 3.2 (3B) works for testing but struggles with reliable instruction following. For production, pull a larger model:

1
# Inside the Ollama container
2
docker exec -it ollama ollama pull llama3.1:8b
3
# Or for 20B params (needs ~12GB VRAM):
4
docker exec -it ollama ollama pull qwen2.5:14b

For testing without local GPU, use Ollama Cloud (create an API key at ollama.com) or OpenRouter. Both provide access to larger open-source models. Configure a new Ollama credential in n8n pointing to the cloud endpoint, then select the model.