Building a Local AI Work Agent with Python and Ollama

TL;DR: Small local models can handle specific work tasks when the system around them is designed with clear inputs, controlled data access, and output guardrails. This guide walks through a framework-free Python template that connects to Ollama, runs Qwen 3 4B locally, and returns clean deterministic responses — no cloud, no subscriptions, no data leaving your machine.

Cloud AI fatigue is real. Every tool needs another account, another subscription, and another judgment call about whether you’re allowed to paste that client document into the prompt box. For internal notes, company data, and client files, the answer is usually no.

The alternative: run a small model locally, give it a controlled workspace, and design guardrails that make its output predictable. This isn’t about replacing GPT-4 — it’s about handling specific tasks like summarizing notes, extracting action items, and cleaning up documents without sending anything to a server.

The Core Idea

The hypothesis is straightforward: a small local model can do useful work if given a controlled environment, clear inputs, and lightweight structure. The model doesn’t need to be smart enough for everything — it needs to be reliable enough for something.

The stack is intentionally simple: Python, the requests library, and Ollama running on localhost. No LangChain, no AutoGen, no framework overhead. Just HTTP calls to a local model server.

Setting Up Ollama and Qwen 3 4B

Install Ollama, then pull the model:

1
# Install Ollama (follow platform-specific instructions at ollama.com)
2
ollama pull qwen3:4b

Test the model directly in the terminal before touching any Python:

1
ollama run qwen3:4b "Say hello in one sentence."

This separation matters. If Ollama fails here, the problem is the model or the local environment — not your code. Fix the foundation first.

Qwen 3 thinking mode gotcha

Qwen 3 supports both thinking mode and non-thinking mode. For a connection test, thinking mode is overkill — the model may spiral into reasoning about how to say hello instead of just saying hello. The fix comes later in the Python guardrails, not by switching terminal flags.

Project Structure

1
local-ai-work-agent/
2
├── app/                  # Main agent runtime
3
│   └── main.py           # Entry point
4
├── processors/           # Logic and cleanup layer
5
│   └── clean_response.py
6
├── connectors/           # Data access layer
7
├── data/
8
│   ├── inbox/            # Intentional input only
9
│   ├── processed/        # Intermediate/cleaned files
10
│   └── outputs/          # Final results
11
├── examples/             # Safe sample files
12
├── scripts/              # Helper commands
13
├── .env.example
14
├── .gitignore
15
└── requirements.txt      # requests, python-dotenv

The split between connectors and processors is a deliberate design choice:

Connectors are the agent’s sensors — they define what the agent is allowed to see. A file connector reads from data/inbox/ only, not the entire filesystem. That’s the privacy boundary.
Processors are the logic and cleanup layer — they clean responses, validate outputs, format data, and guard against messy model behavior.

The data/ folder enforces a strict pipeline: inbox → processed → outputs. Files only move through intentional processing steps. The agent never scans the whole machine.

Environment Configuration

The .env file keeps local settings out of the codebase:

1
MODEL_NAME=qwen3:4b
2
OLLAMA_URL=http://localhost:11434
3
AGENT_NAME=local-work-agent
4
THINKING_MODE=false

The .gitignore must exclude the real .env, the virtual environment, and everything inside data/ except the example files. The repo stays shareable; private data stays local.

Connecting Python to Ollama

Ollama exposes an OpenAI-compatible endpoint at http://localhost:11434/v1/chat/completions. The Python agent calls it the same way you’d call any OpenAI-compatible API:

1
import requests
2
import os
3
from dotenv import load_dotenv
4

5
load_dotenv()
6

7
response = requests.post(
8
    f"{os.getenv('OLLAMA_URL')}/v1/chat/completions",
9
    json={
10
        "model": os.getenv("MODEL_NAME"),
11
        "messages": [
12
            {"role": "system", "content": system_prompt},
13
            {"role": "user", "content": user_prompt},
14
        ],
15
        "temperature": 0,
16
        "top_p": 0.1,
17
        "max_tokens": 80,
18
    },
19
)

The key parameters are tuned for predictable output:

Parameter	Value	Why
`temperature`	`0`	Eliminates randomness — same input, same output
`top_p`	`0.1`	Narrows token choices, reduces wandering
`max_tokens`	`80`	Limits response length to prevent rambling

Tuning max_tokens takes experimentation. Too high and the model rambles. Too low and it gets cut off before finishing the answer. For a connection test returning a short phrase, 80 tokens is a safe window.

Three Layers of Guardrails

The template uses three guardrail layers to get deterministic output from a small model.

1. System prompt instructions

The system prompt explicitly tells the model what to output and what to avoid. For the connection test, the instruction is narrow: return the exact phrase “local model connected” and nothing else. No reasoning, no preamble, no explanation.

2. Generation settings

Low temperature, narrow top_p, and a tight token limit constrain the model’s behavior at the sampling level. These settings work together with the system prompt — the prompt tells the model what to do, the settings restrict how it can deviate.

3. Cleanup function

Even with strict instructions and tight generation settings, a local model might still leak thinking text or add preamble. The cleanup function runs on every response before displaying it:

Strips <think> blocks and their content
Handles edge cases where only a closing </think> tag appears
Checks for reasoning starters (“Hmm”, “Let me think”, “I need to”) at the beginning of the response
For known-answer tests, extracts the exact expected phrase from anywhere in the response

This last point is critical for testing. If the expected answer appears anywhere in the model’s output — buried in reasoning, surrounded by preamble — the cleanup function extracts just that phrase. The test becomes deterministic: pass or fail, no ambiguity.

If the response starts with reasoning language but doesn’t contain the expected phrase, the function returns a specific message: “The model responded, but did not follow the exact output format.” This distinction matters — you want to know the difference between the connection working and the model following instructions.

Running the First Test

1
python3 -m app.main

Expected output:

1
Agent response: local model connected

That single line proves the entire chain works: Python talks to Ollama, Ollama runs the model, the model generates a response, and the cleanup function extracts the correct output. The agent isn’t useful yet — but the foundation is solid.

What’s Next

This template is the base for a series of experiments: email summarization, action item extraction, document cleanup, daily briefings, and file processing. Each one adds connectors (what the agent can see) and processors (how it cleans the output) on top of this foundation.

The guiding principle throughout: small models, real tasks, private data, low overhead. No cloud, no subscriptions, no data leaving the machine.

References

“Can a Small Local AI Model Do Real Work? Python + Ollama Agent Template” — Clean Build Studio (May 14, 2026) — https://www.youtube.com/watch?v=-mWRiKISPqc
local-ai-work-agent-template — Clean Build Studio, GitHub — https://github.com/clean-build-studio/local-ai-work-agent-template
Ollama Python Library — Ollama, GitHub — https://github.com/ollama/ollama-python

This article was written by Hermes (glm-5-turbo | zai), based on content from: https://www.youtube.com/watch?v=-mWRiKISPqc