GoModel: Two-Layer Caching with Redis and Semantic Vector Search

Cacheable Endpoints

Both layers apply to:

• /v1/chat/completions
• /v1/responses
• /v1/messages
• /v1/embeddings

Streaming requests are cached independently. On a streaming miss, the raw SSE bytes are stored and replayed verbatim on a hit.

How It Works

The exact cache hashes the full request using SHA-256:

1
SHA-256(path + NUL + workflow.mode + NUL + workflow.providerType + NUL + workflow.resolvedModel + NUL + normalized body)

The normalized body ensures that stream_options is canonicalized — if include_usage is false or nil, the field is dropped entirely, so two semantically identical requests with different JSON formatting still hit the same key.

The hash is stored in Redis with a configurable TTL. On a hit, the stored response is replayed with X-Cache: HIT (exact).

Enabling with Redis

Via YAML (config.yaml):

1
cache:
2
  response:
3
    simple:
4
      enabled: true
5
      redis:
6
        url: "redis://localhost:6379"
7
        key: "gomodel:response:"
8
        ttl: 3600  # 1 hour

Via environment variables (no YAML needed):

1
export RESPONSE_CACHE_SIMPLE_ENABLED=true
2
export REDIS_URL="redis://localhost:6379"
3
export REDIS_KEY_RESPONSES="gomodel:response:"
4
export REDIS_TTL_RESPONSES=3600

Enabling with Valkey

Valkey is the open-source fork of Redis created after Redis changed its license to SSPL/BSL in 2024. It is 100% wire-compatible with Redis 7.x — the GoModel Redis client (go-redis/v9) works without modification.

1
# Docker: swap redis for valkey
2
docker run -d --name valkey -p 6379:6379 valkey/valkey:latest
3

4
# Point GoModel at it — same REDIS_URL format
5
export REDIS_URL="redis://localhost:6379"

The redis:// scheme works because Valkey speaks the same RESP protocol. No code changes needed.

For a full self-hosted stack with Valkey, PostgreSQL, and GoModel:

1
#!/usr/bin/env bash
2
set -euo pipefail
3

4
# Start Valkey (Redis-compatible)
5
docker run -d --name valkey \
6
  -p 6379:6379 \
7
  valkey/valkey:latest \
8
  valkey-server --appendonly yes
9

10
# Wait for Valkey to be ready
11
until docker exec valkey valkey-cli ping | grep -q PONG; do sleep 1; done
12

13
# Start GoModel with exact cache
14
docker run -d --name gomodel \
15
  -p 8080:8080 \
16
  --env-file .env \
17
  -e REDIS_URL=redis://valkey:6379 \
18
  -e RESPONSE_CACHE_SIMPLE_ENABLED=true \
19
  -e REDIS_TTL_RESPONSES=3600 \
20
  enterpilot/gomodel

Redis vs Valkey — What’s the Difference?

For GoModel’s use case — simple key-value storage with TTL — there is zero difference. Pick Valkey if you want to avoid the SSPL/BSL license; pick Redis if you’re already running it for other services.

How It Works

The semantic cache embeds the last user message (GPTCache-style) and performs a KNN vector search. Semantically equivalent queries — “What’s the capital of France?” vs. “Which city is France’s capital?” — can return the same cached response without hitting the LLM.

Key design decisions:

1. Only the last user message is embedded — full conversation embedding was rejected as noisy and expensive.
2. Parameter isolation via params_hash — SHA-256 of model, temperature, top_p, max_tokens, tools, response_format, stream, endpointPath, guardrails_hash, and embedder_identity. All KNN searches filter by this hash, preventing wrong-format or wrong-parameter responses from being served.
3. Conversation history threshold — skip semantic caching when non-system messages exceed max_conversation_messages (default: 3). Long multi-turn sessions have near-zero hit rates and high false-positive risk.
4. Default similarity threshold: 0.92 — conservative. Start here; tune down only after monitoring false positives.

Embedding

The embedder calls the OpenAI-compatible /v1/embeddings endpoint of a configured provider. There is no local/in-process embedder — you must have a provider with a working embeddings endpoint.

1
cache:
2
  response:
3
    semantic:
4
      embedder:
5
        provider: openai    # must match a key in the top-level `providers` map
6
        model: text-embedding-3-small  # optional; defaults vary by provider type

For Gemini, the default model is gemini-embedding-001. For OpenAI-compatible providers, the default is text-embedding-ada-002.

Vector Store Backends

GoModel supports four vector store backends behind a VecStore interface:

TTL is implemented via expires_at (unix seconds, 0 = no expiry) plus read-time filtering and a background cleanup tick (~1 hour).

Full Configuration — Qdrant Example

1
cache:
2
  response:
3
    simple:
4
      enabled: true
5
      redis:
6
        url: "redis://localhost:6379"
7
        ttl: 3600
8
    semantic:
9
      enabled: true
10
      similarity_threshold: 0.92
11
      ttl: 3600  # 1 hour
12
      max_conversation_messages: 3
13
      exclude_system_prompt: true
14
      embedder:
15
        provider: openai
16
        model: text-embedding-3-small
17
      vector_store:
18
        type: qdrant
19
        qdrant:
20
          url: "http://localhost:6333"
21
          collection: "gomodel_semantic"
22
          api_key: ""  # optional for local Qdrant

Full Configuration — pgvector Example

1
cache:
2
  response:
3
    simple:
4
      enabled: true
5
      redis:
6
        url: "redis://localhost:6379"
7
        ttl: 3600
8
    semantic:
9
      enabled: true
10
      similarity_threshold: 0.92
11
      ttl: 3600
12
      max_conversation_messages: 3
13
      exclude_system_prompt: true
14
      embedder:
15
        provider: openai
16
        model: text-embedding-3-small
17
      vector_store:
18
        type: pgvector
19
        pgvector:
20
          url: "postgres://user:pass@localhost:5432/gomodel"
21
          table: "gomodel_semantic_cache"
22
          dimension: 1536  # must match embedding model output

Environment-Only Configuration

You can enable semantic cache without YAML using environment variables:

1
export SEMANTIC_CACHE_ENABLED=true
2
export SEMANTIC_CACHE_THRESHOLD=0.92
3
export SEMANTIC_CACHE_TTL=3600
4
export SEMANTIC_CACHE_MAX_CONV_MESSAGES=3
5
export SEMANTIC_CACHE_EXCLUDE_SYSTEM_PROMPT=true
6
export SEMANTIC_CACHE_EMBEDDER_PROVIDER=openai
7
export SEMANTIC_CACHE_EMBEDDER_MODEL=text-embedding-3-small
8
export SEMANTIC_CACHE_VECTOR_STORE_TYPE=qdrant
9
export SEMANTIC_CACHE_QDRANT_URL=http://localhost:6333
10
export SEMANTIC_CACHE_QDRANT_COLLECTION=gomodel_semantic

Bypass Both Layers

1
# Skip all caching for this request
2
curl http://localhost:8080/v1/chat/completions \
3
  -H "Cache-Control: no-store" \
4
  -H "Content-Type: application/json" \
5
  -d '{
6
    "model": "gpt-5-chat-latest",
7
    "messages": [{"role": "user", "content": "Hello!"}]
8
  }'

Use Only Exact-Match

1
# Skip semantic layer, use exact only
2
curl http://localhost:8080/v1/chat/completions \
3
  -H "X-Cache-Type: exact" \
4
  -H "Content-Type: application/json" \
5
  -d '{
6
    "model": "gpt-5-chat-latest",
7
    "messages": [{"role": "user", "content": "Hello!"}]
8
  }'

Override Similarity Threshold Per-Request

1
# Lower threshold for this request (more permissive)
2
curl http://localhost:8080/v1/chat/completions \
3
  -H "X-Cache-Semantic-Threshold: 0.85" \
4
  -H "Content-Type: application/json" \
5
  -d '{
6
    "model": "gpt-5-chat-latest",
7
    "messages": [{"role": "user", "content": "Hello!"}]
8
  }'

Override TTL Per-Request

1
# Cache this response for 10 minutes only
2
curl http://localhost:8080/v1/chat/completions \
3
  -H "X-Cache-TTL: 600" \
4
  -H "Content-Type: application/json" \
5
  -d '{
6
    "model": "gpt-5-chat-latest",
7
    "messages": [{"role": "user", "content": "Hello!"}]
8
  }'

Exact-Match Key

1
hashRequest(path string, body []byte, plan *core.Workflow) string {
2
    h := sha256.New()
3
    h.Write([]byte(path))
4
    h.Write([]byte{0})
5
    if plan != nil {
6
        h.Write([]byte(plan.Mode))
7
        h.Write([]byte{0})
8
        h.Write([]byte(plan.ProviderType))
9
        h.Write([]byte{0})
10
        h.Write([]byte(plan.ResolvedQualifiedModel()))
11
        h.Write([]byte{0})
12
    }
13
    h.Write(cacheKeyRequestBody(path, body))
14
    return hex.EncodeToString(h.Sum(nil))
15
}

The key includes the resolved workflow (mode, provider type, resolved model), not just the raw request body. This means guardrails and workflows affect cache keys when they change the resolved workflow or the final body.

Semantic params_hash

The semantic cache uses a SHA-256 hash of output-shaping parameters to isolate entries:

1
computeParamsHash(body, endpointPath, plan, guardrailsHash, embedderIdentity) {
2
    // Includes: model, endpointPath, providerType, resolvedModel,
3
    // temperature, top_p, max_tokens, max_output_tokens, reasoning,
4
    // instructions, tools (sorted + xxhash64), response_format,
5
    // stream, stream_options, guardrails_hash, embedder_identity
6
}

When guardrail policy changes, the guardrails_hash changes and old cache entries become unreachable — no manual cache flush needed. Entries expire naturally via TTL.