# ADR-025: Self-Hosted Ollama Runs Alongside Cloud AI for Vision and Text Inference

## Status

Accepted, 2026-06-25.

## Status History

```yaml
status_history:
  - date: 2026-06-25
    status: Proposed
    changed_by: hkl
    reason: Formalising self-hosted AI inference strategy used across multiple proxy services
    changed_via: adr-kit (360lm)
  - date: 2026-06-25
    status: Accepted
    changed_by: hkl
    reason: Ollama running with llava:7B + gemma3:4b; used by dispatch-ai and ocr-proxy
    changed_via: adr-kit (360lm)
```

## Context

360lm uses AI inference for: dispatch document OCR (extract structured data from dispatch slips), address parsing (turn free-text addresses into structured fields), and expense bill OCR (extract amounts/merchant from photos). These tasks require vision models (for images) and text models (for parsing). Cloud AI APIs (Claude, Gemini, OpenAI) are accurate but have per-call costs and send data to external servers. Self-hosted models (via Ollama on the VPS) have zero per-call cost and keep data on-premise, but are slower and less accurate. The VPS has sufficient CPU/RAM to run 7B parameter models (llava:7B for vision, gemma3:4b for text) in acceptable time for background processing tasks.

## Decision

Ollama runs as a Docker container on the VPS with two models loaded:
- `llava:7B` — vision model for document/image OCR tasks
- `gemma3:4b` — text model for structured data extraction and address parsing

**Usage by service:**
- `dispatch-ai` (port 8767): Ollama is the **primary** inference engine for dispatch document OCR (llava) and address parsing (gemma3). Cloud AI is not used here — dispatch documents are internal operational data with no PII concern, and volume justifies avoiding per-call cost.
- `ocr-proxy` (port 8766): Gemini Flash 2.0 is primary (better accuracy for complex bills); Ollama llava is **fallback** when Gemini is unavailable or quota-exceeded.
- `excel-ai-sidecar` (port 8778): Claude (via claude CLI OAuth) is primary for Excel normalization; Ollama is not used here (reasoning quality required exceeds 7B model capability).

New AI features must choose the appropriate tier: Ollama for high-volume/low-stakes inference, cloud AI for complex reasoning or when accuracy is critical.

**Decision Maker:** hkl

## Alternatives Considered

- **Cloud AI only (Claude/Gemini/OpenAI for all tasks).** Rejected for dispatch-ai: dispatch volume is high (multiple slips per day); per-call cost accumulates; dispatch data is internal and should not leave the VPS unnecessarily; Ollama llava accuracy is sufficient for structured dispatch slip formats.
- **Self-hosted Ollama only (no cloud AI).** Rejected: Ollama 7B models are not accurate enough for complex tasks (multi-language handwritten bills, Excel column detection from arbitrary formats); cloud AI quality is required for those use cases.
- **Larger self-hosted models (13B, 70B).** Rejected: VPS RAM cannot run >7B models without swap, which causes unacceptable latency (>5 min per inference). Current VPS spec is the ceiling. // ponytail: upgrade trigger=VPS upgraded to 32GB+ RAM
- **Dedicated GPU inference server.** Rejected: cost-prohibitive for current scale; cold start time acceptable on CPU for background tasks. // ponytail: upgrade trigger=inference latency becomes user-visible bottleneck

## Consequences

**Positive:**
- Zero per-call cost for dispatch-ai and ocr-proxy fallback — Ollama is always running.
- Dispatch document data never leaves the VPS.
- Ollama warm response time ~30s for llava (cold: ~50s) — acceptable for background processing.
- Two-tier system provides both cost efficiency (Ollama) and accuracy (cloud) where needed.

**Negative / Trade-offs:**
- Ollama cold start (~50s for llava) means first inference after container restart is slow.
- 7B models have lower accuracy than cloud APIs for ambiguous/complex inputs.
- Two AI systems to configure, monitor, and keep running.
- Ollama container must be on `root_default` Docker network to be reachable by proxy containers as `http://ollama:11434`.

**Risks and mitigations:**
- Ollama OOM (out of memory) crash: mitigated by `restart: always`; llava:7B fits in VPS RAM with headroom.
- Model not loaded on first request: Ollama auto-loads model from disk on first call; acceptable latency on first daily use.
- Cloud AI quota exhausted with no Ollama fallback configured: mitigated by always configuring Ollama fallback in proxy services that use cloud AI as primary.

## Related Decisions

- ADR-019 (OCR strategy) — describes the split between Tesseract (client-side) and server-side proxy; this ADR explains the Ollama vs cloud AI split within the server-side proxies.
- ADR-017 (Traefik routing) — dispatch-ai and ocr-proxy containers use Traefik labels.
- ADR-010 (cross-schema proxy pattern) — AI proxy services follow the same proxy architecture.

## References

- `memory/infra_vps.md` — Ollama container config, models, network, timeout note
- `memory/dbt_dispatch.md` — dispatch-ai proxy (llava + gemma3, port 8767)
- `memory/dbt_infra.md` — OCR proxy dual-backend details (Gemini + Ollama)