# ADR-073: Traefik Docker Label Conventions and Middleware Chaining

## Status

Accepted, 2026-06-27.

## Status History

```yaml
status_history:
  - date: 2026-06-27
    status: Proposed
    changed_by: hkl
    reason: Formalising Traefik label conventions and middleware patterns in use across all services
    changed_via: adr-kit (360lm)
  - date: 2026-06-27
    status: Accepted
    changed_by: hkl
    reason: Label-based routing live for 40+ services; dual cert resolvers (mytlschallenge + zerossl) active
    changed_via: adr-kit (360lm)
```

## Context

The 360lm platform uses **Traefik** as the reverse proxy for all services on the VPS (`srv1111289.hstgr.cloud`). Routing configuration is entirely **Docker-label-based** — no static Traefik config files are written for individual services (see ADR-017).

Currently deployed services include:
- **360lm-web** (Apache httpd) — root domain + all static PWAs
- **Dispatch PWA** (nginx) — `/dispatch` with stripprefix middleware
- **Track Proxy** (Python) — `/track-proxy`
- **OCR Proxy** (Python) — `/ocr-proxy`
- **Dispatch AI** (Python) — `/dispatch-ai`
- **Paperclip** (Docker container) — subdomain `paperclip.srv1111289.hstgr.cloud`
- **OpenClaw** (nginx proxy to systemd) — subdomain `openclaw.srv1111289.hstgr.cloud`

### Certificate Resolver Strategy

Two cert resolvers are configured in Traefik:

1. **`mytlschallenge`** — Let's Encrypt HTTP challenge for the primary domain `srv1111289.hstgr.cloud` and all subdomains under it.
   - Used for: 360lm-web (main platform), Dispatch, Track, OCR, Dispatch AI
   - Rate limit: Let's Encrypt shares cert quotas across all domains on `hstgr.cloud` (a shared Hostinger domain)
   - Hit limit once during Paperclip onboarding (2026-05-05); Traefik auto-retried cert issuance successfully

2. **`zerossl`** — ZeroSSL as a backup CA for external/AI services not in the main platform domain.
   - Used for: Paperclip, OpenClaw
   - Reason: Avoids exhausting Let's Encrypt quota for the primary domain when external services need renewal
   - Both services are non-user-facing (internal platform tools), so using a secondary CA is acceptable

### Middleware: stripprefix

When a PathPrefix must be removed before forwarding to the backend service (e.g., `/dispatch/` → service sees `/`), the `stripprefix` middleware is applied. This is necessary for Nginx and Python services that do not understand path prefixes in their routing.

Example: `dispatch-pwa` container serves Dispatch PWA at `/dispatch` but has its own routing logic for `/` (expecting root paths like `/index.html`).

### Network Topology

All services are on the external Docker network `traefik-net`, which maps to the compose network `root_default`. This is the shared network that Traefik auto-discovers for label-based routing.

## Decision

### Label-Based Routing Only

All new services **must** use Docker labels for Traefik routing. No static Traefik config files for individual services will be created or maintained.

**Rationale:** Labels are co-located with the service definition (docker-compose.yml), making routing logic discoverable and maintainable alongside the service itself. Traefik auto-discovers and reloads on container restart.

### Standard Label Template

For a new service, use this label set as a baseline:

```yaml
labels:
  - "traefik.enable=true"
  - "traefik.http.routers.<service_name>.rule=Host(`${WEB_DOMAIN}`) && PathPrefix(`/<path>`)"
  - "traefik.http.routers.<service_name>.entrypoints=websecure"
  - "traefik.http.routers.<service_name>.tls=true"
  - "traefik.http.routers.<service_name>.tls.certresolver=mytlschallenge"
  - "traefik.http.services.<service_name>.loadbalancer.server.port=<PORT>"
```

### Cert Resolver Selection

- **`mytlschallenge` (default):** All platform services under the primary domain `srv1111289.hstgr.cloud`
  ```yaml
  - "traefik.http.routers.<name>.tls.certresolver=mytlschallenge"
  ```

- **`zerossl` (for external/AI services):** New external services or sidecar services not directly part of the 360lm platform core:
  ```yaml
  - "traefik.http.routers.<name>.tls.certresolver=zerossl"
  ```

**When to use ZeroSSL:** If adding a new service that is external-facing, AI-oriented (like Paperclip), or not part of the employee-facing PWA ecosystem, use `zerossl` to avoid exhausting the Let's Encrypt quota for the primary domain.

### Middleware: stripprefix

Use the `stripprefix` middleware when the backend service does not understand or route based on the PathPrefix:

```yaml
labels:
  - "traefik.enable=true"
  - "traefik.http.routers.<name>.rule=Host(`${WEB_DOMAIN}`) && PathPrefix(`/<path>`)"
  - "traefik.http.routers.<name>.entrypoints=websecure"
  - "traefik.http.routers.<name>.tls=true"
  - "traefik.http.routers.<name>.tls.certresolver=mytlschallenge"
  - "traefik.http.routers.<name>.middlewares=<name>-stripprefix@docker"
  - "traefik.http.middlewares.<name>-stripprefix.stripprefix.prefixes=/<path>"
  - "traefik.http.services.<name>.loadbalancer.server.port=<PORT>"
```

**When to apply stripprefix:**
- Nginx services with internal routing (e.g., `dispatch-pwa` expects `/index.html`, not `/dispatch/index.html`)
- Python/Node services that do not understand path prefixes in their URL routing
- Static servers where the root path is hard-coded

**When NOT to apply stripprefix:**
- If the backend service understands and routes based on the full PathPrefix, do not strip it
- PostgREST, custom REST APIs that expect the path as-is
- Reverse proxies that explicitly handle path prefixes

### Shared Network

All services **must** be on the `traefik-net` network (external network mapping to `root_default`):

```yaml
networks:
  - traefik-net
```

This is the single shared network that Traefik monitors for Docker labels.

### Router Priority and Rule Ordering

Traefik automatically assigns priority based on rule complexity:
- **More specific rules (longer PathPrefix, more conditions) have higher priority**
- No manual priority assignment is needed unless two rules are equally specific (overlap)

Example:
- `PathPrefix(/dispatch/admin)` has higher priority than `PathPrefix(/dispatch)`
- `Host(example.com) && PathPrefix(/api)` has higher priority than `Host(example.com)`

If a rule overlap occurs (e.g., two routers with the same prefix and host), explicitly set priority:

```yaml
- "traefik.http.routers.<name>.priority=100"
```

Higher numbers take precedence.

## Implementation Notes

### Adding a New Service (Checklist)

1. **Define the service in docker-compose.yml** (`/root/360lm-web/docker-compose.yml` or a separate compose file like `/opt/<service>/docker-compose.yml`)

2. **Determine routing:**
   - What is the host? (e.g., `${WEB_DOMAIN}` for main domain, or a subdomain)
   - What is the PathPrefix? (e.g., `/my-service`, or empty if subdomain-only)
   - What port does the service listen on internally? (e.g., 3100 for Paperclip)

3. **Decide on cert resolver:**
   - Platform service → `mytlschallenge`
   - External/AI service → `zerossl`

4. **Check if stripprefix is needed:**
   - Does the backend route based on the full path (e.g., `/my-service/index.html`), or does it expect root paths (e.g., `/index.html`)?
   - If it expects root paths, apply stripprefix

5. **Add labels to the service definition**

6. **Test routing:**
   ```bash
   # Verify the service is on traefik-net
   docker network inspect root_default | grep <container_name>
   
   # Check Traefik logs for label parsing errors
   docker logs root-traefik-1 | grep -i "route\|middleware"
   
   # Test the route from the host
   curl https://srv1111289.hstgr.cloud/<path>
   ```

7. **Document the routing in this ADR or a service-specific README** (port, PathPrefix, cert resolver, stripprefix usage)

### Common Pitfalls

1. **Forgetting to add the service to `traefik-net` network**
   - Result: Traefik cannot discover the service; route never comes online
   - Fix: Add `networks: [traefik-net]` to the docker-compose labels section

2. **Applying stripprefix when the backend already understands the path**
   - Result: Routes return 404 (e.g., backend expects `/api/resource` but receives `/resource`)
   - Fix: Remove the stripprefix middleware

3. **Not stripping prefix when the backend expects root paths**
   - Result: Routes return 404 (e.g., backend expects `/index.html` but receives `/dispatch/index.html`)
   - Fix: Add stripprefix middleware

4. **Using `mytlschallenge` for a high-rate-renewal service**
   - Result: Let's Encrypt quota exhausted for the primary domain
   - Fix: Use `zerossl` for external or frequently-renewing services

5. **Reusing a router name across multiple services**
   - Result: Traefik treats them as one route; the last definition wins
   - Fix: Use unique router names per service (e.g., `traefik.http.routers.dispatch.rule=...` vs `traefik.http.routers.dispatchai.rule=...`)

### Monitoring and Debugging

**Check Traefik dashboard:**
```bash
# Traefik dashboard is at http://localhost:8080 (not exposed to internet)
# SSH into the VPS and port-forward to view it
ssh -L 8080:localhost:8080 hkl@srv1111289.hstgr.cloud
# Then visit http://localhost:8080 in your browser
```

**View Traefik logs for label parsing:**
```bash
docker logs root-traefik-1 | grep -E "route|middleware|label" | tail -50
```

**Check if a service is discoverable:**
```bash
docker inspect root-traefik-1 | grep -A 10 "Mounts" # See if config volume exists
```

## Alternatives Considered

### 1. Static Traefik Config Files (Rejected)

**Approach:** Define routes in static config files (e.g., `/etc/traefik/config.d/my-service.yml`).

**Why rejected:**
- Routing logic is decoupled from the service definition, making it harder to understand what networking a service requires
- Config files must be manually created, versioned, and deployed — increases deployment complexity
- Changes require editing two places: service definition + config file
- Error-prone: easy to forget updating one or the other

### 2. Traefik Operator / Kubernetes Ingress (Rejected)

**Approach:** Use Kubernetes Ingress or a Traefik operator for declarative routing.

**Why rejected:**
- 360lm runs on a single VPS, not a Kubernetes cluster
- Docker Compose is sufficient for the scale and complexity
- Operator complexity outweighs the benefit for a small deployment

### 3. Nginx as the Primary Reverse Proxy (Rejected)

**Approach:** Use Nginx instead of Traefik for routing.

**Why rejected:**
- Nginx requires manual config file editing for each route
- No built-in Docker label support; routing is not co-located with services
- Traefik's docker-label-based discovery is superior for dynamic service scaling
- Traefik's automatic cert renewal (ACME) simplifies TLS management

### 4. Single Cert Resolver (Rejected)

**Approach:** Use only Let's Encrypt for all services, including external ones.

**Why rejected:**
- Let's Encrypt enforces a rate limit on the domain and IP
- Hostinger's shared domain (`hstgr.cloud`) already shares the quota across all customers' services
- Adding high-renew-rate services (e.g., Paperclip, future AI agents) exhausts the quota faster
- ZeroSSL as a backup CA is a proven strategy to avoid downtime due to cert issuance delays

## Consequences

### Benefits

1. **Discoverable routing:** Service routing is defined inline with the service, making it easy to understand networking requirements
2. **Automated cert renewal:** Traefik handles ACME challenges and cert renewal without manual intervention
3. **Dual CA strategy avoids outages:** Backup cert resolver reduces risk of Let's Encrypt quota exhaustion blocking new service deployment
4. **Middleware composability:** stripprefix and other middlewares can be combined for complex routing scenarios
5. **Low operational overhead:** No manual config file management; Traefik auto-discovers and reloads on container restart

### Trade-Offs

1. **Label namespace is crowded:** Docker labels for Traefik routing can make `docker-compose.yml` verbose; consider breaking large services into separate compose files
   - **Mitigation:** Use a `labels:` anchor in YAML to reduce repetition:
     ```yaml
     x-traefik-defaults: &traefik-defaults
       - "traefik.enable=true"
       - "traefik.http.routers.myservice.entrypoints=websecure"
     labels:
       <<: *traefik-defaults
       - "traefik.http.routers.myservice.rule=Host(...)"
     ```

2. **Cert resolver choice is manual:** Developers must decide between `mytlschallenge` and `zerossl`; no automatic policy
   - **Mitigation:** Document the decision criteria clearly (as in this ADR). Code review checklist includes cert resolver check.

3. **Traefik auto-discovery lag:** On rare occasions (e.g., high container churn), Traefik can miss a label update until the container is restarted
   - **Mitigation:** Traefik restart is fast (<2s); restart manually if needed: `docker restart root-traefik-1`

### Risks & Mitigations

1. **Risk:** A service misconfigured with a wrong PathPrefix or missing stripprefix returns 404 to users.
   - **Mitigation:** Checklist in "Adding a New Service" section; test routes before merging to master.

2. **Risk:** Dual CA strategy may cause confusion if one CA's cert is revoked or expires.
   - **Mitigation:** Monitor cert expiration; set calendar reminders for renewal. Traefik logs will show cert issues clearly.

3. **Risk:** Let's Encrypt quota exhaustion if external services use `mytlschallenge` too aggressively.
   - **Mitigation:** Enforce `zerossl` for non-core services via code review (ADR-067: Cross-PWA Change Safety Gate).

## Related Decisions

- **ADR-015:** Dev and Prod Are Two Full Stacks on the Same VPS — defines the two-stack architecture that Traefik serves
- **ADR-017:** All Service Routing Uses Traefik + Docker Labels — the foundational routing decision
- **ADR-060:** Production Promotion Uses rsync Allowlist — defines how services are deployed and Traefik configs are reloaded
- **ADR-067:** Cross-PWA Change Safety Gate — emphasizes review for shared infrastructure changes (e.g., cert resolver policy)

## References

- **Traefik Documentation:** https://doc.traefik.io/traefik/
- **Traefik Docker Provider:** https://doc.traefik.io/traefik/providers/docker/
- **Traefik Middleware:** https://doc.traefik.io/traefik/middlewares/overview/
- **Let's Encrypt Rate Limits:** https://letsencrypt.org/docs/rate-limits/
- **ZeroSSL:** https://zerossl.com/
- **360lm docker-compose.yml:** `/root/360lm-web/docker-compose.yml`
- **Paperclip docker-compose.yml:** `/opt/paperclip/docker-compose.yml`
- **OpenClaw docker-compose.yml:** `/opt/openclaw/docker-compose.yml`

---

**Decision Maker:** hkl  
**Changed Via:** adr-kit (360lm)  
**Last Updated:** 2026-06-27