# ADR-069: Production Hotfix Protocol — Bugfix DB Template, Dev-Branch Fix, Deploy via Master

## Status

Accepted, 2026-06-27.

## Status History

```yaml
status_history:
  - date: 2026-06-27
    status: Proposed
    changed_by: hkl
    reason: Formalising hotfix workflow implemented in infra_env_separation.md
    changed_via: adr-kit (360lm)
  - date: 2026-06-27
    status: Accepted
    changed_by: hkl
    reason: Hotfix protocol live since 2026-06-23; safeguards against prod-only edits and cross-PWA bugs
    changed_via: adr-kit (360lm)
```

## Context

The 360lm platform runs two full stacks on the same VPS (ADR-015): dev (`dev.srv1111289.hstgr.cloud`, `lm360` DB, `/var/www/360lm/`) and prod (`srv1111289.hstgr.cloud`, `lm360_prod` DB, `/var/www/360lm-prod/`). The promotion path is one-way: dev → master branch → deploy-prod.sh.

When a production bug is discovered, a developer must never edit prod files directly (`/var/www/360lm-prod/`) or prod DB tables (`lm360_prod`). Direct edits:
- Create divergence between prod and dev code, making future merges and deploys fragile
- Bypass version control, making the bug fix impossible to audit or version
- Risk losing the fix if the developer forgets to backport it to the dev branch

Additionally, prod data and schema may differ from dev (e.g., prod has only the MVP PWA schemas: hub, expense, sales, custodian, activity; dev has all 18+ PWAs). A bug that appears in prod may not reproduce in dev if the bug is a cross-schema interaction (e.g., custodian → expense trigger, or sales → custodian FK dependency).

This ADR formalises the hotfix protocol: clone the prod DB as a temporary template, author the fix in a dev branch against that template, merge to master, and promote with deploy-prod.sh.

## Decision

A hotfix for a production bug follows this workflow:

1. **Never edit prod files or DB directly.** All changes go through git (dev branch) and are promoted via deploy-prod.sh.

2. **Create a temporary bugfix DB from the prod template.** This ensures the bug can be reproduced exactly as it appears in prod (with prod schema closure, prod data, and cross-PWA dependencies intact).

3. **Author the fix in a dev branch,** pointing a temporary PostgREST container at the bugfix DB for testing.

4. **Test the fix against the bugfix DB,** then merge the dev branch to master.

5. **Promote to prod via deploy-prod.sh,** which applies pending migrations and reloads PostgREST.

6. **Clean up the bugfix DB and temporary container** immediately after promotion to prevent resource leaks and confusion in future sessions.

**Decision Maker:** hkl

## Implementation Notes

### Step-by-Step Hotfix Workflow

#### Phase 1: Bugfix DB Setup (Dev Environment)

**Step 1.1: Clone prod DB as a temporary template**

```bash
# On the VPS, as a user with psql access (e.g., via docker exec postgres)
docker exec -i postgres psql -U lmadmin -d postgres << 'EOF'
CREATE DATABASE lm360_bugfix TEMPLATE lm360_prod;
EOF
```

This creates a snapshot of the prod schema and data. The bugfix DB is ephemeral; it is dropped after the fix is merged and deployed.

**Step 1.2: Start a temporary PostgREST container pointing at the bugfix DB**

```bash
docker run -d \
  --name postgrest-bugfix \
  --rm \
  -e PGRST_DB_URI="postgresql://lmadmin:PASSWORD@postgres:5432/lm360_bugfix" \
  -e PGRST_DB_SCHEMA="hub,expense,sales,custodian,activity" \
  -e PGRST_JWT_SECRET="SECRET" \
  -e PGRST_DB_ANON_ROLE="web_anon" \
  --network 360lm-network \
  postgrest:12.2.3
```

The container is named `postgrest-bugfix` so it is distinguishable from the prod and dev PostgREST instances. The `--rm` flag ensures the container is auto-deleted when stopped.

**Step 1.3: Wire the temporary PostgREST to a dev routing domain (optional)**

If the bug requires UI testing via the browser, add a Traefik label to the postgrest-bugfix container so it is routed on `bugfix.dev.srv1111289.hstgr.cloud/db`. This requires temporary docker-compose or docker run label injection; see infra_env_separation.md for Traefik label format.

If testing is only via direct RPC calls or Playwright, this step is optional.

---

#### Phase 2: Author and Test the Fix (Dev Code)

**Step 2.1: Create a hotfix branch from master**

```bash
cd /var/www/360lm
git checkout master
git pull origin master
git checkout -b hotfix/brief-description-of-bug
```

Branch naming convention: `hotfix/<2-3-word-bug-description>` (e.g., `hotfix/custodian-trigger-sync-race`, `hotfix/activity-null-response`).

**Step 2.2: Author the fix in code (HTML, CSS, JS, SQL migrations)**

- **If it's a UI/JS bug:** edit the relevant `/var/www/360lm/[pwa]/index.html` or `.js` files.
- **If it's a schema/DB bug:** create a new migration file in the appropriate PWA schema directory: `/var/www/360lm/[pwa]/migrations/migrate_[schema]_vX.sql`.

Migrations follow the naming convention: `migrate_<schema>_v<number>.sql` (e.g., `migrate_custodian_v011.sql`).

**Step 2.3: Test the fix against the bugfix DB**

For **DB/schema fixes:**
```bash
# Apply the migration to the bugfix DB
docker exec -i postgres psql -U lmadmin -d lm360_bugfix < /var/www/360lm/finance/custodian/migrations/migrate_custodian_vXXX.sql

# Verify the fix (e.g., test trigger logic, FK constraints)
docker exec -i postgres psql -U lmadmin -d lm360_bugfix << 'EOF'
-- Verify the fix: check table state, trigger behaviour, etc.
SELECT * FROM custodian.wallets WHERE id = X;
EOF
```

For **UI/code fixes:**
- Deploy the changed HTML/JS to `/var/www/360lm/` (already in dev environment).
- Point a browser or Playwright test to the dev environment and verify the fix.
- If the bug requires prod schema state (cross-schema FK check, trigger state), use the bugfix PostgREST container to serve RPC calls during UI testing.

**Step 2.4: Verify in Playwright (if applicable)**

```bash
cd /var/www/360lm
npx playwright test tests/[pwa].spec.js
```

All tests must pass before merging the hotfix branch.

---

#### Phase 3: Merge and Promote to Prod

**Step 3.1: Merge hotfix branch → master**

```bash
cd /var/www/360lm
git checkout master
git pull origin master
git merge --no-ff hotfix/brief-description-of-bug -m "Hotfix: brief description [Closes #XXX]"
git push origin master
```

The `--no-ff` (no fast-forward) merge keeps the hotfix branch history visible in the commit graph, aiding future audits.

**Step 3.2: Run deploy-prod.sh to promote**

```bash
/usr/local/bin/deploy-prod.sh --dry-run
# Review the output; if it looks correct:
/usr/local/bin/deploy-prod.sh
```

The script:
1. Pulls the latest master branch
2. Rsyncs MVP PWA files from dev to prod web root (ADR-060)
3. Applies any pending SQL migrations to `lm360_prod`
4. Reloads the PostgREST prod schema cache

If the hotfix includes SQL migrations, they are automatically applied to `lm360_prod` in this step.

**Step 3.3: Verify the fix in production**

- If it's a UI fix, test the PWA in production (`https://srv1111289.hstgr.cloud`).
- If it's a DB fix, query the prod DB to verify the migration applied: `docker exec -i postgres psql -U lmadmin -d lm360_prod -c "SELECT * FROM public.schema_migrations WHERE name LIKE '%vXXX';"`.

---

#### Phase 4: Cleanup

**Step 4.1: Stop and remove the temporary PostgREST container**

```bash
docker stop postgrest-bugfix
# --rm flag auto-deletes it; if not using --rm:
docker rm postgrest-bugfix
```

**Step 4.2: Drop the bugfix DB**

```bash
docker exec -i postgres psql -U lmadmin -d postgres << 'EOF'
DROP DATABASE lm360_bugfix;
EOF
```

**Step 4.3: Remove the hotfix branch**

```bash
cd /var/www/360lm
git branch -d hotfix/brief-description-of-bug
git push origin --delete hotfix/brief-description-of-bug
```

---

### Why Each Step Matters

**Bugfix DB Template (Step 1.1):** Prod data and schema may differ from dev. A cross-PWA bug (e.g., custodian trigger interacting with expense.employees) will only manifest with the full prod schema closure (hub, expense, sales, custodian, activity). The dev DB alone may have incomplete schemas or stale test data, causing the bug to not reproduce. The bugfix DB ensures the exact prod state is available for testing without touching prod directly.

**Temporary PostgREST Container (Step 1.2):** The fix is always authored in code (git branch) and tested against a separate DB instance. This keeps the code change and DB change in sync, and makes the fix auditable in version control. The temporary container serves the bugfix DB during development; it is discarded after the fix is merged, so no long-term infra burden.

**Hotfix Branch (Step 2.1):** Branching from master ensures the fix is based on what is currently in production, not on in-progress dev work. This reduces merge conflicts and ensures the fix can be cleanly merged and promoted without waiting for unrelated features.

**Merge with --no-ff (Step 3.1):** Fast-forward merges erase the hotfix branch from the commit history, making it hard to audit which commits were emergency fixes vs. planned development. A non-fast-forward merge keeps the branch visible.

**Deploy-prod.sh (Step 3.2):** The script handles rsync (ADR-060), migration application, and PostgREST reload atomically. Never run SQL migrations directly on lm360_prod — always route them through deploy-prod.sh so they are logged in schema_migrations and can be audited later.

**Cleanup (Phase 4):** The bugfix DB and temporary PostgREST container consume resources (disk space, memory, network ports). Leaving them behind can cause confusion in future hotfix sessions (e.g., "is lm360_bugfix a real DB or from a previous hotfix?") and pollutes the VPS. Cleanup is a single-line operation; it must always be done.

---

## Alternatives Considered

- **Edit prod files directly, then backport to dev.** Rejected: loses version control for the initial fix; makes audits and rollbacks impossible; the bugfix DB template approach ensures changes go through git first.

- **Test only in dev DB (lm360) without cloning prod schema.** Rejected: cross-PWA bugs (e.g., custodian → expense trigger, sales → custodian FK) require the full prod schema closure. Dev may have incomplete or stale schemas, causing bugs to appear fixed in dev but still broken in prod after deploy.

- **Hotfix directly on master branch without a feature branch.** Rejected: committing directly to master without testing risks pushing broken code to prod; a feature branch enforces testing and code review before merge.

- **Keep the bugfix DB and PostgREST container as permanent fixtures for testing.** Rejected: stale containers and databases confuse future hotfix sessions; they consume resources and add operational debt. A fresh bugfix DB per incident is simpler and prevents cross-contamination of test data.

- **Automated hotfix branching and promotion (e.g., GitHub Actions).** Rejected: hotfixes are infrequent (< 2/week); the manual workflow is simple and keeps the developer in control of testing and verification. Over-engineering with CI/CD is not justified. // ponytail: upgrade trigger = hotfix frequency > 2/week or team size > 3 developers

---

## Consequences

**Positive:**
- All hotfixes are versioned in git; no divergence between prod code and dev code.
- Bugs can be reproduced exactly (via the bugfix DB template with prod schema closure and data).
- The hotfix is tested against the exact prod state before being promoted.
- Migrations are applied atomically via deploy-prod.sh, ensuring lm360_prod stays in sync with code.
- Easy to audit: `git log --oneline --grep="Hotfix"` shows all emergency fixes.

**Negative / Trade-offs:**
- Hotfixes require an extra step (creating the bugfix DB) compared to direct prod edits, adding ~2–3 minutes to the workflow.
- If a developer forgets to clean up the bugfix DB, it will linger until manually dropped.
- Cross-schema bugs may require deeper understanding of the prod schema closure (hub, expense, sales, custodian, activity) to reproduce.

**Risks and mitigations:**
- **Hotfix DB created but not cleaned up:** Mitigated by explicit cleanup checklist in Phase 4. Memory.md notes should include a reminder to verify cleanup at session end.
- **Forgot to apply migration to bugfix DB during testing:** Mitigated by Step 2.3 instructions and the deploy-prod.sh log output (which will show pending migrations after promotion).
- **Hotfix branch diverges too far from master (long-lived):** Mitigated by keeping hotfix branches short-lived (< 1 day); if the fix takes > 1 day, merge master into the hotfix branch to avoid divergence.
- **Bugfix DB data is stale (doesn't match current prod state):** Mitigated by creating the bugfix DB immediately before starting the fix, not days in advance. If the fix takes > 1 day, consider recreating the bugfix DB from prod to capture any new data.

---

## Related Decisions

- **ADR-015** (Dev and Prod Are Two Full Stacks on the Same VPS) — the two-stack architecture that makes this hotfix protocol necessary.
- **ADR-060** (Production Promotion Uses rsync Allowlist) — deploy-prod.sh uses rsync to promote files; migrations are applied separately.
- **ADR-009** (Each PWA Owns a Dedicated PostgreSQL Schema) — hotfixes may span multiple PWA schemas; the bugfix DB includes the full prod schema closure.
- **ADR-040** (Cross-Schema Active-State Sync Uses SECURITY DEFINER Trigger) — hotfixes that touch cross-schema triggers must preserve SECURITY DEFINER declarations.

---

## References

- `memory/infra_env_separation.md` — full dev/prod environment details, two-stack architecture
- `/usr/local/bin/deploy-prod.sh` — production promotion script; Step 3.2 references
- `/var/www/360lm/docs/adr/ADR-015-dev-prod-two-stacks-same-vps.md` — two-stack context
- `/var/www/360lm/docs/adr/ADR-060-prod-deploy-rsync-allowlist-not-full-checkout.md` — rsync allowlist and deploy mechanism
- `docker-compose.yml` (location: `/root/360lm-web/`) — Traefik labels and container configs for reference (Step 1.3)