# Production Hotfix Protocol

Use this when a bug is found **only in prod** and needs fixing faster than the normal dev→master cycle.

## The Rule

- **Data** flows from prod (clone it for reproduction)
- **Code** flows through git (`dev` branch → `master` → deploy)
- The bugfix env is **ephemeral** — spin up, fix, tear down

---

## Step-by-Step

### 1. Clone prod DB to a bugfix environment

```bash
# Create bugfix DB as an exact copy of prod
docker exec postgres psql -U lmadmin -c \
  "CREATE DATABASE lm360_bugfix TEMPLATE lm360_prod;"

# Spin up a temporary PostgREST pointed at bugfix DB
docker run -d --rm \
  --name postgrest-bugfix \
  --network root_default \
  -e PGRST_DB_URI="postgres://authenticator:$(grep PG_PASSWORD /root/360lm-web/.env | cut -d= -f2)@postgres:5432/lm360_bugfix" \
  -e PGRST_DB_SCHEMAS="hub,expense,sales,custodian,activity" \
  -e PGRST_DB_ANON_ROLE="web_anon" \
  -e PGRST_JWT_SECRET="$(grep PGRST_JWT_SECRET /root/360lm-web/.env | cut -d= -f2)" \
  -e PGRST_SERVER_PORT="3000" \
  -l "traefik.enable=true" \
  -l "traefik.http.routers.postgrestbugfix.rule=Host(\`bugfix.dev.srv1111289.hstgr.cloud\`) && PathPrefix(\`/db\`)" \
  -l "traefik.http.routers.postgrestbugfix.entrypoints=websecure" \
  -l "traefik.http.routers.postgrestbugfix.tls=true" \
  -l "traefik.http.routers.postgrestbugfix.tls.certresolver=mytlschallenge" \
  -l "traefik.http.middlewares.bugfix-strip.stripprefix.prefixes=/db" \
  -l "traefik.http.routers.postgrestbugfix.middlewares=bugfix-strip" \
  -l "traefik.http.services.postgrestbugfix.loadbalancer.server.port=3000" \
  postgrest/postgrest:v12.2.3
```

The bugfix environment is now live at `bugfix.dev.srv1111289.hstgr.cloud` with a full clone of prod data. All 5 closure schemas (hub, expense, sales, custodian, activity) are present, so cross-PWA bugs reproduce exactly as they do in prod.

### 2. Reproduce the bug

Visit `https://bugfix.dev.srv1111289.hstgr.cloud/hub/` (or whichever PWA has the bug).
The PWA files are still served from `dev.srv1111289.hstgr.cloud` (same code as master). Only the DB is the bugfix clone.

If the bug is in the **PWA code** (not DB): you can test code changes directly in `/var/www/360lm/` — they immediately show on the dev subdomain, and the bugfix PostgREST gives you prod data to reproduce against.

### 3. Author the fix on the dev branch

```bash
cd /var/www/360lm
git checkout dev          # make sure you're on dev
# ... make the fix ...
git add <files>
git commit -m "fix: <description of bug>"
```

If the fix requires a SQL migration:
```bash
# Create the migration file (increment version number)
vim finance/custodian/migrate_custodian_vNN.sql
# Apply to the bugfix DB first to confirm it works
docker exec -i postgres psql -U lmadmin -d lm360_bugfix < finance/custodian/migrate_custodian_vNN.sql
# Apply to dev DB too
docker exec -i postgres psql -U lmadmin -d lm360 < finance/custodian/migrate_custodian_vNN.sql
```

### 4. Test on bugfix environment

With the fix on `dev` branch and applied to `lm360_bugfix`, verify the bug is gone at the dev subdomain (which serves from `/var/www/360lm/`).

### 5. Promote to production

```bash
# Merge dev → master
git checkout master
git merge dev
# Note: GitHub push is blocked (large binary files in history) — master IS production
# Just run the deploy script
deploy-prod.sh
```

`deploy-prod.sh` will:
- Pull master (latest merged fix)
- Rsync prod PWA files
- Apply any pending SQL migrations to `lm360_prod`
- Reload PostgREST-prod schema cache

### 6. Tear down bugfix environment

```bash
docker stop postgrest-bugfix
docker exec postgres psql -U lmadmin -c "DROP DATABASE lm360_bugfix;"
```

---

## Cross-PWA Bug Notes

Because `lm360_bugfix` is a **full clone** of `lm360_prod` (all 5 closure schemas: hub, expense, sales, custodian, activity), cross-PWA interactions reproduce faithfully:

- Custodian deactivation sync trigger (`expense.sync_custodian_active`) — present and active
- Sales catalog FK (`custodian.transaction_lines` → `sales.catalog`) — present
- Hub auth and pwa_registry — present with prod data

If a bug involves a schema NOT in prod (e.g., `recce`, `vehicle`) — those schemas do not exist in `lm360_bugfix` or `lm360_prod`. Bugs in non-MVP PWAs stay on dev only until those PWAs are promoted.

---

## Quick Reference

| Environment | URL | DB | Code served from |
|---|---|---|---|
| Dev/Test | `dev.srv1111289.hstgr.cloud` | `lm360` | `/var/www/360lm/` |
| Production | `srv1111289.hstgr.cloud` | `lm360_prod` | `/var/www/360lm-prod/` |
| Bugfix (ephemeral) | `bugfix.dev.srv1111289.hstgr.cloud` | `lm360_bugfix` | `/var/www/360lm/` (dev) |

| Action | Command |
|---|---|
| Promote dev → prod | `deploy-prod.sh` |
| Dry-run (see what would change) | `deploy-prod.sh --dry-run` |
| Apply migrations only | `deploy-prod.sh --migrate-only` |
| Create bugfix env | See Step 1 above |
| Tear down bugfix env | `docker stop postgrest-bugfix && docker exec postgres psql -U lmadmin -c "DROP DATABASE lm360_bugfix;"` |