# Nightly Test Summary — 2026-07-05 (IST)

**3RD CONSECUTIVE MASS-FAILURE NIGHT — still infra/environment, not app regressions** (938/1654 failed, 57%)

- Run: `test_reports/nightly_20260704.json` (marker date=20260704, exit_code=1)
- Note on dating: the marker's `date` field is stamped at UTC run-start, not IST — this run's `startTime` is `2026-07-04T18:45:03Z` (00:15 IST Jul 5) and `finished_at` is `2026-07-05T00:01:53Z` (05:31 IST Jul 5), so it belongs to **today (2026-07-05 IST)**, not Jul 4. Same one-day UTC/IST lag was present in the prior night's marker too — flagging the cron's date-stamping for a fix, not treating it as a missed run.
- Totals: **1654 total, 583 passed (expected), 938 failed (unexpected), 132 skipped, 1 flaky**

## Root cause this run

The Playwright browser-binary issue from the prior night (2026-07-03→04, `chromium_headless_shell` missing) is **gone** — no `Executable doesn't exist` errors in this run. But the same underlying signature as both prior nights persists: wholesale failures across unrelated PWAs on basic checks like `manifest.json returns 200`, `sw.js contains current version`, and PostgREST table-existence checks (`client.zones table exists and is accessible via PostgREST`) — consistent with the **dev server / DB / proxy stack being down or unreachable** during the run, cascading into:
- 566 timeout-class failures (`page.waitForSelector` / `beforeEach` hook timeouts / 30s test timeouts)
- 170 element-not-found / content-mismatch failures (`toContainText`, `toBeVisible` — e.g. fundcustodian's `#my-balance` never appearing)
- 202 other failures, mostly downstream of the above: `Cannot read properties of undefined (reading 'paid_amount'/'invoiced_amount'/'balance_due'/'job_name')` — API responses coming back empty/malformed once the stack is unreachable

## Failing spec files (top 20 by failure count)

| Spec file | Failures |
|---|---|
| fundcustodian.spec.js | 154 |
| printing.spec.js | 94 |
| sales_billing.spec.js | 82 |
| vendors.spec.js | 74 |
| upi_pay.spec.js | 68 |
| customers.spec.js | 50 |
| recce.spec.js | 44 |
| custodian.spec.js | 40 |
| hr.spec.js | 36 |
| sales.spec.js | 34 |
| recce_client.spec.js | 32 |
| safe_bottom.spec.js | 28 |
| stores.spec.js | 26 |
| vehicle.spec.js | 20 |
| counters.spec.js | 18 |
| yagya-portfolio.spec.js | 18 |
| expense.spec.js | 16 |
| contacts.spec.js | 14 |
| finance.spec.js | 10 |
| installation.spec.js | 10 |
| learn.spec.js | 10 |
| tour_planner.spec.js | 10 |
| hub.spec.js | 8 |
| oc_admin.spec.js | 8 |
| production.spec.js | 8 |

(remaining spec files also failed with fewer instances each.)

## Sample failing tests (first 20, illustrating the wholesale/infra pattern)

- admin.spec.js — client.zones table exists and is accessible via PostgREST
- admin.spec.js — client.accounts has zone and is_head columns
- admin.spec.js — installation.campaigns has zones column
- client.spec.js — manifest.json returns 200
- client.spec.js — sw.js contains current 360client version
- contacts.spec.js — manifest.json returns 200
- contacts.spec.js — sw.js is versioned
- contacts.spec.js — auth screen and app container present
- contacts.spec.js — hub navigation button present
- contacts.spec.js — search and contact list elements present
- contacts.spec.js — Dexie offline DB used
- contacts.spec.js — Top Management access control in place
- fundcustodian.spec.js — balance card renders and shows shambhu name
- fundcustodian.spec.js — no pending transfers banner shown for shambhu
- fundcustodian.spec.js — outstanding advances section is visible
- fundcustodian.spec.js — admin balance grid is NOT shown for non-admin
- fundcustodian.spec.js — manage custodian button hidden for non-admin
- fundcustodian.spec.js — bottom nav has 3 tabs (advance merged into transfer)
- printing.spec.js — (multiple, same pattern)
- vendors.spec.js — (multiple, same pattern)

(Full list of 938 failing tests is in the JSON; not reproduced here per the "max 20" instruction — pattern is uniform wholesale failure, not isolated regressions.)

## Baseline comparison

Trusted baseline remains **2026-06-20: ~1220+ tests, 186 skipped, 0 failed** (`project_pwa_test_coverage.md`). This is the **3rd consecutive mass-failure night** (2026-07-02→03, 2026-07-03→04, 2026-07-04→05), each with a different proximate error signature (server/DB unreachable → browser binary missing → server/DB unreachable again), but the same underlying pattern of wholesale cross-PWA failure rather than isolated app regressions. Not re-baselined.

## Action needed (not performed by this autonomous task)

- Verify the dev server / DB / proxy stack is actually up and reachable **during** the nightly test window (01:00 IST) — this is now 3 for 3 nights with the stack apparently down/unreachable at that time, worth checking if something in the 01:00/02:00/04:30 IST nightly job sequence (tests → graphify → analysis) is itself causing resource contention or restarting the stack on this 2-core VPS.
- Fix the nightly cron's date-stamping (marker `date` field and `nightly_<date>.json` filename use UTC run-start date, causing a recurring one-day lag vs the IST calendar date the run actually belongs to).
- Re-run Playwright manually against a confirmed-up dev environment to get a clean comparison point.