# ADR-114 Google AI Access Uses OAuth/ADC Service Identity via Vertex AI — Never Static Gemini API Keys

## Status

Accepted, 2026-07-16.

## Status History

```yaml
status_history:
  - date: 2026-07-16
    status: Accepted
    changed_by: hkl
    reason: Implementation is live and operational; decision ratified.
    changed_via: adr-kit (360lm)
```

## Context

360lm services call Google Gemini models for vision (OCR fallback, image analysis) and text inference. Prior static API key deployment (`GEMINI_API_KEY` string hardcoded or env-injected) creates operational and security risks:

1. **Leak surface**: Keys in code, logs, or accidental git commits are difficult to revoke cleanly.
2. **Key rotation burden**: Requires coordinated env var + secrets manager updates; downtime risk.
3. **No audit trail**: Cloud audit logs cannot distinguish key usage; impossible to trace misuse to a specific service.
4. **Token lifecycle**: Static keys never expire; a leaked key remains usable indefinitely until admin action.

**Incident driver**: Health-tracker OCR fallback relied on a static key from GCP project 933872821357. That project was suspended (CONSUMER_SUSPENDED status) without advance notice, silently breaking the fallback. No graceful degradation occurred because the library (`google-generativeai`) doesn't expose provisioning state.

**Decision scope**: Any 360lm service calling Google Gemini models (gemini-1.5-flash, gemini-1.5-pro, or vision models). This includes standalone PWAs (health-tracker, blogsmith image workflows) and shared infrastructure (tour planner, recce, custodian AI chains).

## Decision

**Every 360lm service calling Google Gemini models must authenticate as a GCP service-account identity through Application Default Credentials (ADC), never with a static API key.**

- Use `google.generativeai` SDK with `vertexai=True` parameter, or use `google-cloud-aiplatform` SDK directly (VertexAI).
- Service account credentials are mounted read-only at `/etc/gemini-proxy/sa.json` (or similar isolated path).
- Credentials are loaded once and cached in-memory by the client library; no string in logs or environment.
- OAuth tokens (~60 minutes lifetime) are automatically minted and refreshed by `google-auth` library; no manual token management.
- Revocation happens via IAM: disable/delete the service account or remove the `roles/aiplatform.user` role; takes effect immediately on next token refresh.

**Implementation**: `gemini-proxy` container is the reference pattern:
- **File**: `/opt/gemini-proxy/main.py` (FastAPI service).
- **Config**: Environment variables `GOOGLE_CLOUD_PROJECT` (e.g., `my-gemini-360lm-prod`) and `GOOGLE_CLOUD_LOCATION` (e.g., `us-central1`).
- **Credentials**: Service account key mounted read-only at `/etc/gemini-proxy/sa.json`.
- **Compose service**: Defined in `/root/360lm-web/docker-compose.yml` with labels for Traefik routing.
- **Endpoint**: Accessed at `https://dev.srv1111289.hstgr.cloud/gemini-proxy/analyze` (dev) or prod equivalent.
- **Authorization header**: Clients pass `Authorization: Hub <b64(lm360-session)>` (inherits ADR-105 pattern; see consequences).

**Endpoint contract**:
```
POST /gemini-proxy/analyze
Authorization: Hub <base64(lm360-session)>
Content-Type: application/json

{
  "prompt": "Describe this image",
  "image_url": "https://example.com/image.jpg",
  "model": "gemini-1.5-flash"
}

Response (200 OK):
{
  "analysis": "The image shows...",
  "model": "gemini-1.5-flash",
  "usage": { "input_tokens": 150, "output_tokens": 42 }
}

Response (503 Service Unavailable):
{
  "error": "Provisioning required",
  "provisioned": false
}
```

**Decision Maker:** hkl

## Implementation Notes

- **Files & references**:
  - `/opt/gemini-proxy/main.py` — FastAPI handler; lazy client construction returns 503 `provisioned:false` until credentials exist.
  - `/root/360lm-web/docker-compose.yml` — Service definition with SA key volume mount and environment variables.
  - `google-generativeai>=0.7.0` — SDK version constraint (must support `vertexai=True`).
  - `google-auth>=2.25.0` — Auto-refresh token management.

- **Key identifiers**:
  - Service account email: `gemini-proxy@<gcp-project>.iam.gserviceaccount.com`
  - IAM role: `roles/aiplatform.user` (minimum scope for Gemini inference).
  - Cloud Audit Logs: filter by `resource.type="api"` and `resource.labels.service="aiplatform.googleapis.com"` to trace all Gemini calls by service account.

- **Find all call sites**: `grep -r "gemini-proxy" /var/www/360lm --include="*.html" --include="*.js"` (PWA-side fetch calls); grep for `GOOGLE_CLOUD_PROJECT` or `vertexai=True` in services under `/opt/`.

- **Gotchas**:
  - If the service-account key file is missing, the library silently creates an ADC fallback to user credentials (breaking isolation). Always verify the mount path in Traefik labels and docker-compose.
  - The `google-generativeai` SDK caches the client in memory; restarting the container is required to pick up a new SA key. For zero-downtime key rotation, keep two keys active, swap in compose, and blue-green deploy.
  - Logs must NOT include the raw JWT or access token. The `google-auth` library redacts these by default; verify in deployment logs.

## Alternatives Considered

- **Static `GEMINI_API_KEY` from AI Studio console.** Rejected: keys cannot be rotated without downtime, no audit trail, revocation requires manual admin action and takes time to propagate, and leaks are permanent until discovered and rotated.
- **OAuth 2.0 "Client Credentials" flow with a GCP service account but no ADC.** Rejected: requires manual token refresh logic in application code; error handling is repetitive and buggy. ADC abstracts this away and is the GCP-recommended path.
- **User OAuth (e.g., user's personal Google account via OAuth 2.0 consent flow).** Rejected: PWA end-users should not be asked for permission to call Gemini; only admin/service identities should authorize AI inference.

## Consequences

**Positive:**
- No secret string in code, env vars, or logs—audit surface shrinks to file permissions and IAM.
- OAuth tokens auto-minted and refreshed; no manual token management or hardcoded expiry checks.
- IAM revocation is instant: disable the SA or remove the role, and the next token refresh (within ~60 minutes) fails gracefully with a clear 503 `provisioned: false` response.
- Cloud Audit Logs provide a complete audit trail of every Gemini call by service-account identity.
- Least-privilege: `roles/aiplatform.user` does not grant access to GCP resources outside of Gemini models.

**Negative / Trade-offs:**
- GCP project provisioning and service-account creation is a one-time operational task (tracked in dbt_pending.md; pending user action as of 2026-07-16).
- Requires a read-only mount of the SA key file in docker-compose; if the path is misconfigured, the library falls back silently to user credentials (not a security break, but breaks isolation intent).
- Gemini-proxy container restart is required to pick up a new SA key; not zero-downtime for key rotation without a blue-green deployment strategy.

**Risks and mitigations:**
- **Risk**: Credentials file is accidentally world-readable or logged. **Mitigation**: File permissions are enforced via docker-compose read-only flag and Traefik labels; audit via `docker inspect <container>` to verify mount mode is read-only.
- **Risk**: SA key leaks via container image (e.g., committed in Dockerfile). **Mitigation**: SA key is mounted at runtime, never baked into images; use `docker build --secret` if the key must be available during build (not recommended here).
- **Risk**: Gemini-proxy is unreachable or returns 503 for an extended period, breaking dependent PWAs. **Mitigation**: This ADR does not mandate dependency on gemini-proxy; PWAs should implement a fallback (e.g., client-side Ollama, prompt hand-off per ADR-056).

## Related Decisions

- **ADR-062** (Live AI Pipeline Contract): Governs the fallback order and 503 handling when any AI service is unavailable. This ADR specifies the auth mechanism for Google Gemini; ADR-062 still applies to pipeline orchestration.
- **ADR-105** (Proposed, Signed JWT for Proxy Auth): Endpoint auth currently uses unsigned `Authorization: Hub <b64>` header (inherited from tour-pg-proxy). When ADR-105 lands, this ADR should be amended to use signed JWT instead.

## References

- Incident: GCP project 933872821357 (CONSUMER_SUSPENDED) silently broke health-tracker OCR fallback (2026-07-12).
- GCP service account auth: https://cloud.google.com/docs/authentication/application-default-credentials
- `google-generativeai` SDK with VertexAI: https://ai.google.dev/gemini-api/docs/gemini-api-overview
- Cloud Audit Logs: https://cloud.google.com/logging/docs/audit
- Memory reference: `dbt_pending.md` — GCP project provisioning tasks.

## Amendment — 2026-07-16 (same day): Tier 1 personal-OAuth CLI added

The proxy now implements ADR-062-style tiering INSIDE the Google lane:

- **Tier 1 (free): `gemini` CLI + personal Google OAuth** (`/root/.gemini` creds mounted rw; node22 + @google/gemini-cli baked into the image). Google retired the "Code Assist for individuals" free tier for this client (`UNSUPPORTED_CLIENT` → Antigravity migration notice), but the **standard tier via `GOOGLE_CLOUD_PROJECT` + the free `cloudaicompanion.googleapis.com` API works without billing**. Vision requires the `@file` prompt syntax (a bare "read the file" prompt makes the model refuse). ~28s/call, daily quotas apply.
- **Tier 2 (billed): Vertex AI + service account** as originally decided — wired, verified to the `BILLING_DISABLED` gate, activates when billing is enabled on `gemini-pwa-360dlm`. ~2s/call, no practical quotas, IAM audit.

`/analyze` responses carry a `tier` field. Both tiers remain OAuth identities — the core decision (no static API keys) is unchanged. Live E2E verified 2026-07-16: hub-auth → tier `gemini-cli/oauth` → gemini-2.5-flash → schema-shaped JSON.