den-api production: POST /v1/org, GET /v1/inference-providers(/usage) failing since 14:59Z after #4358 (migration 0097 never applied on PlanetScale)

Summary

Since 14:59Z on 2026-09-11 den-api production returns 500 on POST /v1/org, GET /v1/inference-providers and GET /v1/inference-providers/usage. Root cause: migration 0097_gateway_access_matrix from #4358 has never applied to production PlanetScale — the Den DB Migrate workflow has failed on every run since the #4358 push (14:49Z) and every 30-minute self-heal tick since — while the den-api code from #4358 already reads the renamed/new gateway_* tables.

A second, shorter incident at ~16:41Z (den-api + gateway fatal at boot, DEN-API-3W / INFERENCE-6) was an env change: GATEWAY_ENABLED=true set on both Render services without a valid GATEWAY_PUBLIC_BASE_URL; corrected by ~16:47Z.

Not caused by the API-quality stack (#4475, #4700–#4705, #4707): none of those 8 commits touch any file, table, migration or env var involved.

Sentry issues (project den-api, environment=production)

3V is on release 4fa023235 — i.e. it started on the #4358 deploy itself, before #4888 (667b450fd, merged 15:50Z) existed. #4888 is unrelated.

Evidence: Den DB Migrate failures

gh run list --workflow den-db-migrate.yml: success at 13:29, 13:54, 14:29Z; failure at 14:49:16Z (push, 4fa023235) and every scheduled tick after (14:53, 15:28, 15:53, 16:30, 16:53, 17:26Z).

First run (34612414683) — 0095 and 0096 applied, then 0097's embedded preflight fails because Vitess routes the INSERT … SELECT … FROM information_schema.tables to the primary keyspace and strips the qualifier:

DrizzleQueryError: Failed query: INSERT INTO `__gateway_0097_preflight` (`failure`) SELECT '0097_requires_complete_0096_schema' WHERE (SELECT COUNT(*) FROM information_schema.tables …
cause: target: <db>.-.primary: vttablet: Table '<db>.tables' doesn't exist (errno 1146) (sqlstate 42S02)
  Sql: "select :vtg1 from dual where (select count(*) from `tables` where table_schema = database() …"

Every retry since (e.g. 34627695206) fails one statement earlier — drizzle's MySQL migrator runs inside a transaction and Vitess refuses the temp-table DDL there:

CREATE TEMPORARY TABLE `__gateway_0097_preflight` (…
errno: 1105  sqlMessage: 'target: <db>.-.primary: vttablet: DDL statement executed inside a transaction'

Retries starting at 0097 (not 0095) confirm 0095/0096 are recorded in __drizzle_migrations. So production currently has the inference_* tables from 0095/0096 but not gateway_keys / gateway_providers / gateway_request_logs (created or renamed only by 0097), which is exactly what the three failing queries select from.

drizzle/0097_gateway_access_matrix.md itself notes: "other drivers or serverless sessions that cannot preserve a temporary table across statements need a separate migration runner" and "No database execution or real-data preflight has happened during this batch." 0097 also renames 8 tables in place while the same PR switches code to the new names, contrary to the expand/contract policy in ee/packages/den-db/README.md.

Evidence: 3W env fatal

parseGatewayDeploymentEnv only validates GATEWAY_PUBLIC_BASE_URL when GATEWAY_ENABLED=true; new URL(undefined ?? "") throws the exact message in the alert. den-api and the gateway service failed simultaneously on fresh pods at 16:41Z; no Den API Env workflow ran today, so the flag was set directly in Render. 3Y at 17:08Z is behind the gatewayManagementUnavailable() gate, which proves GATEWAY_ENABLED=true is now live with a valid URL.

Current impact

• POST /v1/org → 500 for every new organization (createOrganizationRecord calls ensureMemberGatewayKey unconditionally). insertMemberIfMissing (bootstrap membership) calls it too.
• GET /v1/inference-providers (any scope) and /usage → 500. Shipped desktop builds (v0.18.46, coworker-v0.1.0-alpha.20260910.1) predate #4358 and do not call these; den-web gateway-providers dashboard and dev builds do.
• Gateway service findActiveGatewayKey reads gateway_keys → any ow_gw_ key traffic fails (none minted yet).
• Legacy Models / inference_keys paths are unaffected.

Remediation

1. Required — make 0097 applicable on PlanetScale, then let the self-heal tick run it (0098/0099 follow):
   • (a) move the ten preflight checks into a Node step in .github/workflows/den-db-migrate.yml (query information_schema via @planetscale/database like the existing pending-check does) and drop the CREATE TEMPORARY TABLE / INSERT … SELECT preflight from the SQL; or
   • (b) apply 0097's persistent DDL (from RENAME TABLE onward) via pscale shell in a safe-migrations window and insert the journal row.
   • Quiesce den-api/gateway writers first (RENAME is not online, per the migration's own .md).
2. Optional stop-gap: revert #4358 on dev (code goes back to inference_*, which exist; nothing to lose since gateway_* don't exist), or skip ensureMemberGatewayKey when !env.gatewayEnabled — but the flag is now true in prod, so (1) is the real fix. Consider setting GATEWAY_ENABLED=false on both services until 0097 lands.
3. Verify GATEWAY_PUBLIC_BASE_URL is an exact https://<public-host> origin on both den-api and gateway (packages/docs/self-host/gateway.mdx L81–99).
4. Follow-up: den-db-migrate.yml should alert on failure instead of silently retrying every 30 min; split future rename migrations expand/contract.

Suggested fix PR: fix(den-db): make 0097 gateway migration runnable on PlanetScale (no temp-table/information_schema preflight in SQL).

Full write-up (local): reports/sentry-denapi-2026-09-11.md.