proxy: Data API keeps validating against a replaced/deleted JWKS provider for ~1h (unknown kid does not refresh auth rules)

Summary

After changing a project's JWKS provider (new JWKS URL serving a new kid), the Data API proxy kept validating against the previous key set for ~60 minutes. Tokens with the new kid got 400 {"message":"jwk not found"} the whole time, while a token signed with the same key under the old kid still passed key lookup — even after the provider had been deleted entirely. Nothing user-facing (remove/re-add provider, pinned vs unpinned URL, compute restart, schema-cache refresh) shortened it. It resolved on its own at almost exactly one hour, which matches MAX_RENEW in proxy/src/auth/backend/jwt.rs.

For a JWKS change this is effectively up to an hour of auth downtime, with no way to force a refresh and no documentation of the delay.

Environment

• Data API (rest broker), us-west-2, endpoint ep-spring-tooth-afr2n6e9
• Custom provider ("Other"), JWKS hosted on a GitHub gist, RS256, kid = RFC 7638 thumbprint
• Token claims: role: authenticated, sub, iat, exp; no aud

Timeline (UTC, 2026-08-22)

Time	Event
~07:58	First Data API requests of the session with the new kid → jwk not found (old JWKS doc at the configured, commit-pinned gist URL still had kid: main)
08:15	Gist updated; provider URL changed in console to the new pinned revision. apply_config + epc_sync finished, 0 failures
08:16–08:44	Requests spaced ≥5 min apart (to clear the 5-min renewal guard): still jwk not found
08:31, 08:39	Provider removed and re-added with the unpinned gist URL; compute restarted; schema cache refreshed. apply_config finished, 0 failures. GET /projects/{id}/jwks shows exactly one entry, correct URL, role_names: [authenticator, authenticated, anonymous]
08:39, 08:49	A token signed with the same key but kid: "main" passes key lookup (fails downstream with a 500, not jwk not found) — i.e. the proxy still holds the old document, including after the provider was deleted
~08:58	Requests with the new kid start succeeding (200). ~60 min after the proxy's first fetch

Expected

Per proxy/src/auth/backend/jwt.rs, an unknown kid should trigger renew_jwks once ≥MIN_RENEW (30s) has elapsed, and renew_jwks re-pulls auth rules from the control plane (GET /endpoints/{id}/jwks) and re-fetches every URL. With requests spaced >5 min apart I'd expect the new rule/document to be picked up within ~5 minutes of the control-plane change, not at the 1-hour MAX_RENEW hard refresh.

Actual

The new key set was not observed until the ~1h mark, regardless of provider changes, unknown-kid requests, or compute restarts. From the outside it looks like either (a) the unknown-kid renewal path isn't refreshing the auth rules for this endpoint, or (b) the control-plane response to /endpoints/{id}/jwks is served stale to the proxy for ~1h. I can't distinguish from outside.

One thing that looked odd while reading main: check_jwt's loop renews when last_retrieved.elapsed() > MIN_RENEW (30s), but renew_jwks returns the cached entry unchanged when last_update < 300s — in the 30s–300s window that appears to loop without making progress. May be unrelated to the 1-hour behaviour, but flagging it since it's the same path.

Request IDs (jwk not found responses)

• 490baa23-28d7-48f3-94b7-f49abdc56c43
• ac59262e-7cee-489c-bccc-ceb5719a4d3a

Asks

1. Make a JWKS provider change (or at least an unknown kid) refresh the proxy's key set promptly — ideally bounded by the documented 5-minute renewal, not the hourly one — or expose a way to force it.
2. Document the propagation delay for JWKS provider changes.
3. Clarify whether an in-place JWKS content change at a fixed URL (the normal key-rotation path: same URL, new kid) is subject to the same delay. That's the case that matters for production rotations.

Happy to share the project ID and more detail privately.