oauth2: refresh token grant deadlocks /token endpoint with SQLite storage (/healthz unaffected)

Summary

With SQLite storage, a single grant_type=refresh_token request deadlocks the token endpoint permanently: the request never completes, every later /token request (any grant type) hangs until the process restarts, and /healthz keeps returning 200 — so orchestrated deployments never recycle the wedged pod.

Present on master (verified at 7ace0e79); introduced by the per-grant /token handler refactor (2c81c4457d). Releases up to v2.45.1 are unaffected.

Reproduction

Minimal config (SQLite + local password DB + public client, rotation is default):

issuer: http://127.0.0.1:15556/dex
storage:
  type: sqlite3
  config:
    file: /tmp/dex-repro/dex.db
web:
  http: 127.0.0.1:15556
enablePasswordDB: true
oauth2:
  passwordConnector: local
staticClients:
  - id: test-client
    redirectURIs: ['http://127.0.0.1:15556/callback']
    name: 'test-client'
    public: true
staticPasswords:
  - email: "[email protected]"
    hash: "$2a$10$2b2cU8CPhOTaGrs1HRQuAueS7JTT5ZHsHSzYiFPm1leZck7Mc8T4W"  # "password"
    username: "admin"
    name: "Admin"
    emailVerified: true

1. Password grant works fine:

$ curl -s http://127.0.0.1:15556/dex/token -d grant_type=password -d [email protected] \
    -d 'password=password' -d client_id=test-client --data-urlencode 'scope=openid offline_access'
{"access_token":"...","refresh_token":"..."}

1. The refresh grant never returns (10 s timeout, 0 bytes):

$ curl -m 10 -w 'HTTP=%{http_code} time=%{time_total}s bytes=%{size_download}\n' \
    http://127.0.0.1:15556/dex/token -d grant_type=refresh_token \
    -d 'refresh_token=<rt>' -d client_id=test-client
HTTP=000 time=10.002080s bytes=0      # curl exit 28

1. While wedged, /healthz still passes and a brand-new password grant also hangs:

$ curl -s http://127.0.0.1:15556/dex/healthz
Health check passed

$ curl -m 5 -w 'HTTP=%{http_code} time=%{time_total}s\n' http://127.0.0.1:15556/dex/token \
    -d grant_type=password -d [email protected] -d 'password=password' ...
HTTP=000 time=5.006826s

Root cause

RefreshStore.Rotate invokes the freshIdentity callback from inside the UpdateRefreshToken transaction. For a user without an offline session (every local-password user), that callback calls refreshConnectorData → GetOfflineSessions, a db-level query. The SQLite storage pins the pool to one connection (SetMaxOpenConns(1)), so the nested query waits forever for the connection the transaction already holds. Every later token request queues behind the exhausted pool. /healthz pings storage on a path that never contends with the rotation transaction.

Before the refactor the connector data was resolved during request verification, outside the transaction (v2.45.1 server/refreshhandlers.go:207); the refactor moved that read into the lazy closure.

Fix

#4991 resolves the connector data before rotation starts (restores the pre-refactor ordering). With the fix, the same reproduction completes three consecutive rotating refreshes at ~6 ms each.

A goroutine dump taken at the deadlock is attached to #4991.