core deadlocks on its own DB pool: audit-log resolver takes a second connection inside transaction middleware (bulk user delete wedges core)

Expected behavior and actual behavior:

Expected: a burst of concurrent write requests (here DELETE /api/v2.0/users/{id} fired in parallel by the portal's bulk user delete) completes, or fails with an error, and core keeps serving.

Actual: harbor-core deadlocks on its own database connection pool and stays wedged until the pods are restarted. Every request that touches the DB hangs forever; /api/v2.0/ping still answers 200, /api/v2.0/health times out (504 at the ingress), jobservice hook calls to core time out, harbor-exporter crash-loops on its startup probe. Postgres shows exactly POSTGRESQL_MAX_OPEN_CONNS sessions per core pod in state idle in transaction, last statement begin, no locks, no active queries. The database itself answers select 1 in 40 ms.

Root cause: every non-GET request runs inside transaction.Middleware (src/core/middlewares/middlewares.go:97), which takes one pool connection with begin and stores the tx ormer in the request context. Inside that transaction, before the handler runs, log.Middleware calls the audit-log resolver's PreCheck, and src/pkg/auditext/event/user/user.go:60-61 does:

// use different context to so that the user is visible before the transaction is committed
user, err := pkgUser.Mgr.Get(orm.Context(), int(id))

orm.Context() builds a fresh non-transactional ormer, so this query needs a second connection from the same database/sql pool. With N = POSTGRESQL_MAX_OPEN_CONNS concurrent such requests on one pod, all N connections are held by begin and all N goroutines wait in database/sql.(*DB).conn for a connection that will never be released. Beego v2.3.10 hardcodes context.Background() into Raw/Read/Insert/Begin, so no request cancellation or deadline reaches the pool wait, and Harbor sets no acquisition timeout anywhere.

The comment's justification is backwards: PreCheck runs before the handler deletes the row, and a transaction sees its own state plus everything committed, so the request ctx would return the same user with zero extra connections.

Cascade: one config loader (security.Middleware → config.AuthMode → FetchOrSave under lib/cache.keyMutex) also ended up waiting for a pool connection while holding the key mutex, so 1300+ further goroutines, including GETs and the health checker, parked on keyMutex.Lock. That is why even read-only requests hung.

Goroutine dump excerpt (25 identical per pod):

goroutine 8080354 [select, 164 minutes]:
database/sql.(*DB).conn
github.com/beego/beego/v2/client/orm.(*DB).QueryContext
github.com/beego/beego/v2/client/orm.(*querySet).All
github.com/goharbor/harbor/src/pkg/user/dao.(*dao).List
github.com/goharbor/harbor/src/pkg/user.(*manager).Get
github.com/goharbor/harbor/src/pkg/auditext/event/user.userIDToName
github.com/goharbor/harbor/src/pkg/auditext/event.(*Resolver).PreCheck
github.com/goharbor/harbor/src/controller/event/metadata/commonevent.(*Metadata).PreCheckMetadata
github.com/goharbor/harbor/src/server/middleware/log.Middleware...
github.com/goharbor/harbor/src/server/middleware/security.Middleware...
github.com/goharbor/harbor/src/lib/orm.WithTransaction.func1
github.com/goharbor/harbor/src/server/middleware/transaction.Middleware...

pg_stat_activity at the time (2 core pods, POSTGRESQL_MAX_OPEN_CONNS=25):

 client_addr   |        state        | count | max_xact_age | last_query
 10.10.18.216  | idle in transaction |    25 | 02:42:00     | begin
 10.10.179.252 | idle in transaction |    25 | 02:42:00     | begin

Same pattern elsewhere (verified in v2.15.2 and current main):

Every other orm.Context() / orm.Copy() call site runs in a goroutine, at startup, or in jobservice, and does not hold a request transaction. Nested orm.WithTransaction uses SAVEPOINT on the same connection and is fine.

Prior art: #15649 (2021) describes this exact mechanism at a different call site ("core hang when a lot request coming at the same time ... not enough db connect to deal with the next dao request") and was closed as fixed in 2.3.3. #21455 reintroduced it in the audit path. #21062 / #23114 / #23703 / #23813 are related pool-exhaustion reports but each involves a single connection held too long, not one request needing two connections.

Steps to reproduce the problem:

1. Harbor v2.15.2, one core replica, POSTGRESQL_MAX_OPEN_CONNS=25 (default in the helm chart is 100; lower values make it trivial, higher values just need more concurrency).
2. Create 30 users.
3. Log in as admin, select all users on the Users page, delete. The portal issues one DELETE /api/v2.0/users/{id} per user in parallel. Equivalent: seq ... | xargs -P 30 -I{} curl -u admin:... -X DELETE https://harbor/api/v2.0/users/{}.
4. Core stops responding to anything except /api/v2.0/ping. select state, query from pg_stat_activity where datname='registry' shows 25 × idle in transaction / begin. It never recovers without a restart.

Versions:

• harbor version: v2.15.2 (code identical on main as of 2026-09-09)
• deployment: helm on EKS, 2 core replicas, external Aurora PostgreSQL
• docker engine version: n/a (API-level trigger)

Additional context:

Suggested fix direction, not implemented:

1. Give IDToNameFunc a ctx parameter and pass the request context through Resolver.PreCheck; same for ensureORMContext, populateGroupsDB and authproxy. A request must never acquire more than one pool connection.
2. Bound pool acquisition (a pool-level timeout, since beego's non-WithCtx methods swallow the request ctx) so a future regression of this class turns into 5xx errors instead of a permanent wedge.
3. Optionally a lint rule forbidding orm.Context() under src/server, src/pkg/auditext, src/pkg/oidc, src/pkg/authproxy.