Transient S3 error during open leaves dbRef{db:nil} cached; GetState returns "unable to get state" until restart

What happened

A transient S3 403 during a new database's first open leaves a dbRef{db: nil} cached in DBManager.dbCache. GetState then fails on every call, the server keeps reporting healthy, and only a restart recovered it.

Creating audit and billing against S3 storage, 3 of 53 S3 calls returned 403 during their first open. Every other call in that window, same bucket and credentials, returned 200, and the failing key succeeded one second later — provider flapping, not auth:

S3 PUT .../<prefix>/audit/aht/commit/00000000.di failed with status code 403 (403 Forbidden)
S3 PUT .../<prefix>/audit/aht/commit/00000000.di 200 OK        <-- same key, one second later

{"caller":"/src/pkg/database/database.go:1739","level":"error",
 "message":"unable to open database: ... 403 (403 Forbidden): could not open indexer"}

billing failed at 12:06:42, audit at 12:09:15. Until the restart at 15:29:52 the log held nothing but this, once per minute per database:

"error getting current state of db audit to update the number of entries metric: unable to get state"

After the restart all four databases opened and 74/74 S3 calls returned 200.

What you expected to happen

A transient remote-storage error during an open should not be terminal. The failed open should clean up so a later access retries, or retry with backoff.

How to reproduce it (as minimally and precisely as possible)

A real provider fault, not a synthetic one, so these are conditions rather than a script:

1. immudb with S3_STORAGE=true.
2. A proxy in front of the endpoint failing a small share of requests with 403.
3. Call CreateDatabaseV2 — databaseList.Put installs a lazyDB, so the open runs on first access.
4. Land one injected 403 inside that open.
5. Expect unable to open database once, then unable to get state per minute. Ours never recovered until we restarted 3 h 20 min later.

Environment

immudb 1.11.0
Commit  : bfdce03649f52d575be46f74425fd18eaf4fa69c
Built at: Thu, 03 Sep 2026 06:46:12 UTC
S3 storage, endpoint https://fsn1.your-objectstorage.com (Hetzner Object Storage, Ceph RGW)
Auth enabled: true   Synced mode: true   Log format: json

Additional info (any other context about the problem)

allocDB caches the dbRef before the open is attempted. When openDB fails, Get calls Release, which zeroes the refcount but leaves the entry in dbCache — a dbRef{db: nil, count: 0}.

The file already knows this state exists. CloseAll:

// ref.db may be nil if the database was never successfully opened
// (e.g. openDB failed in Get and left a db-less ref in the cache).
if ref.db != nil {

GetState calls the same state impossible:

if dbRef := ref.(*dbRef); dbRef != nil && dbRef.db != nil {
    return dbRef.db.CurrentState()
}
// this condition should never happen
return nil, fmt.Errorf("unable to get state")

The cached entry makes that branch permanent — GetState never reaches the m.Get fallback below it that would retry the open. IsActive is wrong for the same reason: it only checks that the cache lookup succeeds, so it reports a database that never opened as active.

What I could not explain

Get should retry openDB on the next access — the cached ref has db == nil, and a failed open never sets dbInfo.closed, so allocDB does not short-circuit. Yet our clients retried for over three hours without connecting, and no S3 traffic appeared in the log during that window, so nothing was re-driving the open. The cached ref and the stuck GetState are established; why no caller re-drove the open is not. I would rather flag that than guess.

Suggested fix

Pop the entry from dbCache on the openDB error path in Get, so a dbRef with a nil db is never observable. That also makes GetState's comment true again.

Related

#1997 — same symptom on an S3 backend, no reproduction; the workaround comment in DatabaseListV2 cites it. Possibly the same root cause with a concrete trigger. #2156 — this incident stayed invisible because Health cannot report a database that failed to open.