#2156·immudb

Health reports healthy while databases are unopened; no readiness signal for failed open

Author: clemlesneCreated Sep 4, 2026Updated Sep 4, 2026

What would you like to be added or enhanced

A health signal reporting whether the databases that should be open actually are — something an orchestrator can use as a readiness probe.

Health cannot report anything but success, and DatabaseHealth returns request statistics, not load state:

go
func (s *ImmuServer) Health(ctx context.Context, _ *empty.Empty) (*schema.HealthResponse, error) {
	return &schema.HealthResponse{Status: true, Version: Version.Version}, nil
}

Why is this needed

On immudb 1.11.0 (commit bfdce03649f52d575be46f74425fd18eaf4fa69c, S3 storage), a transient 403 left two of four databases unopened for 3 h 20 min (#2155). The pod stayed 1/1 Running and the liveness probe passed the whole time.

So the outage surfaced as a crashloop in our client services and sent our on-call after the consumers instead of immudb. A per-minute unable to get state log line was the only tell. A readiness probe with real state would have restarted the pod automatically — which is what fixed it once we did it by hand.

Additional context

DatabaseListV2 looked like the answer, but I do not think it is:

go
Loaded: !db.IsClosed(),

IsClosed returns dbInfo.closed, set only by Close() or Put(..., closed=true) (used by PutClosed for autoload-disabled databases). databaseList.Put passes closed=false, and the openDB error path never touches it — so a database whose open failed should report Loaded: true while unusable. Loaded separates "explicitly unloaded" from "loaded", but neither from "failed to open". Verified by reading, not by running; a maintainer's read would be welcome.

IsActive fails the same way — a failed open leaves dbRef{db: nil} in dbCache, and it only checks that the lookup succeeds:

go
func (m *DBManager) IsActive(idx int) bool {
	_, err := m.dbCache.Get(idx)
	return err == nil
}

Either direction would work for us:

  1. Let Health (or a new endpoint) fail when a database expected to be open is not.
  2. Make Loaded reflect real open state and document it as the readiness signal.

Happy to test a patch against the S3 setup that produced this.