/health reports healthy while the ClickHouse migration has failed and logs, traces and profiles are all unavailable

This relates to https://github.com/coroot/coroot-operator/issues/57 as it was discovered in the same debugging session.

Summary

When create or update clickhouse tables fails, Coroot serves no logs, no traces and no profiles - but /health continues to return healthy, so the readiness probe never flips and nothing external can tell that the instance is degraded. Metrics keep working via Prometheus, so the UI looks broadly normal.

In our case this ran for 17.5 hours across two production clusters before a user happened to report that traces were missing.

Version

Coroot EE 1.26.3 (current release at time of writing), deployed by coroot-operator 1.9.6.

Detail

The migration failed at 16:13:48. The pod reported Ready=True at 16:13:57 - nine seconds later - and stayed ready, with zero restarts, until the underlying ClickHouse problem was fixed by hand the following morning:

readinessProbe: httpGet { path: /health, port: http }

Ready=True   since 2026-09-15T16:13:57Z
restarts=0

Throughout that window the container logged, once per minute:

E migration.go:33] failed to create or update clickhouse tables for project <id>: ...
E migration.go:40] clickhouse tables migration failed, next attempt in 1m0s

and roughly once per second:

E traces.go:249]   clickhouse tables not ready for project <id>
E logs.go:193]     clickhouse tables not ready for project <id>
E profiles.go:166] clickhouse tables not ready for project <id>
E space_manager.go:211] failed for cluster ...: could not get cluster topology: code: 722

So the failure was known internally, continuously, for seventeen hours, and was reported nowhere an operator would see it.

What we would like

Not a failing readiness probe - that would remove the pod from service and lose the metrics that still work, making things worse. The request is that the degraded state becomes observable:

1. Expose ClickHouse availability as a distinct signal - a field on /health, a separate endpoint, or a Prometheus metric such as coroot_clickhouse_migration_failed. Anything an operator can alert on.
2. Surface it in the UI. The Traces, Logs and Profiles views currently render as though there is simply no data. Saying the backend is unavailable, and ideally showing the underlying ClickHouse exception, would have turned a 17-hour outage into a 5-minute fix - the exception named the exact table and the exact setting needed.

Related

• #910 (open) - official Grafana dashboards for self-monitoring Coroot components. Same underlying gap; this issue is the minimum signal those dashboards would need.
• #712 (closed) - same clickhouse tables not ready message, but caused by running multiple Coroot replicas. This report is a different cause: a single replica, with ClickHouse itself refusing to attach a table.