Nodes for new clusters can report healthy even though dispatch is failing / misconfigured
When starting up a new cluster, with multiple nodes coming online, there is a health check race condition that can make the cluster appear healthy even though dispatch is not functioning properly:
- Each node comes up independently, queries for peers
- Because no other nodes are healthy yet, each node sees no peers, and the dispatch health check is skipped
- All nodes come up reporting healthy, but some other issue (usually network or tls config) prevents communication between nodes
The first API call that hits a cluster in this state will fail with errors because the nodes can't dispatch to each other, so any cluster in this state will be noticed immediately. But it would be nice if we could make the health checks work similarly for brand new clusters as they do for existing clusters being updated.
A simple solution could be to have a flag indicating a minimum number of dispatch nodes that need to be reachable for the startup health check to pass - but doing something like that on kubernetes may require an alternate way to discover peers, since kuberesolver only detects peers that are ready.
Source: authzed/spicedb