TiDB: Support Graceful Shutdown behavior of TiDB Pods
Description
When a pod of a TiDB cluster is deleted, then it enters a graceful shutdown phase.
- Its HTTP health endpoint returns a failure status — signaling load balancers to reroute traffic.
- It stops responding normally to COM_PING commands
- After configured timeout the pod is terminated
Following Keycloak configuration would play nice with TiDB Graceful Shutdown behavior:
Set QUARKUS_DATASOURCE_JDBC_VALIDATE_ON_BORROW to true Before a connection from the DB Pool is used, it will be validated. If the connection is using a tidb pod in graceful shutdown phase it will return an error during validation and the connection will be removed from the pool.
ConnectionValidator via PING However, the standard SQL validation query (SELECT 1, a COM_QUERY command) continues to return success during this phase, making it unsuitable for detecting graceful shutdown. We should implement a custom connection validation mechanism (Agroal ConnectionValidator ) in keycloak. Its only activated if TiDB is used. This custom ConnectionValidator implements low-level ping validation (COM_PING). An additional benefit is, that its not that "expensive" than a SQL query.
New connection requests are already routed to "healthy" tidb pods automatically.
Value Proposition
Zero downtime during TiDB pod restarts, rolling upgrades, or node rebalancing. Keycloak's connection pool lifecycle is aligned with TiDB's graceful shutdown protocol No transaction failures due to TiDB pod terminations.
Goals
Ensure Keycloak handles TiDB pod graceful shutdown transparently by validating connections before use (QUARKUS_DATASOURCE_JDBC_VALIDATE_ON_BORROW) and implementing a TiDB-specific COM_PING-based ConnectionValidator via the Agroal SPI
Non-Goals
Discussion
No response
Notes
No response
Source: keycloak/keycloak