Fail Fast on bad credentials
HikariPool takes 30 seconds to report that a database password is bad. This seems wrong. I've searched online for how to "fail fast" when I pass bad credentials to hikari. The closest I've found is: https://github.com/brettwooldridge/HikariCP/issues/32
which introduced initializationFailFast (later replaced by initializationFailTimeout).
That seems to control how quickly hikari will fail when first starting up. My situation is different--I am looking at a case where my service (and hikari) are already running, and the password on the database gets changed. I can recover dynamically by looking to our secret management solution (vault) to get an updated password. But it takes 30 seconds for hikari to let my code know that it failed.
The stack trace I examined:
at com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:696) at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:181) at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:146) at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:100) caused by org.postgresql.util.PSQLException: FATAL: password authentication failed for user "exd_system" at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:659) at org.postgresql.core.v3.ConnectionFactoryImpl.tryConnect(ConnectionFactoryImpl.java:180) at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:235) at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49) at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:247) at org.postgresql.Driver.makeConnection(Driver.java:434) at org.postgresql.Driver.connect(Driver.java:291) at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:138) at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:359) at com.zaxxer.hikari.pool.PoolBase.newPoolEntry(PoolBase.java:201) at com.zaxxer.hikari.pool.HikariPool.createPoolEntry(HikariPool.java:470) at com.zaxxer.hikari.pool.HikariPool$PoolEntryCreator.call(HikariPool.java:733) at com.zaxxer.hikari.pool.HikariPool$PoolEntryCreator.call(HikariPool.java:712)
and the HikariPool is pretty obviously waiting 30 seconds (configurable) for a connection, no matter what connection error occurs.
public Connection getConnection(final long hardTimeout) throws SQLException { suspendResumeLock.acquire(); final var startTime = currentTime(); try { var timeout = hardTimeout; do { var poolEntry = connectionBag.borrow(timeout, MILLISECONDS); if (poolEntry == null) { break; // We timed out... break and throw exception } final var now = currentTime(); if (poolEntry.isMarkedEvicted() || (elapsedMillis(poolEntry.lastAccessed, now) > aliveBypassWindowMs && isConnectionDead(poolEntry.connection))) { closeConnection(poolEntry, poolEntry.isMarkedEvicted() ? EVICTED_CONNECTION_MESSAGE : DEAD_CONNECTION_MESSAGE); timeout = hardTimeout - elapsedMillis(startTime); } else { metricsTracker.recordBorrowStats(poolEntry, startTime); return poolEntry.createProxyConnection(leakTaskFactory.schedule(poolEntry)); } } while (timeout > 0L); metricsTracker.recordBorrowTimeoutStats(startTime); throw createTimeoutException(startTime); } catch (InterruptedException e) { Thread.currentThread().interrupt(); throw new SQLException(poolName + " - Interrupted during connection acquisition", e); } finally { suspendResumeLock.release(); } }
Waiting for a connection to get through a flaky network requires a timeout, so that code makes sense from a fault tolerance perspective. I cannot imagine a reason to retry a bad password more than once. Once Hikari knows it has a bad password, it should fail trying to get a connection immediately.
Source: brettwooldridge/HikariCP