[Bug] JDBC connection leak when BaseJdbcExecutor initialization fails after getConnection() succeeds

Search before asking

• I had searched in the issues and found no similar issues.

Version

The issue was reproduced on:

• Apache Doris: 2.1.11
• JDBC Catalog: Microsoft SQL Server
• HikariCP: 4.0.3

I also reviewed the current Doris master branch.

The current master still uses the same general initialization pattern in BaseJdbcExecutor: a JDBC connection is borrowed from HikariCP before statement initialization is completed.

If initialization fails after getConnection() succeeds, the borrowed connection needs to be explicitly released.

The current master uses HikariCP 6.0.0, but HikariCP still has the aliveBypassWindowMs optimization, so upgrading HikariCP itself does not eliminate this trigger condition.

What's Wrong?

There appears to be a JDBC connection leak path in BaseJdbcExecutor when a connection has already been successfully borrowed from HikariCP, but subsequent executor initialization fails.

The relevant initialization flow is conceptually:

conn = hikariDataSource.getConnection();
initializeStatement(conn, config, sql);

If:

getConnection() succeeds
        ↓
initializeStatement() fails
        ↓
BaseJdbcExecutor initialization fails

the already borrowed connection must always be returned to HikariCP.

Otherwise, because initialization did not complete successfully, the normal BaseJdbcExecutor.close() lifecycle may never be reached and the corresponding Hikari PoolEntry can remain permanently IN_USE.

Production symptom

We reproduced a connection pool exhaustion problem using a Microsoft SQL Server JDBC Catalog.

The Hikari active connection count gradually increases over time.

Eventually the pool reaches:

total = maximumPoolSize
active = maximumPoolSize
idle = 0

After that, JDBC Catalog queries fail because HikariCP cannot provide another connection.

Changing connection_pool_max_size causes Doris to create another Hikari pool, which temporarily restores service.

Runtime evidence

Using Arthas attached to the JVM embedded in Doris BE, we inspected the leaked Hikari PoolEntry instances.

The leaked entries consistently have this state:

PoolEntry state = IN_USE
physical connection closed = true
openStatements = 0

Examples observed in our environment:

ConnectionID:13599 | physicalClosed=true | statements=0
ConnectionID:13633 | physicalClosed=true | statements=0
ConnectionID:13557 | physicalClosed=true | statements=0
ConnectionID:13578 | physicalClosed=true | statements=0
ConnectionID:13581 | physicalClosed=true | statements=0
ConnectionID:13680 | physicalClosed=true | statements=0
ConnectionID:13662 | physicalClosed=true | statements=0
ConnectionID:13665 | physicalClosed=true | statements=0
ConnectionID:13580 | physicalClosed=true | statements=0
ConnectionID:13595 | physicalClosed=true | statements=0
ConnectionID:13627 | physicalClosed=true | statements=0
ConnectionID:13660 | physicalClosed=true | statements=0

These PoolEntry objects remain IN_USE indefinitely even though the underlying SQL Server physical connection has already been closed and there are no open statements.

Therefore each occurrence permanently consumes one connection-pool slot.

Confirmed SQL Server abort path

We also traced the physical connection lifecycle with Arthas.

Doris explicitly invokes abort() on SQL Server JDBC connections through this runtime call chain:

BaseJdbcExecutor.close()
  -> SQLServerJdbcExecutor.abortReadConnection()
  -> HikariProxyConnection.abort()
  -> SQLServerConnection.abort()

Therefore the closed physical connections are not caused only by an external network or SQL Server event. Doris itself can intentionally abort the SQL Server connection as part of the executor close path.

Afterwards, Hikari handles recycling/eviction of that connection.

Hikari alive bypass window

We additionally traced Hikari connection borrow/recycle events.

For one connection:

16:59:50.549  borrow ConnectionID:13595
16:59:50.554  recycle ConnectionID:13595
16:59:50.964  borrow ConnectionID:13595 again

The second borrow occurred approximately 410 ms after recycle.

Later this PoolEntry was observed as:

ConnectionID:13595
IN_USE
physicalClosed=true
statements=0

Another example:

16:59:53.451  borrow ConnectionID:13599
16:59:53.456  recycle ConnectionID:13599
16:59:53.930  borrow ConnectionID:13599 again

The second borrow occurred approximately 474 ms after recycle.

It later remained:

ConnectionID:13599
IN_USE
physicalClosed=true
statements=0

HikariCP has an aliveBypassWindowMs optimization whose default value is 500 ms. A recently used connection can therefore be borrowed again without a liveness check during this window.

This provides a plausible trigger for the issue:

Doris aborts SQL Server physical connection
        ↓
connection is recycled/handled by Hikari
        ↓
connection is borrowed again very quickly
        ↓
Hikari liveness check may be bypassed
        ↓
Doris receives an already closed physical connection
        ↓
executor initialization fails
        ↓
borrowed PoolEntry is not released
        ↓
IN_USE + physicalClosed=true

The final initialization-failure step above is the suspected connection leak path based on the current BaseJdbcExecutor resource lifecycle.

The important underlying issue is independent of why the connection is invalid:

After hikariDataSource.getConnection() succeeds, every subsequent initialization failure path must guarantee that the borrowed connection is released.

What You Expected?

Once Doris successfully obtains a JDBC connection from HikariCP, every subsequent failure during BaseJdbcExecutor initialization should release all partially initialized JDBC resources before propagating the exception.

Expected lifecycle:

Hikari getConnection()
        ↓
initialize executor
        ↓
initialization fails
        ↓
close Statement / ResultSet if created
        ↓
close/return Connection
        ↓
throw JdbcExecutorException

A failed initialization should never leave a Hikari PoolEntry permanently in this state:

IN_USE
physicalClosed=true
openStatements=0

Even if the physical JDBC connection has already been closed or aborted, the Hikari proxy still needs to be properly closed/released so that the PoolEntry can be evicted instead of permanently consuming a connection-pool slot.

How to Reproduce?

How to Reproduce?

The issue is timing-sensitive, but it can be reproduced with a Microsoft SQL Server JDBC Catalog by repeatedly executing simple queries.

1. Create a SQL Server JDBC Catalog

Using a relatively small connection pool makes the problem easier to observe.

For example:

CREATE CATALOG sqlserver_catalog
PROPERTIES (
    "type" = "jdbc",
    "user" = "xxx",
    "password" = "xxx",
    "jdbc_url" = "jdbc:sqlserver://xxx:1433;databaseName=testdb;encrypt=false",
    "driver_url" = "xxx",
    "driver_class" = "com.microsoft.sqlserver.jdbc.SQLServerDriver",
    "connection_pool_max_size" = "10"
);

Use the default HikariCP aliveBypassWindowMs configuration.

Do NOT start BE with:

-Dcom.zaxxer.hikari.aliveBypassWindowMs=0

2. Repeatedly execute a simple query through the JDBC Catalog

For example:

SELECT TOP 100 *
FROM test_table;

Execute the query repeatedly.

Running multiple queries concurrently makes the timing issue easier to trigger.

The important point is not the specific SQL statement. The issue is related to the JDBC connection lifecycle rather than any specific table or business SQL.

3. Observe the Hikari pool inside BE

Attach Arthas to the JVM embedded in the Doris BE process.

List Hikari pools:

vmtool --action getInstances \
  --className com.zaxxer.hikari.pool.HikariPool \
  --limit 500 \
  --express 'instances.{#this.toString()+" | total="+#this.getTotalConnections()+" | active="+#this.getActiveConnections()+" | idle="+#this.getIdleConnections()+" | waiting="+#this.getThreadsAwaitingConnection()}'

During reproduction, active connections may gradually accumulate.

Eventually the affected pool may reach:

total=10
active=10
idle=0

At this point, subsequent JDBC operations fail to acquire a connection.

4. Inspect the leaked PoolEntry objects

Identify the affected pool, for example HikariPool-XXX, and execute:

vmtool --action getInstances \
  --className com.zaxxer.hikari.pool.PoolEntry \
  --limit 500 \
  --express 'instances.{?#this.getPoolName().equals("HikariPool-XXX") && #this.getState()==1}.{#this.connection.toString()+" | physicalClosed="+#this.connection.isClosed()+" | statements="+#this.openStatements.size()}'

The leaked entries can be observed in the following state:

IN_USE
physicalClosed=true
statements=0

These entries remain IN_USE indefinitely even though the underlying physical JDBC connection is already closed.

5. Trace the SQL Server abort path

The following Arthas watch can be used to observe SQL Server connection aborts:

watch com.microsoft.sqlserver.jdbc.SQLServerConnection abort \
'{target.toString(),@java.lang.Thread@currentThread().getName()}' \
-b -x 3

The observed call chain in our environment is:

com.microsoft.sqlserver.jdbc.SQLServerConnection.abort()
com.zaxxer.hikari.pool.HikariProxyConnection.abort()
org.apache.doris.jdbc.SQLServerJdbcExecutor.abortReadConnection()
org.apache.doris.jdbc.BaseJdbcExecutor.close()

6. Verify the workaround

Restart BE with:

-Dcom.zaxxer.hikari.aliveBypassWindowMs=0

Ensure that a new Hikari pool is created after applying the property.

Verify the actual value:

vmtool --action getInstances \
  --className com.zaxxer.hikari.pool.HikariPool \
  --limit 500 \
  --express 'instances.{#this.toString()+" | aliveBypassWindowMs="+#this.aliveBypassWindowMs}'

The new pool should report:

HikariPool-XXX | aliveBypassWindowMs=0

Repeat the same query workload.

In our environment, after aliveBypassWindowMs=0 was actually applied to a newly created pool, the IN_USE + physicalClosed=true leak could no longer be reproduced.

Anything Else?

Workaround

The following JVM system property currently prevents the issue in our environment:

-Dcom.zaxxer.hikari.aliveBypassWindowMs=0

This should only be considered a workaround.

It prevents HikariCP from quickly handing an invalid recently used connection back to Doris without a liveness check, but it does not fix the resource management issue in BaseJdbcExecutor.

It may also add additional connection validation overhead.

Why this should be fixed in Doris

A JDBC connection can become invalid for many reasons:

• Doris intentionally aborting the physical connection
• database-side disconnect
• socket/network failure
• database restart
• firewall/NAT timeout
• JDBC driver error
• other transient connection failures

Therefore Doris should not rely on HikariCP always returning a usable connection.

The lifecycle should guarantee:

getConnection() succeeds
        ↓
ownership of the borrowed connection belongs to BaseJdbcExecutor
        ↓
ANY later initialization failure
        ↓
connection must be released

Possible fix

A possible fix would be to make BaseJdbcExecutor.init() clean up all resources that may have been created before the initialization failure.

Conceptually:

try {
    conn = hikariDataSource.getConnection();
    initializeStatement(conn, config, sql);
} catch (Exception e) {
    try {
        if (resultSet != null) {
            resultSet.close();
        }
    } catch (Exception closeException) {
        LOG.warn("Failed to close ResultSet after JDBC initialization failure",
                closeException);
    }

    try {
        if (stmt != null) {
            stmt.close();
        }
    } catch (Exception closeException) {
        LOG.warn("Failed to close Statement after JDBC initialization failure",
                closeException);
    }

    try {
        if (conn != null) {
            conn.close();
        }
    } catch (Exception closeException) {
        LOG.warn("Failed to release JDBC connection after initialization failure",
                closeException);
    }

    resultSet = null;
    stmt = null;
    conn = null;

    throw new JdbcExecutorException("Initialize datasource failed: ", e);
}

This is only an example. The actual fix should reuse the existing Doris JDBC resource cleanup mechanism where appropriate and preserve the original exception.

The essential requirement is:

After a successful getConnection(), every failure path during executor initialization must guarantee that the Hikari connection is released.

Current master

Although our runtime reproduction was performed on Doris 2.1.11, the issue is being reported against the current code path because the same resource ownership problem can still occur if statement initialization fails after a Hikari connection has already been borrowed.

Current master uses HikariCP 6.0.0.

HikariCP still has the alive-bypass optimization, so upgrading from the HikariCP version used by Doris 2.1.11 does not by itself eliminate this type of trigger.

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct