BUG: Connection loss / utility crash is unrecoverable — RPC layer has no timeout and never rejects in-flight requests

Summary

When a database connection dies or the utility process crashes, the app becomes permanently unrecoverable: you cannot switch connections, cannot ROLLBACK an open transaction, and cannot cancel the hung query. The only escape is closing the window.

There are already several issues describing the symptoms of this (#389, #2705, #2845, #3616, and the "Utility Process Crashed" cluster #3180 / #3347 / #3437 / #3556 / #4387), and #3958 adds detection + a reconnect prompt. This issue is about a distinct, lower-level cause that none of those cover: the renderer↔utility RPC layer has no timeout and never rejects in-flight promises when the transport dies. Because almost every UI state flag is reset in a finally block, a promise that never settles leaves the UI wedged forever — regardless of how good connection-loss detection gets.

All references below are against 3abb5a8 (paths relative to apps/studio/).

Root cause 1: UtilityConnection has no timeout and never rejects on transport death

src/lib/utility/UtilityConnection.ts:19 parks every request in a map:

private replyHandlers: Map<string, { resolve: any, reject: any }> = new Map();

send() (lines 112-132) adds an entry and posts to the port. There is no setTimeout, no Promise.race, no port.onclose and no onmessageerror anywhere in the file, and replyHandlers.delete() only runs when a reply actually arrives (lines 61, 85).

Consequences when the socket or the utility process dies mid-request:

• Every in-flight await $util.send(...) becomes a permanently pending promise. It is never rejected.
• setPort() (line 39) overwrites this.port on restart and flushes only messageQueue; the previously-pending replyHandlers entries are orphaned but still block their callers.
• this.port is never nulled, so calls made after the crash take the else branch at line 123 and postMessage into a dead port instead of being queued for the replacement port.
• portsRequested (lines 24, 121) is latched true forever, so the request-ports recovery path can fire at most once per window.

Because callers reset their state in finally, this is what wedges the UI:

• src/components/ConnectionInterface.vue:585-598 — connecting = false is in finally, so Connect/Test stay disabled (:disabled="testing || connecting", lines 209, 217).
• src/components/TabQueryEditor.vue:1763-1850 — running / tab.isRunning reset only in finally (1847-1850), so the tab shows "Running" forever.

Related: hasWorkingPort() (lines 30-37) is the only health-check hook that exists and it cannot work — this.port.postMessage({name: test}) at line 35 references an undefined identifier test, and the if (!this.port) reject() on line 32 has no return. It is never called.

Additionally, src-commercial/entrypoints/main.ts:70-79 only restarts on a non-zero exit code:

utilityProcess.on('exit', async (code) => {
  log.log("UTILITY DEAD", code)
  if (code) {
    ...

A clean or signal-reported-as-0 exit restarts nothing and notifies nobody, while the stale utilityProcess reference is retained.

Root cause 2: a failed disconnect() permanently blocks connection switching

src/store/index.ts:598-609:

async disconnect(context) {
  if (context.state.connection) {
    await context.state.connection.disconnect();
  }

  window.main.disableConnectionMenuItems();

  context.commit('clearConnection')
  ...

clearConnection is what resets connected, usedConfig, server and tables. If line 600 throws or hangs, it never runs and the app believes it is still connected. state.connection is a permanent client instance, so the guard on 599 is always true.

Line 600 can hang (BasicDatabaseClient.ts:158 awaits sshTunnel.connection.shutdown() on a dead tunnel; it also awaits the shared module-level knex.destroy()) or throw — see root cause 3.

Every "switch connection" call site awaits this outside its try/catch, so a single failure kills the switcher:

• src/components/sidebar/core/ConnectionButton.vue:288
• src/components/quicksearch/QuickSearch.vue:275
• src/components/sidebar/CoreSidebar.vue:122
• src/components/UtilDiedModal.vue:41 and src/components/LostConnectionModal.vue:77 — dispatched without await and without catch, so the modal closes on a floating rejected promise and nothing is cleaned up.

This is very likely the mechanism behind #3616 and #2705.

Root cause 3: getDriverHandler null-derefs after a utility restart

src/handlers/handlerState.ts:92-96:

export function getDriverHandler(name: string) {
  return async function({sId }: { sId: string }): Promise<any> {
    return await state(sId).connection[name]();
  }
}

There is no null check, and this deliberately bypasses checkConnection (lines 98-102). A restarted utility has a fresh State with connection = null, so conn/disconnect (registered via getDriverHandler('disconnect') in src-commercial/backend/handlers/connHandlers.ts:277) throws a raw TypeError: Cannot read properties of null.

The result is that the Disconnect button on the crash modal is itself broken by the crash it is reporting, and it feeds directly into root cause 2.

Also worth noting: 'conn/clearConnection' (connHandlers.ts:253-259) is never called from the renderer, so state(sId).connection / server are never cleared.

Root cause 4: ROLLBACK is pinned to the connection that just died

Transactions reserve one physical client keyed by tab (BasicDatabaseClient.ts:82, src/lib/db/clients/postgresql.ts:1352-1384, mysql.ts:1523-1565):

async rollbackTransaction(tabId) { await this.runQuery(this.peekConnection(tabId), 'ROLLBACK', {}); }

So a user-initiated rollback has no path to success:

1. Socket dead, process alive — ROLLBACK is written to a dead socket with no statement_timeout and no keepalive, so it hangs. If a statement is still outstanding, pg.Client queues the ROLLBACK behind it and it can never be dispatched.
2. Utility restarted — reservedConnections is empty, but the renderer still has hasActiveTransaction = true, so the button stays enabled (TabQueryEditor.vue:180, 189) and the user gets peekConnection's message from BasicDatabaseClient.ts:807-809: "Could not retrieve reserved connection, please report this issue on our GitHub."

manualRollback / manualCommit (TabQueryEditor.vue:1954-1979) have no try/catch, so releaseConnection never runs and the pooled client leaks. With maxReservedConnections = 2 (default.config.ini:98), two wedged tabs block all future transactions via postgresql.ts:1356-1357.

This is the same user-visible complaint as #2845 ("the only way to retry is to close the tab"), reached by a different route.

Root cause 5: Postgres query cancel deadlocks against itself

src/lib/db/clients/postgresql.ts:830-850:

cancel: async (): Promise<void> => {
  if (!pid) { throw new Error('Query not ready to be canceled'); }
  canceling = true;
  try {
    const data = await this.driverExecuteSingle(`SELECT pg_cancel_backend(${pid});`, { tabId });
    ...
    cancelable.cancel();

Passing { tabId } routes the cancel through rawExecuteQuery → peekConnection(tabId) (lines 1386-1401), i.e. the very connection that is blocked running the query. pg_cancel_backend is queued behind the hung statement and never executes, so cancelable.cancel() on line 845 is never reached and the Promise.race in execute() never settles.

Meanwhile cancelQuery() (TabQueryEditor.vue:1408-1416) has already set running = false and displayed "Query Execution Cancelled" before awaiting the cancel, so the UI claims success while the backend statement is still live.

(MySQL avoids this by accident — mysql.ts:1157 omits tabId, so the kill takes a different pool connection. But mysql.ts:146-161 sets no queueLimit and no acquire timeout, so once the pool is exhausted getConnection() waits indefinitely.)

Why #3958 doesn't fully resolve this

#3958 is a good step and I don't want to duplicate it — it wires up connection-lost and the reconnect prompt, which is real progress on #389 / #2370. But its scope is detection and re-establishment:

• It doesn't touch src/lib/utility/UtilityConnection.ts at all, so root cause 1 is unchanged.
• store/index.ts disconnect appears only as unchanged context in its diff, so root cause 2 is unchanged.
• Root causes 3, 4 and 5 are untouched.

The PR description frames the residual problem as detection latency ("hangs for roughly 60 seconds before the socket error is thrown"). That's accurate, but it understates the issue: even once the socket error does fire, pool.on('error') at postgresql.ts:179-181 only logs it, and the already-pending replyHandlers promise is still never rejected. So the UI stays wedged after detection, not just before it.

Supporting evidence that detection alone was never wired up: the auto-reconnect path in BasicDatabaseClient.ts:677-688 and 716-727 is commented out, checkIsConnected() (line 566) is referenced only from those comments, and setConnError is committed in exactly one place — with null (LostConnectionModal.vue:66). No keepAlive, enableKeepAlive, statement_timeout or query_timeout appears anywhere under src/lib/db/clients/.

Suggested direction

Roughly in order of leverage:

1. Add a timeout and transport-death rejection to UtilityConnection. On port.onclose / utility exit, reject all outstanding replyHandlers with a distinguishable error, clear the map, null this.port, and reset portsRequested. This alone converts most of these permanent hangs into catchable errors that let existing catch/finally blocks run. Long-running queries need an exemption or a heartbeat rather than a flat deadline.
2. Make disconnect unconditionally reach clearConnection — wrap line 600 in try/finally — and add null guards to getDriverHandler, so a dead backend can always be abandoned.
3. Make main.ts restart on exit code 0 as well, and notify the renderer on every death.
4. Offer a "reset session" / forced local rollback that discards the reserved connection and clears hasActiveTransaction when the pinned connection is known dead, instead of trying to speak to it (addresses #2845).
5. Route pg_cancel_backend over a fresh pool connection, not the reserved one, and only update the UI after the cancel actually confirms.
6. Set keepAlive on pg and enableKeepAlive on mysql2 to shorten the OS-level detection window noted in #3958.

I'm happy to open a PR for items 1-3 if maintainers agree with the direction — they're small and mostly independent of #3958, so they shouldn't create conflicts.

Environment

Analysis is static, against master at 3abb5a8. The behaviour is not driver-specific: root causes 1-3 are in the transport and store layers, and apply to every connection type.