#7804·redash

MySQL query runner has no read_timeout — a dropped connection makes queries hang forever and exhausts the worker pool

Author: deemoowoorCreated Sep 7, 2026Updated Sep 7, 2026

Issue Summary

The MySQL query runner (redash/query_runner/mysql.py) has no read_timeout/write_timeout option. When the underlying TCP connection to MySQL dies silently after a query is sent (e.g. a proxy or network layer drops it without sending RST), the worker thread blocks forever in cursor.execute(). The job never reaches finished or failed — it stays in executing_query state permanently, and the worker slot is never freed. Enough of these accumulate and they exhaust the whole worker pool, so unrelated queries start timing out too.

Steps to Reproduce

  1. Point a MySQL data source at a host/proxy that will silently drop an established connection mid-session (e.g. an idle-timeout on a proxy/load balancer sitting in front of MySQL that doesn't send a TCP RST on expiry — ProxySQL, some cloud NLBs, and some CNI conntrack GC configs can all do this).
  2. Run a query on that data source. If the drop happens after the query is sent but before the result is read back, the query never completes.
  3. Watch the RQ job: redash.tasks.queries.execution.execute_query logs state=executing_query and never logs state=finished or state=failed. It stays that way indefinitely (observed >50 minutes before we intervened).
  4. Repeat step 2 a few times (e.g. a user re-clicking "refresh" because the query "seems slow") — each attempt permanently consumes one more worker, since the job that owns it never exits. Eventually all workers are wedged and the whole install stops executing any query, including unrelated ones on unrelated data sources.

Root cause (from reading redash/query_runner/mysql.py on master)

configuration_schema() only exposes connect_timeout (bounds the initial TCP handshake). _connection() only forwards that one timeout to the driver:

python
params = dict(
    host=..., user=..., passwd=..., db=..., port=...,
    charset=..., use_unicode=...,
    connect_timeout=self.configuration.get("connect_timeout", 60),
    autocommit=...,
)
connection = MySQLdb.connect(**params)

mysqlclient's C extension (_mysql.c) does support read_timeout/write_timeout kwargs — they map directly to mysql_options(MYSQL_OPT_READ_TIMEOUT/WRITE_TIMEOUT, ...) — but mysql.py never passes them through, so there is no way to bound how long cursor.execute() can block waiting on a read that will never arrive.

It's worse than "the query just runs long," because the runner's own cancellation path can't recover from this state either. run_query() runs the query in a background thread and polls an Event in the main thread, so RQ's JobTimeoutException can interrupt the main thread. But its handler calls _cancel(thread_id), which opens a new connection and issues KILL <thread_id> on the server, then does t.join() on the background thread. If the original connection is already dead, there's no live server-side thread left for KILL to act on, the background thread is still blocked in its (timeout-less) socket read, and t.join() has no timeout of its own — so the main thread now hangs too, waiting on the join.

Proposed fix

Add read_timeout (and optionally write_timeout) to Mysql.configuration_schema() and forward it in _connection(), the same way vertica.py already does (read_timeout was added there in #1528). With a read timeout set, a dead connection makes cursor.execute() raise inside the background thread on its own — caught by the existing except MySQLdb.Error in _run_query() — which sets the error and wakes the main thread's poll loop normally. No KILL/t.join() path needed at all for this failure mode.

Happy to open a PR for this if a maintainer confirms the approach — it looks like a small, additive change to configuration_schema() and _connection().

Technical details

  • Redash Version: 25.8.0 (Docker)
  • Data source: MySQL, behind ProxySQL in front of a Percona XtraDB Cluster
  • How installed: Docker on Kubernetes

Related: #7061 (same symptom family — silent connection loss causing a permanent hang with no error — reported against the Postgres query runner, still open, root cause not identified there).