Expose `connectAttributes` to PostgreSQL
Is your feature request related to a problem? Please describe.
When running Temporal with a Postgres datastore via the temporalio/auto-setup image, there is no
way to set connect_timeout on the database connection.
config_template.yaml exposes connectAttributes only in the MySQL8 datastore block. The
postgres12 / postgres12_pgx blocks (both default and visibility) have no
connectAttributes and no equivalent environment variable, so there is no supported way to pass
driver connection parameters to Postgres.
This is not a missing Temporal feature — common/config.SQL already has
ConnectAttributes map[string]string yaml:"connectAttributes"\ and it is plugin-agnostic, and
the Postgres buildDSN already appends those attributes to the DSN as query parameters. The gap
is purely that the Docker config template never surfaces it for Postgres.
Why this matters in practice: with no connect_timeout, lib/pq applies no deadline when opening
a connection, so a connection attempt can block indefinitely. We hit this in production on with temporalio/auto-setup:1.27.2. Postgres reset our connections
(read: connection reset by peer), Temporal's DatabaseHandle.reconnect() attempted to rebuild
the pool, and the new connection hung in the SSL negotiation read
(pq.(*conn).ssl → io.ReadFull) and never returned.
Because DatabaseHandle.reconnect() holds its mutex while calling connect(), every goroutine
needing the database blocked behind it. Goroutine profiles showed ~140 goroutines waiting on that
mutex via DB(), Conn() and ConvertError(), including all 82 task-queue reader/writer loops.
Matching then deadlocked against itself: 66 goroutines in userDataManagerImpl.fetchUserData
waiting on a GetTaskQueueUserData gRPC call, and 63 in HandleGetUserDataRequest — the server
side of that same call — blocked on the mutex. The result was 622 error fetching user data from parent / context deadline exceeded errors over 52 minutes.
Temporal's own deadlock detector fired 9 times and produced the profiles above. The process stayed alive the whole time and only recovered when we restarted the pod manually.
Describe the solution you'd like
Expose connection attributes for the Postgres datastores in config_template.yaml, the same way
MySQL8 already does.
Simplest version — a dedicated variable mirroring the existing SQL_* naming, added to both the
default and visibility postgres blocks:
connectAttributes:
connect_timeout: "{{ default .Env.SQL_CONNECT_TIMEOUT "10" }}"A non-empty default would also give every Postgres user protection against this class of hang out of the box. If a default change is unwelcome, leaving it unset and only rendering the block when the variable is present would still solve our problem.
A more general alternative would be a passthrough for arbitrary attributes (e.g.
SQL_CONNECT_ATTRIBUTES as comma-separated key=value pairs), which would cover future driver
parameters too. We'd be happy with either.
Describe alternatives you've considered
Replacing config_template.yaml with our own copy. This would work, but the cost is that we take ownership of a ~445-line upstream file, pinned to one Temporal version, in order to add two lines.
Additional context
- Image:
temporalio/auto-setup:1.27.2
Source: temporalio/temporal