[Bug]: POST /rpc/* intermittently 500s with "socket hang up" — stale-socket replay is capped at one attempt for non-idempotent methods
Author: thinkenvyCreated Sep 9, 2026Updated Sep 10, 2026
Labelsbugneeds-triage
What's the bug?
Database RPC calls intermittently return 500 with "socket hang up". The request never reaches PostgREST, so nothing appears in the Postgres or PostgREST logs.
{"error":"INTERNAL_ERROR","message":"socket hang up","statusCode":500}
It's always a POST. Since every RPC is a POST, this hits read-only functions too, and a
signed-in user sees a broken page for no reason they can act on.
The asymmetry is visible in the gateway's own logs. A GET on a dead pooled socket
recovers:
{"errorCode":"ECONNRESET","message":"PostgREST request failed, retrying (attempt 1/3) socket hang
up","method":"GET","staleSocketRetry":true,"url":"http://postgrest:3000/some_table"}
{"errorCode":"ECONNRESET","message":"PostgREST request failed, retrying (attempt 2/3) socket hang
up","method":"GET","staleSocketRetry":false,"url":"http://postgrest:3000/some_table"}
A POST in the same second does not:
{"errorCode":"ECONNRESET","message":"PostgREST stale-socket replay of non-idempotent request (attempt 1/3) socket hang
up","method":"POST","staleSocketRetry":true,"url":"http://postgrest:3000/rpc/some_function"}
{"error":"socket hang up","level":"error","stack":"Error: socket hang up\n at AxiosError.from ..."}
{"duration":"25ms","method":"POST","path":"/api/database/rpc/some_function","status":500}
The 25ms duration is consistent with writing to an already-closed socket.
Analysis. In backend/src/services/database/postgrest-proxy.service.ts,
staleSocketRetryUsed allows exactly one stale-socket replay per request. Idempotent
methods can then continue through the general retry path, which is why the GET above got a
second attempt and the POST didn't. With maxFreeSockets at 10, more than one pooled
socket can be stale at the same moment, so the single replay lands on another dead one and
the request fails.
The reasoning already in your code comment looks right to us: a connection reset with no
bytes written is safe to replay regardless of method, because the server never saw the
request. The gap seems to be only the attempt count.
What we tried. We set POSTGREST_FREE_SOCKET_TIMEOUT_MS=1000, down from the default
4000, so idle sockets are reaped well before the server can close them. It reduced the
frequency but did not eliminate it. That suggests the window isn't only about idle
duration, so tuning this value isn't a fix on its own. We haven't measured PostgREST's own
idle timeout, so we can't say how the two relate precisely.
Suggested fix. Allow the stale-socket replay to retry more than once for non-idempotent
methods, since a pre-write ECONNRESET carries no risk of duplicate execution. Bounding it
by maxFreeSockets, or retrying until the agent returns a socket that isn't a reused one,
would both cover the case where several pooled connections expire together.
Happy to test a patch against a live install.
### How to reproduce
```markdown
It's intermittent, and we haven't found a deterministic trigger. Roughly 1 in 15 loads of a
page that calls an RPC.
What does NOT reproduce it: a serial loop. We ran 70 sequential POSTs to the same RPC with
5-second gaps, which should be ample for the 4-second free-socket timeout to reap between
calls, and got 70/70 success.
What does: ordinary browser traffic, where a single page load fires several requests
concurrently. That fills the pool with several free sockets at once, and our reading is
that it takes several of them going stale together to get past the one permitted replay.
So the shape that reproduces it is:
1. Run the stack from the repo's docker-compose with default socket settings.
2. Load a page that issues several parallel authenticated requests, at least one of them a
POST to /api/database/rpc/<function>.
3. Wait 10 to 30 seconds so the pooled connections go idle.
4. Reload. Repeat 15 to 20 times.
5. Watch the gateway container logs for `stale-socket replay of non-idempotent request`.
The 500 follows immediately after it, in the same millisecond range.
`docker logs -f <gateway container> 2>&1 | grep -i "socket hang up"` is the quickest way to
see it, since the browser only shows a generic 500.
---
### Environment (optional)
- InsForge v2.3.1 and v2.3.2, self-hosted from the repo's docker-compose
- PostgREST 12.2.12, same Docker network, POSTGREST_BASE_URL=http://postgrest:3000
- POSTGREST_MAX_SOCKETS and POSTGREST_MAX_FREE_SOCKETS at defaults (50 and 10)
- POSTGREST_FREE_SOCKET_TIMEOUT_MS set to 1000 (default 4000), which reduced but did not
remove the problem
- Node 20 Alpine, Linux x86_64
Note: backend/src/services/database/postgrest-proxy.service.ts is byte-identical between
the v2.3.1 and v2.3.2 tags, so upgrading doesn't change this behavior.
Notes
- I genericized the two schema names to some_table and some_function, since you asked about disclosure last time and they add nothing for the maintainer.
- The "what we tried" and "what does not reproduce" parts are the highest-value bits. They save a maintainer from suggesting the socket-timeout knob or a serial repro,
both of which we've already ruled out.Source: InsForge/InsForge