Client wedges permanently when a server accepts a TCP stream and never answers (unbounded ReadTCPResponse)

Summary

clientImpl.TCP writes the TCP request and then reads the response with no deadline. If a server accepts the stream and then never answers — while its QUIC connection stays perfectly healthy — the call never returns.

That alone would be a stuck goroutine. What makes it an outage is the second half: reconnectableClientImpl only drops the connection when the call returns a ClosedError. A call that never returns never reaches that check, so the client never reconnects. It stays wedged until the process is restarted.

We hit this twice on 2.12.2, two days apart, with an identical signature.

What we observed

Two SIGQUIT goroutine dumps taken while the client was wedged:

goroutine population	dump A	dump B
internal/poll.runtime_pollWait (IO wait)	171	428
socks5.(*Server).handleUDP	99	200
client.(*udpConn).Receive	78	213
quic-go.(*ReceiveStream).readImpl	40	60
socks5.(*Server).handleTCP	7	4
runtime housekeeping	11	11

Scale differs, shape does not — the runtime set is identical line for line.

Only one population was actually stuck, and age is what shows it. Blocked for longer than a minute: pollWait 6/171 and 5/428, handleUDP and udpConn.Receive 4 and 1 — all cycling normally. But readImpl: 34 of 40 (85%) and 46 of 60 (77%), averaging 93 and 60 minutes.

Their stack is identical frame for frame in both dumps:

socks5.(*Server).Serve -> dispatch -> handleTCP
  -> client.(*reconnectableClientImpl).TCP -> clientDo -> TCP.func1
    -> client.(*clientImpl).TCP
      -> protocol.ReadTCPResponse          <- blocked here
        -> io.ReadFull -> utils.QStream.Read
          -> quic-go Stream.Read -> ReceiveStream.readImpl [chan receive]

33 goroutines in dump A, 56 in dump B.

Their ages give a sharp onset rather than a smear. Dump A: 24 goroutines at exactly 4 minutes, 3 at 3, 6 under a minute. Dump B: 30 at exactly 7 minutes, 3 at 5, 9 at 3, 14 under a minute. A cohort arrives the moment the condition starts, then a trickle — every new connection joining the pile.

Nothing was blocked on the send side in either dump: zero goroutines in SendStream, flowcontrol, OpenStreamSync or blockedFrame. So this is not flow control and not stream-limit exhaustion. The client opens a stream and sends; the peer simply never answers.

Why MaxIdleTimeout does not cover this

defaultMaxIdleTimeout is 30s and defaultKeepAlivePeriod is 10s. Had the connection gone silent, it would have been torn down in 30 seconds and every blocked stream would have errored out — and clientDo would have reconnected.

The goroutines were blocked 4 and 7 minutes: 8x and 14x the idle timeout. The connection was demonstrably alive the whole time; keep-alives were being answered. Only the application layer was silent. That is precisely the case connection-level failure detection cannot see.

Why the client never recovers

core/client/reconnect.go:

ret, err := f(client)
if _, ok := err.(coreErrs.ClosedError); ok {
    rc.client = nil   // reconnect next time
}

The recovery path is healthy and well armed — wrapIfConnectionClosed turns everything except quic.StreamLimitReachedError into a ClosedError. It is simply unreachable, because f(client) never returns.

What the server was doing beforehand

Both incidents had the same three-phase prelude in the client's own log, and the counts are nearly identical (35 minutes before each dump):

	incident A	incident B
dial error: ... i/o timeout	52	50
stream N canceled by remote with error code 0	8	7

1. the server answers, but cannot reach targets — its own egress degrading;
2. the server starts cancelling streams with NO_ERROR;
3. the server goes silent entirely, and from that minute every new connection wedges.

The onset of the hang coincides with the first stream cancellation, within a minute in both cases. We cannot say why the server stopped servicing streams — that is its side — but the client's behaviour in response is the same either way.

Suggested fix

Bound the response read. The machinery already exists and is simply never used on this path: utils.QStream implements SetReadDeadline and forwards it to the stream. ReadTCPResponse takes a plain io.Reader, so it cannot set one itself.

Two things seem worth getting right:

The deadline should be longer than connection-level failure detection, so that a connection which genuinely died is still diagnosed by the idle timeout exactly as today, and the new deadline only covers what the idle timeout cannot see. Deriving it from the configured MaxIdleTimeout alone is not enough: QUIC requires an endpoint to raise its effective idle period to at least three times the current PTO (RFC 9000 §10.1), so a short configuration does not buy short detection. Measured against a closed server, a client configured for maxIdleTimeout: 4s still detected the loss at 30.0s, the same as one left at the default. A deadline of max(MaxIdleTimeout, defaultMaxIdleTimeout) + grace keeps the existing behaviour intact; a naive MaxIdleTimeout + grace changes it for every config below the default, and breaks TestClientServerServerShutdown.

The grace has to clear the worst legitimate answer. The server's outbound dial is capped at 10s (defaultDialerTimeout, with the dual-stack dials run in parallel), so a legitimate answer cannot take much longer than that. Measured against our own exit: a normal answer takes 55-175ms (n=120), a resolver failure 68-161ms, and a dial that runs to the cap 10059-10405ms. We used 15s of grace, i.e. 45s total with default settings.

The FastOpen path has the same unbounded read, deferred into tcpConn.Read. We left it alone deliberately: by then the caller owns the stream and may have set its own deadline, and quic-go offers no way to read that deadline back, so bounding it there would silently clobber the caller's. It may still be worth addressing, but it needs the deadline to be tracked rather than overwritten.

Reproduction

An integration test against the existing harness reproduces it exactly: an Outbound whose TCP() blocks forever, so the server accepts the stream and never answers while its connection stays up.

serverOb.EXPECT().TCP(mock.Anything).RunAndReturn(func(string) (net.Conn, error) {
    <-release           // never released during the test
    return nil, net.ErrClosed
}).Maybe()

Unpatched, client.TCP() never returns and the test fails on its own budget. Patched, it returns a ClosedError wrapping a deadline error after max(MaxIdleTimeout, 30s) + 15s.

We are running this as a local patch. Happy to open a PR if the approach looks right — in particular whether you would rather have the grace configurable than constant.

Environment

• hysteria 2.12.2 (app/v2.12.2), built from source
• client mode, SOCKS5 inbound, single server
• no fastOpen, no maxIdleTimeout / keepAlivePeriod overrides