codec-quic: STOP_SENDING is not surfaced without pending writes or connection send capacity

Netty version: 4.2.17.Final (mainline netty-codec-classes-quic / netty-codec-native-quic); reproduced on the 4.2 branch at d64f6a2ea8.

This is an issue that was discovered while stress testing a product that uses netty's quic/quiche integration.

Expected behavior

When the remote peer sends STOP_SENDING for a stream, the local QuicStreamChannel should learn that its output is gone: queued writes fail with ChannelOutputShutdownException, isOutputShutdown() becomes true, isWritable() becomes false (a further write cannot succeed because quiche returns STREAM_STOPPED). Later writes are refused as output-shutdown, and the pipeline is told (ChannelOutputShutdownEvent) whether or not a write is pending at that moment, and whether or not the connection-level send window happens to be open.

Actual behavior

Two gaps in how QuicheQuicChannel.handleWritableStreams and QuicheQuicStreamChannel.writable(long) cooperate:

1. No pending writes: silent. QuicheQuicStreamChannel.writable(capacity) receives QUICHE_ERR_STREAM_STOPPED from the parent's writable pass and, with an empty write queue, simply return false. Nothing is fired, isOutputShutdown() stays false, and isWritable() keeps its cached value. A user that gates writes on isWritable() therefore either waits indefinitely (cached false, the stream was created or last visited under zero capacity) or, with a stale true, writes into quiche and only then discovers the stop through the STREAM_STOPPED error from quiche_conn_stream_send.

2. Connection window exhausted: never found. handleWritableStreams drives everything from quiche_conn_writable(...). quiche keeps a stopped stream in its writable set so the application observes StreamStopped when it asks, but Connection::writable() returns an empty iterator whenever the connection-level send capacity (tx_cap) is zero. A stream stopped while another stream holds the window is therefore never visited: queued writes stay pending until the connection closes, and a stream without queued writes gets no signal either. The intent in writeQueued() ("Once its signaled that the stream is stopped we can just fail everything") is only reached if some write is attempted later.

We hit both at once in production: a 2-byte SOCKS handshake reply was held behind a writability gate on a stream created under zero connection capacity (isWritable()==false, gap 1); the peer timed out after 3 s and closed the stream with STOP_SENDING while other streams kept the window exhausted (gap 2); the local channel reported isWritable()==false and nothing else for the next 120 s until our own watchdog reset it.

Proposed changes

I had a couple of agents work on a fix for this. I'm not very familiar with the quiche integration in netty, and since I didn't personally author the changes I didn't want to clutter up netty's PRs with something I wasn't sure about. So I opened the PR against my personal fork, which can be found here: https://github.com/jbaldassari/netty/pull/1 . If this looks like the correct fix, feel free to grab it, or I can re-point that PR at netty as the base.

Reproduction

Six tests added to QuicStreamShutdownTest (in the linked PR). All pass with the fix; all but the freed-connection guard fail on 4.2 for both newSslTaskExecutors variants:

• testShutdownInputByServerSignalsOutputShutdownWithoutQueuedWrites : client writes 4 bytes; the server calls shutdownInput() on read (sends STOP_SENDING). Expects ChannelOutputShutdownEvent, isOutputShutdown()==true, isWritable()==false, bytesBeforeUnwritable()==0, and a later write refused with ChannelOutputShutdownException. On 4.2: STOP_SENDING from the peer was never signaled to the stream channel.
• testShutdownInputByServerFailsWritesQueuedWhileConnectionWindowIsExhausted : server advertises initialMaxData(1024); stream 0 sends 8 bytes the server reads, stream 4 sends 1016 bytes the server never reads (per-stream ChannelInitializer, autoRead=false), so the window is spent and stream 4 reports isWritable()==false; a further 8-byte write on stream 0 is queued; the server calls shutdownInput() on stream 0. Expects the queued write to fail with ChannelOutputShutdownException. On 4.2: queued write was not failed after the peer sent STOP_SENDING.
• testShutdownInputByServerWhileConnectionWindowIsExhaustedSignalsOutputShutdown : same exhausted window, nothing queued on stream 0; the server stops stream 0. On 4.2: STOP_SENDING for a stream without queued writes was not signaled while the connection window was exhausted.
• testShutdownInputByServerIsFoundAmongManyStreamsWithoutFurtherTraffic : 66 streams share the exhausted window; the server stops the last one created; nothing but that packet and the acknowledgements it provokes follows. On 4.2: STOP_SENDING for stream 260 was not signaled among 66 streams with no further traffic.
• testStopSendingArrivingWithFirstDataOfRemoteStreamIsSignalledToItsHandler : the server opens a stream to the client, stops it, then sends the first bytes, so the client's channel is created already stopped, before its user handler is installed. Expects the handler to receive ChannelOutputShutdownEvent. On 4.2 (and on an earlier version of this fix that fired the event immediately): STOP_SENDING for a stream that arrived already stopped was not signaled to its handler.
• testHandlerClosingConnectionWhileSeveralStopsAreSurfacedDoesNotTouchFreedConnection : a handler closes the whole connection from the first of two stops surfaced by one scan; guards that the scan stops at the freed native connection. Reading freed memory is undefined rather than reliably fatal, so this test does not fail deterministically on the old code; it fixes the required behavior.

Mechanism (code paths)

• QuicheQuicChannel.handleWritableStreams: iterates Quiche.quiche_conn_writable(connAddr) and calls streamChannel.writable(quiche_conn_stream_capacity(...)) per id. quiche's Connection::writable() returns an empty iterator when tx_cap == 0 (lib.rs, pub fn writable), so a stopped stream is invisible to this pass while the window is exhausted.
• QuicheQuicStreamChannel.writable(long): capacity == QUICHE_ERR_STREAM_STOPPED with an empty queue returns without any side effect.
• QuicheQuicStreamChannel.Unsafe.writeQueued() / writeWithoutCheckChannelState: the only other places STREAM_STOPPED is observed; both require a write attempt.

The only direct query for a received STOP_SENDING, without attempting a write, is quiche_conn_stream_capacity() on that stream, and writable() withholds stopped streams while tx_cap == 0. quiche does count received stops (stopped_stream_count_remote in quiche_stats, filled by quiche_conn_stats() as part of the full stats snapshot), which the fix uses to know when to ask: one stats snapshot per processed datagram, measured at 24 ns per call. A quiche change to report stopped streams from writable() regardless of tx_cap would make that polling unnecessary; that is a separate request to the quiche project.

A second ordering detail matters for remotely-initiated streams: netty creates the channel for such a stream only when its first data arrives, and if the STOP_SENDING arrived earlier the channel is born stopped, before any user handler exists. The event for that case has to be delivered after registration.