`WebSocket` over `wss:` can leave the end of a message unsent forever: `bufferedAmount` reaches 0 but the last ~64 KiB only arrive when the client sends something else

Written by AI (Claude Fable 5.1), reviewed by Gideon Wald before filing. Everything below was run on the official 2.9.6 release binary and on the official canary build of main (2.9.6+057a3da), on macOS 26.6 (aarch64) and on Linux aarch64 (the denoland/deno:2.9.6 Docker image); the output is copied from those runs.

Version: Deno 2.9.6 (release) and main at 057a3da (canary). First seen on 2.9.4.

What happens

A native WebSocket client sends one 4 MiB binary message over wss: to a server that reads slowly. bufferedAmount drops to 0, which per the spec means every byte has been handed to the network. The server, still reading, stops receiving about 56 or 72 KB short of the end of the message and never gets the rest: the connection stays open and nothing more arrives. As soon as the client sends any further message, the missing bytes arrive, followed by that message.

We hit this in production with ~160 KB messages: the client waits forever for a reply to a message the server never fully received.

Reproduction

server.ts: a raw TLS listener that does the WebSocket handshake by hand, then reads 16 KiB at a time with a 1 ms pause and prints how many bytes it has received so far:

// Raw TLS server that accepts one WebSocket connection by hand and reads it slowly.
const listener = Deno.listenTls({ port: 8443, cert: Deno.readTextFileSync("cert.pem"), key: Deno.readTextFileSync("key.pem") });
const conn = await listener.accept();
const buf = new Uint8Array(1 << 16);
const request = new TextDecoder().decode(buf.subarray(0, (await conn.read(buf))!));
const key = request.match(/Sec-WebSocket-Key: (\S+)/i)![1] + "258EAFA5-E914-47DA-95CA-C5AB0DC85B11";
const accept = btoa(String.fromCharCode(...new Uint8Array(await crypto.subtle.digest("SHA-1", new TextEncoder().encode(key)))));
await conn.write(new TextEncoder().encode(`HTTP/1.1 101 Switching Protocols\r\nUpgrade: websocket\r\nConnection: Upgrade\r\nSec-WebSocket-Accept: ${accept}\r\n\r\n`));
let received = 0;
setInterval(() => console.log(`server: received ${received} bytes`), 500);
for (let n; (n = await conn.read(buf.subarray(0, 16384)).catch(() => null)) !== null;) {
  received += n;
  await new Promise((r) => setTimeout(r, 1)); // read slowly so the client's socket backs up
}
Deno.exit(0);

client.ts: sends one 4 MiB message, reports bufferedAmount until it reaches 0, and 3 s later sends one more message:

const ws = new WebSocket("wss://localhost:8443");
ws.onopen = () => {
  ws.send(new Uint8Array(4 * 1024 * 1024)); // one 4 MiB binary message (4194318 bytes on the wire)
  const timer = setInterval(() => {
    console.log(`client: bufferedAmount=${ws.bufferedAmount}`);
    if (ws.bufferedAmount > 0) return;
    clearInterval(timer);
    setTimeout(() => { console.log("client: sending one more message"); ws.send("x"); }, 3000);
    setTimeout(() => Deno.exit(0), 5000);
  }, 500);
};

openssl req -x509 -newkey rsa:2048 -nodes -keyout key.pem -out cert.pem -days 30 -subj /CN=localhost \
  -addext subjectAltName=DNS:localhost -addext basicConstraints=critical,CA:FALSE
deno run --allow-net --allow-read server.ts &
sleep 1
deno run --allow-net --cert cert.pem client.ts

Expected

Once bufferedAmount is 0, the server receives the rest of the message on its own: 4 MiB of payload plus a 14-byte frame header, 4194318 bytes in total.

Actual

The last lines of a failing run (2.9.6 release):

server: received 3502318 bytes
client: bufferedAmount=0
server: received 4122542 bytes
server: received 4122542 bytes
server: received 4122542 bytes
server: received 4122542 bytes
server: received 4122542 bytes
client: sending one more message
server: received 4155310 bytes
server: received 4194325 bytes
server: received 4194325 bytes

After the client reports bufferedAmount=0, the server is stuck at 4122542 bytes, 71776 short of the end, and stays there. The one-byte text message the client sends 3 s later (7 bytes on the wire) releases everything: 4194318 + 7 = 4194325. In the runs below the stuck tail was between 20 KB and 73 KB, most often about 56 KB or 72 KB.

Whether a run stalls depends on the socket being full at the moment the last bytes of the message are handed to TLS, so not every run does. On this machine, running the two files above ten times each:

Deno	stalled	runs
macOS, 2.9.6 release (official binary)	4	10
macOS, main at 057a3da (official canary binary)	6	10
macOS, main at 685eb3b (local debug build)	4	10
Linux, 2.9.6 (denoland/deno:2.9.6), server and client in two containers on a Docker bridge network	1	6
Linux, main at 057a3da (official Linux canary binary in the same image), same setup	1	6

Every stalled run released the whole tail on the follow-up message. On Linux the two files in one container over loopback did not stall in 12 runs: Linux loopback has a 64 KiB MTU and autotunes the socket buffers into the megabytes, so the whole 4 MiB was accepted before the client's send could block. Across a bridge network (veth, MTU 1500) it stalls like a real network path does. Two controls with the same server: the same client over plain ws: (Deno.listen instead of Deno.listenTls) completed 3 of 3 times, and Node 26's built-in WebSocket over wss: against this TLS server completed 3 of 3 times.

Why, as far as I can tell

send() → op_ws_send_binary → send_binary spawns a task that awaits ServerWebSocket::write_frame and then subtracts the message length from the buffered count (ext/websocket/lib.rs, send_binary). ServerWebSocket::write_frame awaits fastwebsockets::WebSocketWrite::write_frame and returns; that ends with write_all on the underlying stream, and nothing in ext/websocket ever calls flush.

For a wss: connection the underlying stream is tokio_rustls::client::TlsStream (the connection comes from deno_fetch, whose connector is hyper_rustls). tokio-rustls' poll_write encrypts into rustls' internal buffer, tries to write that ciphertext to the socket, and when the socket would block it returns Ready(Ok(n)) for the plaintext it accepted while the ciphertext stays buffered (tokio-rustls-0.26.0/src/common/mod.rs, poll_write, the (n, true) => Poll::Ready(Ok(n)) arm). Its poll_read only reads; it never writes out pending ciphertext. tokio-rustls' crate documentation says exactly this, under "Why do I need to call poll_flush?" and "Why don't we write during poll_read?": https://docs.rs/tokio-rustls/0.26.0/tokio_rustls/#why-do-i-need-to-call-poll_flush

So once the send task has resolved, up to rustls' 64 KiB of ciphertext can sit in user space with no future left that polls the write half: op_ws_next_event only polls the read half. The bytes are never lost, just never sent, until some later write drives them out, while bufferedAmount already says they were sent.