Let an addon complete a group of HTTP/2 requests in a single write

Gap

This is useful for testing race conditions that are only reachable when several requests become processable by the server at nearly the same moment. This synchronization technique is described in James Kettle's research on modern web race conditions, including Smashing the State Machine: The True Potential of Web Race Conditions (Black Hat USA 2023).

Today mitmproxy can multiplex requests concurrently over a single HTTP/2 connection, but request completion is still handled independently for each stream. Http2Connection drains h2's data_to_send() as HTTP events are processed, so multiple streams completing independently result in separate transport writes.

For ordinary proxying this is exactly the desired behavior. For race-condition testing, however, even small gaps between request completions can make narrow race windows difficult or impossible to reproduce reliably.

Proposed mechanism

A common way to reduce that timing variance is to stage every request up to—but not including—its final completion fragment, wait until all members of the group are ready, and then emit all of those completions in a single transport write.

For a bodyless HTTP/2 request, this can be done by sending its headers without END_STREAM and later emitting a zero-length DATA frame carrying END_STREAM. For requests with bodies, a small final fragment can similarly be retained until the group is ready.

mitmproxy already appears to provide most of the machinery needed for this:

• Requests are buffered by default. Unless request streaming is enabled, HttpStream already has the complete request before forwarding it. This provides a natural place to introduce staging without changing the normal streaming architecture.

• End-of-message is already represented separately. HttpStream ultimately emits RequestEndOfMessage separately from request data. On HTTP/2, stream completion can therefore be delayed independently from the bulk of the request.

• Http2Client already owns the relevant upstream state. This is where mitmproxy maps logical streams onto upstream HTTP/2 stream IDs and interacts with h2's send buffers and flow-control state. Group registration, staging, and release can therefore remain local to the HTTP/2 implementation.

• The existing send path can already coalesce generated frames. If the final fragments for all staged streams are generated before the next data_to_send() drain, h2 can return them together and mitmproxy can emit them through the normal SendData path. No separate transport/write path should be necessary.

What appears to be missing is a way for an addon to declare that several requests belong to the same synchronized group.

One possible API would be a typed request attribute set no later than the request hook:

from mitmproxy import http

class RaceGroup:
    def request(self, flow: http.HTTPFlow) -> None:
        if flow.request.path.startswith("/redeem"):
            flow.request.race_group = http.RaceGroupSpec("redeem", expected=3)

RaceGroupSpec(id, expected) would identify the group and declare how many members are expected. mitmproxy would forward each member normally except for its final completion fragment, stage it, and once all expected members are ready, release all retained completions together through a single HTTP/2 send-buffer drain.

Scope

Scope is HTTP/2 only, on purpose: the HTTP/1 equivalent needs one connection per request and belongs nowhere near this code.

I have this implemented and would be glad to open a PR.

References

• Kettle, J. (2023). Smashing the State Machine: The True Potential of Web Race Conditions. PortSwigger Research, Black Hat USA 2023.
• Kettle, J. (2023). The single-packet attack: making remote race-conditions 'local'. PortSwigger Research.