Add a per-frame HTTP/2 observation hook to Pingora

Apologies for the AI-generated request below; I'm not too sure of the exact detail, but the overall request seems sensible to me. Our use-case is true byte-counting per-request (including headers) and this feels like a generic enough solution that may be useful to others also...

Summary

Expose a callback invoked for every HTTP/2 frame as it crosses the connection, in both directions, with the frame's raw payload bytes as Pingora saw them and their decoded detail — a HEADERS block's compressed length and flags, SETTINGS entries, GOAWAY/RST_STREAM error codes, PING payloads, WINDOW_UPDATE increments, and so on.

This is a Pingora-only change: it is implemented by observing the bytes on the stream handed to h2::server::handshake, so no h2 release is required. The one thing that route cannot provide is the decoded header fields; see "Raw bytes and decoded headers" below.

Why

Pingora gives applications Session::body_bytes_read()/body_bytes_sent() (decoded body bytes) and connection-level Digest/socket counters (whole TCP connection). Neither can describe an individual HTTP/2 stream's wire cost, and nothing exposes framing at all.

Use cases:

• Per-stream byte accounting. HTTP/2 multiplexes streams onto one connection, so socket counters cannot be attributed per stream. Billing, quotas, and per-request reporting need each stream's header-block and frame bytes.
• HPACK observability and diagnostics. Compression ratio per stream, dynamic-table behaviour, and detecting a client that has stopped compressing are all invisible today.
• Security research and hardening. Header-list-size abuse (SETTINGS_MAX_HEADER_LIST_SIZE), HPACK/BREACH-style compression side-channels, frame-level protocol anomalies, and smuggling or reset abuse are frame properties. A callback gives defenders the same view an attacker has.
• Tracing and troubleshooting. Per-frame timing (inter-frame gaps, drip feeds, stalls), stream lifecycle, and why a connection was reset.
• Adaptive policy. Priority hints, per-stream rate limiting, and traffic shaping driven by what actually went on the wire.
• Fuzzing and conformance. A generic frame stream is a test oracle for any HTTP/2 stack.

Current state

Nothing like this exists in 0.9.0. In pingora-core/src/protocols/http/v2/server.rs there are no occurrences of hpack, header_block, encoded, or compressed; the only header-size symbol is the decoded max_header_list_size receive limit. handshake is fixed to Stream, decoded request extensions are dropped, Digest is a fixed struct, and HttpPersistentSettings::user_context is HTTP/1-only.

Proposed interface

A single observer with a default no-op method, so an implementation overrides only the frames it handles:

pub enum Direction { Read, Write }

/// Everything known about one frame. `raw` is the frame's payload exactly as
/// Pingora observed it on the wire — padding included, the 9-byte frame header
/// excluded — so an observer can inspect the bytes directly. `kind` is the
/// decoded view of the same frame.
pub struct FrameEvent<'a> {
    pub direction: Direction,
    pub stream_id: u32,
    pub at: Instant,
    pub raw: &'a [u8],
    pub kind: FrameKind<'a>,
}

pub enum FrameKind<'a> {
    Headers {
        /// Compressed HEADERS + CONTINUATION octets, padding/priority excluded.
        header_block_len: u32,
        end_headers: bool,
        end_stream: bool,
    },
    Data { end_stream: bool },
    Settings { ack: bool, entries: &'a [Setting] },
    WindowUpdate { increment: u32 },
    Priority { dependency: u32, weight: u8, exclusive: bool },
    RstStream { error_code: u32 },
    Ping { ack: bool, opaque: [u8; 8] },
    GoAway { last_stream_id: u32, error_code: u32, debug: &'a [u8] },
    PushPromise { promised_id: u32, header_block_len: u32 },
    Unknown { frame_type: u8 },
}

pub trait FrameObserver: Send + Sync + 'static {
    fn on_frame(&self, _event: FrameEvent<'_>) {}
}

Ergonomic variant: one default method per frame type (on_headers, on_data, on_settings, ...) instead of a single on_frame, backed by the same enum.

Registration is opt-in and defaults off, e.g. through Pingora's HTTP/2 options:

H2Options::default()
    .max_header_list_size(64 * 1024)
    .frame_observer(Arc::new(MyObserver::new()));

Where it lives

Pingora already owns the Stream it passes to h2::server::handshake (apps/mod.rs), and the same seam exists for the upstream client (connectors/http/v2.rs). A transparent AsyncRead/AsyncWrite wrapper around that stream observes frames without touching h2:

• It sits above TLS, so it sees cleartext HTTP/2 framing for both h2c and TLS.
• It parses the 9-byte frame header and reassembles each frame's payload so it can hand the raw bytes to the observer. One frame is buffered per direction, bounded by the connection's SETTINGS_MAX_FRAME_SIZE, and the buffer is reused; capture can be restricted to selected frame types (for example, control frames and HEADERS) so a bulk DATA path does not pay for a copy.
• Non-HPACK frames are trivially decoded from their payloads (SETTINGS, WINDOW_UPDATE, PRIORITY, RST_STREAM, PING, GOAWAY); HEADERS/CONTINUATION contribute their accumulated payload length and flags, and their raw compressed bytes.

Details the implementation must handle:

• Partial writes — parse only the accepted prefix of buf; never account for bytes the inner IO did not take.
• Vectored writes — handle poll_write_vectored without double-counting.
• Partial reads — the parser is a byte-stream state machine, so chunk boundaries do not matter.
• Client preface — skip the 24-byte PRI * HTTP/2.0... preface (h2c try_peek does not consume it).
• Trailers — a later HEADERS on a known stream is a trailer block; report it as its own event.
• PUSH_PROMISE — include it, or skip by length so framing stays in sync.

Raw bytes and decoded headers

FrameEvent::raw carries each frame's payload exactly as observed, so HEADERS and CONTINUATION events expose the raw compressed block bytes and every other frame type exposes its raw payload. What the wrapper cannot produce is the decoded header fields, because HPACK dynamic-table state lives inside h2; the decoded request/response headers remain available as they are today via Session::req_header() / response_written().

A small h2 change could go further — store the fragment length in Headers::load and surface the decoded HeaderMap on the callback, at the point the frame is processed. That is the clean way to expose decoded fields per frame, but it is not needed for the accounting, observability, and security use cases above, so it can be left as future work.

Compatibility and cost

• Purely additive and opt-in; no behaviour change and no cost when disabled.
• One callback per frame on the connection task; observers must be cheap and non-blocking (batch into atomics/counters if needed).
• Reassembling raw payloads copies at most one frame per direction into a reused buffer, bounded by SETTINGS_MAX_FRAME_SIZE (16 KiB by default; cap it rather than trusting a peer's advertised value). Capture filtering keeps bulk DATA off that path when it is not needed.