#3386·bbot

HTTP_RESPONSE event data is built by two divergent functions

Author: liquidsecCreated Aug 21, 2026Updated Aug 21, 2026

Two functions build the dict for an HTTP_RESPONSE event, and they disagree:

  • response_to_event_dict() in bbot/core/helpers/web/response_event.py, used by http and lightfuzz
  • WebHelper.response_to_json() in bbot/core/helpers/web/web.py, used by wayback

They're near-duplicates but not duplicates, so the same response yields different event data depending on which module found it.

Key sets differ:

  • only in response_to_event_dict: input, content_length, title, location, cert_info
  • only in response_to_json: timestamp, port, scheme

raw_header is a different shape. For one response:

response_to_event_dict: 'HTTP/1.1 200 \r\nContent-Type: text/html\r\nServer: nginx\r\n\r\n'
response_to_json:       'Content-Type: text/html\r\nServer: nginx'

HTTP_RESPONSE.raw_response is raw_header + body, so on the response_to_json path it has no status line and no blank line between headers and body.

Duplicate header names. response_to_event_dict joins them, response_to_json keeps the last. For two Set-Cookie headers:

response_to_event_dict: 'a=1, b=2'
response_to_json:       'b=2'

response_to_json recomputes hashes blasthttp already has. It runs md5/mmh3/sha256 over the body and headers itself; response.hash is the same values, computed lazily in Rust and cached. (The values do match, I checked. It's wasted work, not a discrepancy.)

Smaller ones: body is response.body vs smart_decode(response.content); content_type is the raw header value vs the value split on ; and stripped; header keys are k.lower().replace("-", "_") vs tagify(k, delimiter="_").

Net effect: wayback's HTTP_RESPONSE events are missing title, location, content_length and cert_info, and their raw_response is malformed.

Noticed while adding decode_error in #3385, which had to be written into both.

Source: blacklanternsecurity/bbot