HTTP_RESPONSE event data is built by two divergent functions
Two functions build the dict for an HTTP_RESPONSE event, and they disagree:
response_to_event_dict()inbbot/core/helpers/web/response_event.py, used byhttpandlightfuzzWebHelper.response_to_json()inbbot/core/helpers/web/web.py, used bywayback
They're near-duplicates but not duplicates, so the same response yields different event data depending on which module found it.
Key sets differ:
- only in
response_to_event_dict:input,content_length,title,location,cert_info - only in
response_to_json:timestamp,port,scheme
raw_header is a different shape. For one response:
response_to_event_dict: 'HTTP/1.1 200 \r\nContent-Type: text/html\r\nServer: nginx\r\n\r\n'
response_to_json: 'Content-Type: text/html\r\nServer: nginx'HTTP_RESPONSE.raw_response is raw_header + body, so on the response_to_json path it
has no status line and no blank line between headers and body.
Duplicate header names. response_to_event_dict joins them, response_to_json keeps
the last. For two Set-Cookie headers:
response_to_event_dict: 'a=1, b=2'
response_to_json: 'b=2'response_to_json recomputes hashes blasthttp already has. It runs md5/mmh3/sha256
over the body and headers itself; response.hash is the same values, computed lazily in
Rust and cached. (The values do match, I checked. It's wasted work, not a discrepancy.)
Smaller ones: body is response.body vs smart_decode(response.content); content_type
is the raw header value vs the value split on ; and stripped; header keys are
k.lower().replace("-", "_") vs tagify(k, delimiter="_").
Net effect: wayback's HTTP_RESPONSE events are missing title, location,
content_length and cert_info, and their raw_response is malformed.
Noticed while adding decode_error in #3385, which had to be written into both.
Source: blacklanternsecurity/bbot