safe_requests_get decodes charset-less UTF-8 pages as ISO-8859-1, corrupting every accented character

Author: amenocriCreated Sep 13, 2026Updated Sep 13, 2026

Summary

url_safety.safe_requests_get returns a requests.Response whose .text is decoded as ISO-8859-1 whenever the server sends a text/* Content-Type with no charset parameter. The document's own <meta charset="UTF-8"> is ignored, so every non-ASCII character becomes mojibake.

This affects every non-English site audited with the toolkit. On a French page, Specialiste (with an acute e) comes back doubled into Ã© sequences.

Why it happens

requests implements RFC 2616 literally: for a text/* response with no charset in the header, get_encoding_from_headers returns ISO-8859-1, and resp.text uses that. HTML5 says the in-document <meta charset> should win when the header is silent, but requests never looks at the body.

Serving content-type: text/html with no charset is common, so this is not an edge case.

Scope

Seven scripts read HTML through this helper and are all affected:

render_page.py, parse_html.py, nlp_analyze.py, preload_check.py, parasite_risk.py, ucp_check.py, gbp_deprecation_lint.py

apparent_encoding appears nowhere in scripts/.

Downstream, the corruption is silent rather than loud. Word counts, title-length checks, readability scoring, duplicate detection and E-E-A-T passage scoring all keep producing plausible numbers from corrupted text. In one audit a lane concluded the client's site had a character-encoding defect and filed a high-severity finding against it.

Reproduction

Minimal check, no network access needed:

python

import requests
html = "<html><head><meta charset='utf-8'><title>" + "Spécialité" + "</title></head></html>"
r = requests.Response()
r._content = html.encode("utf-8")
r.status_code = 200
r.headers["content-type"] = "text/html"          # no charset, as many servers send
r.encoding = requests.utils.get_encoding_from_headers(r.headers)   # -> 'ISO-8859-1'
assert "Spécialité" in r.text          # fails

Note the get_encoding_from_headers line. Without it a hand-built Response leaves encoding as None, requests then sniffs on its own, and the bug does not reproduce, which makes this easy to mis-test.

Against a live host that responds with a bare content-type: text/html while serving UTF-8 with <meta charset="UTF-8">:

bash

claude-seo run render_page.py <such-a-url> --mode auto --json

Every accented character in the returned raw_content and <title> arrives as a two-character Ã-prefixed sequence.

Suggested fix

One guard at the shared seam in scripts/url_safety.py covers all seven callers:

python

    with _pin_dns(parsed.hostname, pinned_ip, port):
        resp = requests.get(norm_url, timeout=timeout, **kwargs)
    # requests follows RFC 2616 and decodes a text/* body with no charset in the
    # header as ISO-8859-1, ignoring the document's own <meta charset>.
    # apparent_encoding reads the meta declaration and the byte distribution.
    # Restricted to texty bodies so we never charset-detect an image.
    ctype = resp.headers.get("content-type", "").lower()
    if "charset" not in ctype and (
        ctype.startswith("text/") or "xml" in ctype or "json" in ctype or not ctype
    ):
        resp.encoding = resp.apparent_encoding or resp.encoding
    return resp

A server that genuinely declares charset=iso-8859-1 is untouched, since charset is then present in the header.

Verified against a live French site: 158 correctly decoded accented characters, zero mojibake, zero replacement characters. Happy to open a PR if useful.

Version

claude-seo 2.2.5, Python 3.14, Windows 11.

Source: AgriciDaniel/claude-seo

View original on GitHub View discussion on GitHub