citability_scorer: answer_block_quality regexes are English-only, and <dl>/<dd> FAQ answers are invisible to the block splitter
Summary
Two blind spots in scripts/citability_scorer.py that make its absolute score unusable for non-English sites, found while scoring a Spanish/English bilingual site (same templates, same content, translated):
answer_block_quality(30 of 100 points) is matched with English-only regexes, so no non-English page can ever earn most of that category.- The block splitter only reads
h1-h4,p,ul,olandtable, so FAQ answers rendered as<dl>/<dd>are invisible to the scorer even though real crawlers read them fine.
1. English-only patterns cap non-English pages ~15-25 points below their translations
The definition patterns (\w+\s+is\s+(?:a|an|the), refers? to, means?, defined as, ...), the early-answer check (\b(?:is|are|was|were|means?|refers?)\b in the first 60 words) and the citation patterns (according to|research shows|studies show...) are all English lexical matches. A Spanish page saying the exact same thing ("X es una...", "X se refiere a...", "segun...") scores 0 on every one of them.
Measured on translation pairs of the same site, same structure, same block count (15 blocks each):
| Page (ES / EN) | Total | answer_block_quality |
|---|---|---|
/servicios/rescate-apps-ia/ |
28.1 | 5.1 / 30 |
/en/services/vibe-code-rescue/ |
50.6 | 22.0 / 30 |
/diseno-web-caceres/ |
27.5 | 6.8 / 30 |
/en/web-design-caceres/ |
41.5 | 22.4 / 30 |
The ~20-point spread is language, not content. As it stands, the score is only safe to read as before/after on the same page; as an absolute grade it silently penalizes every non-English site, and nothing in the output warns about it.
Suggested fixes, in increasing effort:
- Document the limitation in the output (a one-line warning when the page
langis notenwould already prevent misreads). - Make the pattern lists per-language, keyed off
<html lang>, starting with the handful of big languages (Spanish equivalents:es un/una/el/la,se refiere a,significa,se define como,segun,en otras palabras). - Or score structure-only categories for non-English pages and renormalize.
2. <dl>/<dd> FAQ answers never reach the scorer
extract_content_blocks walks soup.find_all(["h1", "h2", "h3", "h4", "p", "ul", "ol", "table"]). A FAQ implemented as a definition list (<dl><dt>question</dt><dd>answer</dd></dl>) contributes nothing: the answers are in dd elements whose inner text has no p wrapper, so entire FAQ sections are missing from the block list. Pages whose main citable content is exactly that kind of Q&A block score as if the content did not exist.
Adding dl (or dd) to the element list, or falling back to any block-level element with direct text, would cover it.
Happy to provide the full per-block outputs for the four pages above if useful.
Source: zubair-trabzada/geo-seo-claude