report_audit.py extract: regex-based extraction produces false-positive 'data points' from narrative table cells
Summary
tools/report_audit.py extract walks Markdown tables and pulls out (label, value, unit) triples to build the 15% audit sample. The extraction regex has no word-boundary or context awareness, so it also grabs numbers embedded in ordinary prose/identifiers inside table cells, producing audit sample items that aren't actually verifiable external claims.
Repro
Given a report table like:
| Metric | Value | Source | Verified |
|---|---|---|---|
| FY2026E non-GAAP EPS guidance | $1.85–$2.25 | company guidance (raised twice in 2026) | — || Strategy | Recommendation |
|---|---|
| If you already hold | ...if you haven't independently checked the RPO gap against the 10-K, that's the single highest-value verification to do next. |Running extract on a report containing these produces sample items like:
ID 12 FY2026E non-GAAP EPS guidance · Source 2026.00
ID 34 If you already hold · Recommendation 10.00- ID 12 comes from the literal word "2026" inside "raised twice in 2026" in the Source column — not a data point at all.
- ID 34 comes from the "10" in "10-K" (a filing name) inside a free-text recommendation cell.
Neither is a claim that should go into a fetch-and-cross-check sample; asking someone to "verify" that ID 34 = 10 against macrotrends/stockanalysis is meaningless.
Root cause (as far as I can tell from _parse_md_tables / extract_data_points)
_is_valid_label/ the column-header skip-list (_SKIPset, and thecol_header.upper() in (...)check inextract_data_points) only contains Chinese terms (来源,说明,备注, etc.). English report headers likeSource,Notes,Verifiedaren't filtered, so entire narrative columns get scanned for numbers.- The number regex (
[\d,,\.]+) has no requirement that the match be a standalone number — it happily matches digits that are part of an alphanumeric token like10-K,Q4,COVID-19, etc.
Suggested fixes
- Extend the header skip-list to include common English equivalents:
source,sources,notes,note,verified,recommendation,commentary,summary. - Require the matched number to be at a word boundary not immediately followed by a letter (e.g. reject matches like
10-Kwhere digits are immediately followed by-+ letter forming a known-identifier pattern, or more simply: skip cells where the "number" is immediately adjacent to a letter with no space, since that's almost always an identifier/citation rather than a standalone value). - Optionally: let a report opt a specific column out of extraction (e.g. treat any column whose header matches a broader stoplist, or a markdown convention like a column literally titled "Assumption"/"Model Input", as non-factual/non-verifiable, since these are often self-computed scenario outputs rather than externally-sourced claims).
Happy to submit a PR if a maintainer can confirm which fix direction is preferred — the header-skip-list extension is low-risk, but the "assumption vs. fact" distinction is more of a design call.
Source: xbtlin/ai-berkshire