extract_numbers.py: a CORRECT multi-year deck reports 5 'high severity' inconsistencies — grouping ignores the period
Summary
find_inconsistencies in skills/ib-check-deck/scripts/extract_numbers.py
groups figures by category alone and flags anything more than 5% from the
largest group. A deck carrying FY2023, FY2024 and FY2025E revenue is therefore
self-contradictory by construction — which is every deck with a financial
history.
Measured on a four-line deck: a correct deck produced 5 "high severity" inconsistencies, against 1 for a deck containing a genuine contradiction. Precision ≈ 1/6. The true finding is indistinguishable from the noise.
Reproduce
clean.md — correct, no contradiction:
## Slide 2 — Historical Performance
Revenue grew from $100.0 million in FY2023 to $120.0 million in FY2024.
EBITDA was $20.0 million in FY2023 and $25.0 million in FY2024.
## Slide 5 — Projections
Revenue of $145.0 million is forecast for FY2025E.planted.md — one genuine contradiction (FY2024 revenue stated twice):
## Slide 2 — Historical Performance
Revenue was $120.0 million in FY2024.
## Slide 7 — Transaction Summary
FY2024 revenue of $135.0 million supports the valuation.$ python extract_numbers.py clean.md --check -> 5 inconsistencies (all severity "high")
$ python extract_numbers.py planted.md --check -> 1 inconsistencyOn the clean deck it reports revenue $100.0million vs $120.0million,
vs $145.0million, and ebitda $20.0million vs $25.0million — all correct
figures for different periods. (One of the five is 2023t, the separate
parsing defect filed alongside this.)
Cause
# lines ~192–210
for num in numbers:
if num.category != 'other':
by_category[num.category].append(num) # period is discarded
...
if diff_pct < 0.05: # 5% tolerance on VALUE onlyNothing distinguishes "the same metric stated twice inconsistently" from "the same metric across three periods".
Suggested fix
Key the comparison on (category, period), not category alone:
- parse a period marker near each figure —
FY2024,FY24,2025E,2023A; - attribute a figure to the period that follows it before the next figure,
falling back to the preceding one. Nearest-by-distance is wrong on the
commonest sentence there is: in "EBITDA was $20.0m in FY2023 and $25.0m in
FY2024",
25.0sits 13 chars afterFY2023and 18 beforeFY2024, so distance alone assigns both figures to FY2023 and re-creates the false positive; - only report when two values share a category and a period;
- skip figures with no resolvable period rather than pooling them.
With that change the two fixtures above give 0 and 1 respectively.
Happy to open a PR.
Source: anthropics/financial-services