feat(scan): domain-level aggregator repost list, complementing the SimHash cross-listing warning
Time: ~2-3h · Difficulty: Intermediate · Area: scanner
What
Add a domain-level aggregator/repost detection list to the scanner, complementing the existing SimHash cross-listing warning (FAQ §5).
Today the scanner catches agency reposts by content: a SimHash fingerprint of each job description flags near-identical duplicates. A domain list catches them by source, which is cheaper, works when the text has been rewritten, and needs no fingerprint at all. They're complements, not competitors.
Seed (with credit)
@gup-abhi enumerated ~20 aggregator boards that repost other people's listings in #1944 (Recruit.net, Zippia, Expertini, Jooble, Adzuna, Neuvoo, JobisJob, Jobrapido, …), including the hard part most lists get wrong: Indeed is carved out as a primary board, not a scraper, because companies post there directly. That list is the starting point, and landing it earns the credit it deserves: keep the carve-out reasoning in a short note per entry.
Acceptance criteria
- A data file (e.g.
data-static/aggregator-domains.txtor similar, wherever the scanner keeps static lists) with one domain per line and a one-line reason each - The scanner surfaces a "possible aggregator repost" note when a listing's host matches, alongside (not replacing) the SimHash warning
- The Indeed carve-out preserved and documented
- A test covering match / no-match / carve-out
First refusal is @gup-abhi's (the list is his): if you're someone else and want it, comment /assign and it's yours if nobody has it yet.
Source: santifer/career-ops