#3577·career-ops

feat(scan): domain-level aggregator repost list, complementing the SimHash cross-listing warning

Author: santiferCreated Aug 31, 2026Updated Sep 17, 2026
Labelsgood first issue

Time: ~2-3h · Difficulty: Intermediate · Area: scanner

What

Add a domain-level aggregator/repost detection list to the scanner, complementing the existing SimHash cross-listing warning (FAQ §5).

Today the scanner catches agency reposts by content: a SimHash fingerprint of each job description flags near-identical duplicates. A domain list catches them by source, which is cheaper, works when the text has been rewritten, and needs no fingerprint at all. They're complements, not competitors.

Seed (with credit)

@gup-abhi enumerated ~20 aggregator boards that repost other people's listings in #1944 (Recruit.net, Zippia, Expertini, Jooble, Adzuna, Neuvoo, JobisJob, Jobrapido, …), including the hard part most lists get wrong: Indeed is carved out as a primary board, not a scraper, because companies post there directly. That list is the starting point, and landing it earns the credit it deserves: keep the carve-out reasoning in a short note per entry.

Acceptance criteria

  • A data file (e.g. data-static/aggregator-domains.txt or similar, wherever the scanner keeps static lists) with one domain per line and a one-line reason each
  • The scanner surfaces a "possible aggregator repost" note when a listing's host matches, alongside (not replacing) the SimHash warning
  • The Indeed carve-out preserved and documented
  • A test covering match / no-match / carve-out

First refusal is @gup-abhi's (the list is his): if you're someone else and want it, comment /assign and it's yours if nobody has it yet.