Baike.dev
Log in
> 返回资讯列表
news_article.exe
📰

Craigslist's JSON-LD has no ID field — we join 290 of 325 listings by title alone

2026年9月8日3 次浏览来源:Dev.to 阅读原文

Quick answer Craigslist search pages ship two copies of every listing: a static HTML list, and a JSON-LD block with images, currency, and geo-coordinates. The obvious move is to join them by ID. Don't — Craigslist's JSON-LD carries no shared identifier at all, not a bare post ID, not a URL, not a SKU. The only field both copies reliably share is the listing's title, and titles repeat. We joined by title through a per-title FIFO queue and measured it recovering 290 of 325 listings (89%) end-to-end on a captured 298-item page. That number is the ceiling of what a title-only join can do on this page shape — plan your field completeness around it, don't assume 100%. Why can't you just match the JSON-LD by ID? 🧩 When we built the Craigslist Multi-City Listings Scraper, the first design...

Quick answer Craigslist search pages ship two copies of every listing: a static HTML list, and a JSON-LD block with images, currency, and geo-coordinates. The obvious move is to join them by ID. Don't — Craigslist's JSON-LD carries no shared identifier at all, not a bare post ID, not a URL, not a SKU. The only field both copies reliably share is the listing's title, and titles repeat. We joined by title through a per-title FIFO queue and measured it recovering 290 of 325 listings (89%) end-to-end on a captured 298-item page. That number is the ceiling of what a title-only join can do on this page shape — plan your field completeness around it, don't assume 100%. Why can't you just match the JSON-LD by ID? 🧩 When we built the Craigslist Multi-City Listings Scraper, the first design assumed what almost every JSON-LD block on almost every e-commerce-shaped site provides: a , a , a , an — something that lines up a JSON entry with its DOM counterpart deterministically. Live inspection of a captured Craigslist search page found none of those. Each entry has exactly , , , , and a field that looks like it should solve the problem — until you check it against the static list past the first ~18 entries, where some static-list rows have no JSON-LD counterpart at all and the position numbering drifts out of alignment. So the join key that's actually usable, live, is — and titles aren't unique. The fix is a FIFO queue per title: walk the static list in document order, and for each title pop the next unconsumed JSON-LD entry with a matching name. This handles the ordinary case of duplicate titles reasonably — two identical "iPhone 14 Pro 256GB Unlocked" postings get their JSON-LD entries handed out in the same document order they appear in the static list — but it is not a guarantee. When the queue for a title runs dry (more static-list rows carry that title than JSON-LD has entries for), those listings still land with a title, price, and location from the static HTML — they just come back with , , , and as /empty, rather than being dropped. Fault isolation happens at the field level here, not just the row level: a partial join beats no row. What changed under the URL scheme? 🔗 The other live-wire finding: Craigslist has migrated off its old numeric-ID, per-subdomain URL scheme onto a unified shape, and the segment in that new scheme is not purely numeric anymore — fixture capture found opaque alphanumeric strings like . Any post-ID regex written against the legacy numeric-only assumption silently drops every listing on the new scheme: A spec or scraper written from documentation or memory rather than a live fixture capture is exactly the kind of place this assumption survives unchallenged — Craigslist's own help pages don't advertise the migration, you only find it by pulling a real page. What do you deliberately never touch? 🚧 Craigslist's disallows the reply flow (, , and a handful of related paths) outright, and this Actor checks every detail URL against that list before any network call — not as a courtesy, as a hard refusal baked into the client: The public still ships as a data field — it's on the page, it's useful context — but it is captured exactly as rendered and never opened, and no email de-obfuscation happens on top of it. That's a scope boundary, not a missing feature. The part that generalises 🧭 Structured data (JSON-LD, , embedded RSC payloads — pick your target) is usually more complete than the visible DOM, but "more complete" doesn't mean "keyed the way you need." When a target's structured block has no shared identifier with its rendered markup, resist the urge to assume or array order will hold past the first page of results — verify it against a real, large fixture before you trust it as a join key, and design the fallback (partial row, not dropped row) for when the join comes up short. What the Actor gives you Up to 20 explicit metro areas plus a curated top-25-metro preset, across all seven Craigslist top-level categories, in one run instead of one call per city. Title, price, currency, location, coordinates, images, posted/updated timestamps, structured attributes, description, and the public (never-followed) reply link per listing. Per- combo fault isolation — one blocked or slow combo is skipped and logged, the rest of the run keeps going. A best-effort (normalized title + area + category) for tracking reposts and price drops. Honest limitations 🚧 Each combo returns up to 300 listings — Craigslist's observed single-page ceiling, no further pagination in v1. Total combos are capped at 150 per run. covers a curated 25-metro list, not the full ~700-area directory. One search query per run. FAQ Can I get a seller's phone number or email? No. is surfaced exactly as Craigslist renders it — never opened, never de-obfuscated. disallows that flow and we respect it. Why one run instead of one run per city? Every incumbent we benchmarked requires a separate run per metro. This Actor fans out across areas and categories in one call, into one normalized dataset. What happens if a city or category is unreachable mid-run? That combo is skipped and logged; the rest of the run continues, and a status message reports what finished. Do I need Residential proxy? Not by default — standard Apify Proxy with session rotation and fingerprint impersonation clears this target. Pricing $0.20 per run plus $0.002 per result — $2.01 per 1,000 listings. A search that matches nothing costs only the start fee. → Craigslist Multi-City Listings Scraper on Apify Built by Devil Scrapes. We handle the fingerprint rotation, the multi-city fan-out, and the JSON-to-DOM join Craigslist doesn't make easy, so you get a flat table instead of a weekend. 😈

> 分享:
Baike.dev

baike.dev helps you discover great languages, frameworks, databases, DevOps and cloud-native tools.

Quick links

About

Contribute

Found a great developer tool? Share it with the community.

Submit a tool
© 2026 baike.dev Developer EncyclopediaUpdated daily · Discover great developer tools