CL search pages: browser UA gets JS shell that never hydrates under V8

CL search pages: browser UA gets JS shell that never hydrates under V8

Summary

Craigslist serves two different responses for the same URL depending on the client's User-Agent:

• Non-browser UAs (curl, node fetch, no UA): full server-rendered HTML with all listings embedded in the page. ~14 KB.
• Browser UAs (Chrome/145, Obscura's default): a lightweight JS shell (~2.4 KB) that requires client-side hydration to populate results.

Obscura identifies as Chrome/145.0.0.0, so CL serves the browser version. But the hydration step never completes under V8 — results stay at "retrieving..." → "no results found" regardless of wait time.

Reproduction

# This works (server-rendered, listings in HTML):
curl -s "https://www.craigslist.org/search/area/scranton?cat=sss&query=iphone" | grep -c "iPhone 17promax"
# → 3

# This fails (browser shell, no hydration under V8):
obscura fetch "https://www.craigslist.org/search/area/scranton?cat=sss&query=iphone" \
  --wait 12 \
  --eval "document.querySelectorAll('.cl-search-result').length"
# → 0

# Same URL in real Chrome: 60 results render fine.

Tested variants (all fail)

Root cause hypothesis

CL's hydration JS likely depends on a Blink-specific API or behavior that standalone V8 doesn't replicate. The page loads, JS executes, but the XHR that fetches/populates results never completes. In Chrome (Blink + V8) the same page hydrates in <2s.

Suggested fix

One of:

1. Add a --no-browser-ua or --plain-fetch flag to obscura fetch that skips the browser engine and does a plain HTTP GET (like curl), returning the server-rendered HTML. This would work for CL and similar sites that serve full content to non-browser clients.
2. Allow overriding the User-Agent so CL serves the server-rendered version instead of the JS shell.
3. Fix V8 hydration for CL's specific pattern (if it's a known gap).

Option 1 is the most broadly useful — many sites serve full content to non-browser clients and only gate behind JS for real browsers.

Environment

• Obscura 0.2.2, Linux x86_64
• Test URL: https://www.craigslist.org/search/area/scranton?cat=sss&query=iphone
• V8-Version reported by Obscura: 14.5.0.0