Instagram: /v2/instagram/reels/search returns "No scrapeable reels found" for every query — working lane is search/profiles → user/reels

Version: 3.21.0 · Platform: macOS (Darwin 24.6.0), Python 3.12 · Host: Claude Code

Summary

The Instagram keyword lane is dead at the provider, not in the enrichment chain. /v2/instagram/reels/search returns HTTP 404 — "No scrapeable reels found for that query. You were not charged for this request." for every query tested. Because the engine only ever calls that endpoint for topical search, Instagram contributes 0 items on every keyword run.

This refines #986, which hypothesised a secondary enrichment call was 404-ing and zeroing the source. The primary search itself is the one returning 404.

Evidence — primary search is empty for everything

Direct calls with a valid key, 2026-08-18:

query	reels	credits_charged
ai	0	0
ai automation	0	0
aiautomation	0	0
нейросети	0	0
business	0	0
fitness	0	0
coffee	0	0

/v1/instagram/search/hashtag?hashtag=aiautomation is equally dry (No posts found, 0 credits).

Note the silver lining: these failures are free, so the dead lane costs no credits — only coverage.

A working topical lane exists (verified end to end)

ScrapeCreators exposes 19 Instagram endpoints; the engine uses 4. Two unused ones are alive:

GET /v1/instagram/search/profiles?query=AI%20automation   → 200, 10 profiles, 1 credit
GET /v1/instagram/user/reels?handle=<handle>              → 200, 9-12 reels, 1 credit each
GET /v2/instagram/post/comments?url=<reel_url>            → 200, comments, 1 credit
GET /v1/instagram/reels/trending                          → 200, 31 reels, 1 credit

Chained keyword → profiles → reels → comments returned real dated reels with view/like counts for ~4-5 credits total.

Caveat worth encoding: profile search is loose. AI automation returned a 35.6K-follower automation account, a 1.5K-follower agency, and a 225-follower smart-home installer in India. A follower floor plus engagement sort is needed.

Second bug, same source: the media envelope (#1017)

The two endpoints disagree on item shape:

Endpoint	Item shape
/v2/instagram/reels/search	flat reel object
/v1/instagram/user/reels	{"media": { ...reel... }}

_user_reels() returns items still wrapped, so _parse_items() sees no pk/caption/play_count and drops all of them. Users see Instagram: 0 items (error: HTTP 404) even when the fetch succeeded — the 404 in that message comes from an unrelated handle in the same batch, which makes the real cause invisible.

Patch

Locally applied and working (10 dated reels for AI automation, correct authors, views and likes):

1. _user_reels() — unwrap: it.get("media", it).
2. search_instagram() — when the v2 search yields nothing (empty or HTTPError), fall back to a _reels_via_profiles() helper: profile search, drop accounts under a follower floor, sort by followers, fetch reels for the top N, truncate to results_per_page.
3. Route the v2 HTTPError into that fallback instead of returning early, and only surface the original error if the fallback is also empty.

Happy to open a PR with this if the approach looks right.