Instagram source always returns HTTP 404 while ScrapeCreators endpoints return 200 (v3.18.4)

Summary

The Instagram source always fails with HTTP 404: Not Found (Instagram: 0 items (error: HTTPError: HTTP 404: Not Found)), even though the exact ScrapeCreators endpoints the engine uses return HTTP 200 with valid data when called directly with the same API key.

• Plugin version: 3.18.4
• Host: Claude Code on macOS (Python 3.14)
• ScrapeCreators key: valid, plenty of credits remaining (confirmed via direct calls)
• Reproduced on multiple runs with different topics/handles

Repro

Run any research with an Instagram creator + keyword, e.g.:

last30days.py "Prohern tattoo brand" --plan <plan> \
  --ig-creators=prohern_official --tiktok-hashtags=prohern

Engine output (every run):

## Source Coverage
- Instagram: 0 items (error: HTTPError: HTTP 404: Not Found (run doctor for fixes))

## Source Errors
- Instagram: HTTPError: HTTP 404: Not Found

doctor --postmortem only prints instagram — error: HTTPError: HTTP 404: Not Found with fix: doctor (a self-referential loop, no actionable fix).

Evidence the endpoints themselves are fine (HTTP 200)

Both Instagram endpoints referenced in scripts/lib/instagram.py return 200 with real data using the same SCRAPECREATORS_API_KEY:

1. /v1/instagram/user/reels (used by --ig-creators)

GET https://api.scrapecreators.com/v1/instagram/user/reels?handle=prohern_official
→ HTTP 200
{"success":true,"credits_charged":1,"items":[{"media":{...}}, ...]}   # 12 reels returned

2. /v2/instagram/reels/search (keyword search)

GET https://api.scrapecreators.com/v2/instagram/reels/search?query=prohern
→ HTTP 200
{"success":true,"reels":[{"shortcode":"...","caption":"Prohern Black – the black that doesn't fade...", ...}]}

For contrast, the profile endpoint also confirms the account is public and exists:

GET https://api.scrapecreators.com/v1/instagram/profile?handle=prohern_official → HTTP 200 (is_private:false)

So: account is public, key is valid, and both source endpoints return 200 with data — yet the engine surfaces the Instagram source as a hard 404. This points to a bug inside the plugin's Instagram code path (scripts/lib/instagram.py) — likely a chained/secondary call (e.g. an enrichment endpoint such as /v2/instagram/post/comments or /v2/instagram/media/transcript, or a paging/response-shape edge) that raises 404 and is bubbled up as the whole source's error, rather than the primary reels calls actually 404-ing.

Impact

Instagram is unusable via the engine even when correctly configured and paid. For brand/creator topics whose conversation lives primarily on Instagram, the run returns "Nothing solid this window" and misses the one source that has the signal. The direct SC endpoints are a working manual workaround, but the engine path is broken.

Suggested next step

Log the exact request URL/status for each Instagram sub-call in instagram.py (or catch per-sub-call 404s so a failing enrichment call doesn't zero out the whole source). Happy to provide more detail from a run if useful.