Instagram profile lane (--ig-creators) parses to empty items: _parse_items doesn't unwrap the media envelope

Summary

_user_reels() (the --ig-creators profile lane) returns items that the ScrapeCreators /v1/instagram/user/reels endpoint wraps in a media envelope, but _parse_items() reads every field off the top level. Result: the fetch succeeds, the log prints -> 12 reels from @handle, and then every parsed item is empty (url="", text="", date=None, views/likes/comments=0) and is dropped downstream.

The profile lane is the only reliable Instagram path (keyword search is flaky by design — see the 500-retry-with-hashtag-form comment at instagram.py:313-317), so in practice this makes Instagram return near-zero on every run that relies on --ig-creators.

Impact

Three consecutive runs on Windows / v3.18.4, all with --ig-creators and valid SCRAPECREATORS_API_KEY:

run	profile lane fetched	items in report
brand topic, default depth	12 + 12 reels	2 (both from keyword search)
two brands, --deep --search instagram	12 + 12 reels	0 (source marked errored)
short topic, --deep --search instagram	12 + 12 reels	3 (all off-topic keyword noise)

Fetching the same two handles directly and unwrapping media yields 24 usable reels, all within the 30-day window, with captions, engagement and comments.

Root cause

scripts/lib/instagram.py

_user_reels() returns the raw list unchanged:

raw_items = data.get("items") or data.get("reels") or data.get("data") or []
_log(f"  -> {len(raw_items)} reels from @{handle}")
return raw_items

_parse_items() then reads fields at the top level (instagram.py:182-221):

reel_pk    = str(raw.get("id", raw.get("pk", "")))
shortcode  = raw.get("shortcode", raw.get("code", ""))
caption_obj= raw.get("caption", "")
play_count = raw.get("video_play_count") or raw.get("video_view_count") or raw.get("play_count") or 0

But /v1/instagram/user/reels returns:

{
  "success": true,
  "items": [
    { "media": { "pk": "...", "code": "...", "taken_at": 1785596409,
                 "caption": {"text": "..."}, "play_count": 0,
                 "like_count": 0, "comment_count": 0, "user": {...} } }
  ],
  "paging_info": {...}
}

Every lookup misses, so each item parses to zeros/empties. The search lane (/v2/instagram/reels/search) returns unwrapped objects, which is why keyword results parse fine and only the profile lane is affected.

Suggested fix

Unwrap in _user_reels() so _parse_items() keeps one contract:

raw_items = data.get("items") or data.get("reels") or data.get("data") or []
raw_items = [(it.get("media") or it) for it in raw_items]

Note taken_at is a unix timestamp on this shape, so the date path may need the same treatment as the search shape for the hard date filter at instagram.py:346 to work.

Secondary issue (same investigation)

When one search variant 404s, the whole Instagram source is marked failed and items that other lanes already returned are discarded. Observed with topic Guday e Gummy Original no Instagram:

[Instagram]   -> 12 reels from @guday.br
[Instagram]   -> 12 reels from @gummy
[Instagram] ScrapeCreators error: HTTP 404: Not Found     <- variant "... reaction edit"
...
Source Coverage
- Instagram: 0 items (error: HTTPError: HTTP 404: Not Found)

A per-lane failure probably should degrade to partial rather than zero a source whose other lanes succeeded.

Environment

• last30days v3.18.4 (Claude Code plugin, user scope)
• Windows 11, Python 3.13.14
• INCLUDE_SOURCES=tiktok,instagram,youtube_comments,tiktok_comments,instagram_comments