YouTube: ScrapeCreators search backstop never fires when yt-dlp is bot-gated but returns 1-2 items

Summary

The ScrapeCreators YouTube search backstop is gated on not result.get("items"). When yt-dlp is bot-gated by YouTube it often still returns 1–2 results instead of zero, so the backstop never fires and the run silently reports a degraded YouTube lane as a success.

Engine v3.18.4, commit 1004324, macOS 15 (Apple Silicon), Python 3.13.2, yt-dlp 2026.07.04 (Homebrew), SCRAPECREATORS_API_KEY present and healthy (doctor reports the transcript backstop as armed).

The code

skills/last30days/scripts/lib/pipeline.py:3778

# Fall back to SC YouTube search if yt-dlp failed or isn't installed.
if (result is None or not result.get("items")) and sc_token:

1 item is not falsy, so a partially bot-gated yt-dlp run skips the fallback entirely.

Reproduction

Any run where YouTube bot-gates yt-dlp. Observed on two consecutive real runs:

[YouTube] yt-dlp transcript failed for 24hStKkUY48 (exit 1): ERROR: [youtube] 24hStKkUY48:
  Sign in to confirm you're not a bot. Use --cookies-from-browser or --cookies ...

Resulting coverage:

- YouTube: 1 item
✅ All agents reported back!
├─  YouTube: 1 video │ 29 views │ 1/1 with transcripts

No warning, no Source Errors entry for YouTube — the run reads as healthy. The per-video transcript fallback did work correctly (Got transcripts for 6/6 videos (0 failed, 6 rescued via ScrapeCreators fallback)), which is what masks the problem: transcripts are rescued, but the search result set was never widened.

Note that --cookies-from-browser is not an available workaround here, since youtube_yt.py passes --no-cookies-from-browser unconditionally (lines 465, 707, 819) — by design, as far as I can tell.

Verifying the backstop itself works

With the threshold temporarily forced high, the SC search path runs fine and returns good results, so this is purely a trigger-condition issue:

[YouTube] SC YouTube: 8 videos returned
- YouTube: 4 items | channels: KernelCare, TuxCare, Learn Linux TV

Suggested fix

Trigger on a low item count rather than only on zero. One caveat worth flagging: the current code replaces result with the SC result, which would discard yt-dlp items (and their already-fetched transcripts) whenever the fallback fires on a non-empty result. Merging by video_id avoids that regression:

# Fall back to SC YouTube search if yt-dlp failed, isn't installed, or came back thin.
_YT_SC_MIN_ITEMS = 3
_yt_items = list((result or {}).get("items") or [])
if (result is None or len(_yt_items) < _YT_SC_MIN_ITEMS) and sc_token:
    try:
        _sc_result = youtube_yt.search_youtube_sc(
            yt_query, from_date, to_date, depth=depth, token=sc_token,
        )
        if _sc_result.get("error"):
            youtube_failure = str(_sc_result["error"])
        _seen = {i.get("video_id") for i in _yt_items if i.get("video_id")}
        for _i in _sc_result.get("items") or []:
            if _i.get("video_id") and _i["video_id"] not in _seen:
                _seen.add(_i["video_id"])
                _yt_items.append(_i)
        result = {"items": _yt_items}
    except Exception as exc:
        youtube_failure = str(exc)
        result = {"items": _yt_items} if _yt_items else None

Running this locally now and it behaves as expected. Happy to open a PR if the approach looks right — and happy to make the threshold configurable (e.g. LAST30DAYS_YT_SC_MIN_ITEMS) instead of a constant if you'd prefer.

Secondary observation

Separately, a bot-gated YouTube lane produces no warning at all in the output. Something like a ⚠ degraded marker on the source line when yt-dlp reports a bot-gate would make this visible to users, the same way the Reddit HTTP 429 partial is surfaced today.