LinkedIn 404 means ZERO RESULTS, not a query-length limit - and MAX_SOURCE_FETCHES=1 makes it silently zero the source

Author: grahamgourlay-progCreated Aug 31, 2026Updated Sep 12, 2026

Summary

ScrapeCreators' /v1/linkedin/search/posts returns HTTP 404 to mean "this search had zero results". linkedin.py treats it as an HTTPError, so a source that was merely quiet is reported as broken: LinkedIn: 0 items (error: HTTP 404: Not Found) in Source Coverage, and Failed in doctor --postmortem.

This breaks the skill's own synthesis contract. SKILL.md tells the model that no-results means "the source completed cleanly with zero matches", while an error state "means the run did not establish that the source was quiet", and instructs it never to write "nothing on X" for the latter. Here the run did establish quietness, and the model is told to report the opposite.

  • Version: 3.22.0 (Windows 11, Python 3.14, INCLUDE_SOURCES=linkedin, valid SC key)
  • Related: #939 (reports this symptom with a different root cause - see correction below), #986 (Instagram 404, plausibly the same misclassification in instagram.py)

Correction to #939, defect 3

#939 states: "Queries over ~80 characters return HTTP 404. Verified: 108 chars fails, 79 passes." Length is not the predictor. Same key, same date_posted=last-month:

chars query result
31 Twenty7tec Mortgage Brain Iress 404
39 zxqwvbn qqjjkl nonexistent phrase 12345 404
63 mortgage broker sourcing system Twenty7tec Mortgage Brain Iress 404
90 the mortgage broker industry and mortgage brokers working across the mortgage market today 200, 20 posts
92 mortgage broker mortgage brokers mortgage industry mortgage market mortgage lending mortgage 200, 20 posts

A 31-char query 404s; a 92-char query returns 20 posts. The two-point sample in #939 (108 fails, 79 passes) is consistent with those queries differing in result count, not length.

The controlled case is the nonsense query: 39 chars, syntactically fine, guaranteed no matches, returns 404. Narrowing confirms it tracks specificity - Twenty7tec alone returns 20 posts, Twenty7tec Mortgage Brain Iress returns 404.

This matters for the fix: a length-based guard (truncating or splitting long queries) would not help, and would degrade good long queries that work today.

Why it bites harder than it looks

MAX_SOURCE_FETCHES['linkedin'] = 1 (pipeline.py:110) gives LinkedIn exactly one subquery per run, and it draws the primary - by SKILL.md convention the longest, most-ANDed subquery, i.e. the one most likely to return zero. On a normal multi-term topic LinkedIn therefore contributes nothing, reports as broken, and gets no second attempt.

Observed on a real run: three subqueries, LinkedIn issued only the primary (mortgage broker sourcing system Twenty7tec Mortgage Brain Iress), 404, lane finished in 2.1s. The second subquery - which returns 10 posts when run directly - was never issued.

With --max-source-fetches 3 on the same plan: the long primary 404s, two short subqueries return 10 posts each, the engine pools 6 items and correctly reports partial - 6 items returned. Pooling and partial-status handling are already correct - the 404 classification is the only defect.

Secondary observation (possibly intended)

A capped source appears to consume subqueries in plan order and ignore the per-subquery sources array. Given a plan whose primary listed only ["reddit"] and whose second subquery listed ["linkedin"], LinkedIn still searched the primary's text. Verified both with --search linkedin,reddit and via INCLUDE_SOURCES with no --search.

Consequence: a caller cannot hand a capped source a purpose-built short query - the only lever is raising the global cap. If intended, worth documenting, since the natural authoring instinct is to target a subquery at a source.

Suggested fix

linkedin.py:search_linkedin(), the except http.HTTPError at line 66:

except http.HTTPError as exc:
    if exc.status_code == 404:
        # ScrapeCreators returns 404 for "no results", not endpoint-missing.
        _log("No results (HTTP 404)")
        return {"posts": []}
    _log(f"Search failed (HTTP {exc.status_code}): {exc}")
    return {"posts": [], "error": str(exc)}

Restores the no-results vs error distinction the synthesis rules depend on, and stops doctor showing a working source as Failed. search_profile() (line 293) likely wants the same. If #986 is the same shape, instagram.py may too.

Source: mvanhorn/last30days-skill