LinkedIn 404 means ZERO RESULTS, not a query-length limit - and MAX_SOURCE_FETCHES=1 makes it silently zero the source
Summary
ScrapeCreators' /v1/linkedin/search/posts returns HTTP 404 to mean "this search had zero
results". linkedin.py treats it as an HTTPError, so a source that was merely quiet is
reported as broken: LinkedIn: 0 items (error: HTTP 404: Not Found) in Source Coverage, and
Failed in doctor --postmortem.
This breaks the skill's own synthesis contract. SKILL.md tells the model that no-results
means "the source completed cleanly with zero matches", while an error state "means the run
did not establish that the source was quiet", and instructs it never to write "nothing on X"
for the latter. Here the run did establish quietness, and the model is told to report the
opposite.
- Version: 3.22.0 (Windows 11, Python 3.14,
INCLUDE_SOURCES=linkedin, valid SC key) - Related: #939 (reports this symptom with a different root cause - see correction below),
#986 (Instagram 404, plausibly the same misclassification in
instagram.py)
Correction to #939, defect 3
#939 states: "Queries over ~80 characters return HTTP 404. Verified: 108 chars fails, 79
passes." Length is not the predictor. Same key, same date_posted=last-month:
| chars | query | result |
|---|---|---|
| 31 | Twenty7tec Mortgage Brain Iress |
404 |
| 39 | zxqwvbn qqjjkl nonexistent phrase 12345 |
404 |
| 63 | mortgage broker sourcing system Twenty7tec Mortgage Brain Iress |
404 |
| 90 | the mortgage broker industry and mortgage brokers working across the mortgage market today |
200, 20 posts |
| 92 | mortgage broker mortgage brokers mortgage industry mortgage market mortgage lending mortgage |
200, 20 posts |
A 31-char query 404s; a 92-char query returns 20 posts. The two-point sample in #939 (108 fails, 79 passes) is consistent with those queries differing in result count, not length.
The controlled case is the nonsense query: 39 chars, syntactically fine, guaranteed no
matches, returns 404. Narrowing confirms it tracks specificity - Twenty7tec alone returns
20 posts, Twenty7tec Mortgage Brain Iress returns 404.
This matters for the fix: a length-based guard (truncating or splitting long queries) would not help, and would degrade good long queries that work today.
Why it bites harder than it looks
MAX_SOURCE_FETCHES['linkedin'] = 1 (pipeline.py:110) gives LinkedIn exactly one subquery
per run, and it draws the primary - by SKILL.md convention the longest, most-ANDed
subquery, i.e. the one most likely to return zero. On a normal multi-term topic LinkedIn
therefore contributes nothing, reports as broken, and gets no second attempt.
Observed on a real run: three subqueries, LinkedIn issued only the primary
(mortgage broker sourcing system Twenty7tec Mortgage Brain Iress), 404, lane finished in
2.1s. The second subquery - which returns 10 posts when run directly - was never issued.
With --max-source-fetches 3 on the same plan: the long primary 404s, two short subqueries
return 10 posts each, the engine pools 6 items and correctly reports
partial - 6 items returned. Pooling and partial-status handling are already correct -
the 404 classification is the only defect.
Secondary observation (possibly intended)
A capped source appears to consume subqueries in plan order and ignore the per-subquery
sources array. Given a plan whose primary listed only ["reddit"] and whose second
subquery listed ["linkedin"], LinkedIn still searched the primary's text. Verified both
with --search linkedin,reddit and via INCLUDE_SOURCES with no --search.
Consequence: a caller cannot hand a capped source a purpose-built short query - the only lever is raising the global cap. If intended, worth documenting, since the natural authoring instinct is to target a subquery at a source.
Suggested fix
linkedin.py:search_linkedin(), the except http.HTTPError at line 66:
except http.HTTPError as exc:
if exc.status_code == 404:
# ScrapeCreators returns 404 for "no results", not endpoint-missing.
_log("No results (HTTP 404)")
return {"posts": []}
_log(f"Search failed (HTTP {exc.status_code}): {exc}")
return {"posts": [], "error": str(exc)}
Restores the no-results vs error distinction the synthesis rules depend on, and stops
doctor showing a working source as Failed. search_profile() (line 293) likely wants the
same. If #986 is the same shape, instagram.py may too.
Source: mvanhorn/last30days-skill