Semantic Scholar retriever has an unbounded request and unguarded JSON parsing
Summary
SemanticScholarSearch.search() currently performs its external request without a timeout and parses the response body outside the request-error guard:
try:
response = requests.get(self.BASE_URL, params=params)
response.raise_for_status()
except requests.RequestException as e:
print(...)
return []
payload = response.json()This leaves two reliability gaps in the same retrieval call path:
- an unresponsive Semantic Scholar endpoint can block the research step indefinitely at the HTTP layer;
- an HTTP 200 response with malformed/non-JSON content raises from
response.json()instead of degrading to[].
Expected behavior
The scholarly retriever should have a bounded request duration and treat response-decoding failures the same way it already treats transport/HTTP failures.
A narrow fix would:
try:
response = requests.get(self.BASE_URL, params=params, timeout=10)
response.raise_for_status()
payload = response.json()
except (requests.RequestException, ValueError) as e:
print(...)
return []This follows the existing timeout/error-handling patterns used by other retrievers without changing happy-path result normalization.
Regression coverage
Focused offline tests can verify:
requests.getreceives a finite timeout;.json()raisingValueError("invalid json")returns[]rather than propagating;- the existing happy path remains unchanged.
This is also a concrete instance of the broader external-call timeout concern in #1764, but scoped to the Semantic Scholar retriever so it can be fixed and tested independently.
I searched the PR tracker for a Semantic Scholar timeout / malformed-JSON fix and did not find an existing implementation. #2036 touches Semantic Scholar sort validation and SearXNG timeout, not this request path.
AI-assisted review disclosure: an AI coding assistant was used to inspect the retriever request path and help prepare this report.
Source: assafelovic/gpt-researcher