Semantic Scholar retriever has an unbounded request and unguarded JSON parsing

Author: anupamking01Created Sep 13, 2026Updated Sep 13, 2026

Summary

SemanticScholarSearch.search() currently performs its external request without a timeout and parses the response body outside the request-error guard:

python
try:
    response = requests.get(self.BASE_URL, params=params)
    response.raise_for_status()
except requests.RequestException as e:
    print(...)
    return []

payload = response.json()

This leaves two reliability gaps in the same retrieval call path:

  1. an unresponsive Semantic Scholar endpoint can block the research step indefinitely at the HTTP layer;
  2. an HTTP 200 response with malformed/non-JSON content raises from response.json() instead of degrading to [].

Expected behavior

The scholarly retriever should have a bounded request duration and treat response-decoding failures the same way it already treats transport/HTTP failures.

A narrow fix would:

python
try:
    response = requests.get(self.BASE_URL, params=params, timeout=10)
    response.raise_for_status()
    payload = response.json()
except (requests.RequestException, ValueError) as e:
    print(...)
    return []

This follows the existing timeout/error-handling patterns used by other retrievers without changing happy-path result normalization.

Regression coverage

Focused offline tests can verify:

  • requests.get receives a finite timeout;
  • .json() raising ValueError("invalid json") returns [] rather than propagating;
  • the existing happy path remains unchanged.

This is also a concrete instance of the broader external-call timeout concern in #1764, but scoped to the Semantic Scholar retriever so it can be fixed and tested independently.

I searched the PR tracker for a Semantic Scholar timeout / malformed-JSON fix and did not find an existing implementation. #2036 touches Semantic Scholar sort validation and SearXNG timeout, not this request path.

AI-assisted review disclosure: an AI coding assistant was used to inspect the retriever request path and help prepare this report.

Source: assafelovic/gpt-researcher