[Bug] Google Search/SERP requests return HTTP 429 and the specialist finishes without results

Summary

The google_search specialist is being invoked correctly, but native Google SERP requests are repeatedly blocked with HTTP 429.

The scraper exhausts its retry/IP budget without obtaining SERP HTML. Despite having no usable search results, the specialist task exits with outcome=ok after several minutes.

This makes the feature appear to hang or makes the assistant stop responding without showing a clear error.

Environment

• Project: SurfSense
• Deployment platform: Railway
• Environment: Production
• Specialist: google_search
• Backend deployment: d3ed8a73-65fc-4003-9ac4-885145ce670c
• Approximate occurrence: 2026-08-25 16:51–16:56 UTC
• Relevant implementation areas:
  • surfsense_backend/app/proprietary/platforms/google_search/fetch.py
  • surfsense_backend/app/proprietary/platforms/google_search/scraper.py
  • surfsense_backend/app/proprietary/platforms/google_search/searxng.py
  • surfsense_backend/app/capabilities/google_search/scrape/executor.py

Proxy credentials are intentionally omitted from this issue.

Important finding

This is not a case where the specialist was simply skipped.

The logs confirm that:

• subagent_type='google_search' was invoked.
• The specialist was compiled with its search tools.
• Search queries were submitted.
• The browser fetch layer attempted to retrieve Google SERP pages.
• Google returned repeated HTTP 429 responses.

Representative logs

[google_search][perf] status=429 bytes=6571 has_results=False vet_ms=531 render_ms=4138 from_pool=False pool=0

The same pattern was repeated for multiple attempts:

status=429
has_results=False
from_pool=False
pool=0

The scraper then reported:

[google_search] no SERP HTML for https://www.google.com/search?q=women+over+40+health
[google_search] gave up on https://www.google.com/search?q=women+over+40+health (deadline/24-IP budget)

Other affected queries included searches similar to:

joint pain women over 40
menopause weight loss
menopause weight loss causes

Additional browser failures:

[google_search] no SERP HTML
[google_search] gave up ... (deadline/24-IP budget)

[google_search] render failed: TimeoutError(
  Page.goto: Timeout 30000ms exceeded,
  waiting until "load"
)

Task duration and incorrect success status

The specialist eventually exited with outcome=ok, even though no SERP HTML or usable search results were returned.

Observed invocation durations included approximately:

114 seconds
209 seconds
287 seconds

Examples from the task logs:

atask EXIT subagent_type='google_search'
outcome=ok
ainvoke=114.333s

atask EXIT subagent_type='google_search'
outcome=ok
ainvoke=209.419s

atask EXIT subagent_type='google_search'
outcome=ok
ainvoke=287.233s

Actual behavior

• The google_search specialist starts correctly.
• Google SERP requests receive HTTP 429.
• No valid SERP HTML is obtained.
• The scraper retries across the configured IP/deadline budget.
• Browser navigation may also hit a 30-second timeout.
• The specialist can take several minutes to finish.
• The task is reported as successful despite having no usable results.
• The user receives no clear explanation that Google SERP access failed.

Expected behavior

When Google SERP access is available:

• The specialist should return search results normally.

When Google blocks the request or no SERP HTML can be obtained:

• The request should fail within a bounded timeout.
• The error should be explicit and structured.
• The specialist should report a failure status instead of outcome=ok.
• The UI should show that Google Search is temporarily unavailable.
• The assistant should not silently continue as if the search succeeded.

Suspected cause

Google is actively blocking the automated browser SERP requests with HTTP 429.

The logs show an application-level response from Google, rather than:

• proxy authentication failure;
• DNS failure;
• TLS failure;
• connection refusal;
• backend exception.

The configured proxy may be contributing through IP reputation, rotation behavior, or Google anti-bot detection, but the logs do not prove that the proxy credentials are invalid.

The following values are also significant:

from_pool=False
pool=0

This suggests that no warmed internal SERP IP/session was available for these attempts.

Possible secondary issue: fallback is not active

The code contains a SearXNG fallback path controlled by SEARXNG_URL.

However, there were no corresponding fallback success logs, such as:

[google_search][searxng] ...

This means one of the following should be verified:

• SEARXNG_URL is not configured;
• the fallback is disabled for this request type;
• the fallback is not reached after the Google failure;
• the fallback itself is failing without sufficient logging.

Earlier successful Google News/RSS crawler requests are not evidence that native Google SERP is working. They use a different code path.

Steps to reproduce

1. Deploy SurfSense in Railway production.
2. Configure the Google Search specialist.
3. Run a chat request that requires live Google SERP research.
4. Ask a query that triggers the google_search specialist.
5. Observe the specialist invocation in the backend logs.
6. Observe repeated HTTP 429 responses from Google.
7. Wait for the retry/deadline budget to be exhausted.
8. Observe that the task may finish as outcome=ok without search results.

Impact

• Google Search/SERP specialist is currently unreliable or unusable.
• Requests may take several minutes before returning.
• Users may believe that the model is frozen or has stopped responding.
• The assistant may produce no answer because the research step returned an empty result.
• The current success status makes monitoring and debugging misleading.
• Repeated retries may consume proxy bandwidth and IP quota without producing value.

Suggested investigation

1. Improve error classification

Distinguish at least the following cases:

• HTTP 429 rate limit or anti-bot block;
• CAPTCHA/challenge page;
• empty SERP page;
• browser navigation timeout;
• proxy connection failure;
• proxy authentication failure;
• DNS/TLS failure;
• internal parser failure.

2. Fail fast on repeated HTTP 429

Do not continue the full retry budget when multiple consecutive attempts return the same blocking status.

Return a structured failure containing fields similar to:

{
  "status": "error",
  "provider": "google",
  "error_type": "rate_limited",
  "http_status": 429,
  "results_available": false,
  "retryable": true
}

The exact schema can follow the existing project conventions.

3. Fix task success reporting

The specialist must not finish with outcome=ok when:

has_results=False

and no fallback result was obtained.

The task should either:

• return a structured error;
• return a clearly marked empty result with an error reason;
• or successfully use a fallback provider.

4. Verify proxy/browser-pool behavior

Investigate:

• why pool=0;
• why all attempts report from_pool=False;
• whether sticky sessions are being created;
• whether IP rotation is occurring as expected;
• whether Google is blocking the current proxy IP range;
• whether the browser session has the required cookies;
• whether challenge pages are being detected;
• whether proxy authentication is working independently of Google blocking.

Proxy credentials and secrets should never be logged.

5. Verify SearXNG fallback

If SearXNG is supported as a fallback:

• document the required SEARXNG_URL;
• confirm that it is configured in Railway;
• ensure it is called after Google SERP failure;
• log fallback success/failure without exposing secrets;
• return fallback results when available.

6. Improve observability

Add structured logs containing:

• query hash or opaque query ID;
• provider;
• attempt number;
• HTTP status;
• timeout category;
• whether a proxy pool IP was used;
• whether SERP HTML was detected;
• whether a fallback was attempted;
• final result status.

Do not log raw user queries if they may contain sensitive information.

Acceptance criteria

• A simple Google SERP query returns valid results in production when the provider is available.
• Repeated HTTP 429 responses do not consume several minutes of retries.
• A failed SERP request returns a clear structured error.
• The specialist never reports outcome=ok when no results and no fallback result are available.
• The UI clearly communicates that Google Search is unavailable instead of appearing frozen.
• SearXNG fallback is either verified to work or explicitly documented as unavailable.
• Proxy/IP-pool health is visible through safe structured logs.
• No proxy credentials, tokens, cookies, or authorization headers appear in logs.
• An automated test covers:
  • successful SERP response;
  • HTTP 429;
  • browser timeout;
  • empty SERP response;
  • fallback success;
  • fallback failure.