Baike.dev
All toolsAI codingTrendingOpen sourceNewsSubmit
Log in
Back to tool/Back to issues
#2096·crawlee-python

Rendering type detection treats a failed static run as 'client only'

Author: ayush27316Created Jul 28, 2026Updated Aug 11, 2026
Labelst-tooling

Crawlee 1.8.4, _adaptive_playwright_crawler.py:431:

python
static_run = await self._crawl_one('static', context=context, state=old_state_copy)
if static_run.result and self.result_comparator(static_run.result, pw_run.result):
    detection_result = 'static'
else:
    detection_result = 'client only'

self.rendering_type_predictor.store_result(context.request, detection_result)

If the static sub-crawler raises (connection reset, timeout, SessionError from retry_on_blocked), static_run.result is None, so this falls to else and trains the predictor with 'client only'. static_run.exception isn't checked.

What confuses me is that the browser side, six lines earlier, does the opposite:

python
if pw_run.exception is not None:
    raise pw_run.exception

A browser failure is never turned into a label, but a static failure is.

Was this deliberate?

What made me look was that the label sticks: once 'client only' is stored, detection_probability_recommendation drops toward detection_ratio, so the URL shape gets re-tested less often. A transient failure during one detection seems like it can keep similar URLs on the browser path for a while, with nothing logged.

Am I reading that right?


Possibly related, same store_result call: with the default push_data_only_comparator, if both sub-crawlers succeed but the handler extracts nothing from either (both got served a challenge page), [] == [] compares equal and the request is labeled 'static'. Hit this on a Google search URL that returns a soft-block with HTTP 200. Happy to open separately if it's a different discussion.

Source: apify/crawlee-python

View original on GitHubView discussion on GitHub