`ThrottlingRequestManager` does not stop a 429 from retiring the session
Author: janbucharCreated Aug 9, 2026Updated Aug 10, 2026
Labelsbugt-tooling
_raise_for_session_blocked_status_code records the domain delay and then falls through to
Session.is_blocked_status_code, and 429 is in _DEFAULT_BLOCKED_STATUS_CODES. So on a domain the
throttler covers, a 429 still raises SessionError, still retires the session and still spends a
max_session_rotations slot — the churn #1437 was filed about. Only ignore_http_error_status_codes
avoids it, and nothing says so.
This wants fixing together with a stall timeout, in one PR:
- Nothing bounds a permanently rate-limiting domain except the retry accounting the fix removes.
_should_retry_requestcapsSessionErroratmax_session_rotationsand refusesHttpClientStatusCodeErroroutright, so today such a domain drains and the crawl ends. Make a throttled 429 free and it never terminates —run()hangs while the statistics report nothing, which is what the TS port walked into. - So the fix also needs a per-domain stall timeout that gives up once a domain with queued requests
has been rate-limiting for that long, and a
keep_aliveexemption.max_domain_stall_secsis a constructor argument, so landing it separately ships a release where the knob does nothing.
Both halves are in apify/crawlee#3741. From #1762.
Source: apify/crawlee-python