ThrottlingRequestManager has no effect behind RequestManagerTandem

Author: vdusekCreated Sep 16, 2026Updated Sep 16, 2026
Labelsbugt-tooling

BasicCrawler finds the throttler with a bare isinstance(self._request_manager, ThrottlingRequestManager) check in three places: the respect_robots_txt_file startup warning (_basic_crawler.py:708), the 429 recording (:1644), and the robots.txt crawl-delay (:1710).

RequestLoader.to_tandem() wraps the manager in a RequestManagerTandem, so await loader.to_tandem(throttler) silently drops the per-domain backoff and the crawl-delay. Both warnings then say the crawler is not using ThrottlingRequestManager, while one is configured.

RequestManagerTandem exposes no accessor for the manager it wraps, so a fix needs new public API. #2158 left this out of scope on purpose.

✍️ Drafted by Claude Code