Share one concurrency budget between multiple crawlers
Author: vdusekCreated Aug 7, 2026Updated Aug 7, 2026
Labelsenhancementt-tooling
Each crawler builds its own AutoscaledPool and SystemStatus, so several crawlers in one process oversubscribe the host - there is no way to cap their combined concurrency. JS v4 split the scaling logic out into a ConcurrencySystem that can be constructed once and passed to several crawlers.
JS implementation: apify/crawlee#3917
✍️ Drafted by Claude Code
Source: apify/crawlee-python