BasicCrawler.teardown() during startup can reject or leave run() pending

Author: nikitachapovskii-devCreated Aug 28, 2026Updated Aug 28, 2026

Labelst-tooling

Context

I'm not sure if it's a v4 regression. Seems like teardown-before-run race already existed in v3, but the v4 migration exposed it more clearly and added a new failure mode involving ConcurrencySystem. Right know it affects tests for wcc sitemap crawler. The test is green because of catch (lol) but crawler fails.

Which package is this bug report for?

@crawlee/basic

Issue description

If teardown() is called while run() is still finishing initialization, but before AutoscaledPool.run() starts, the result depends on ConcurrencySystem ownership:

With the default crawler-owned system, run() rejects because teardown() stops the system before the pool starts.
With an injected system, run() remains pending indefinitely.

In the second case, AutoscaledPool.abort() marks the pool as stopped before its run resolver exists. When AutoscaledPool.run() starts later, the stopped pool never resolves or rejects its newly created promise.

Expected behavior: once crawler.run() has started, calling await crawler.teardown() should always make that run settle, including during initialization.

Code sample

javascript

import { setTimeout as delay } from 'node:timers/promises';

import { BasicCrawler } from '@crawlee/basic';
import { ConcurrencySystem } from '@crawlee/core';

async function reproduce(concurrencySystem) {
    const initialized = Promise.withResolvers();
    const resume = Promise.withResolvers();

    class DelayedCrawler extends BasicCrawler {
        async init() {
            await super.init();
            initialized.resolve();
            await resume.promise;
        }
    }

    const crawler = new DelayedCrawler({
        keepAlive: true,
        ...(concurrencySystem && { concurrencySystem }),
        requestHandler: async () => {},
    });

    const runPromise = crawler.run();
    const outcomePromise = runPromise.then(
        () => 'resolved',
        (error) => `rejected: ${error.message}`,
    );

    await initialized.promise;
    await crawler.teardown();
    resume.resolve();

    const outcome = await Promise.race([
        outcomePromise,
        delay(1_000, 'still pending'),
    ]);

    console.log(concurrencySystem ? 'injected:' : 'owned:', outcome);

    // Cleanup for the pending case.
    if (outcome === 'still pending') {
        await crawler.teardown();
        await outcomePromise;
    }
}

await reproduce();

const concurrencySystem = new ConcurrencySystem({ maxConcurrency: 1 });
await concurrencySystem.start();
await reproduce(concurrencySystem);
await concurrencySystem.stop();

Output:

owned: rejected: The ConcurrencySystem this AutoscaledPool borrows has not been started...
injected: still pending

The pending abort-before-run behavior can also be reproduced in Crawlee v3. The owned ConcurrencySystem rejection is the new v4 manifestation of the same lifecycle race.

Package version

@crawlee/[email protected]
@crawlee/[email protected]

Node.js version

v24.15.0

Operating system

macOS 26.2 (arm64)

reproduced locally.

Source: apify/crawlee

View original on GitHub View discussion on GitHub