page_timeout is silently capped at 60s over HTTP with no way to raise it

Author: damusixCreated Aug 28, 2026Updated Sep 14, 2026

Labels🐞 Bug⚙️ In-progress📌 Root caused

Problem

_clamp_untrusted caps page_timeout and wait_for_timeout at _MAX_TIMEOUT_MS = 60_000 for any config arriving over HTTP. The Docker server passes Provenance.UNTRUSTED at every /crawl call site, so a client asking for more is silently given 60s — no warning, no field in the response. A page that legitimately takes longer always fails with Page.goto: Timeout 60000ms exceeded, quoting a number the caller never sent.

The cap itself is reasonable for a public server. Not being able to change it is the problem: it isn't in config.yml, no env var overrides it, and the value is a module-level literal.

crawler.base_config looks like the intended escape hatch and isn't — it's applied post-deserialization with setattr, genuinely bypassing the clamp, but only for fields that are None or ''. page_timeout always carries its 60000 default, so setting it there does nothing, silently.

Expected

An operator running their own container can raise the ceiling — an env var, a limits: entry alongside wall_clock_s, or a base_config that applies to defaults too. Failing that, a clamped request should say so rather than report a timeout the caller didn't ask for.

Reproduce (docker, 0.9.2)

A server that answers after 90s:

python

import time
from http.server import BaseHTTPRequestHandler, HTTPServer
BODY = b"<html><body><h1>Slow</h1><p>" + b"x"*600 + b"</p></body></html>"
class H(BaseHTTPRequestHandler):
    def do_GET(self):
        time.sleep(90)
        self.send_response(200); self.send_header("Content-Length", str(len(BODY))); self.end_headers()
        self.wfile.write(BODY)
HTTPServer(("0.0.0.0", 8080), H).serve_forever()

bash

docker network create repro
docker run -d --name slowsrv --network repro -v $PWD/slow_server.py:/srv/s.py:ro python:3.12-slim python /srv/s.py
docker run -d --name c4ai --network repro -p 11235:11235 --shm-size=2gb \
  -e CRAWL4AI_API_TOKEN=dev-token -e CRAWL4AI_ALLOW_INTERNAL_URLS=true unclecode/crawl4ai:0.9.2

curl -X POST http://localhost:11235/crawl -H 'authorization: Bearer dev-token' \
  -H 'content-type: application/json' \
  -d '{"urls":["http://slowsrv:8080/"],"crawler_config":{"page_timeout":300000,"wait_until":"domcontentloaded"}}'

Before — asked for 300s:

HTTP 500 in 60.7s
Page.goto: Timeout 60000ms exceeded

After — same request, against an image built with _MAX_TIMEOUT_MS = 600_000:

HTTP 200 in 90.9s
success: true, markdown: "# Slow page\nxxxx…"

Suggested patch

Make the ceiling configurable rather than removing it:

python

_MAX_TIMEOUT_MS = int(os.environ.get("CRAWL4AI_MAX_TIMEOUT_MS", 60_000))

An env var keeps the safe default for a public deployment while letting an operator raise it for a trusted one. A limits.max_timeout_ms key in config.yml, next to wall_clock_s, would fit the existing "0 = unbounded" convention equally well.

Separately, base_config skipping any field with a non-empty default is surprising enough to be worth documenting.

Source: unclecode/crawl4ai

View original on GitHub View discussion on GitHub