Baike.dev
Log in
Back to tool/Back to issues
#4145·crawlee

`BrowserFetchHttpClient`: use a logged-in page as the HTTP client

Author: janbucharCreated Sep 18, 2026Updated Sep 18, 2026
Labelst-tooling
  • related to #3040, #3283, discussion #3217; prior art in crawlee-python: PlaywrightHttpClient (apify/crawlee-python#1134)
  • Recurring pattern: open a site once in a browser (login, anti-bot), then fetch many API/HTML URLs from that page with in-page fetch() because CheerioCrawler gets blocked where the browser's TLS/HTTP2 fingerprint, cookies and service workers do not. Today people hand-roll a crawler loop inside a PlaywrightCrawler handler (fetchNextRequest / reclaimRequest / retry counting).
  • With the v4 BaseHttpClient contract this is a ~50-line client: fetch() runs page.evaluate(fetch(...)), rebuilds a ResponseWithUrl, and re-runs a login callback on 401. Plugged into CheerioCrawler as httpClient it gives back the queue, retries, stats, transactions, $ / json, and real concurrency (concurrent evaluate(fetch) calls on one page are fine — nothing navigates). Verified locally with retries, 4-way concurrency and a mid-run re-login.
  • Constraints worth documenting: same-origin (or CORS) with the page the login left behind; forbidden headers (Cookie, User-Agent) and Set-Cookie are invisible, so the browser jar is authoritative; proxyUrl / fingerprint from CustomFetchOptions are ignored (the browser context decides).
  • Python already has half of this: PlaywrightHttpClient is the default http_client of PlaywrightCrawler for send_request, bound to the current page via a contextvar and calling browser_context.request.fetch() (APIRequestContext). It shares the context's cookies and proxy and has no same-origin constraint, but it is Playwright's own HTTP stack rather than the browser's (fingerprint TODO: apify/crawlee-python#1055), and crawl() / stream() are NotImplementedError, so it cannot drive a crawler. Two flavours worth distinguishing here: APIRequestContext-backed (parity with Python, good default for sendRequest inside browser crawlers) and in-page-fetch-backed (browser TLS, origin-bound) — the second is the one that replaces the hand-rolled loop.
  • Either ship it next to FetchHttpClient or document the pattern in the HTTP clients guide; it answers the "scraping behind login" asks more directly than a page-level login helper.

Source: apify/crawlee

View original on GitHubView discussion on GitHub