A fetch extension that fails over between scraping providers

Author: scarsamCreated Sep 18, 2026Updated Sep 18, 2026

extensions/firecrawl/install.sh wires Firecrawl as the fetch layer. Something measured this week that affects it: Firecrawl's v2 scrape returns the same HTTP 500 SCRAPE_ALL_ENGINES_FAILED for a target 404, a 503, a 429 and a block page. The status carries no information about which, so a skill that reads it can't tell "gone" from "blocked".

I maintain Proxlane, a self-hosted gateway in front of Firecrawl, ScraperAPI, ScrapingBee, Scrapfly and Bright Data. It fails over between them and gives each response a real outcome, with a block-page check before anything counts as a success. ScraperAPI-shaped parameters, one container, AGPL.

https://github.com/proxlane/proxlane

Would a Proxlane extension fit next to the Firecrawl one? I can draft it.