A fetch extension that fails over between scraping providers
extensions/firecrawl/install.sh wires Firecrawl as the fetch layer. Something measured this week that affects it: Firecrawl's v2 scrape returns the same HTTP 500 SCRAPE_ALL_ENGINES_FAILED for a target 404, a 503, a 429 and a block page. The status carries no information about which, so a skill that reads it can't tell "gone" from "blocked".
I maintain Proxlane, a self-hosted gateway in front of Firecrawl, ScraperAPI, ScrapingBee, Scrapfly and Bright Data. It fails over between them and gives each response a real outcome, with a block-page check before anything counts as a success. ScraperAPI-shaped parameters, one container, AGPL.
https://github.com/proxlane/proxlane
Would a Proxlane extension fit next to the Firecrawl one? I can draft it.
Source: AgriciDaniel/claude-seo