Feature request: anybrowse MCP integration for Cloudflare-protected scraping
Problem
GPT-Researcher's web retrieval fails silently on Cloudflare-protected sites (major news outlets, LinkedIn, Amazon, government sites). The scraper gets a 403 or returns empty content without the research loop knowing.
Proposed integration
anybrowse is an MCP-native web scraper with Cloudflare bypass via residential Chrome. It fits naturally into the GPT-Researcher retrieval chain as a fallback scraper when standard HTTP fails.
It exposes a scrape tool that returns clean LLM-ready markdown -- the exact format GPT-Researcher needs for its context window.
Quick test
Add as an MCP server:
{
"mcpServers": {
"anybrowse": {
"type": "streamable-http",
"url": "https://anybrowse.dev/mcp"
}
}
}Then: "Research the latest AI regulation news from EU government sites" -- anybrowse handles the Cloudflare-protected .europa.eu domains that standard scrapers bounce off.
Details
- Free tier: 10 scrapes/day, no API key
- Paid: $5 for 3,000 credits (never expire)
- MCP tools:
scrape,crawl,batch_scrape,extract,search - Docs: https://anybrowse.dev/docs
Would be happy to contribute a retriever class if this seems like a good fit for the project.
Source: assafelovic/gpt-researcher