[Request] Add Unbrowse as an opt-in crawler impl (CRAWLER_IMPLS=unbrowse)

Feature Description

Self-hosters who set CRAWLER_IMPLS get one of seven crawlers: browserless, exa, firecrawl, jina, naive, search1api and tavily. I'd like to propose adding an eighth, unbrowse, and ask whether you'd accept one before I write it. No new crawler has been added since #8033 (June 2025), so I'm asking first instead of sending code.

Disclosure: I maintain Unbrowse (https://unbrowse.ai, MIT source at https://github.com/unbrowse-ai/unbrowse). It is a hosted service that turns websites into APIs that agents can call. This proposal only covers the crawl slot. It would not add a search provider, and Unbrowse is not a general web search engine. I drafted this with an AI coding assistant, then checked the API call below myself.

The part I think matters for LobeHub is JS-heavy pages. #14899 and #15180 report browserless timing out, or returning empty markdown, on modern frontend pages. Unbrowse returns markdown whether or not the page needs a browser: render: "auto" only starts one when plain fetching isn't enough.

Proposed Solution

This would follow crawImpl/firecrawl.ts closely:

• packages/web-crawler/src/crawImpl/unbrowse.ts: a CrawlImpl that uses the existing withTimeout, toFetchError, createHTTPStatusError and parseJSONResponse helpers. Its config is UNBROWSE_API_KEY, plus an optional UNBROWSE_URL (default https://unbrowse.ai/api/mcp).
• Register it in crawImpl/index.ts, and add unbrowse: 'Unbrowse' to src/features/ToolSetting/const.tsx.
• crawImpl/__tests__/unbrowse.test.ts, mirroring firecrawl.test.ts: success, 4xx/5xx, JSON-RPC error, empty content, timeout.
• A row in the CRAWLER_IMPLS table plus an UNBROWSE_API_KEY section in docs/self-hosting/advanced/online-search.mdx and .zh-CN.mdx.

The request is a single POST:

curl https://unbrowse.ai/api/mcp \
  -H "Authorization: Bearer $UNBROWSE_API_KEY" \
  -H "Content-Type: application/json" -H "Accept: application/json" \
  -d '{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"unbrowse.scrape","arguments":{"url":"https://example.com","formats":["markdown"]}}}'

It returns result.content[0].text, a JSON string containing { url, finalUrl, metadata: { title, description, statusCode }, markdown, links }. That maps directly to CrawlSuccessResult: content = markdown, title, description, url = finalUrl, siteName = hostname, contentType: 'text'. Errors come back as { error: { code, message } } and would go through the existing error mapping. I ran it against https://example.com today and got a 200 with the expected title and markdown.

It would be opt-in only. Nothing changes in DEFAULT_CRAWL_IMPLS, and nothing runs unless someone sets CRAWLER_IMPLS and UNBROWSE_API_KEY.

Additional Information

If you're open to it, I'll send a PR against canary that follows the checks in AGENTS.md (bun run check on the changed files). If you'd rather not add more third-party crawlers to the tree, please say so and I'll close this. Thanks.