Proposal: Add scrape-quality evals under evals/

Problem

I'd like to propose adding a structured eval layer for Firecrawl outputs under evals/.

Right now, when a scrape/crawl/search output is poor, it is hard to separate:

• What went wrong? — missing headings, tables, links, code blocks, prices, dates, noisy boilerplate, paywall text, etc.
• Why did downstream usage fail? — agent hallucinated, answer was not grounded, scraped content was incomplete/noisy, or trajectory failed later.

I think separating these into two eval layers would make regressions easier to catch and debug.

Proposed Eval Design

Layer 1 — Firecrawl Output Evals

This layer evaluates the raw Firecrawl output. Different endpoints need different metrics:

• /scrape: headings, tables, links, code blocks, price/date extraction, noise ratio, paywall detection
• /crawl: sitemap coverage, duplicate rate, broken/failed pages
• /search: relevance, freshness, result usefulness
• /map: URL discovery precision/recall, nDCG

This layer answers: what was wrong with the extracted output?

Layer 2 — Agent / Downstream Evals

This layer evaluates how the scraped content performs in an agent or downstream LLM workflow:

• Did the agent get a relevant answer from the scraped content?
• Did it hallucinate because the scrape output was noisy or incomplete?
• How many steps did the agent take?
• Did it recover from failures?
• Where did the trajectory fail?

This layer answers: why did the downstream task fail?

What I Built So Far

A first version focused on Layer 1: /scrape quality evals.

Pipeline:

Deterministic code checks first

→ Call LLM judge only for borderline/ambiguous cases

→ Add uncertain cases to human review queue

Key design choices:

• Deterministic checks catch obvious failures without LLM cost
• LLM judge only used for ambiguous outputs
• Final score = 60% deterministic + 40% LLM judge
• Missing required ground-truth content caps the score at 70
• Cohen's Kappa to compare LLM judge behavior against human labels

Initial Results

Tested on 30 URLs across 7 categories.

Overall: 56.3/100 across 30 URLs

Category findings:

• Adversarial: 35/100 — NYT appears unsupported, WSJ partial
• Ecommerce: 54.5/100 — price/spec fields often missing
• News: 57.5/100 — date extraction inconsistent
• Docs: 60.3/100 — code blocks often missing
• Jobs: 60/100 — inconsistent across job boards
• SPA: 62.5/100 — some SPAs work well, others fail
• Research: 62.5/100 — Wikipedia good, arXiv inconsistent

Observed gaps:

• Tables detected: 9/30
• Code blocks detected: 6/30
• Paywall detection: 0/30
• NYT returned "Website Not Supported" error from Firecrawl

Next Steps

• Expand benchmark to 50+ URLs with richer ground truth
• Add /crawl and /search eval modules
• Add agent/downstream eval layer for trajectory, relevance, hallucination
• CI integration to run evals on PRs and/or nightly builds

Code

https://github.com/AchuthReddy-16/firecrawl/tree/feature/scrape-quality-evals

Would the team be open to adding this type of scrape-quality eval framework under evals/?