#5057 Feature Request: RFC: Standardized Evaluation/Benchmarking Harness for LLM Tool Calling

What is the problem that your feature request solves?

Hi team - as Browser Use scales to support more models (Groq, local LLMs, GPT-4o), ensuring agent reliability across updates and different architectures becomes difficult without a baseline metric.

I’d like to contribute a lightweight, standardized benchmarking suite in a new /evals directory to help the team catch silent regressions before they merge.

Proposed Implementation:

A standalone script that runs the browser-use agent against 3-5 static, local HTML files (testing basic forms, deep DOM trees, and dynamic popups).
Automatically tracks and outputs a clean CLI table comparing:
Task success rate
Step count / Routing accuracy
Latency and token usage per run
Designed to be easily hooked into GitHub Actions/CI later so PRs can be tested against a known benchmark.

Why me: I specialize in building custom evaluation harnesses for tool-calling agents. I recently built a similar suite for a LangGraph support system that successfully tracked routing accuracy and caught reproducible function-calling failures across different models (like malformed tool calls from Llama-3.3-70B).

Before I start writing the core logic, is this an architectural direction you'd like merged into the main repo? Happy to draft the initial PR this week if so.

What is your proposed solution?

I propose building a native, lightweight evaluation pipeline. The specific solution includes:

Creating an /evals directory containing 3-5 static HTML mockups (e.g., a dummy e-commerce checkout, a deep/nested DOM tree, and a dynamic modal).
A test runner script that executes the browser-use agent against these static targets with a predefined set of instructions.
An automated logging output that tracks: Task Success (boolean), Steps Taken, Token Usage, and Latency.

Broader scope: Right now, this acts as a local benchmarking tool for developers to run before committing. In the broader scope, this lays the architectural foundation for CI/CD integration, allowing maintainers to run a simple make eval to instantly see if a new PR or parser update regresses the agent's core capabilities.

What hacks or alternative solutions have you tried to solve the problem?

When building my previous tool-calling evaluation harnesses, I encountered the standard workarounds, which all fall short for an open-source agent:

Manual / Live-Site Regression Testing: Developers run the agent on real websites (like Google or Amazon) to test updates. The problem is that live DOMs change daily (A/B tests, varying network latency, captchas), making the benchmarks completely non-deterministic. A static local-HTML approach isolates the LLM's logic from network/DOM volatility.
Standard Unit Tests: Testing the Pydantic parsers or LLM wrappers in isolation. While necessary, this fails to catch emergent "agentic" failures—like the model successfully parsing JSON but getting stuck in an infinite loop of clicking the wrong coordinate.
Heavy Third-Party Eval Platforms (LangSmith, Braintrust): These are great for internal enterprise teams, but they introduce friction for open-source contributors who don't want to set up external API keys just to test a PR. A native, lightweight script removes that friction entirely.

What version of browser-use are you currently using?

0.13.2

How badly do you want this new feature?

It's an urgent deal-breaker, I can't live without it
It's important to add it in the near-mid term future
It would be nice to add it sometime in the next 2 years
I'm willing to start a PR to work on this myself
My company would spend >$5k on Browser-Use Cloud if it solved this reliably for us