A language model can perform well on a clean benchmark and still struggle with the cases that matter in production.
Benchmarks and curated datasets are useful when prototyping an LLM-based system.
They help teams compare models, test an initial prompt, and determine whether an idea is technically plausible.
But as a system moves closer to production, the evaluation problem changes.
Real inputs are often ambiguous.
Labels may be inconsistent.
Important context may be missing or truncated.
The evaluation set may not reflect the production distribution.
Edge cases that rarely appear in benchmarks can become common sources of failure.
Even when offline metrics improve, those results may not translate cleanly into production behavior.
We encountered these challenges while evaluating an LLM-based system designed to reduce false positives in GitHub secret scanning.
Secret scanning identifies credentials such as tokens and keys that may have been committed to a repository.
Because some candidate strings resemble secrets, but don’t actually represent real credentials, developers may spend time investigating alerts that don’t require remediation.
Rather than determine whether an LLM could classify a string correctly, we needed to understand whether the system could reduce noisy alerts while preserving enough recall to remain safe for a security workflow.
In this post, we share the practices that helped us move from promising prototype results to production.
The lessons apply broadly to LLM-powered systems in code analysis, developer tools, security, data analysis, and other production workflows.
1.
Start with the product decision, not the model When an LLM system doesn’t perform as expected, the first instinct is often to adjust its technical components.
Teams may rewrite the prompt, add context, introduce another reasoning step, adjust the surrounding pipeline, or switch models.
Before making any of these changes, they should define the decision the evaluation is meant to support.
For our secret-scanning work, we asked: Can the system reduce false positives while preserving enough recall to be safe in a production security workflow?
To answer this question, teams must decide which mistakes are acceptable, which metrics should drive the product decision, and which guardrails must remain within their defined thresholds.
In secret scanning, incorrectly suppressing a real credential can be more consequential than asking a developer to review an additional alert.
We therefore did not treat precision and recall as equally interchangeable metrics.
Our primary objective was to reduce false positives and improve precision.
Recall served as a safety constraint: an experiment could advance only if any decrease remained within a predefined acceptable range.
This gave us a clear way to evaluate tradeoffs.
We selected the configuration that achieved the strongest false-positive reduction while satisfying the recall requirement and meeting our operational guardrails.
We organized the evaluation criteria into three levels: Primary outcome This measured the user benefit we were trying to improve: False-positive reduction Precision Safety constraint This prevented an apparent improvement from introducing unacceptable security risk: Recall Operational guardrails These determined whether the result was practical to deploy: Latency Cost Reliability Production compatibility This distinction prevented us from treating every metric as interchangeable.
A change that reduced false positives but significantly lowered recall wasn’t automatically an improvement.
Neither was a change that improved quality while making the system too slow, expensive, or difficult to integrate.
Consider two hypothetical experiment results: Experiment Precision Recall Latency Decision Experiment A Large improvement Falls below the safety guardrail Acceptable Don’t advance Experiment B Moderate improvement Remains within the guardrail Acceptable Continue t