Stabilize and Enhance Internal Project Evaluations

Goal: To significantly improve the reliability, visibility, and actionability of our project's internal (non-benchmark) evaluations.

Problem: Our current small project evals are inconsistent ("bleed"), making it difficult to trust their results, track quality trends, and quickly address regressions. This slows down development and impacts confidence in changes.

Vision: A streamlined evaluation system where internal project evals are stable, provide clear signals, and enable rapid feedback loops. Developers can confidently rely on these evals to catch issues early and understand the impact of their changes.

Key Outcomes:

• Reduced flakiness and increased reliability of existing project evals.
• Clear visibility into eval results and historical trends.
• Faster detection and easier debugging of regressions.
• Basic automation to flag issues and guide developers.
• "Approve" button to increase visibility and reduce excessive runs

Out of Scope:

• Large-scale benchmark evals (e.g., SWE-bench, terminal-bench).
• Overhauling the entire testing infrastructure beyond the scope of these specific project evals.