Proposal: detect tool-trajectory regressions before deployment

Describe the feature or improvement

Add an opt-in trajectory regression detector for tool-using evaluations. A run should be able to capture a normalized per-sample trajectory, compare it with a checked-in known-good baseline, and fail validation when the tool-call sequence or terminal state drifts unexpectedly.

This is deliberately narrower than a personality or cross-model behavior metric: the target is observable execution behavior before deployment (which tools were called, with what normalized inputs, and whether the run reached the expected terminal state).

Why this is useful

The current recorder persists individual function_call, sampling, metrics, and extra events, while the tool conversation runner keeps parsed tool calls and the final task state in memory. Scalar metrics can remain green even when an agent starts calling tools in a different order, omits a required call, or stops in a different terminal state. A stable trajectory artifact would make this class of regression reproducible and bisectable in CI.

Proposed shape

• Add an opt-in capture hook at the tool-conversation runner boundary, or an equivalent recorder API, so existing evals keep their current behavior by default.
• Serialize a versioned, deterministic per-sample record containing normalized tool name/input, tool result status (not secrets), turn index, and an explicit terminal-state value.
• Provide a comparator that reports the first divergent step and distinguishes tool-call drift from terminal-state drift. Keep raw outputs out of the default comparison unless an eval opts in.
• Support baseline generation and validation as separate operations, so CI can review intentional baseline updates rather than silently rewriting them.
• Add focused known-good/known-bad regression coverage and a small benchmark showing time-to-bisect with and without the detector.

The existing evals/elsuite/solver_tools_convo.py Runner already has the integration boundary: it parses ToolCall values, executes them, tracks turns, and returns RunnerResult; evals/record.py provides the event/recorder serialization boundary. The maintainer-facing question is which boundary and schema should become the supported public contract.

Scope and compatibility

This should be additive and opt-in. It should not change existing event schemas, default eval results, or require model/API calls during tests. Baselines should be explicit test fixtures and should avoid user data or tool secrets.

Related discussions exist around broad behavioral drift (#1589, #1590), but this proposal is distinct: it checks deterministic agent execution trajectories and terminal states rather than personality, empathy, or cross-model claims.

Relevant source paths:

• evals/elsuite/solver_tools_convo.py
• evals/record.py
• evals/elsuite/bugged_tools/eval.py

Would maintainers prefer the capture/comparison API to live on RecorderBase, on Runner, or as a separate utility consumed by both? Once that contract is agreed, I can prepare a small implementation and regression fixtures.