[BFCL] Several multi_turn_base items have ground truth that cannot be produced from the environment

Several multi_turn_base items have ground truth that cannot be produced from the environment the model is given. They are unpassable regardless of model capability.

How these were found

We benchmark local open-weight models on multi_turn_base. 27 of the 200 items are failed by all 18 models we have measured. Since multi_turn_checker executes the ground truth live against the same API classes, a malformed answer key is ruled out by construction, so we had a frontier model sit 8 of the 27 blind — it saw only the user turns, the tool docs, and the execution results of its own calls, never possible_answer — and graded it with BFCL's own checker.

It passed 2 and failed 6. Four of those six are defects rather than capability gaps: in three of them every call, argument, tag and mention matched ground truth exactly and only a string literal differed.

The defect: gold contains text the tools never produce

multi_turn_base_4 — report.txt is a single line. sort() sorts lines, so it is a no-op and returns the input unchanged. Gold's tweet is an alphabetised version that no sequence of calls yields:

report.txt          : "Initial report content Unsorted data More unsorted data"
sort() returns      : "Initial report content Unsorted data More unsorted data"
gold post_tweet     : "Initial report content More unsorted data Unsorted data"

multi_turn_base_8 — the user asks to post "the verbatim results of diff". Gold's tweet content is about research topics and literature reviews, which appears nowhere in this scenario; it looks copy-pasted from another item:

diff() returns  : "- Observation 1: Normal Observation 2: Anomaly detected ...\n+ Observation A: ..."
gold post_tweet : "- Research topic selection+ Data analysis- Literature review+ Data collection..."

multi_turn_base_35 — gold writes a truncated, newline-stripped fragment of what diff() returns:

diff() returns : "- Initialization of the system Error in module Setup complete Initialization successful Error detected\n+ Real Config."
gold echo      : "- Initialization of the system+ Real Config."

multi_turn_base_18 — same shape, newline stripped from the diff() output. Also inconsistent mention formatting: gold here uses mentions=['Jerry'] where multi_turn_base_4's gold uses mentions=['@Julia'].

A separate defect: prompt and answer key disagree

multi_turn_base_10 — turn 3's user says *"get cracking on a new file, call it 'notes.md'"*. Gold is touch(file_name='note.md'). Every other turn of our run matched gold exactly. An agent that follows the user's literal instruction cannot pass.

Free-text exact matching (already reported as #914, still present)

multi_turn_base_178, _180, _187 require an exact free-text string the user never dictates. In 178 the user says "insurance-related issues that haven't been resolved"; gold's ticket description is 'Insurance-related issues that need resolution'. Semantically identical, lexically different, graded as wrong.

This is the same problem #914 raised for multi_turn_base_186 in February 2025, which was closed without a documented response.

These are current, not stale

We diffed the packaged dataset between bfcl_eval 2025.8.6.2 and 2026.3.23:

question (user turns) : 0 of 200 changed
initial_config        : 0 of 200 changed
involved_classes      : 0 of 200 changed
ground_truth          : 0 of 200 changed

multi_turn_base is byte-identical across those seven months, including multi_turn_base_154, which the changelog lists as fixed in #1177 — that fix does not appear in the v3 file.

Why it matters beyond these items

At least 10 of the 27 hardest items are unsound, a floor of 37%, and our detector is blind to two of the failure modes so the real figure is likely higher. Rankings across models survive, since everyone meets the same broken items, but absolute multi_turn accuracy is understated for every model on the leaderboard, and per-item error analysis is unreliable.

Happy to share the screening script. It executes ground truth turn by turn and flags any string literal in a gold call that cannot be derived from the user's words, the initial config, or an earlier tool output. It catches 4 of the 6 problems above, with one false positive in five flags, so it needs a human pass rather than being trusted alone.