Irrelevance detection credits a tool call that failed to parse as a correct abstention

In BFCL v4 the irrelevance check credits a tool call that failed to parse as if it were a correct abstention. Falcon3-1B-Instruct (FC) is scored correct on 229 of 240 non-live irrelevance entries, and 228 of those 229 stored outputs contain a <tool_call> with a function name and arguments. Nothing was rerun to see this: the numbers come from the published score files in HuanzhiMao/BFCL-Result.

The branch is the one in the irrelevance path of eval_runner.py:

success = not contain_func_call

and contain_func_call is set to False whenever decode_ast raises. So "abstained" and "tried to call a function and got the syntax wrong" arrive at that line as the same event, and both are credited.

Reproduction (snapshot 2025-12-16 of HuanzhiMao/BFCL-Result)

Line 1 of a *_score.json is the category summary; every later line is an entry marked wrong. So the credited-correct set is the entries absent from lines 2+. No judgement is involved anywhere in this.

Falcon3-1B-Instruct (FC), rank 106:

irrelevance        229 of 240 credited correct; 228 of those 229 (99.6%) contain
                   "<tool_call", e.g.
                   '<tool_call>\n<tool_call name="math.solve" arguments={"a": 1,
                   "b": 2, "c": 3}/>\n</tool_call>'
live_irrelevance   700 of 884 credited correct; 588 of those (84.0%) contain
                   "<tool_call"

The 11 non-live entries it is marked WRONG on are the ones whose syntax happened to come out valid. Within the category the metric runs backwards.

Llama-3.1-Nemotron-Ultra-253B-v1 (FC), rank 108, is the limit case. Every stored result is [], and one run records the same failure three different ways:

BFCL_v4_simple_python      accuracy 0.0    0 of 400   error: ["Invalid syntax.
                           Failed to decode AST. expected string or bytes-like
                           object, got 'list'"]
BFCL_v4_live_relevance     accuracy 0.0    0 of  16   error: ["Invalid syntax.
                           Failed to decode AST when it should have. ..."]
BFCL_v4_irrelevance        accuracy 1.0  240 of 240   (silent)
BFCL_v4_live_irrelevance   accuracy 1.0  884 of 884   (silent)

The two silent ones are the ones that reach the score. 10% x 100.00 is its entire published Overall Accuracy of 10.00, with every other block at zero.

What this does not touch

The top 10 draw 10.9% to 13.4% of their overall from the hallucination block, which is roughly its nominal weight, and their relevance runs 62.50 to 93.75. Competitive rankings are unaffected and I am not claiming otherwise.

The signature -- relevance 0.00 together with a near-perfect irrelevance -- appears in exactly 3 of the 109 published rows: Ministral-8B (105), Falcon3-1B (106), Nemotron (108). What it does distort is the bottom: all three outrank Gemma-3-1b-it (109, 7.17), which actually calls functions correctly about a fifth of the time.

Ministral-8B is the weakest of the three and I will say so: only 84 of its 240 outputs are empty and many of the rest are genuine prose refusals that deserve to score. It is in the list because of its 0.00 relevance, not because I have shown it is wrong.

A fix, and the version of it I could not make work

The obvious fix is to make a decode exception a distinct third outcome and count it as incorrect for irrelevance, since a malformed call is still an attempt to call. I tried to break that before filing it and I did: a model that genuinely abstains in prose that happens to trip the decoder would then be marked wrong, and scanning the raw text for <tool_call only works because I already know Falcon's format.

The version that survived my own attempt to break it needs no format knowledge at all. A model whose irrelevance is at or near 1.000 while its relevance is 0.000 is reporting a decoder that never succeeds, not a model with perfect judgement -- the two categories disagree in a way no real model can. Flagging that pair, rather than trusting either number, catches all three rows above and would have caught Nemotron from the score files alone.

Separately, one word on the leaderboard page

leaderboard.html currently says: "Overall Accuracy is the unweighted average of all the sub-categories." The code weights them [10, 10, 10, 30, 40]. Reconstructing the published table with the weights matches all 109 rows to within 0.007 pp; unweighted misses all 109, by 15.07 pp on average and by 24.04 pp at worst (Amazon-Nova-2-Lite, published 27.10 against 51.14 unweighted).

I think I see how it survived: the V4 blog says "Within-Category Calculation: Unweighted Average", which is true inside a category and not across them. The "details on score composition" link next to that sentence points at the V3 blog, which opens by saying its composition is now outdated. The correct formula is in the V4 blog, which that sentence does not link to.

Disclosure

I emailed this to [email protected] on 26 August 2026 before filing here, and I am filing here as well because the README names an issue as the first channel. The write-up with the scripts and the controls is at https://greenthatlies.com/bfcl.html . In fairness I should say that page went public before the report went out rather than pretend otherwise -- if anything on it is wrong I will correct or remove it the same day you tell me.

I am not asking for anything for this. The README invites criticism and the data being open is the only reason I could check any of it.

Igor Voytyuk, [email protected]