#8276·autogen

gaia_question_scorer vacuously scores a degenerate/empty agent answer as correct when the ground truth normalizes to empty

Author: shaurya416Created Sep 23, 2026Updated Sep 28, 2026

python/packages/agbench/benchmarks/GAIA/Scripts/custom_tabulate.py:121-122

python
    # if gt is a str
    else:
        return normalize_str(model_answer) == normalize_str(ground_truth)

normalize_str (same file, lines 57-77) strips all whitespace and punctuation and lowercases. It has no check that the normalized ground truth is non-empty before comparing. gaia_question_scorer is the function actually wired into scoring: scorer() calls it directly at line 157.

python
        # Return true if they are equal after normalization
        # return in_house_question_scorer(final_answer, expected_answer)
        return gaia_question_scorer(final_answer, expected_answer)

expected_answer reaching this call is expected_answer_file content after .strip() (line 135); final_answer is the text captured by re.search(r"FINAL ANSWER:(.*?)\n", ...) after .strip() (line 149) — both can legitimately be "" (an empty expected_answer.txt, or an agent that emits FINAL ANSWER: with nothing after it).

What happens (reproduced on the real module)

In gaia_question_scorer's string-comparison branch (custom_tabulate.py:121-122, wired in via scorer() at line 157), a ground truth that normalizes to the empty string is never rejected before the equality check, so a degenerate/blank agent answer (empty, whitespace-only, or punctuation-only) is scored as a correct match whenever the reference answer is empty or normalizes to empty — e.g. any GAIA test-split task, whose reference answers are withheld/placeholder values, or a hand-authored task with a blank expected_answer.txt. This does not make the scorer accept arbitrary wrong answers (a real, non-empty answer still correctly fails against such a ground truth); it specifically converts "the agent failed to produce a real answer" into a recorded pass. The file's own commented-out in_house_question_scorer (lines 27-33) already carries the right guard (n_gt != ""); the minimal fix is porting that same guard into gaia_question_scorer's final else-branch: compute n_gt = normalize_str(ground_truth) once and return n_gt != "" and normalize_str(model_answer) == n_gt.

Happy to open the PR.