`mmlu_*_generative` `get_response` compares the whole first line to the gold letter — models that prefix or explain their answer score exactly 0.000

Minimal reproduction (pure filter + metric, no API calls):

from lm_eval.api.metrics import exact_match_hf_evaluate
from lm_eval.filters.extraction import RegexFilter, WhitespaceFilter
from lm_eval.filters.selection import TakeFirstFilter

chain = [RegexFilter(r"^(.*?)(?=\n|$)"), WhitespaceFilter(), RegexFilter(r"^(.*?)\s*$"), TakeFirstFilter()]
# filter a raw response through `chain`, then:
exact_match_hf_evaluate(predictions=[pred], references=["B"], ignore_case=True, ignore_punctuation=True)

(this mirrors lm_eval/tasks/mmlu/generative/_default_template_yaml → filter_list: get_response)

measured:

Observed at scale: mmlu_*_generative, 10 subjects × 30 items, --apply_chat_template, two models (deepseek-flash, deepseek-v4-pro), num_fewshot 0:

• reported exact_match = 0.000 (0/298) for both models;
• 0% of the 298 responses are empty — the models answered, e.g. B. 4 (letter plus option text) or ANSWER: B;
• judging the same raw responses with a letter-extraction scorer yields ≈0.73 for that run.

Impact: the task silently reports "the model cannot do MMLU" for any model that does not reply with a bare letter, and the run gives no hint that the predictions were unparseable. Related: this is also why models that enter a reasoning preamble before answering score 0 here.

Suggested direction: give the generative MMLU task the letter-extraction fallback used elsewhere in the harness (or have get_response capture a leading ^[A-D]\b group), and/or warn when a large fraction of filtered predictions are not valid choices.