lm-evaluation-harness · Issues· 994 open

`mmlu_*_generative` `get_response` compares the whole first line to the gold letter — models that prefix or explain their answer score exactly 0.000

Python API guide passes unsupported num_fewshot to evaluate()

Metadata n-shot override is ignored in collected results

Issue with `bigbench_gender_inclusive_sentences_german_multiple_choice`

An apostrophe in a CLI value silently swallows every following key=value argument

Acc vs acc_norm

longbench2_multi group omits longbench2_legal_multi, so it disagrees with the longbench2 tag roll-up

Shared tasks are double-counted in nested group aggregation

Finite retry exhaustion returns None instead of raising

Add New Lambada Translations

Group rows keep a 0.0 stderr when every subtask is entirely on the boundary, so the row with the most evidence carries no bound

GGUF backend's echo=true handling doesn't work against current llama-server — every loglikelihood request fails

MELA group omits Icelandic and lists Arabic twice

At 0 (or 100%) accuracy the reported stderr is exactly 0.0 at any N, so null results claim infinite precision; a boundary-aware interval or flag would fix it

CLI key-value parser preserves whitespace in argument names