`gsm8k_postprocess` breaks numeric answers at thousands separators (`\boxed{$9{,}500}` → `500`, `\boxed{8,000}` → `000`)
Minimal reproduction:
from opencompass.datasets.gsm8k import gsm8k_postprocessmeasured (opencompass installed from PyPI, 0.5.4):
| input | output |
|---|---|
\boxed{9500} |
9500 ✅ |
\boxed{$9{,}500} |
500 ❌ |
\boxed{8,000} |
000 ❌ |
\boxed{10,000} |
000 ❌ |
The answer is $9{,}500. |
500 ❌ |
Impact: any answer containing a comma-separated number is extracted as a different number, and the failure is silent — it is indistinguishable from the model answering wrongly. In our GSM8K run (300 items, deepseek-flash, one leg per harness) this was one of the measurable sources of OpenCompass's losses against the other two harnesses on the same raw answers: the same responses were graded correctly by the other extractors.
Also worth noting: the same postprocessor handles \boxed{...} fine when the number has no separator,
so the defect is specifically the , inside a numeric token.
Suggested direction: treat , as a thousands separator when it sits between digits
(i.e. strip separators from the numeric token instead of splitting on them), or reuse the numeric
normalisation the metric applies.
Note on LaTeX: models often emit \,{,}-style escapes (9{,}500) precisely to avoid a math-mode
space, so this input shape is common, not exotic.
Source: open-compass/opencompass