`gsm8k_postprocess` breaks numeric answers at thousands separators (`\boxed{$9{,}500}` → `500`, `\boxed{8,000}` → `000`)

Author: laa1991Created Sep 18, 2026Updated Sep 18, 2026

Minimal reproduction:

python
from opencompass.datasets.gsm8k import gsm8k_postprocess

measured (opencompass installed from PyPI, 0.5.4):

input output
\boxed{9500} 9500
\boxed{$9{,}500} 500
\boxed{8,000} 000
\boxed{10,000} 000
The answer is $9{,}500. 500

Impact: any answer containing a comma-separated number is extracted as a different number, and the failure is silent — it is indistinguishable from the model answering wrongly. In our GSM8K run (300 items, deepseek-flash, one leg per harness) this was one of the measurable sources of OpenCompass's losses against the other two harnesses on the same raw answers: the same responses were graded correctly by the other extractors.

Also worth noting: the same postprocessor handles \boxed{...} fine when the number has no separator, so the defect is specifically the , inside a numeric token.

Suggested direction: treat , as a thousands separator when it sits between digits (i.e. strip separators from the numeric token instead of splitting on them), or reuse the numeric normalisation the metric applies.

Note on LaTeX: models often emit \,{,}-style escapes (9{,}500) precisely to avoid a math-mode space, so this input shape is common, not exotic.

Source: open-compass/opencompass