Several reported scores are incompatible with the released test split sizes
Hi, thanks for releasing SkillOpt. I found a possible mismatch between the paper results and the currently released dataset splits.
The released code computes the hard score as the mean of per-example scores. For LiveMath, the evaluator is binary exact match, and the released manifest contains 124 test items:
data/livemathematicianbench_id_split/split_manifest.jsonskillopt/envs/livemathematicianbench/evaluator.pyskillopt/utils/scoring.py
However, several LiveMath results in Table 1 cannot be obtained from 124 binary examples after rounding to one decimal:
| Score | Implied result |
|---|---|
| 22.4 | 28/125 |
| 27.2 | 34/125 |
| 52.0 | 65/125 |
| 31.2 | 39/125 |
| 29.6 | 37/125 |
| 41.6 | 52/125 |
These values all fall exactly on a 125-item grid, suggesting that some experiments may have used a 125-item test split rather than the released 124-item split.
I found a similar issue for DocVQA. Its released test split contains 374 items, but the reported score 87.6 is not attainable in a single binary run:
- 327/374 = 87.4%
- 328/374 = 87.7%
There are also some denominator-incompatible cells for OfficeQA and SpreadsheetBench under the released hard-score implementation.
Could you please clarify:
- Did the paper use an earlier version of the dataset splits?
- Were the reported values averaged across multiple runs?
- Were any failed or invalid examples removed before aggregation?
- Could the exact paper splits and aggregation protocol be released?
Publishing per-example results or the exact result manifests would make the reported table much easier to reproduce.
Related: #108
Source: microsoft/SkillOpt