LiveMath scores imply 125 test items, but the released split has 124
Author: KhanColdCreated Jul 19, 2026Updated Jul 26, 2026
Hi SkillOpt team,
I found a possible mismatch between the LiveMath split and the Qwen results in Table 1.
The released manifest contains 35/18/124 train/val/test items. However, several reported scores exactly match a denominator of 125:
- Qwen3.5-4B: No skill
22.4 = 28/125, Trace2Skill27.2 = 34/125, SkillOpt52.0 = 65/125 - Qwen3.6-35B-A3B: No skill
31.2 = 39/125, Trace2Skill29.6 = 37/125, SkillOpt41.6 = 52/125
With 124 binary exact-match examples, neither 52.0% nor 41.6% is attainable after rounding.
For 177 total items, a 2:1:7 split can be either:
35/17/125if train and val are floored and the remainder goes to test;35/18/124with the currently released largest-remainder split.
Could you confirm whether Table 1 used the former 35/17/125 split, and release the original LiveMath manifest or per-item results?
Related: #108.
Thanks!
Source: microsoft/SkillOpt