#157·SkillOpt

LiveMath scores imply 125 test items, but the released split has 124

Author: KhanColdCreated Jul 19, 2026Updated Jul 26, 2026

Hi SkillOpt team,

I found a possible mismatch between the LiveMath split and the Qwen results in Table 1.

The released manifest contains 35/18/124 train/val/test items. However, several reported scores exactly match a denominator of 125:

  • Qwen3.5-4B: No skill 22.4 = 28/125, Trace2Skill 27.2 = 34/125, SkillOpt 52.0 = 65/125
  • Qwen3.6-35B-A3B: No skill 31.2 = 39/125, Trace2Skill 29.6 = 37/125, SkillOpt 41.6 = 52/125

With 124 binary exact-match examples, neither 52.0% nor 41.6% is attainable after rounding.

For 177 total items, a 2:1:7 split can be either:

  • 35/17/125 if train and val are floored and the remainder goes to test;
  • 35/18/124 with the currently released largest-remainder split.

Could you confirm whether Table 1 used the former 35/17/125 split, and release the original LiveMath manifest or per-item results?

Related: #108.

Thanks!