mcp-builder: evaluation.py defaults to claude-3-7-sonnet-20250219, and the low-score advice points back at that same model
The evaluation harness in mcp-builder still defaults to claude-3-7-sonnet-20250219, and the troubleshooting section recommends that same snapshot as the upgrade path. Line numbers are against main at the time of writing.
Where
skills/mcp-builder/scripts/evaluation.py
223: model: str = "claude-3-7-sonnet-20250219",
324: parser.add_argument("-m", "--model", default="claude-3-7-sonnet-20250219", help="Claude model to use (default: claude-3-7-sonnet-20250219)")
skills/mcp-builder/reference/evaluation.md
488: -m, --model Claude model to use (default: claude-3-7-sonnet-20250219)
599:- Use a more capable model (e.g., `claude-3-7-sonnet-20250219`)
Why it matters
The harness exists to tell an MCP author whether their tool descriptions and schemas are good enough for an agent to use. Scoring them against a model three generations old measures the wrong thing: descriptions that a current model reads fine can fail on 3.7, and descriptions that 3.7 needs spelled out are over-specified for the models people actually deploy against. Anyone who runs evaluation.py without -m gets that older baseline silently, because nothing in the output says which model produced the score.
Line 599 compounds it. It sits under troubleshooting for low scores and offers the pinned default as the more capable alternative, so a reader following the advice moves nowhere. When that snapshot is eventually retired, the same default turns into a hard failure for every invocation that omits -m.
Suggested fix
- Point both defaults in
evaluation.pyat a current Sonnet-class snapshot, and update the two lines inreference/evaluation.mdto match. - Reword the troubleshooting bullet so it names a step up from the default rather than the default itself, or drop the model suggestion and keep the description-quality advice around it.
- Consider printing the model id in the run header, so a score is always attributable to a baseline.
Happy to send a PR if the direction looks right.
Source: anthropics/skills