feat: public validation benchmark — reproducible probe suite (announced in #900)
Author: czlonkowskiCreated Jul 2, 2026Updated Jul 11, 2026
Labelsdocumentationenhancementhelp wantedroadmap
Context
Announced in Discussion #900 as the first near-term deliverable: a public, reproducible validation benchmark so anyone can verify the validation claims in the July 2026 competitive analysis themselves, against both the official n8n MCP server and n8n-mcp.
The seed already exists: the five broken-workflow probes from the analysis (§6.1, §10) — unknown node type, impossible typeVersion, missing required field, duplicate node names, AI Agent without a model — all of which the official validator currently accepts as valid:true while n8n-mcp rejects all five.
Scope (v1 — validation honesty only)
- Probe suite: the five seed probes plus an expanded matrix of invalid configurations — community-node configs, expression syntax errors, connection/graph errors (invalid input index, orphaned nodes), per-node required-field cases across common nodes.
- A small harness that runs the suite as an MCP client against (a) an official n8n MCP endpoint and (b) an n8n-mcp endpoint, and emits a scored report (caught / warned / silently passed, per probe).
- Edit-cost measurement shapes from the analysis (insert-node-mid-flow; one-line large-field edit) as a secondary, optional module.
- Methodology doc: what each probe tests, why it matters for agent loops (the
valid:truestop-signal problem), how to run it against your own instance. - Published results for the current n8n release, with the intent to re-run per n8n release.
Design decisions to settle
- Location: standalone repo (neutral optics, own README) vs
scripts/benchmark/here. Leaning standalone. - Result format: machine-readable (JSON) + rendered markdown table.
Out of scope (v2+)
End-to-end task-success benchmarking (build/edit/debug scenarios scored by outcome) — larger effort, separate issue when v1 ships.
Source: czlonkowski/n8n-mcp