#7959 replaced the frozen conformance corpus with artifacts generated by the library under test — the standard lost its source of truth

System information

Summary

#7959 solved a real problem (~37 MB of node-test artifacts in the wheel, #5925). But the way it was solved changed something much bigger than package size: what the ONNX node tests are checked against.

Before the PR, the expected inputs/outputs and model files under onnx/backend/test/data/node/ were frozen, versioned files. docs/OnnxBackendTest.md called them "the source of truth" — a sentence that was removed in the same PR. When a backend said "we pass the ONNX backend tests", that claim referred to a fixed, reviewable set of files tracked in git.

After the PR, both the test models and the expected outputs are generated at runtime, by the installed onnx package itself. What a backend is tested against now depends on which onnx and numpy versions happen to be installed. For a project whose purpose is to be an interchange standard, this is the wrong trade-off. My ask is concrete: revert #7959 quickly, then reopen the question in a proper design discussion. The wheel-size win can be recovered afterwards without giving up the corpus — options are listed at the end.

The node tests are part of the standard, not just test code

This is the core of this issue: the node test data is not internal test infrastructure of this repository. Together with docs/Operators.md, the frozen input/output pairs were the only machine-checkable definition of operator semantics that ONNX has. The prose in Operators.md describes what an operator does; the golden files defined what a conforming implementation must produce, down to dtype and tolerance. That makes them part of the standard itself. Changes to them deserve the same care and visibility as changes to the operator schemas or the IR spec.

Treating them as throwaway test files, regenerated at will by whatever library version is installed, turns the relationship upside down: the implementation now defines the standard, instead of being checked against it.

Khronos, W3C, and Unicode all maintain frozen, implementation-independent conformance suites — because a standard that cannot be verified independently is not a standard, it is a suggestion. ONNX spent almost a decade building exactly such a corpus, operator by operator, release by release. #7959 deleted it in a single unreviewable commit. That is not cleanup; that is a standard regressing into a library — a regression in what ONNX is.

A conformance corpus is allowed to be big. 37 MB — or 37 GB — of frozen, reviewed operator semantics is not bloat; it is the standard itself in its most precise form, and it is worth every byte. The mistake was never the size of the corpus, it was shipping it inside a pip package. #7959 solved a packaging problem by deleting the standard.

Why this matters for a standard

1. A conformance claim must be reproducible. "Runtime X at commit Y passes the ONNX node tests for opset Z" used to point to one fixed set of bytes. Now the same sentence can mean different tests, depending on the environment that generated them.
2. Spec changes must be reviewable. A change to an operator's behavior used to show up as a visible diff in the golden .pb files — reviewable and bisectable with git log. That history ended with the merge. Changes to expected values are now invisible side effects of code changes.
3. The test data must be independent of the implementation. A conformance artifact should not be produced by the implementation it is meant to check. That independence is now lost in two ways, shown below.

Concrete regressions

1. Circular test generation for several operators

At least 12 node test cases (Resize, Attention, FlexAttention, LinearAttention, MaxPool, AveragePool, LpPool, RotaryEmbedding, RMSNormalization, AffineGrid, CausalConvWithState, Cast) do not compute their expected outputs with standalone numpy code — they import the reference implementation under test, e.g.:

onnx/backend/test/case/node/resize.py → from onnx.reference.ops.op_resize import _interpolate_nd, ...

This is the same function the reference evaluator calls in its own _run. While the goldens were checked in, this circular dependency was frozen: the .pb files came from an earlier state of the code, so a bug in _interpolate_nd produced a visible diff and a failing test. Now expectation and actual value come from the same code in the same process — for these operators, the reference-evaluator backend test can no longer fail.

2. Model bytes are no longer stable

_make_test_model_gen_version (onnx/backend/test/case/node/__init__.py) takes the model's ir_version from onnx.helper.VERSION_TABLE[-1] and the opset version from onnx.defs.get_schema(...).since_version — both are properties of the installed build. The removed CI step ("Test backend test data" in main.yml) used to check that regenerated model.onnx files were identical to the checked-in ones (git diff --exit-code, only input_*.pb/output_*.pb excluded). That guard is gone and nothing replaces it.

3. Expected outputs depend on the installed numpy

The removed CI step carried this comment: "the test data should be generated with numpy>=2.0. numpy 1.x and numpy 2.0 produce slightly different numerical values." The project itself documented that expected values drift across numpy versions — which was exactly the reason to freeze them. (The input data is stable thanks to np.random.seed(0) and NEP 19, but the expected-output computation — accumulation order, casts, float32 rounding — is not.)

4. Non-Python consumers lost the corpus with no migration path

The PR discussion considered downstream users of the Python Runner class. But an important group of consumers read onnx/backend/test/data/node/ as files: non-Python runtimes, code generators, and independent conformance harnesses. For them there is no replacement — cmd_tools.py generate-data no longer emits node data, and the --clean/--op_type/--diff options were removed.

A process concern: this change was not effectively reviewable

I raise this not to assign blame, but because it explains how a spec-level change could land without discussion:

• The diff touched ~10,000 files. The automated reviewer declined to review it five times ("exceeds the maximum number of files (300)"). No human can meaningfully review a diff of that size either; the three approvals came without any recorded discussion of the semantic consequences.
• The PR description framed the change purely as a packaging/size fix ("removes all the node-test artifacts but recreates them on-the-fly"). It did not mention: that the deleted files were documented as "the source of truth", that this wording was removed from docs/OnnxBackendTest.md in the same diff, that some expected outputs are now computed by the implementation under test, and that model bytes now depend on the installed build. The redefinition of what is normative happened silently, buried in the file deletions.
• By the project's own norms, spec-affecting changes are supposed to be conservative and deliberate. A change that dissolves the conformance corpus should have been an explicit design decision (issue or discussion thread), not a side effect of a size optimization.

For future changes of this scope, splitting the mechanical deletion from the semantic change would keep the reviewable part reviewable.

Requested action: revert quickly, then discuss properly

My concrete ask is that #7959 be reverted quickly — before the next release ships without the corpus — and that the underlying question be reopened as an explicit, visible design discussion (issue or discussion thread, with the relevant SIGs involved) rather than decided as a side effect of a size optimization. Every day of delay makes the revert more expensive: new operator work is already landing on top of the new mechanism, and once a release goes out without the frozen corpus, downstream conformance tooling will start to diverge.

Reverting is not a judgment that the wheel-size problem was invalid — it is the recognition that a change to what is normative must not come before the discussion about it.

What the discussion should then consider

The wheel-size problem and the corpus question are separable: the frozen goldens can stay in the repository (or in a separate versioned artifact) while being excluded from the wheel — e.g. via wheel.exclude in pyproject.toml — with the CI regeneration check restored. Independently, the test cases listed above should stop importing from onnx.reference.ops.

I'm happy to contribute the revert PR and follow-up PRs for whichever direction the maintainers prefer.

References

• #7959 (merged as a8f77b5b), #5925, #5970
• Removed wording in docs/OnnxBackendTest.md ("exported to protobuf files ... as the source of truth")
• Removed CI step "Test backend test data" in .github/workflows/main.yml