[meta] No CI runs the harness test suites - the 2,461 passed / 100% pass rate claim is unverified

[meta] No CI runs the harness test suites — the "2,461 passed / 100% pass rate" claim is unverified

Severity: process — this is the root cause behind the other four reports

Summary

The README advertises per-harness test counts with ✅ marks and a summary line:

TOTAL        2,461 passed  ✅   100% pass rate

No CI job reproduces any of it. .github/workflows/ contains six workflows — check-codex-skill.yml, check-root-skills.yml, deploy-pages.yml, pr-labeler-tests.yml, pr-labeler.yml, publish-cli-hub.yml — and none references pytest, test_core, or test_full_e2e:

$ grep -rn "pytest\|test_core\|test_full_e2e" .github/workflows/
# (no matches)

So the numbers are a snapshot from a maintainer's machine, pasted into the README, with nothing keeping them true as the repo and the upstream applications move.

Every runner that does exist is ubuntu-latest on Python 3.10 / 3.12.

Why it matters

Testing five harnesses on Windows 11 / Python 3.14 with current app versions found:

Harness	README	Measured here
Inkscape	✅ 202	207 passed, clean
Audacity	✅ 161	165 passed, clean
Blender	✅ 208	211 passed / 17 failed
GIMP	✅ 107	hung indefinitely with GIMP 3 installed
ComfyUI	70 passed	69 passed / 1 failed

The suite sizes match closely, so these are real suites that really did pass once. What has drifted is the pass rate, and nothing was watching.

Notably, none of these are exotic. Two are structurally impossible to see on Linux (\U path escapes), one is version drift in a dependency the registry pins loosely (requires: gimp now installs GIMP 3), and one — the ComfyUI --json break — is fully platform-independent and should be failing in any environment.

Suggested minimum

A single workflow that runs the unit suites would have caught the ComfyUI regression on the PR that introduced it:

strategy:
  matrix:
    os: [ubuntu-latest, windows-latest]
    python: ['3.10', '3.12']

Even Linux-only would help; adding windows-latest would have caught reports #1 and #4 before release. E2E tests needing the real application could stay gated behind an opt-in flag, as several harnesses already support via CLI_ANYTHING_FORCE_INSTALLED.

A second, cheaper win: derive the README table from a generated report rather than maintaining it by hand, so a count can never claim more than the last run proved.

Related reports

• #343 blender — Windows path escape breaks every render (open since 2026-06-05)
• #400 gimp — Script-Fu targets the GIMP 2.10 PDB
• #401 comfyui — --json contract broken
• #402 blender — EEVEE name drift and symlink-dependent previews

#343 is the clearest illustration: a one-character bug that breaks every render on Windows, reported independently at least twice, open for seven weeks. A windows-latest job would have caught it before release.