Deterministic fact-consistency check for document skills — from standalone tool to a required post-step of /document-release

Problem

gstack's own docs already name this failure class. docs/howto-document-a-shipped-feature.md lists, as a known failure scenario:

A generated reference doc has hallucinated API signatures.

And document-release/sections/release-body.md records a real incident:

A real incident occurred where an agent replaced existing CHANGELOG entries when it should have

The defenses against this today are all prose: "NEVER do: Overwrite, replace, or regenerate CHANGELOG entries", "never fabricate", and Step 6's cross-doc consistency pass, which is the model reading its own output and comparing. Prose guards are advisory; gstack-verify-gate exists precisely because gstack learned that lesson for tests — a declared verify command blocks the turn, while "remember to run tests" does not. Documentation facts have no equivalent structural gate.

Three source-level observations make this concrete:

1. The model is trusted to write hard facts without stopping. /document-release explicitly says "Never stop for: … Updating paths, counts, version numbers … Cross-doc factual inconsistencies (e.g., version number mismatch)". That is the right call for legitimate syncs — but it means version numbers, paths, counts, and dates land in docs on the model's word alone, with no deterministic check behind them.

2. On the /ship path there is no human interception point at all. ship/sections/pr-body.md dispatches /document-release as a spawned subagent (GSTACK_SESSION_KIND=spawned) so that "document-release's interactive gates auto-choose their recommended options", and adds "Do not block /ship on subagent failure or slowness". So in the highest-volume path, hard facts flow model → commit → PR with zero eyes on them. The CHANGELOG incident above happened in exactly this kind of flow.

3. Count drift is live in the repo right now. The README headline reads "Twenty-three specialists and eight power tools", while the command tables in the same file now list considerably more than both. Nothing is wrong with the prose — it just drifted as the codebase grew, because nothing checks it. This is the same failure mode, already happened, in gstack's own flagship doc.

The failure mode is bounded and mechanical — semver strings, port numbers, dates, command names, API paths, counts — which is exactly the class of thing a deterministic tool can check cheaper and more reliably than a model can be trusted to.

Proposal

A two-channel deterministic checker, adopted in two phases.

Channel A — entity extraction against ground truth

Extract checkable entities from generated/updated docs and verify them against ground-truth sources:

Entity class	Examples	Tolerance
Versions / IDs	semver strings, PR numbers, flags	exact match or it's wrong
Dates	release dates, ship dates	±1 day → minor; >3 days → critical
Numbers / counts	"23 specialists", "8 tools", byte counts	relative delta >10% → critical
Paths & commands	CLI names, flags, file paths, API routes	must exist in the actual CLI surface / tree

Ground truth is tiered, and the tier decides what the checker is allowed to do:

• Tier A — authoritative sources (source code, git diff, actual tool output such as --help): may drive automatic correction of the doc.
• Tier B — derived sources (other docs): may only produce a signal for the user to adjudicate. Never auto-correct from Tier B, or one wrong doc propagates its error into every other doc — the correction path becomes the contamination path.

Dispositions: minor mismatches auto-corrected (with the correction recorded), critical mismatches block, and — the hallucination case — an entity that appears in a doc but has no Tier A source at all is flagged as a suspected fabrication and always blocks.

Channel B — structural command validation

gstack already does this for its own SKILL.md files: validateSkill / the skill-parser tests extract declared commands, permissions, and references from skill frontmatter and check them against actual registries. The proposal is to generalize that pattern to user-project docs: pull shell commands, flags, and code blocks out of generated docs and verify them against the real CLI surface (--help output, command registry, package.json scripts). A doc that says my-cli --force-deploy is checkable; whether --force-deploy exists is a fact, not a judgment.

Two-phase adoption path

Phase 1 — standalone tool: bin/gstack-fact-check. Modeled on gstack-issue-guard: a focused, testable binary with stable semantics:

• exit 0 — clean
• exit 1 — minor issues found and auto-corrected (diff printed)
• exit 2 — critical mismatches / suspected fabrications; blocks

Runs get recorded in the evidence ledger (gstack-evidence run --label fact-check) so fact-check results ride the same trust chain as tests. This phase is useful on its own and carries zero risk to existing flows.

Phase 2 — required post-step of /document-release. Append a STOP-gate-style delegation to the skill definition — the same pattern the skill already uses for its own sections ("STOP. Before auditing each doc file … Read sections/release-body.md and execute it"), and the same pattern /ship uses at Step 18. No internal logic changes: the workflow simply does not complete until gstack-fact-check exits 0 or 1. This mirrors gstack's own line between structural gates (verify-gate blocks the turn) and advisory prose ("never fabricate") — the point is to move doc facts from the second category into the first.

On the spawned /ship path, a critical exit becomes the one structural interception that exists today between a hallucinated fact and the PR. What a human override means in that context (proceed-with-recorded-exception vs. hard stop) is left as an open question for maintainers — the conservative default is hard stop.

Also applies to

• /document-generate — same class of output, fresh generation rather than sync
• /retro — weekly statistics (counts of PRs, review pass rates) are exactly the numeric facts that drift silently

Related prior art in this repo

• #973 (open) established the failure class for the spec-review skills: design docs score 8/10 while containing wrong file paths, wrong math, and fabricated CLI flags, because review subagents see only document text. This proposal is the document-release sibling — a different skill, a different failure surface (cross-doc sync facts: versions, counts, paths, dates, rather than design-doc premises), and a structural gate as the deliverable rather than a post-approval recovery pattern.
• #1644 (open, fix in flight via #1738) is the same class inside /office-hours: a design doc reached APPROVED on narrative quality while its premises were false against the codebase. Same lesson, independently reported: review passes on prose quality; the facts under the prose go unchecked.

Both issues stay valuable as evidence that the class is real and recognized; this proposal covers the third, highest-volume instance — release-path doc sync — where today there is no interception point at all.

How this fits gstack

• Boil the Ocean: "AI makes completeness cheap." A deterministic pass over a docs diff is seconds of compute for a class of error models demonstrably make. The complete thing — docs whose facts are actually checked — is the goal; the only thing out of scope is semantic truth, which stays with the model and the user.
• AI models recommend. Users decide.: the tiering keeps judgment with the user — Tier A mechanical corrections are applied (they're not judgments), Tier B signals and suspected fabrications are surfaced for the human. Nothing silent, nothing judgmental automated.
• Precedents in-repo: validateSkill (Channel B is its generalization), gstack-issue-guard (the tool shape), gstack-evidence (the ledger), gstack-verify-gate (the structural-gate philosophy), and gstack-wtree (content-bound identity — fact-check results should bind to the doc content hash, so a changed doc is re-checked).
• Test pyramid fit: the checker is Tier 1 — static, free, <2s on a docs diff — with a small fixture set of known-drift cases (the CHANGELOG incident makes a good regression fixture).

Non-goals / open questions

• No semantic fact-checking ("is this claim true") — out of scope, stays with review.
• Does not replace Step 6's narrative cross-doc pass; it runs before it so the model starts from verified facts.
• Open: override semantics on the spawned path; whether --budget-style ceilings make sense for "max tolerable minor corrections per run".

Suggested TODOS.md entry

What: Build bin/gstack-fact-check (entity extraction vs. tiered ground truth + structural command validation) and wire it as a required post-step of /document-release, /document-generate, and /retro statistics.

Why: gstack's own docs record a hallucinated-API-signature failure and a real CHANGELOG-clobbering incident, and the README's own headline counts have drifted; every current defense is advisory prose, while the /ship doc-sync path is spawned and auto-choosing, so no human sees doc facts before the PR.

Context: Two-channel design: regex/structural entity checks with Tier A (authoritative — may auto-correct) vs. Tier B (derived — signal only) ground-truth tiering; two-phase rollout (standalone gstack-issue-guard-style tool first, STOP-gate delegation into the skills second, same pattern as verify-gate vs. prose). Channel B generalizes the existing validateSkill/skill-parser pattern to user-project docs. Exit semantics 0/1/2; results recorded via gstack-evidence.

Effort: M Priority: P2 Depends on: None