writing-plans: self-review misses task-state/DAG closure in executable plans
What problem does this solve?
writing-plans currently requires each task to produce a self-contained, independently testable deliverable and gives tasks explicit Consumes / Produces interfaces. It also asks for concrete test code, commands, implementation snippets, and frequent commits.
At current main (b36e0829c6d0140e93cfef2ca599b1b07d4a7797), the final Self-Review checks three things: spec coverage, placeholders, and type/name consistency:
What it does not check is whether the repository state at each task boundary can actually satisfy the plan: whether consumers exist before use, deleted inputs have no surviving consumers, promised RED tests are discoverable, workflow/runtime values exist in the phase where they are consumed, and each commit is runnable under its stated verification.
A real planning session produced a plan-only PR where this missing class repeated many times before implementation began:
https://github.com/TeaShaman-cyber/theseus-repo-search-lab/pull/7
The PR changes one Markdown plan file only. Its review history contains 24 top-level Codex findings. Representative task-state/DAG failures:
Matrix rows before their consumers exist: a workflow task enabled LongGaps/PrimeGaps rows before their replay scripts were created in later tasks. https://github.com/TeaShaman-cyber/theseus-repo-search-lab/pull/7#discussion_r4033330349
Delete input before replacing consumer:
producer/zeta23.jsonwas deleted in Task 2 while the legacy workflow that reads it survived until a later task; deleting it would itself trigger that workflow and fail. https://github.com/TeaShaman-cyber/theseus-repo-search-lab/pull/7#discussion_r4033399751Runtime value consumed before transport makes it visible: the plan wrote
SOURCE_ROOTto$GITHUB_ENVand then read it in the same Actions step. https://github.com/TeaShaman-cyber/theseus-repo-search-lab/pull/7#discussion_r4028363646Promised RED not actually in the test graph:
def test_*(self)appeared at module scope, sounittestskipped the intended cases. https://github.com/TeaShaman-cyber/theseus-repo-search-lab/pull/7#discussion_r4033112363 https://github.com/TeaShaman-cyber/theseus-repo-search-lab/pull/7#discussion_r4033399759Workflow replacement not tested while legacy state still existed: the structure assertion was added only after replacement, so it started GREEN instead of proving it detected the old workflow. https://github.com/TeaShaman-cyber/theseus-repo-search-lab/pull/7#discussion_r4033330353
The repo-wide audit grouped these under a recurring invariant: state N must remain valid until state N+1 is independently ready; a task/commit/step may consume only state already produced and verified in an earlier valid phase.
Full public checkpoint: https://github.com/TeaShaman-cyber/theseus-repo-search-lab/issues/10#issuecomment-5709497603
This is consistent with model error being part of the cause, but the recurrence suggests a framework-level review blind spot: the plan format explicitly models task interfaces and independent testability, while the inline self-review never validates those properties.
Suggested change
Not another reviewer subagent. The history in #229 / #1962 / #1130 already documents why the heavier review loop was removed after evals.
A cheap fourth inline self-review lens would be enough to test the hypothesis:
4. Execution-state closure
For every task boundary, simulate the repository state at task start and end:
- every consumed file/interface/runtime value exists before it is used;
- every produced output precedes all consumers;
- deleting or renaming an input leaves no surviving consumer;
- every promised RED command actually discovers the intended test and fails for the intended reason;
- the task's commit is independently runnable under its stated verification;
- cross-step environment/runtime values name the producer, transport, and consumer phase.
This is deliberately narrow. It does not add a new skill, subagent, plan format, or generalized workflow engine.
Why this is distinct from nearby issues
I searched open and closed issues before filing.
- #895 is about how much implementation detail belongs in plans. This issue assumes the current detailed-plan design and asks whether its existing
independently testable/Consumes/Producescontract is actually self-reviewed. - #229 / #1962 / #1511 cover the old plan-document reviewer subagent loop, which was deliberately removed after evals. This proposal does not restore it.
- #1130 proposes an adversarial reviewer. Again, this is only one cheap deterministic review lens.
- #642 discusses loose skill wiring generally. This is a focused observed failure in
writing-plansitself with a public transcript/artifact trail.
Duplicate searches for writing-plans with task-boundary / independently-runnable / consumer-producer / DAG-closure terms did not find a focused issue for this class.
Reproduction / environment
- Upstream skill inspected at exact
obra/superpowersmain commitb36e0829c6d0140e93cfef2ca599b1b07d4a7797(v6.3.0 release commit). - Planning/review harness: ChatGPT project runtime using the bundled Superpowers skills, with GitHub Codex review on the resulting public plan PR.
- Exact bundled-plugin package commit is not exposed by this runtime, so no claim is made that it is byte-identical to upstream
main; the missing self-review lens described above is independently observable in upstreammainat the exact commit linked above. - Public plan/review corpus:
TeaShaman-cyber/theseus-repo-search-lab#7. - Consolidated causal audit:
TeaShaman-cyber/theseus-repo-search-lab#10.
If a before/after eval is required for behavior-shaping skill text, the PR #7 history provides concrete planted/replayed failure shapes rather than a synthetic prompt: missing consumer, premature deletion, same-step env lifetime, undiscovered RED, and post-change-only assertion.
Source: obra/superpowers