#3408·ECC

ci(catalog): README tree skills unchecked, second tree block never read, and AGENTS.md undercounts pass

Author: YuhaoLin2005Created Oct 2, 2026Updated Oct 2, 2026

Summary

scripts/ci/catalog.js runs in CI as a named gate, but for README.md it validates the project-tree agents count and ignores the project-tree skills count, and it only ever looks at the first of the two project-tree blocks in that file. Separately, the AGENTS.md skills check is mode: 'minimum', so an undercount there passes silently.

All of this is on main at ef648e0, verified by execution rather than by reading the source. Real catalog on that commit: 68 agents, 94 commands, 293 skills.

Correction to an earlier revision of this issue

My first version claimed there is no skills/ project-tree pattern anywhere in catalog.js, and cited that the coverage hole was repo-wide. That was wrong, and I have not edited the framing away quietly — it is stated here because anyone who read the earlier revision would otherwise act on a false premise.

It was wrong because I searched for README.md's wording (reusable workflows) and generalised from the absence of that string. AGENTS.md uses different wording, and the pattern exists. Setting AGENTS.md's skills project-structure line to 999 fails the gate today:

- AGENTS.md project structure (skills): skills documented >= 999, actual 293

The hole is README-specific, not repo-wide. The line numbers in the earlier revision were also wrong; they are corrected throughout below.

What the gate sees vs. what it should see

Every row below is the result of mutating one line on a clean main checkout at ef648e0 and re-running the real gate, then reverting and re-verifying the file hash.

# Mutation Gate result Meaning
1 README.md tree skills line 293 -> 284 exit 0 unchecked
2 README.md:851 agents 67 -> 9999 exit 0 unchecked
3 README.md quick-start 293 -> 999 exit 1 checked
4 AGENTS.md tree skills 293 -> 999 exit 1 checked, overcount only
5 AGENTS.md tree skills 293 -> 100 exit 0 unchecked undercount
— unmodified main exit 0 green baseline

Row 3 matters: the gate is alive, this is a coverage gap, not a dead check.

Root cause

One: the README project tree has an agents pattern and no skills counterpart. scripts/ci/catalog.js:106 matches the README project-tree agents line and :414 repairs it. There is no skills/ counterpart for README.md, so a stale skills count in that block is invisible to both the check and to --write.

For contrast, the other two tree surfaces are covered: catalog.js:250 checks the AGENTS.md skills project-structure line, :329 the docs/zh-CN/AGENTS.md one, with repairs at :457 and :549. README is the odd one out.

Two: String.prototype.match with a non-global regex returns the first match only. README.md has two project-tree agents lines:

L827:  |-- agents/           # 68 specialized subagents for delegation   <- the only one checked
L851:  |-- agents/           # 67 specialized subagents for delegation   <- stale by one

The catalog really is 68. Line 851 has been wrong on main and the gate has been green throughout. Setting it to 9999 still exits 0, so this is not theoretical. That line is a live one-line documentation defect and does not need to wait on a catalog.js refactor.

Three: the AGENTS.md skills check is mode: 'minimum', so it only fails closed on overcounts. AGENTS.md's project-structure skills expectation is a minimum, and 293 is the actual count, so advertising 100 passes. An undercount is arguably the more misleading direction of the two, because it tells a reader the repo contains less than it does.

The gate

.github/workflows/ci.yml:226-227:

yaml
- name: Validate catalog counts
  run: node scripts/ci/catalog.js --text

Reproduce

On a clean main checkout at ef648e0:

bash
# 1. the unchecked README skills tree line
sed -i 's/# 293 reusable workflows/# 284 reusable workflows/' README.md
node scripts/ci/catalog.js --text; echo "exit=$?"
#   - skills: 293
#   Documentation counts match the repository catalog.
#   exit=0

# 2. the repair tool cannot reach it either
node scripts/ci/catalog.js --write --text
grep -n 'reusable workflows' README.md
#   |-- skills/           # 284 reusable workflows loaded on demand   <- unchanged

# 3. the second tree block
sed -i '851s/# 67 specialized/# 9999 specialized/' README.md
node scripts/ci/catalog.js --text; echo "exit=$?"
#   exit=0

# 4. negative control
sed -i 's/Access to 68 agents, 293 skills/Access to 68 agents, 999 skills/' README.md
node scripts/ci/catalog.js --text; echo "exit=$?"
#   README.md quick-start summary: skills documented = 999, actual 293
#   exit=1

Node v24.16.0 on Windows, but the gate is plain Node with no platform-specific paths; the same mutations behave identically on Linux CI.

Why it matters

README.md and AGENTS.md are the standing instruction layer. An agent reading AGENTS.md to decide what this repo contains is told a smaller catalog than exists; a human skimming README.md sees two project trees in one file that disagree about the agent count. Neither is caught by the gate written specifically to catch this.

Suggested fix

  1. Add a skills/ counterpart to catalog.js:106 and :414 for the README project tree. The AGENTS.md pattern at :250 is the shape to copy.
  2. Use matchAll for the tree patterns, or assert every tree line agrees, so a second block cannot hide behind the first.
  3. Consider making the AGENTS.md skills check mode: 'exact' so undercounts fail too — though that may be deliberate if the field is meant as "at least N"; if so, a comment saying so would help.
  4. README.md:851 says 67 agents and should say 68. Independent of the above, and worth doing now.

Items 1 and 2 are independent of each other and each is small. I have not touched catalog.js in #2496 — it is shared CI, a new fail-closed assertion can redden branches already in flight, and the sequencing is your call, not mine.

Relation to #3017

#3017 found the same weakness from the other direction — files the checker never opens — and was fixed by #3060, merged 2026-09-29, which added SOUL.md and .gemini/GEMINI.md. That half is landed, so there is nothing to sequence against it any more.

The distinction still stands, though: #3060 added files, and this is a missing branch in a pattern that already exists for a file the gate already reads. The glob-plus-allowlist approach in #3017 would not have caught it, and a reviewer's eye skips it for the same reason mine did.

Happy to open a PR for 1, 2 and 4, or to coordinate first if you would rather sequence it yourself.