release.yml repeats work the same run already did, and one 20-minute step verifies nothing

Summary

release.yml spends most of its wall time re-doing work, and one of its steps verifies nothing at all. Two release runs timed out today (33957952281, 33963757913) with no test failure and no build error — both quality_static and quality_build were killed by their timeout-minutes caps while still succeeding.

Raising the caps would hide the problem. Three concrete things inside release.yml are worth fixing first, and none of them require a change to branch protection, the merge queue, or the runner fleet:

1. Format Project runs prettier --write and nothing checks the result — ~20 minutes of guaranteed-zero-signal work.
2. Formatting is not gated anywhere in the repository, at any stage. The PR-CI copy has the same --write bug.
3. The same SHA is fully built five times in one release run, while a packed build artifact already exists and is consumed by exactly one job.

The trigger

Neither failing run had a failing check. Both jobs were cancelled by the job timeout:

The job has exceeded the maximum execution time of 45m0s   — Quality Checks (Build)
The job has exceeded the maximum execution time of 30m0s   — Quality Checks (Static)

quality_build was cancelled inside Pack Build Outputs, after check:serve-fast-path-bundle and npm run build had both succeeded. quality_static was cancelled 6m30s into npm run lint:ci, after npm run format had succeeded.

The caps have been out of headroom for a while:

Run 33941624603 is the last full green release: 127 minutes end to end.

Those four rows are the window this issue was filed in, not a trend. Across the last 20 release runs that ran it, quality_build spans 6.7m to 44.4m with a median of 20.2m and zero cap failures — 7.7m sits between 31.6m and 13.6m in that series — and run 33963757913 itself passed on re-run at 39.6m. The spread is placement (#10879 measures 5.4x for identical work), so the caps are not the thing to change: see Finding 4.

The ~40% step-level slowdown between the first and last row is placement, not the tree — that is #10879, and it is the reason a jobs-at-the-cap design fails today rather than next month. The step timings below are given for both a quiet host and a contended one so the two effects stay separable.

Finding 1 — Format Project is a 20-minute no-op

# release.yml
- name: 'Format Project'
  run: |-
    npm run format          # prettier --experimental-cli --write .

--write rewrites files in place and exits 0 whether or not anything changed. Nothing in release.yml inspects the tree afterwards — there is no git diff --exit-code and no git status --porcelain anywhere in the file. The rewritten files are discarded when the job ends.

The only failure this step can produce is Prettier itself crashing on an unparsable file, which ESLint reports anyway in the next step.

quality_static has three substantive steps: npm ci (1m44s), format (20m45s), lint:ci (~8m20s on a quiet host). Deleting the middle one puts the job at roughly 10 minutes against a 30-minute cap, and the timeout stops being reachable.

The step appears to be transcribed from preflight:

preflight = clean && npm ci && format && lint:ci && build && typecheck && test:ci && check:serve-fast-path-bundle

format writing is correct in a local preflight — it repairs your tree before you commit. Carried into CI, where the tree is thrown away, it is pure cost.

Finding 2 — formatting is gated nowhere, at any stage

The same --write bug is in the PR-CI path, so this is not a case of "the check already happened earlier":

// scripts/lint.js — invoked by ci.yml as `node scripts/lint.js --prettier`
export function runPrettier() {
  if (!runCommand('prettier --write .')) {   // --write, not --check
    process.exit(1);
  }
}

ESLint does not cover the gap either: eslint.config.js loads eslint-config-prettier, which disables formatting rules rather than enforcing them.

So the complete list of places that enforce formatting is:

• the husky pre-commit hook plus lint-staged (prettier --write over staged files), which git commit --no-verify bypasses.

That is all. #10573 (N6) already records the mechanism in passing — "Non-gating (scripts/lint.js --prettier runs --write)" — as a footnote explaining why a pending reformat did not block. It has not been raised as a defect.

The fix belongs in ci.yml, not in release.yml: prettier --check at PR time is the gate, and the release copy should simply go away. Sequencing matters, since --check will turn a batch of in-flight PRs red — see Proposal.

Finding 3 — the same SHA is built five times in one release run

Measured cost of the two inside quality_build:

quality_build already packs its outputs and uploads them as release-quality-build, and quality_typecheck is the only consumer — it downloads and unpacks via the download_release_build / unpack_release_build anchors. integration_none and integration_docker rebuild from scratch instead, on the same host label, for the same commit.

e2e.yml solved exactly this and documents the reasoning:

One build feeds every leg. Each of the eleven legs used to build and bundle on its own runner: 4–8 minutes on a hosted VM, 10–17 on a busy pool host, so the six Linux shards spent more of the shared pool rebuilding one commit than testing it. The outputs are platform-independent JavaScript (native modules come from each leg's own npm ci), so a hosted VM builds once, off the pool, and every leg downloads the result.

The pack/unpack helpers already exist as .github/scripts/e2e-build-pack.sh and .github/scripts/e2e-build-unpack.sh.

publish is not part of this. Its build runs after Update package versions and after the audio-capture prebuilds are downloaded, so it produces a genuinely different tree and must stay.

Finding 4 — the caps carry no headroom

quality_static (30) and quality_build (45) are hardcoded, unlike workspace_tests, which reads vars.QWEN_RELEASE_WORKSPACE_TIMEOUT_MINUTES. Findings 1 and 3 remove far more time than any plausible cap increase would buy, so this is listed for the record rather than as the fix. Raising a cap is also actively counterproductive while #10879 is open: a release failing on a starved host is the evidence that ask rests on, and muting it leaves the 5.4x placement spread in place for PR CI and E2E too. If the caps are touched at all, it should be after the work above, and as configurable variables for parity with workspace_tests.

What this is not

This issue is deliberately scoped to work that release.yml does to itself. The adjacent problems already have owners and better write-ups:

• #10254 — merge queue dead since 2026-07-02, main has contexts = []. This is why the release lane is the only place a full suite ever completes on one SHA. Nothing proposed here weakens a gate, so it does not depend on #10254 landing.
• #10820 — reuse a green ci.yml verdict for the duplicated unit lane. Complementary and explicitly disjoint: that proposal states that "typecheck, build, bundle, prepare:package, integration_none, integration_docker, publish" stay unconditional, so Finding 3 is untouched by it.
• #10879 — hk4 still carries the shared ecs-qwen label, so releases compete with PR CI. That is the multiplier on every number above. Fixing it makes these jobs faster; it does not make the duplicated work necessary.
• #10908 — unit-suite cost is bound by core-barrel import cost. Different lane.
• #11103 — tolerate a starved vitest onTaskUpdate RPC so the Test job stops going red with zero failing tests. Adjacent but a different mechanism: there the job finishes and exits 1 on a vitest IPC artifact; here the job is killed by its cap while every step is succeeding. The two are complementary ends of the same capacity problem — that PR reduces the blast radius of contention, this reduces the load that causes it. Note that nothing here is contention-driven at root: on the quietest run in the table above, Format Project still burned 13m58s verifying nothing and the same SHA was still built five times. For that reason this does not belong under #10490 either.

Proposal

Three independent changes, smallest first:

A. Delete Format Project from release.yml. Two lines. Zero behaviour change, because the step gates nothing. quality_static drops to roughly a third of its cap.

B. Make Prettier a real gate at PR time. Change runPrettier() to prettier --check ., and land it together with one npm run format commit so main starts clean.

The size of that commit is known, not guessed: the Format Project step in run 33963757913 printed the files it rewrote, and there are 35 of them, all tracked on main and none generated (npm run generate only writes packages/*/src/generated/git-commit.ts, which .prettierignore covers via **/generated). Twelve are .github/workflows/, five are .github/scripts/, thirteen are under packages/, two under scripts/. Line count not measured.

Once main is clean, --check can only fail a PR that touches one of those files or that bypassed the pre-commit hook — a small and well-defined blast radius, and the case the gate exists for.

C. Deduplicate the build — but not by pointing the integration jobs at release-quality-build. An earlier revision of this issue proposed exactly that. Measuring it first shows it makes the release slower, and the correction matters more than the original suggestion.

integration_none and integration_docker today declare needs: 'prepare' and therefore run in parallel with quality_build. Consuming its artifact means needs: quality_build, which serialises them. From run 33941624603:

today:      max(quality_build 43m40s, integration_docker 1h39m34s)   ~100 min
serialised: 43m40s + npm ci 1m30s + unpack + docker 76m              ~122 min

So the naive form trades ~42 machine-minutes for ~22 minutes of added release latency. Worth knowing which one the project wants to spend; it is not the free subtraction the earlier text implied.

The shape that wins both is e2e.yml's: a dedicated build job that fans out to every consumer, so the build happens once at the head of the graph rather than once per leg. That is blocked on the first open question below — whether check:serve-fast-path-bundle can consume a prebuilt tree. If it can, three builds collapse to one (~42 machine-minutes saved, wall clock unchanged). If it cannot, a shared build job only replaces two builds while adding one of its own (~21 saved), and the case is much weaker.

Two things that were open when this issue was filed are now settled, and both point the same way:

• integration_docker does not need the host build for its image. The release workflow calls npm run build:sandbox -- -s …, and -s is skip-npm-install-build in scripts/build_sandbox.js — the host-side npm install / npm run build / bundle / pack block is skipped outright. .dockerignore excludes dist and **/dist ("rebuilt from scratch inside the container"), and the Dockerfile's builder stage runs its own npm ci && npm run build && npm run bundle && npm run prepare:package && cd dist && npm pack. The host build in that job exists only to give the vitest harness a dist/cli.js to launch (integration-tests/test-helper.ts:211, integration-tests/globalSetup.ts:203).
• The host build cannot simply be dropped in favour of npm run bundle alone. esbuild's entry is packages/cli/src/cli.ts, but @qwen-code/qwen-code-core resolves through that package's exports to dist/index.js, so packages/*/dist must exist first. Whatever supplies it — a build in the job or an unpacked artifact — has to come from somewhere.
• Also settled, and it removes one hazard from any artifact reuse: the DEV=true in check:serve-fast-path-bundle only writes dist/esbuild.json (the metafile). The dist/cli.js in release-quality-build is byte-identical to a non-DEV bundle.

Shape

A and B and the release-side deletion are unambiguous and ship together; C is deferred until the fan-out question is answered, because the only version of it that can be written today makes the release slower.

chore: format the 35 files CI had been silently rewriting
fix(ci): make the Prettier lane a real gate
fix(release): drop Format Project, it writes and discards

That is enough on its own to put quality_static back inside its cap: npm ci (1m44s) + lint:ci (~8m20s) against 30 minutes, instead of 30m24s and cancelled.

Open questions

• Can the two builds inside quality_build be merged? This is now the gating question for C, not a nice-to-have. check:serve-fast-path-bundle starts with clean-package-build-artifacts.js and then builds --cli-only; Build Project then does the full build. If the fast-path check can run against the full build's output, that is another ~15–20 minutes, but the clean step suggests the isolation is deliberate and the comment above the step does not say. Wants an answer from whoever wrote check-serve-fast-path-bundle.js.
• Should the caps become repo variables (QWEN_RELEASE_STATIC_TIMEOUT_MINUTES / ..._BUILD_...) for parity with workspace_tests, or stay hardcoded so that a regression in build time surfaces as a failure rather than being absorbed?
• Is Verify Prepared Package's npm run bundle still needed once C lands, given the artifact already contains a bundle?

Related

• #10254, #10820, #10879, #10908, #10573 (N6), #10038, #11103
• Release-failure notifications for the two runs above: #11098, #11104 — both report only Failed job(s): quality, with no indication that the cause was a timeout rather than a failing check. Worth a separate look at whether the notifier can distinguish the two.

概要

release.yml 的大部分墙钟时间花在重复劳动上，其中一步完全不验证任何东西。今天有两次 release run 超时失败（33957952281、33963757913），没有任何测试失败、没有任何构建错误——quality_static 和 quality_build 都是在正常执行的过程中被 timeout-minutes 上限杀掉的。

调高上限只是把问题藏起来。release.yml 内部有三件事更值得先修，而且都不需要改动分支保护、merge queue 或 runner 集群：

1. Format Project 跑的是 prettier --write，之后没有任何检查——约 20 分钟的、必然零信号的工作。
2. 格式检查在整个仓库的任何阶段都不 gate。PR CI 那份有同样的 --write 缺陷。
3. 同一个 SHA 在一次 release run 里被完整构建了五次，而一份打包好的构建产物已经存在，却只有一个 job 消费它。

触发点

两次失败的 run 都没有任何检查真的失败，两个 job 都是被 job 超时取消的：

The job has exceeded the maximum execution time of 45m0s   — Quality Checks (Build)
The job has exceeded the maximum execution time of 30m0s   — Quality Checks (Static)

quality_build 是在 Pack Build Outputs 里被取消的，此前 check:serve-fast-path-bundle 和 npm run build 都已成功。quality_static 是在 npm run lint:ci 跑了 6m30s 时被取消的，此前 npm run format 已成功。

上限已经贴边很久了：

33941624603 是最后一次完整通过的 release：端到端 127 分钟。

上面这四行是本 issue 提交时所处的窗口，而不是一条趋势线。在最近 20 次跑过它的 release 中，quality_build 的跨度是 6.7m 到 44.4m，中位数 20.2m，零次上限失败——该序列中 7.7m 就夹在 31.6m 与 13.6m 之间——而 run 33963757913 本身在重跑后以 39.6m 通过。这个跨度来自落点（#10879 测得同样的工作相差 5.4 倍），因此该改的不是上限：见发现 4。

第一行到最后一行之间约 40% 的单步变慢来自落点而非代码树——那是 #10879，也正是"贴着上限设计"今天就出事而不是下个月才出事的原因。下面的单步耗时同时给出安静主机和拥挤主机两组数据，以便把两种效应分开看。

发现 1 —— Format Project 是 20 分钟的空转

# release.yml
- name: 'Format Project'
  run: |-
    npm run format          # prettier --experimental-cli --write .

--write 就地改写文件，无论是否有改动都返回 0。release.yml 之后没有任何一步检查代码树——整个文件里既没有 git diff --exit-code 也没有 git status --porcelain。改写过的文件在 job 结束时被丢弃。

这一步唯一可能产生的失败是 Prettier 自己在某个无法解析的文件上崩溃，而那种情况下一步的 ESLint 同样会报出来。

quality_static 有三个实质步骤：npm ci（1m44s）、format（20m45s）、lint:ci（安静主机约 8m20s）。删掉中间那个，job 大约落在 10 分钟，对 30 分钟的上限，超时不再可达。

这一步看上去是从 preflight 抄过来的：

preflight = clean && npm ci && format && lint:ci && build && typecheck && test:ci && check:serve-fast-path-bundle

在本地 preflight 里 format 写盘是正确的——它在你提交前修好你的代码树。搬进 CI 之后代码树被丢弃，它就只剩成本。

发现 2 —— 格式检查在任何阶段都不 gate

同样的 --write 缺陷也在 PR CI 那条路径上，所以这并不是"更早的阶段已经检查过了"：

// scripts/lint.js —— ci.yml 通过 `node scripts/lint.js --prettier` 调用
export function runPrettier() {
  if (!runCommand('prettier --write .')) {   // 是 --write，不是 --check
    process.exit(1);
  }
}

ESLint 也补不上这个缺口：eslint.config.js 加载的是 eslint-config-prettier，它的作用是关闭格式相关规则，而不是执行它们。

于是，真正强制格式的地方只剩下：

• husky 的 pre-commit 钩子加 lint-staged（对已暂存文件跑 prettier --write），而 git commit --no-verify 可以绕过。

仅此而已。#10573（N6）其实已经顺带记录了这个机制——"Non-gating（scripts/lint.js --prettier runs --write）"——但那只是用来解释某个待重排为何不阻塞的脚注，从未被当作缺陷提出。

修复应落在 ci.yml 而非 release.yml：PR 阶段的 prettier --check 才是那道门，release 里那份直接删掉即可。顺序很重要，因为 --check 会让一批在飞的 PR 变红——见"提议"。

发现 3 —— 同一个 SHA 在一次 run 里被构建五次