CI structure amplifies flakes: fail-fast matrix + test-complete mirror check
What happens
Two aspects of the fixture-test workflow (.github/workflows/test-pr.yaml) inflate the blast radius of every flaky job and hide the results of unrelated languages.
fail-fast: trueon the fixture matrix. Thetestjob's matrix (strategy.fail-fast: true, line 34) runs every language fixture as a matrix leg. When any one leg fails — including a pure infra flake — GitHub cancels all sibling legs. A single flaky job therefore turns into a wall of "cancelled" jobs, and we lose the signal of whether the other languages would have passed.test-completeonly mirrors the matrix. Thetest-completejob (line 211) runsif: cancelled() || contains(needs.*.result, 'cancelled') || contains(needs.*.result, 'failure')and simply prints "Some workflows have failed!" andexit 1. It never fails independently — it only reflects whatever the matrix did. During triage its failure carries no information beyond "some matrix leg failed or was cancelled."
Evidence
.github/workflows/test-pr.yamlline 34:fail-fast: trueon thetestmatrix..github/workflows/test-pr.yamllines 211-218:test-completegate that mirrors the matrix result.- Today's triage of 31 failing PRs: several PRs showed many "cancelled" sibling jobs stemming from a single flaky leg, and
test-complete"Some workflows have failed!" was never an independent failure — it only mirrored whichever matrix job failed.
Impact
- One flaky job (Scala Bloop timeout, Elm warmup deadlock, cjson download reset) cancels the entire language matrix.
- Triage is harder: the real failing leg is buried among cancellations, and passing languages never report.
- The
test-completered X is noise on its own; you must open the matrix to learn anything.
Suggested solutions
- Set
fail-fast: falseon the fixture matrix so each language leg runs to completion independently and one flake stops masking the rest. - During triage, treat
test-completeas a mirror only — inspect the individual matrix legs, not the aggregate gate.
Related flaky areas
These are the specific flakes this structure amplifies:
- #3063 — scala3-upickle: Bloop startup timeout + maven-nightlies download failures
- #3064 — elm: Warmup.elm setup races the compiler (file locks / MVar deadlock)
- #3065 — cjson: cJSON.c downloaded from raw.githubusercontent.com at test time
Interim policy
Per CLAUDE.md ("Known CI flakiness"), we accept these flakes for now and retry the failed jobs (gh run rerun <run-id> --failed) when they occur.
Source: glideapps/quicktype