CI flake inventory: three distinct flake classes behind the 2026 failures

I scanned all 411 failed ci-workflow runs from 2026 and classified failures by test and cause. The 25 candidate single-leg flakes reduce to four sites, three actionable.

Method. Pool: gh api .../runs?status=failure&created=2026-01-01..2026-12-31. Filter: exactly one Run pytest leg failed while ≥2 passed → 25 runs. Two gotchas inflated early counts: NLTK runs pytest -rsx, so there are no FAILED lines in short test summary info; and a module-scoped fixture failure produces one ERRORS block but N test-level errors.

Finding 1 — CoreNLP readiness: two failure modes (in scope for #3894)

Together these account for 14 of the 25 shortlist entries.

1a. Port collision on 9000 — 6 entries, macOS / Python 3.12, test_corenlp.py. setup_module constructs CoreNLPServer(port=9000), and try_port(9000) fails at nltk/parse/corenlp.py:235 with OSError: [Errno 48] Address already in use because another xdist worker or a leftover server holds the port. This is the original #3429 signature.

1b. Server readiness failure — 8 entries, macOS / Python 3.10, test_corenlp_options_security.py::TestRealServerLaunch. The fixture uses port=try_port() with an ephemeral port — deliberately avoiding the 9000 race — and still fails: srv.start() raises CoreNLPServerError: Could not connect to the server at nltk/parse/corenlp.py:366, under the heavier preload -preload tokenize,ssplit,pos,lemma,ner,parse,depparse -srparser -maxCharLength=-1. The readiness loop retries only on requests.exceptions.ConnectionError; a non-ok response (503 while parse,depparse load) raises immediately. That is likely the actual macOS failure mode here.

Implication for #3894: if the PR only switches test_corenlp.py to an ephemeral port, it fixes 1a but not 1b. The PR should either cover both call sites or fix the readiness loop itself.

Finding 2 — Wall-clock DoS/ReDoS bounds (proposed: separate PR)

~9 tests asserting wall-clock or linear-scaling bounds: test_hostile_pattern_over_single_long_token_is_bounded, test_reviews_reader_does_not_hang_on_crafted_corpus, TestTimeoutIsTheGuarantee, TestSyllableTokenizerDoS, TestPaiceQuadratic, and similar. They pass on idle runners and flake under --numprocesses auto contention. Correct tests with a measurement problem. Proposed fix: retry decorator, or rewrite bounds to count operations rather than elapsed time. Different mechanism from #3894; different PR.

Finding 3 — Thread exhaustion on free-threaded Python (follow-up if it recurs)

test_everygrams_alloc.py::test_oversized_default_does_not_allocate, Python 3.14t / ubuntu. everygrams raises correctly; the test's own worker dies in multiprocessing.Queue.put → RuntimeError: can't start new thread, then dies again in the except handler, so the parent reports worker produced no result. The assertion message ("worker raised") is misleading, and a pipe would avoid the feeder thread entirely. Seen once.

Finding 4 — Downloader tests (unresolved)

test_downloader_redownload and TestDownloaderCycle.test_mutual_referential_index, one hit each. Cannot distinguish flake from regression without branch inspection. No action proposed.

Proposal

1. Keep #3894 narrow — confirm the fix covers both CoreNLP failure modes (the port-9000 collision and the readiness loop at corenlp.py:366); don't bundle unrelated flakes.
2. Open a separate PR for Finding 2, scoped to the timing-bound DoS/ReDoS tests.
3. No PRs yet for Findings 3–4; open an issue if either recurs.
4. Consider a CI advisory that flags the pattern: at least one Run pytest leg failed while ≥2 passed. The current -rsx + short-summary behavior hides these from anyone scanning for FAILED.

Script and per-run TSV available on request.