Flaky timing-sensitive sync-service tests: ShapeCacheTest await/startup timeouts under CI load
Summary
A small family of timing-sensitive tests fails intermittently in the sync-service CI workflow — hopping between Postgres matrix versions, passing on rerun with no code change, and burying real signal for PR authors. Two recent PRs (#4711 and its review) each spent effort separating these from genuine failures.
Evidence
Electric.ShapeCacheTest — "await_snapshot_start/4 should wait for consumer to come up"
- Failed on pg15: https://github.com/electric-sql/electric/actions/runs/29346393580/job/87131112285 —
Task.await(..., 5300)timeout - Passed on rerun of the same commit (all matrices green)
- Failed on pg17 two commits later: https://github.com/electric-sql/electric/actions/runs/29348549229/job/87138530389 — same test, same timeout
- The failing branch's changes don't execute in this test (verified: the changed code is HTTP-serve-path only, and the branch's heavy
async: falseintegration tests run after the async phase this test runs in — no load coupling)
Electric.ShapeCacheTest — "get_or_create_shape_handle/2 against real db crashes when initial snapshot query fails to return data quickly enough"
Electric.Plug.TraceContextPlugTest — "electric member is found among other tracestate members"
ShapeLogCollectorTest — FlushTracker timing out in Repatch.cleanup
- Reported during review of #4711 (first CI run of that branch)
Common signature
- ~5s
Task.awaittimeouts on tests that race consumer/snapshot startup - Postgres connection churn in the surrounding logs (
tcp connect timeout,econnrefused,tcp recv (idle): closed) suggesting the shared PG instance is briefly saturated by concurrent async tests - Not reproducible locally on fast machines; version-hopping across the PG matrix
Suggested directions
- Widen or parametrize the tight awaits in the affected tests (they're asserting eventual startup, not 5-second startup)
- Or address the underlying PG saturation (connection pool sizing / readiness in the test env)
- At minimum, tag the known-flaky set so authors and reviewers can distinguish them from real regressions without re-deriving this analysis
Filed while triaging CI on #4711, where these flakes twice muddied the signal.
Source: electric-sql/electric