Live build event streams can stall after DB failover (fly/browser stop updating until manual pg_notify or refresh)

Summary

We are seeing intermittent live-stream stalls after database failover in ConvergedCloud (likely AWS based on #2209). Builds continue to run and complete, and build events continue to be written to Postgres, but live consumers (fly execute, fly watch and browser live logs) stop receiving new events in real time. aka. streaming is broken.

A key signal: running pg_notify('build_events_<build_id>', 'poke') during a stall immediately refreshes the stream and populates newer logs. This suggests a wakeup/notification delivery issue in the LISTEN/NOTIFY path under failover turbulence, rather than a build execution failure or missing event persistence.

Current workaround:

Recreate the web processes

Steps to Reproduce

1. Deploy Concourse with Postgres in ConvergedCloud (possibly AWS to be confirmed).
2. Trigger DB failover
3. Observe live streaming behavior in:

• fly execute
• fly watch -b <id>
• browser build page (live log stream for a running build)

1. During a stall, verify events are still being written:

• select max(event_id), count(*) from build_events where build_id=<id> or build_id_old=<id>;

1. During the same stall, test manual wakeup:

• select pg_notify('build_events_<id>', 'poke');

1. Observe that stream refreshes immediately after manual notify

Additional checks used during investigation:

Listener sessions:

select pid, client_addr, query from pg_stat_activity where query like 'LISTEN build_events_%' order by client_addr, pid;

External SSE validation:

curl -v -N --http1.1 -H "Authorization: Bearer <token>" "https://<host>/api/v1/builds/<id>/events" curl -v -N --http2 -H "Authorization: Bearer <token>" "https://<host>/api/v1/builds/<id>/events" fly variants: fly watch -b <id> --verbose GODEBUG=http2client=0 fly watch -b <id> --verbose fly watch -b <id> --verbose --ignore-event-parsing-errors

Expected Results

Live event streams fly execute, fly watch, browser live logs) continue updating through/after failover, or recover automatically without manual intervention. No manual DB notify should be required. If events are persisted, active stream consumers should receive them promptly.

Actual Results

Live streams stall during/after failover while builds continue and complete. Browser may show full logs only after refresh. DB confirms events are still persisted (build_events grows). Manual pg_notify('build_events_', 'poke') immediately unsticks fly watch live stream. In affected environments, this behavior is reproducible for active builds, indicating missed/lost stream wakeups rather than missing events.

Web Node(s) configuration

No response

Worker(s) configuration

No response

Concourse Version

v8.0.1

Browser (if applicable)

No response

Did this use to work?

No response