Election restarts every ~5 s on an idle libp2p cluster and pauses the API for each campaign (~3 s stall on most requests)

Summary

On an idle, healthy 4-node cluster (v1.0.71, macOS, libp2p), the master election never settles: a new campaign starts every ~5 s, indefinitely. That would be harmless noise except that the API is paused for the entire duration of every campaign, so roughly 60% of chat-completion requests stall ~3 s before prefill begins. It shows up as time-to-first-token alternating between ~0.5 s and ~4 s on back-to-back identical requests, and as latencies quantised at 5 / 10 / 15 s.

Mechanism

1. rust/networking/src/discovery.rs re-dials every known peer every RETRY_CONNECT_INTERVAL = Duration::from_secs(5), whether or not a connection already exists. The duplicate connection is established and immediately closed (ToSwarm::CloseConnection, discovery.rs:134).
2. swarm.rs:134-140 maps discovery::Event::ConnectionEstablished → FromSwarm::Discovered and ConnectionClosed → FromSwarm::Expired; exo_pyo3_bindings/src/networking.rs:156-163 turns those into ConnectionMessage{connected: true|false}. So every 5 s, per peer, Python receives an "up, DOWN" pair for a peer that never actually left. Logged on one node (peer ids abbreviated):

   18:00:25.408  connection updates: tS5n9H=up, tS5n9H=DOWN | peers before=[]
   18:00:26.447  connection updates: gtE9s4=up, gtE9s4=DOWN | peers before=[]
   18:00:27.456  connection updates: 7QAt28=up, 7QAt28=DOWN | peers before=[]
   18:00:30.416  connection updates: tS5n9H=up, tS5n9H=DOWN | peers before=[]
   

   The TCP sessions themselves were stable throughout (15/15 one-second lsof -iTCP:51290 -sTCP:ESTABLISHED samples: same three peers, same ports).
3. shared/election.py _connection_receiver starts a fresh campaign for every batch of connection messages, unconditionally.
4. api/main.py _pause_on_new_election sets self.paused = True on any election message with a newer clock, and _send blocks every command on it until unpause() at campaign resolution — DEFAULT_ELECTION_TIMEOUT = 3.0 plus the 0.2 s collect delay later.

Net effect: three peers × one re-dial each per 5 s ⇒ a campaign is (re)started roughly every 5 s and the API is paused for ~3.2 s of each cycle, forever.

Measurements

Master log over 45 minutes: 717 "elected master" events, always the same node, one actual promotion. 12 per minute, steady.

Streaming probe, sequential identical requests, temperature=0, reasoning_effort=low, 8 tokens — request send time correlated against the master log:

sent (relative to campaign)	TTFT
1.2 s after "Cancelling other campaign"	2.79 s — first token at "Unpausing API"
0.2 s before campaign start	4.04 s
1.6 s after "Unpausing API" (idle window)	0.49 s
0.9 s after campaign start	2.90 s
1.6 s after "Unpausing API"	0.50 s
0.9 s after campaign start	2.90 s
1.6 s after "Unpausing API"	0.49 s
0.9 s after campaign start	2.87 s

Decode itself is unaffected: median inter-token gap 37 ms (~27 tok/s), no gap over 0.12 s, 50 ms from last token to stream end. The whole penalty is admission. Six concurrent short requests all returned at exactly 5.0 s.

Ruled out first: prefix-cache state (slow with cached_tokens=22 and with 0), prompt identity (same prompt ×4 alternates), and inter-request gap (6 s pauses between requests do not help).

Fix on v1.0.71

Two layers. The root cause is in Rust: the discovery behaviour should not re-dial peers it is already connected to. Independently, the election should only campaign when cluster membership changes, and must judge that on the batch as a whole — an "established, closed" pair for one peer is a net no-op:

def _apply_connection_updates(self, messages: list[ConnectionMessage]) -> bool:
    before = frozenset(self._connected_peers)
    for message in messages:
        if message.connected:
            self._connected_peers.add(message.node_id)
        else:
            self._connected_peers.discard(message.node_id)
    return frozenset(self._connected_peers) != before

with _connection_receiver doing continue when it returns False. (A first attempt that flagged "changed" if any step changed the set did not help, for exactly this reason.)

Result on the same cluster after deploying it to all four nodes and restarting: campaigns during cluster formation only (19 in the first 20 s), then zero elections in the following minutes (previously 12/min, indefinitely). Time-to-first-token on the same streaming probe went from alternating 0.5 s / 4 s to a flat 0.55–0.61 s across 8 sequential requests (0 of 8 over 1.5 s); four concurrent short requests completed in 0.86–1.45 s instead of all landing at exactly 5.0 s; the same 4-prompt correctness suite went from 15.1 to 22.4 tok/s end-to-end with no change in decode speed — the entire gain is the removed admission stall.

Why this is not a PR against main

main has the same two halves — _connection_receiver still campaigns per event, and api/main.py still pauses on every newer-clock election message — but the networking layer was rewritten for zenoh and FromSwarm::Discovered {} / Expired {} carry no peer identity (rust/networking/src/swarm.rs; discovery.rs has the zid but it is dropped before reaching Python). Deduping on main needs that identity threaded through the Rust bindings first, and I cannot verify whether zenoh re-emits Discovered for known peers the way libp2p re-emits Connection. If it does, main has the same stall. Either way, pausing the API for a full election timeout on every redundant discovery event seems worth guarding against independently of the transport.

Happy to turn this into a PR if a maintainer can confirm the intended shape (thread zid into Discovered, or make the API pause conditional on the peer set changing).