Unhandled RegistrationException resulting in stale member
Summary
ClusteredServiceAgent.joinActiveLog() clears the triggering activeLogEvent before
attempting aeron.addSubscription(...). If that call throws (e.g. a transient
RegistrationException), the exception propagates out of the service agent's duty cycle
uncaught, and the service is left permanently unable to join/rejoin the log — with no
automatic retry — until an unrelated new leadership-term event happens to deliver a fresh
ActiveLogEvent. In a stable cluster where leadership doesn't change again, the service can
be stuck indefinitely with no path to recovery short of an external restart.
This is inconsistent with the rest of the cluster module, which already has an established
pattern for exactly this kind of transient registration failure (see
ClusterMember.tryAddPublication, below) — it just wasn't applied to this call site.
Steps to reproduce (conceptual)
- Start a multi-node Aeron cluster (
ConsensusModule+ClusteredServiceContainer) where at least one member's log/consensus endpoint is configured viaMediaDriver.Context's nativeresolverName/resolverInterface/resolverBootstrapNeighborgossip resolver rather than a literal/static address. - Dispose one member's
MediaDriver/ConsensusModule/ClusteredServiceContainerwhile the other members keep running (simulating a pod restart), then immediately relaunch that member's stack (freshMediaDriver, fresh gossip cache) without waiting for gossip convergence. - If the disposed member's
ClusteredServiceContainerreceives itsActiveLogEvent(to rejoin the log / catch up) before its own driver has finished resolving the relevant peer endpoint via gossip,aeron.addSubscription(...)injoinActiveLogthrows aRegistrationException. - Observe: the agent thread does not crash (
ClusteredServiceImpl'sonStartfires, no further errors are thrown from the agent runner), but the service never actually joins the log.logAdapter.image()staysnullandactiveLogEventstaysnull, sopollServiceAdapter()never attempts the join again. The only way this member rejoins is if a subsequent, unrelated leadership-term change happens to hand it a newActiveLogEvent.
Expected behavior
A transient failure to (re)join the active log should be retried — either automatically
(e.g. don't clear activeLogEvent until the subscription is successfully established, or
catch the registration failure and re-arm activeLogEvent for the next duty cycle, mirroring
ClusterMember.tryAddPublication's pattern) or with an explicit, documented backoff/retry
policy. At minimum, a failure here should not be silently swallowed by "wait for a future,
unrelated leadership change."
Actual behavior
The log-join attempt is made exactly once per ActiveLogEvent. If it throws, the event is
already gone (cleared before the attempt), the exception is swallowed as non-fatal by
AgentRunner, and the service is left permanently stalled until some other event happens to
produce a new ActiveLogEvent.
Suggested fix
Something along these lines in pollServiceAdapter() / joinActiveLog():
if (null != activeLogEvent && null == logAdapter.image())
{
try
{
joinActiveLog(activeLogEvent);
activeLogEvent = null;
}
catch (final RegistrationException ex)
{
ctx.countedErrorHandler().onError(new ClusterEvent(
"failed to join active log, will retry: " + ex.getMessage()));
// leave activeLogEvent set so the next duty cycle retries
}
}Source: aeron-io/aeron