#2099·aeron

Unhandled RegistrationException resulting in stale member

Author: matyasberryCreated Jul 23, 2026Updated Aug 8, 2026

Summary

ClusteredServiceAgent.joinActiveLog() clears the triggering activeLogEvent before attempting aeron.addSubscription(...). If that call throws (e.g. a transient RegistrationException), the exception propagates out of the service agent's duty cycle uncaught, and the service is left permanently unable to join/rejoin the log — with no automatic retry — until an unrelated new leadership-term event happens to deliver a fresh ActiveLogEvent. In a stable cluster where leadership doesn't change again, the service can be stuck indefinitely with no path to recovery short of an external restart.

This is inconsistent with the rest of the cluster module, which already has an established pattern for exactly this kind of transient registration failure (see ClusterMember.tryAddPublication, below) — it just wasn't applied to this call site.

Steps to reproduce (conceptual)

  1. Start a multi-node Aeron cluster (ConsensusModule + ClusteredServiceContainer) where at least one member's log/consensus endpoint is configured via MediaDriver.Context's native resolverName/resolverInterface/resolverBootstrapNeighbor gossip resolver rather than a literal/static address.
  2. Dispose one member's MediaDriver/ConsensusModule/ClusteredServiceContainer while the other members keep running (simulating a pod restart), then immediately relaunch that member's stack (fresh MediaDriver, fresh gossip cache) without waiting for gossip convergence.
  3. If the disposed member's ClusteredServiceContainer receives its ActiveLogEvent (to rejoin the log / catch up) before its own driver has finished resolving the relevant peer endpoint via gossip, aeron.addSubscription(...) in joinActiveLog throws a RegistrationException.
  4. Observe: the agent thread does not crash (ClusteredServiceImpl's onStart fires, no further errors are thrown from the agent runner), but the service never actually joins the log. logAdapter.image() stays null and activeLogEvent stays null, so pollServiceAdapter() never attempts the join again. The only way this member rejoins is if a subsequent, unrelated leadership-term change happens to hand it a new ActiveLogEvent.

Expected behavior

A transient failure to (re)join the active log should be retried — either automatically (e.g. don't clear activeLogEvent until the subscription is successfully established, or catch the registration failure and re-arm activeLogEvent for the next duty cycle, mirroring ClusterMember.tryAddPublication's pattern) or with an explicit, documented backoff/retry policy. At minimum, a failure here should not be silently swallowed by "wait for a future, unrelated leadership change."

Actual behavior

The log-join attempt is made exactly once per ActiveLogEvent. If it throws, the event is already gone (cleared before the attempt), the exception is swallowed as non-fatal by AgentRunner, and the service is left permanently stalled until some other event happens to produce a new ActiveLogEvent.

Suggested fix

Something along these lines in pollServiceAdapter() / joinActiveLog():

java
if (null != activeLogEvent && null == logAdapter.image())
{
    try
    {
        joinActiveLog(activeLogEvent);
        activeLogEvent = null;
    }
    catch (final RegistrationException ex)
    {
        ctx.countedErrorHandler().onError(new ClusterEvent(
            "failed to join active log, will retry: " + ex.getMessage()));
        // leave activeLogEvent set so the next duty cycle retries
    }
}