FNN keeps running after critical actor terminates and only logs ractor messaging failures

Summary

When a critical actor such as the CKB chain actor terminates, fnn can keep running instead of exiting. Later actor sends/calls then fail with ractor's generic messaging error:

Messaging failed to enqueue the message to the specified actor, the actor is likely terminated

For an operator, the process still appears alive even though a critical subsystem is gone. This should be treated as fatal and the node should exit, rather than continuing in a degraded state.

Why this is a problem

A node that has lost a critical actor can keep serving as if it were healthy. In particular, funding tx tracing/broadcasting paths can continue to tick and only log ractor errors, while the process does not shut down. This can leave channel/funding workflows stuck and makes service health checks misleading.

Code path

The observed log text is produced by ractor for MessagingErr::SendErr:

Self::SendErr(_) => {
    write!(f, "Messaging failed to enqueue the message to the specified actor, the actor is likely terminated")
}

In Fiber, one concrete path is InFlightCkbTxActor::send_tx:

match ractor::call_t!(
    self.chain_actor,
    CkbChainMessage::SendTx,
    self.timeout_ms(),
    tx
) {
    Ok(Ok(_)) => {
        // repeat sending the tx on success to let the CKB node broadcasts it
    }
    Ok(Err(err)) => {
        tracing::error!(
            "failed to send tx {} because of rpc error: {}",
            self.tx_hash,
            err
        );
        if is_permanent_error(&err) {
            let _ = self
                .chain_actor
                .send_message(CkbChainMessage::ReportSendTxError(self.tx_hash, err));
        }
    }
    Err(err) => {
        tracing::error!(
            "failed to send tx {} because of ractor error: {}",
            self.tx_hash,
            err
        )
    }
}

The Err(err) branch logs the ractor messaging failure but then returns normally, so the actor/process continues.

There are also process-level reasons this does not become fatal:

• crates/fiber-bin/src/main.rs spawns the ckb actor with Actor::spawn_linked(...) but discards the join handle and only keeps the ActorRef.
• run_node waits for OS signals via signal_listener().await; it does not race signal handling against critical actor termination.
• RootActor::handle_supervisor_evt logs SupervisionEvent::ActorTerminated at debug! only.
• NetworkActor::handle_supervisor_evt also only logs child termination at debug!.

This conflicts with the code comment in network.rs which says the chain actor is currently assumed to always be alive and failure should panic:

// This is a temporary way to document that we assume the chain actor is always alive.
// We may later relax this assumption. At the moment, if the chain actor fails, we
// should panic with this message, and later we may find all references to this message
// to make sure that we handle the case where the chain actor is not alive.
const ASSUME_CHAIN_ACTOR_ALWAYS_ALIVE_FOR_NOW: &str =
    "We currently assume that chain actor is always alive, but it failed. This is a known issue.";

Expected behavior

If a critical actor (ckb, network, built-in watchtower when enabled, CCH when configured as required, etc.) terminates unexpectedly, fnn should fail fast:

• log a clear fatal error naming the actor that stopped,
• cancel the node task tree / stop remaining actors,
• return a non-zero process exit status.

Similarly, messaging failures to actors that are currently treated as mandatory, especially the chain actor, should not be silently downgraded to a normal loop continuation.

Actual behavior

The node can continue running after the critical actor is gone. Subsequent actor sends/calls log errors like:

failed to send tx <tx_hash> because of ractor error: Messaging failed to enqueue the message to the specified actor, the actor is likely terminated

but the process remains alive.

Suggested fix direction

One robust fix is to make the native fnn entry point retain join handles for critical actors and race them against signal_listener():

• keep the join handles returned by Actor::spawn_linked for critical actors,
• make run_node return Err(ExitMessage(...)) when any critical actor handle resolves before a shutdown signal,
• cancel the task tracker and stop the remaining actor tree before returning,
• keep test-only/shared-root actor behavior unchanged so normal per-node test shutdowns do not become global panics.

In addition, paths such as InFlightCkbTxActor::send_tx should not just log and continue when the mandatory chain_actor cannot be reached.

Regression test idea

A minimal regression test can be added around a shutdown arbiter extracted from run_node:

#[tokio::test]
async fn exits_when_critical_actor_stops_before_signal() {
    let actor_handle = CriticalActorHandle::new("ckb", tokio::spawn(async {}));

    let err = wait_for_shutdown(std::future::pending::<()>(), vec![actor_handle])
        .await
        .expect_err("critical actor stop should exit the node");

    assert!(
        err.0.contains("critical actor ckb stopped unexpectedly"),
        "unexpected exit message: {:?}",
        err,
    );
}

This should fail on the current structure because run_node waits only for OS signals and does not observe critical actor termination.