runtime-rs: Sandbox::wait() can miss the stop notification and never return

Description of problem

VirtSandbox announces its stop through a tokio::sync::watch channel (exit_notify_tx). The sandbox keeps the sender only: the receiver made by watch::channel(false) is dropped at once, and wait() subscribes when it is called.

record_stop() announces the stop with

let _ = self.exit_notify_tx.send(true);

watch::Sender::send() fails when no receiver exists at that moment, and then it does not store the value either. The error is discarded. wait() does

{
    let inner = self.inner.read().await;
    if inner.state == SandboxState::Stopped {
        return Ok(inner.exit_info.clone().unwrap_or_default());
    }
}

let mut exit_notify_rx = self.exit_notify_tx.subscribe();
while !*exit_notify_rx.borrow() {
    exit_notify_rx.changed().await...
}

A waiter that reads the state before record_stop() and subscribes after it finds false in the channel and waits for a change. record_stop() returns early once the state is Stopped, so no change ever comes.

The window is the few instructions between the release of the read lock and subscribe(). It is met when the sandbox container exits: the two wait_process() tasks (spawned at create and at start) and the handler of the Wait request all call Sandbox::stop() at the same moment, each ends in self.wait(), and the task that waits for the VM calls record_stop() just then.

Expected result

Every Sandbox::wait() returns once the sandbox has stopped, whenever it subscribed.

Actual result

Seen once, with Cloud Hypervisor, on a build of 4.2.0 plus downstream patches that do not touch this code. record_stop() and wait() are the same on main.

The container was killed and exited. Shim log (timestamps UTC):

02:12:44.145694 begin stop sandbox
02:12:44.145719 begin stop sandbox
02:12:44.145860 begin stop sandbox
...
02:12:44.289606 wait sandbox
02:12:44.289643 wait sandbox
02:12:44.289669 sandbox stopped
02:12:44.289702 wait sandbox
02:12:44.289745 sandbox stopped
02:12:44.289770 receive action Event(TaskExit { ... })
02:12:44.289844 receive action Event(TaskExit { ... })
...
02:22:44.020619 method handle /containerd.task.v2.Task/Wait got error timed out

Three stops entered wait(), two returned. Those two are the spawned wait_process() tasks, which sent their TaskExit events. The third is the one of the Wait request: the task was reported STOPPED within a second, and the request was answered only by its deadline, ten minutes later. A client that waits for the exit of the task it killed (containerd's task.Wait()) hangs for that time.

It cannot be reproduced on demand. The lost notification itself is deterministic and is shown by a unit test: announce the stop while nobody is subscribed, then wait.

Further information

watch::Sender::send_replace() stores the value whether or not a receiver exists; with it a later subscriber finds true and returns.

A fix will follow in a PR.