[Bug Report] Paused sandbox with default on_timeout=kill is deleted after the pause-state TTL expires

Summary

A sandbox paused via Sandbox.pause() / POST /sandboxes/{id}/pause is silently deleted by the lifecycle manager if it was created with the default lifecycle policy (on_timeout="kill"). After the pause, the sandbox is destroyed once two things have elapsed: the internal pause-state marker's 60-second lifetime and the sandbox's own idle timeout. Resuming it afterwards returns 404.

Environment

• CubeSandbox version / commit: v0.7.0 (d0081641)
• Host OS and kernel version: Ubuntu 22.04 / 6.8.x
• KVM info (modinfo kvm): n/a
• Deployment mode: single-node and cluster
• Relevant component: cube-lifecycle-manager (pause-state sync + idle sweeper); CubeMaster lifecycle event publishing

Steps to Reproduce

1. Create a sandbox with the default lifecycle policy, i.e. without passing a lifecycle argument and without on_timeout="pause". This is the documented default (on_timeout="kill").
2. Pause it explicitly through the SDK or the pause API.
3. Wait until both of the following have passed:
   • about 60 seconds (the lifetime of the internal pause-state marker), and
   • the sandbox's idle timeout, measured from its last activity.
4. Try to resume/connect the sandbox, or list it.

Expected Behavior

A paused sandbox preserves its state and remains resumable until it is explicitly deleted (kill() / DELETE /sandboxes/{id}). This matches the lifecycle documentation, which presents pause as a reversible state and delete as a separate, explicit action. The idle timeout is expected to govern a running sandbox, not to reap one the user has deliberately paused.

Actual Behavior

The paused sandbox is destroyed by the lifecycle manager's idle sweep, with no warning to the caller, and its pause snapshot is removed. A subsequent resume/connect fails with 404.

The behaviour is asymmetric across lifecycle policies:

• Sandboxes created with on_timeout="kill" (the default) are destroyed.
• Sandboxes created with on_timeout="pause" survive, but only because the same expiry causes the manager to re-issue a pause (a harmless no-op) rather than a destroy.

Additional Context

Mechanism. When a sandbox is paused, CubeMaster publishes a runtime-state event and the lifecycle manager records the "paused" state in Redis as a marker with a 60-second TTL. Nothing refreshes that marker afterwards. The idle sweeper treats this marker as its only evidence that a sandbox is parked; once the marker expires, a paused sandbox becomes indistinguishable from an ordinary idle running sandbox. For a sandbox whose policy is the default on_timeout="kill", the sweeper's idle action is to destroy — so the paused sandbox is deleted. For on_timeout="pause", the same path re-issues a pause instead, which is why that configuration does not lose data. The code already acknowledges the periodic nature of this expiry for the on_timeout="pause" case (described there as "pointless RPC churn"); the default-policy case, where the same expiry turns into a destroy, appears not to have been considered.

Timing. The destroy happens at approximately the later of (last activity + idle timeout) and (pause time + 60 seconds). The idle clock is not reset by the pause. As a result, a sandbox with a short timeout can be destroyed roughly a minute after it is paused, while a sandbox with a long timeout is destroyed around that timeout after its last activity. Sandboxes configured with timeout = NEVER_TIMEOUT are not affected.

Why it is easy to miss. The pause-state marker expiring is silent, and the only outward sign is a timeout-killed sandbox entry in the lifecycle manager's log. On the client side the deletion simply surfaces later as a 404. The existing test coverage also cannot represent an expired marker, because the sweeper tests use an in-memory store that discards the marker's TTL — so only the "marker still present" case is exercised.

Impact. Any user who pauses a sandbox created with the default lifecycle policy can lose it after the idle timeout without any indication at pause time. This contradicts the documented pause/resume contract.