[Bug Report] Template commit snapshots rootfs before freezing memory, corrupting in-flight downloads (tar: Skipping to next header)

Summary

When committing a running sandbox to a template, CommitSandbox takes the rootfs (disk) snapshot before freezing guest memory, so a download still in flight at commit time is silently corrupted in sandboxes restored from that template (tar: Skipping to next header).

Environment

• CubeSandbox version / commit: v0.7.1 (also reproduces on v0.7.0; the affected code path in Cubelet/services/cubebox/template_ops.go is unchanged between the two tags and on current master)
• Host OS and kernel version: not environment-specific — this is a deterministic ordering bug in the commit/snapshot logic, independent of host OS/kernel
• KVM info (modinfo kvm): N/A (not hardware/KVM dependent)
• Deployment mode: both single-node and cluster (same CommitSandbox code path)
• Relevant component: Cubelet (services/cubebox); CubeShim is involved in the proposed fix

Steps to Reproduce

1. Build a template image whose default container command downloads a large tar.gz and then extracts it, e.g.:
   bash复制

   wget --tries=3 --timeout=20 -q -O /tmp/payload.tar.gz "$URL"
   tar -xzf /tmp/payload.tar.gz -C /tmp/extracted

   Use a large file / slow link so the download stays in flight for a while.
2. Commit the running sandbox to a template while the download is still in progress. This is easy to hit in practice when the readiness probe (--probe) points at the envd port (default 49983) instead of the application's own "download+extract finished" readiness endpoint, so the snapshot is taken before the download completes.
3. Create a new sandbox from that template.
4. In the restored sandbox, wget resumes the download and tar -xzf runs against the resulting file.

Expected Behavior

The tar.gz in the restored sandbox is byte-for-byte identical to the source and extraction succeeds — the template snapshot should capture disk and memory from the same instant.

Actual Behavior

Intermittently, extraction fails:

tar: Skipping to next header
tar: Exiting with failure status due to previous errors

The failure is a silent corruption: wget exits 0 and the file size matches the source exactly, but there is a zero-filled hole in the middle of the archive, which breaks the gzip stream. A plain size check or exit-code check does not catch it; only a byte comparison (or verifying the file prefix) does.

Root cause

Cubelet/services/cubebox/template_ops.go — CommitSandbox snapshots the disk before freezing memory:

• template_ops.go:140 — storage.CommitRootfsFor(...) takes the rootfs CoW snapshot (call it T1) while the VM is still running and writing.
• template_ops.go:201 — s.executeCubeRuntimeSnapshot(...) freezes + dumps memory later (T2).

So the template's disk image comes from T1 and its memory image from T2. On restore, the filesystem reflects T1 but process memory reflects T2. wget's in-memory byte counter (T2) is ahead of what actually reached disk (T1); when it resumes it issues Range: bytes=N- (N from T2) and seeks past the true end of the shorter T1 file, leaving the bytes in [T1, T2) as an unwritten zero hole → corrupt gzip → the tar error above.

The pause/resume path already does this correctly and documents the invariant. Cubelet/services/cubebox/pause_cow.go:294-297:

// Disk after memory freeze: live rootfs volume is still present until
// keep_tombstone Destroy.
rootfsObject, err = storage.CommitRootfsFor(workCtx, backend, sourceRootfs, snapID)

CommitSandbox violates the invariant that pause_cow.go enforces: the disk snapshot and the memory snapshot must be produced inside the same frozen window.

Why it is intermittent

Three conditions must line up, which is why it only happens "sometimes":

• the download is actively in flight during the commit window (if it finished first, disk and memory agree);
• the downloader resumes via HTTP Range — wget --tries=N does this automatically and hits the bug; curl --retry N without --retry-all-errors fails cleanly instead, and --retry-all-errors re-downloads from 0 (safe);
• it correlates strongly with probe timing: probing the envd port rather than an app-level "ready" endpoint makes the snapshot much more likely to land mid-download.

Additional Context

Suggested fix direction (aligning CommitSandbox with pause_cow.go): keep the VM paused after the memory snapshot (e.g. a --keep-paused option on the shim snapshot command, since do_app_snapshot currently pauses → snapshots → resumes unconditionally and exposes no held freeze window), take the rootfs CoW snapshot inside that same frozen window, then resume explicitly (detached from the request context so a client timeout can't leave the guest frozen). AppSnapshot keeps keep_paused=false and is unaffected.

Workaround until fixed: point the template's readiness probe at an application endpoint that only reports ready once the download + extraction have completed, so the snapshot is always taken after the archive is fully on disk.

Docs suggestion: add a troubleshooting note — tar: Skipping to next header with a correct file size after creating a sandbox from a template points at snapshot ordering, not a bad download; verify the file prefix rather than trusting size/exit code.