execd isolated sessions: upper_max_bytes enforced only at allocation — writes uncapped, then all new sessions wedge

Summary

upper_max_bytes (documented as "Hard limit on total upper directory size across all sessions") is enforced only in UpperManager.Allocate(): before creating a new session's upper dir, it walks the upper root and rejects the new session if current usage ≥ limit. Writes made by an already-running session are not capped at all, and once usage crosses the limit every subsequent session creation fails — so a single runaway session both escapes the quota and then wedges all new sessions in that sandbox.

Observed behavior

Config: upper_max_bytes = 33554432 (32 MiB), upper_root on an emptyDir.

# inside an isolated session with overlay workspace:
dd if=/dev/zero of=/workspace/big bs=1M count=45   # rc=0, 45 MiB lands on disk

# then every new session create fails:
{"code":"RUNTIME_ERROR","message":"allocate upper: upper: total usage exceeds configured limit: 47185920 >= 33554432 bytes"}

No ENOSPC is ever delivered to the writing process; the limit only gates future allocations.

Code reference

• components/execd/pkg/isolation/upper.go — Allocate() is the only enforcement point: usageLocked() (a filepath.Walk size sum) compared against maxBytes; nothing else consumes the limit.
• components/execd/pkg/isolation/config.go — default 8 GiB; config comment says "Hard limit".

Impact

1. The quota does not actually bound disk usage — a session can write until the backing emptyDir/node disk fills.
2. After the limit is crossed, all new isolated sessions in the sandbox fail (self-DoS), and the error does not identify the session that consumed the space.
3. Recovery requires killing/rebuilding the sandbox (Pod) since an in-session process owns the data.
4. A true write-time cap needs fs-level enforcement (project quota / overlay quota), which is silently absent on common ext4 nodes (no prjquota) — so even a future "hard" mode would need detection.

Suggestions

1. Document the allocation-time-only semantics in the config comment and the isolation guide ("new sessions are rejected once usage ≥ limit; running sessions are not capped").
2. Expose UpperManager.Usage() (already implemented) via an endpoint or metric so operators can alert before the wedge happens.
3. Consider soft per-session enforcement: a periodic usage walk that flags/kills sessions disproportionately consuming the upper root, or project-quota-based hard enforcement with a startup check that warns when the backing filesystem cannot enforce it.
4. On Allocate rejection, include the top contributing session dirs in the error/message to speed up triage.

Happy to provide more environment details if useful.