Multi-tenant: renew intents are silently dropped after a server restart; in-use sandboxes are deleted at expiry

Summary

With the agent-sandbox provider in multi-tenant mode (control plane and data plane deployed separately), a server restart silently disables access renewal: every renew intent published after the restart is consumed from the Redis queue but processed as a no-op, so in-use sandboxes are deleted when their original shutdownTime arrives. There is no log line at any point in the drop path.

Environment

Item	Value
Version	current main (reproduced at 09e3584c)
Runtime	Kubernetes, workload_provider = agent-sandbox
Mode	multi-tenant ([tenants] provider = "http")
Renewal	[renew_intent] enabled, Redis source
Sandboxes	tenant namespaces, separate from the server's configured kubernetes.namespace

Observed timeline (controlled reproduction, identifiers redacted)

Time	Event	Evidence
11:37:58	Sandbox created via API (tenant key), Running	CR .spec.shutdownTime = 11:47:57 (= create + 600 s)
11:38:12	Server process restarted	graceful shutdown log
11:39:18	Renew intent published to queue	Redis LPUSH, queue length 1
11:39:19	Intent consumed	queue length 0 — and no log line is emitted
11:39:19	CR .spec.shutdownTime	unchanged (11:47:57)
11:47:57	Sandbox deleted at the original expiry while the session is still active	shutdown policy Delete

Root cause

1. HTTPTenantProvider keeps its tenant information purely in memory (server/opensandbox_server/tenants/http_provider.py) and only learns tenants from per-key lookups made by control-plane API calls. A restart empties it, and supports_enumeration is False.
2. Background renew workers have no tenant ContextVar, so KubernetesSandboxService._resolve_namespace_for_lookup (kubernetes_service.py) falls into _find_sandbox_namespace: the server's configured namespace misses, no tenant information is available, and the method returns None.
3. The caller then falls back to the configured namespace, where the sandbox does not exist → get_sandbox raises 404.
4. AccessRenewController._try_renew_sync swallows it: except HTTPException: return False (integrations/renew_intent/controller.py) — no logging. The intent has already been consumed from the queue.

Redis is working correctly throughout — delivery, consumption, and staleness gating all behave; only the sandbox-id → namespace resolution fails.

Impact

• Any multi-tenant deployment where tenant sandboxes live outside the server's configured namespace.
• The tenant information is repopulated on the tenant's next control-plane API call, so the failure window is restart → next API call by that tenant. A quiet session (long-running agent command, no API traffic) stays broken until expiry.
• A mass restart (rollout, OOM, node drain) drops renewal for all in-use sandboxes at once, silently.
• Not affected: single-tenant/direct-API-key mode (sandboxes in the configured namespace), and the in-process proxy path (has tenant context).

Reproduction

1. Configure as in Environment above (any tenant namespace ≠ kubernetes.namespace).
2. Create a sandbox with timeout: 600 and a renew extension (e.g. 600 s) using a tenant API key; confirm Running and read .spec.shutdownTime from the tenant namespace.
3. Restart the server process.
4. Push a renew intent — either from a real gateway session, or deterministically by LPUSHing {"sandbox_id": "...", "observed_at": "<now>", ...} to the configured queue key.
5. Observe: the queue drains, no renew/failure log appears, .spec.shutdownTime does not advance, and the sandbox is deleted at the original expiry.

Expected behavior

Renew intents should survive a server restart: resolve the sandbox id to its actual namespace via a fallback (e.g. a cluster-wide label-selector lookup), or at minimum fail loudly with a structured WARNING instead of a silent drop.