Decision spike: persistent per-user sandboxed executor behind the trusted host kernel

Problem

The current Reborn architecture keeps the complete canonical agent loop in the trusted host process and sends only builtin.shell commands into a persistent per-user Docker sandbox. This preserves a strong authority boundary, but every new CLI requires host-to-sandbox command plumbing, credential binding, lifecycle handling, and result transport.

A potentially simpler steady-state shape is to run the agent-loop executor beside the user's workspace and CLIs inside the existing persistent per-user sandbox, while retaining a small trusted IronClaw kernel outside it for identity, scheduling, durable state, authorization, approvals, secrets, model-provider credentials, and delivery.

This exact option is not settled by the existing epics:

• #7732 chooses a persistent per-user sandbox but keeps the canonical loop host-side (D1).
• #7482 chooses a sandboxed loop/harness executor but specifies disposable per-run containers (decision 5).
• #7622 elaborates the disposable foreign-harness implementation.
• Experimental PR #7648 proves an ACP TurnRunExecutor slot and Docker placement, but not the production trust membrane or persistent per-user placement.

Before expanding the CLI story after #7810, decide whether IronClaw should keep investing in a host-side loop plus remote shell boundary, or converge on a persistent per-user sandboxed executor behind a trusted host kernel.

Status

Landed on main:

• #7764: persistent container per (tenant,user), Docker Exec, process supervision, adoption/recycle, cross-thread reuse, and cross-user isolation.
• #7732 Phase 2 substrate: per-user proxy/network and default-deny managed egress are part of the current sandbox direction.

Open implementation PRs:

• #7810: manifest-declared direct-exec credential binding, sealed authorization evidence, invocation-scoped placeholder credentials, proxy-only secret substitution, and GitHub/gh canary. This is useful substrate under either architecture, although some builtin.shell-specific enrichment/plumbing may become transitional if the loop moves.
• #7648: experimental ACP executor with host/Docker placement and session continuity; default-off and intentionally missing the production membrane.

Existing follow-up:

• #7825: provider-neutral proxy recipes and runtime CLI binding registration. Its credential-broker work remains necessary for a sandboxed executor.

Proposed target to evaluate

Trusted IronClaw host kernel
  - ingress and typed identity
  - durable threads, turns, runs, events, checkpoints
  - scheduling, leases, redrive, triggers
  - authorization, approvals, reservations
  - secret and credential broker
  - model-provider credentials / mediated model gateway
  - product and channel delivery
              |
              | authenticated, scope-stamped capability protocol
              v
Persistent per-user sandbox
  - one untrusted executor process
  - agent loop or selected harness
  - persistent workspace and selected HOME/tool caches
  - local shell and curated CLIs
  - invocation placeholders only; no real credentials
              |
              v
Per-user iron-proxy
  - default-deny egress
  - approved credential transforms
  - bounded audit stream back to the host

This is not “run the complete trusted IronClaw service with its secrets inside each user container.” The executor is untrusted. The host kernel remains the sole authority and database source of truth.

User/operator outcome

A user gets one persistent sandboxed computer where the loop and CLI environment are colocated. CLI execution, local files, caches, and background tools are natural local operations, while a compromised command still cannot read real credentials, mint authority, mutate another user's state, bypass approvals, or become the durable source of truth.

Operators retain one central control plane for migrations, ingress, scheduling, triggers, policy, observability, upgrades, and rollback.

Scope

Included:

• Compare three concrete placements against the same trust and product requirements:
  1. current trusted host loop + sandboxed shell;
  2. #7482 disposable per-run sandboxed executor;
  3. persistent per-user sandboxed executor proposed here.
• Define the minimum trusted-host kernel and the exact executor protocol.
• Reuse the existing TurnRunExecutor seam; do not add a second routing abstraction unless a concrete implementation requires it.
• Spike one full caller-path turn in the persistent per-user sandbox. The executor must invoke a real installed CLI locally and call host authority only through the proposed membrane.
• Determine how the canonical native Rust loop, ACP harnesses, and future harnesses fit the same placement.
• Decide the disposition of #7810: merge unchanged as shared substrate, split out lower proxy/credential foundations, or keep its shell-specific surface as a supported compatibility path.
• Produce an explicit architecture decision and migration/rollback plan.

Out of scope for the spike:

• Moving secret storage, database migrations, inbound listeners, channel delivery, or durable policy authority into user containers.
• Retiring the native Rust loop.
• Building every capability-socket operation before the placement decision.
• Weakening the sandbox by mounting Docker sockets, host credentials, unrestricted database credentials, or host filesystem paths.
• Treating same-container process separation as sufficient protection for a trusted secret-bearing process.

Questions the decision must answer

Trust boundary

• Which components remain host-trusted and why?
• Can sandbox code signal, inspect, debug, or impersonate the executor or its capability client?
• Does the executor require a distinct UID, PID namespace, seccomp/AppArmor profile, sidecar, gVisor, or microVM boundary?
• How are scope and actor identity minted host-side rather than accepted from the container?
• How are every proxy request and host-capability call correlated to one authorized run/invocation?

Durability and recovery

• What state is authoritative after executor/container loss?
• Which loop checkpoint boundaries are safe to redrive?
• Can the host kill the executor mid-turn, recreate/adopt it, and resume without losing or duplicating a side effect?
• Does a persistent user container remain an optimization/cache, or does correctness accidentally depend on resident process state?

Product and multi-user behavior

• A shared Slack/WebUI thread may contain several acting users. Which user's executor runs each message while preserving one canonical thread?
• How do triggers and automations run when the owning user's executor is stopped or absent?
• How do setup, approval, and auth gates park and resume across executor restart?
• How are per-user reset, image upgrade, quarantine, and workspace recovery exposed to operators?

Runtime and operations

• Does one executor per active user improve or worsen memory, connection, and deployment costs compared with host-loop workers plus Docker Exec?
• How are rolling upgrades coordinated without running database migrations or listeners per user?
• What remains of builtin.shell after local execution: compatibility capability, executor-local implementation, or retired remote path?
• What code and contract mass disappears, and what new protocol/lifecycle mass appears?

Spike

Drive the production-shaped path, not a helper-only prototype:

1. The host TurnRunScheduler claims a real run and selects the sandboxed executor through TurnRunExecutor.
2. The executor starts or reconnects inside the existing (tenant,user) sandbox.
3. It reads the durable thread/run context through a bounded host API or host-prepared input; it does not query durable stores with a broad database credential.
4. It performs one model turn through a host-mediated model call or placeholder-authenticated proxy path.
5. It executes gh pr list locally using #7810-style placeholder mediation.
6. It emits progress and final output into the existing durable event/timeline path.
7. The host kills the executor after a checkpoint, restarts/reconnects it, and proves coherent continuation.
8. A negative test from the shell attempts to read real credentials, call a host capability under another scope, reach a denied network target, signal/inspect a trusted host process, and replay an expired placeholder; every attempt fails.

Acceptance criteria

• One production-shaped full turn runs through a persistent per-user sandboxed executor and completes through the normal scheduler, durable run state, and product reply path.
• The sandbox contains no real user, provider, model, database, channel, or host credentials in environment, argv, files, logs, events, checkpoints, or process-readable state.
• All host-capability calls are authenticated, payload-bounded, rate-limited, scope-stamped by the host, and pass through the existing authorization/approval/obligation/sealed-dispatch path.
• Direct egress remains structurally unavailable; all network traffic goes through the per-user proxy and its authorized policy.
• Killing and replacing the executor proves the declared checkpoint/redrive behavior. No durable thread/run/event data depends solely on container state.
• Two users remain structurally isolated. Two threads for the same user follow an explicit concurrency policy.
• A shared canonical conversation with messages from two acting users has a deterministic executor-selection rule without splitting or duplicating the thread.
• A trigger/automation owned by an idle user starts the correct executor without trusting transport metadata or display strings.
• The spike measures end-to-end latency, idle/active memory, process count, host-to-executor calls, new protocol surface, and code removed/added versus the current architecture.
• Security review documents same-UID/process-namespace attacks and names the required isolation wall for production.
• The architecture decision records the chosen placement, rejected alternatives, compatibility impact, rollback, and the exact disposition of #7810, #7732 D1, and #7482 decision 5.

Required validation

• Caller-level integration through TurnRunScheduler and the product reply seam.
• Real-Docker lifecycle test using the existing persistent per-user container.
• Kill/restart/redrive test at a declared safe checkpoint.
• Credential leak-negative and expired-placeholder tests.
• Cross-user, cross-scope, and shared-thread actor tests.
• Proxy default-deny and capability-protocol authentication tests.
• Architecture tests for new dependency edges or public contracts.
• Threat-model review before any production profile can select this executor.

Close criteria

Close when the spike evidence supports one explicit decision:

1. keep the host-side canonical loop and continue the shell-sandbox roadmap;
2. adopt a persistent per-user sandboxed executor and create the production implementation issues;
3. adopt #7482's disposable per-run executor model instead;
4. reject both sandboxed-loop options with measured reasons.

No option may move real secrets or durable authority into the untrusted executor merely to simplify implementation.

Related context

• #7732 — persistent per-user sandbox epic; this issue re-evaluates D1 while retaining its lifecycle and proxy substrate.
• #7482 — kernel plus sandboxed pluggable-loop epic; this issue evaluates a persistent per-user placement rather than its disposable per-run decision.
• #7622 — production foreign-harness executor work item.
• #7648 — experimental ACP executor PR and paired evaluation.
• #7764 — merged persistent per-user Docker Exec implementation.
• #7810 — open credentialed direct-exec and GitHub canary PR.
• #7825 — proxy authentication recipes and runtime CLI bindings.