epic: build the Astrid universal application substrate

Summary

Build the first coherent standalone Astrid operating substrate:

• an independently bootable Astrid kernel and signed System Generation with no host operating system beneath its authority;
• a distribution-neutral boot, resource, Capsule, storage, generation, update, and recovery contract over which independently selected distributions compose products;
• a host-neutral typed state/action boundary that allows consuming hosts to build graphical, A2UI-like, terminal, or accessibility experiences;
• one immutable Hermes application closure shared physically across principals, with separate principal-affine state, authority, budgets, handles, lifecycle, and receipts;
• a provider-neutral application/runtime boundary that supports Capsules directly and existing non-WASM programs through a principal-owned Linux Realm;
• Astrid-owned storage, networking, secrets, devices, lifecycle, accounting, and administration beneath both execution forms; and
• a native proof on a qualified machine contract, while hosted Astrid remains a supported deployment for incremental adoption and differential conformance with explicitly inherited host guarantees.

This is the execution epic for the architecture specified in:

• docs/astrid-universal-application-substrate.md
• docs/astrid-resource-ownership-model.md

The goal is not a themed Linux desktop, “Hermes in a VM,” or a daemon that may someday become an OS. Astrid is analogous to an independently bootable kernel and universal operating substrate; a distribution is analogous to a Linux distribution at a broader agent-native level. The distribution selects the signed system/service/application composition and product experience while Astrid owns boot authority, identity, resources, storage, Capsules, generations, lifecycle, and evidence.

Reference consumer outcome — outside Astrid

A consuming host may present a graphical Home rather than a launcher, terminal, infrastructure dashboard, or capability editor. Astrid does not own or render that Home; it supplies the governed substrate that makes the experience truthful and safe.

The default experience answers four questions without Astrid vocabulary:

1. Where am I? — Personal, Studio, Company, or another owned/delegated Space.
2. What is happening? — live work, services, people, agents, devices, and consequential changes.
3. What can I continue? — durable activities resume where they were left.
4. What can I ask? — Hermes and other agents collaborate inside the current Space.

Astrid exposes typed objects, state, and bounded actions. The consuming host owns presentation, personalization, layout, components, and graphical confirmation chrome. Astrid owns identity, authority validation, exact pending-operation data and challenges, receipts, and action dispatch.

The same immutable Hermes closure may serve many principals economically. Mutable process state, writable storage, credentials, caches with privacy implications, handles, budgets, lifecycle, and receipts are never shared across principals.

Locked architecture

One substrate, multiple execution providers

• Capsules remain the Astrid-native application and system-service form.
• Linux Realm is the first compatibility provider for Python, Hermes, native libraries, SQLite, subprocesses, and existing applications.
• Future native or hardware-virtualized providers implement the same application, resource, storage, lifecycle, and receipt semantics.
• Linux/POSIX concepts remain inside the Realm. They do not enter ring 0 or become Astrid authority identifiers.

Astrid does not make the agent

An agent is an ordinary application composition inside Astrid, not an operating-system singleton or kernel concept. A distribution or user may choose an existing harness such as Hermes, or compose a native agent from replaceable Capsules for a loop, model access, memory, tools, ingress, and presentation. Astrid supplies verified component identity, typed links, resource attachment, state roots, admission, lifecycle, and receipts; it does not prescribe ReAct, prompts, providers, memory architecture, tool brokers, or one canonical harness.

The same composition format supports both ends of the product spectrum:

• hackable: inspect, replace, rewire, or locally derive parts; every executable change creates a new candidate application generation and fresh admission; or
• frozen: run an exact reproducible signed closure whose parts change only through reviewed generation replacement.

The distribution chooses that policy. Mutable principal state remains separate in either mode. Hermes is one compatibility and economics workload, not “the Astrid agent” or a mandatory system service.

Share bytes, never mutable authority

Astrid installs a content-identified application closure once and admits separate instances:

shared:   verified closure, runtime image, immutable pages, and content-addressed
          artifact caches only when the provider proves no cross-principal
          content, key-set, timing, or microarchitectural disclosure
separate: principal state, writable filesystem, secrets, handles, sessions,
          budgets, network authority, lifecycle generation, audit chain

“One Hermes for all” therefore means one canonical application identity and deduplicated immutable representation, not one mutable Python process serving unrelated principals.

Host projections are not authority

Normative invariant: any UI is a host-owned projection of Astrid state and proposed actions. It is never Astrid identity, authority, consent, or policy. A Space is higher-layer presentation and navigation composition with explicit Astrid attachments, not a principal, capability set, policy realm, filesystem namespace, owner, or ambient context switch. Personalization is presentation, not policy.

The architecture must separate:

1. Astrid projection boundary — typed objects, relationships, state, eligible actions, and opaque action references;
2. host-owned experience — scenes, components, layout, density, modality, accessibility, theme, and personalization; and
3. Astrid enforcement boundary — authenticated context, action validation, dispatch, receipts, and revocation.

An object name, icon, path, label, scene node, JSON field, or receipt ID is not authority. Hosts receive opaque, scoped action handles and cannot synthesize broader operations. Astrid supplies typed pending-operation facts and a bound challenge; a trusted host owns their graphical presentation.

Entering or sharing a Space never changes the kernel-stamped principal or widens the accessible resource set. A Space attachment is a non-authoritative, generation-bound reference that must be reauthorized on every interaction. Missing or stale attachments become non-enumerating unavailable placeholders; they never silently retarget. Cross-Space drag/drop proposes a typed copy/move/share/delegation and cannot itself transfer authority.

This does not revive the retired design in #629/#630 where an in-runtime agent owned the complete TUI. It reuses the completed schema groundwork from #643/PR #644 where useful, but introduces only a host-neutral Astrid state/action boundary. astrid-rfcs PR #25 was closed unmerged as superseded by this host-owned projection direction. A presentation owner may define an A2UI-like frontend contract separately; Astrid does not become the UI framework.

Authority remains intersection-only

Every effect is admitted under exactly one host-stamped invocation or durable service lease and intersects principal, application, provider, Realm/job, portal, and budget authority. Manifests and host projection/action descriptions request operations; neither grants them.

Providers run with attenuated backing capabilities and receive scoped resource objects, never broad authority plus caller-selected identifiers. UI dispatch resolves an admitted action to a fresh stamped invocation; it does not trust principal, owner, or resource selectors supplied by the rendered payload.

Dependency graph

architecture/specification ─┬─> portable resource types ─> native admission/table
                            ├─> native boot/kernel/IPC ─> native system services
                            ├─> host-neutral state/actions ─> external consumers
                            └─> application/provider types

#1535 owner storage/mounts ─────> durable Spaces/state ───> Realm writable volume
#1535 landed ─> selected #1562 WAL/cache forward-port ───> publication economics

native boot + native system services + storage + provider contract
    ─> Linux Realm forward-port
    ─> Hermes H0/H1
    ─> supervised Hermes + tools + host-neutral projection
    ─> hostile multi-principal/economic proof

hosted Astrid deployment ───────> supported hosted claims + differential evidence

Storage is a blocker for production persistence, Realm writable state, checkpoints, rollback, cross-device continuity, and Hermes SQLite/WAL claims. It is not a blocker for the native machine contract, loader/kernel work, sealed boot bundle, protection domains, IPC, recovery reserves, portable resource types, application/provider domain types, or conformance harness construction.

Work packages

0. Adopt the architecture and freeze vocabulary

• Review and merge the two architecture specifications after refreshing their #1535/#1562 evidence snapshot. Record the exact heads, required-check state, integration conflict, and final disposition; historical green runs and stale branch snapshots are not merge evidence.
• Before Work Package 3, add the host-neutral projection/action invariant above to the universal-substrate document or a linked ADR without assigning graphical UI ownership to Astrid.
• Freeze owner/holder/provider/payer distinctions and generation vocabulary.
• Record the exact disposition of #629, #630, #643, #1317, #1365, #1535, #1562, preserved Realm draft PR #77, and its artifacts.
• Link rather than duplicate #1406 dynamic namespaces, #1297 generation enforcement, #1563 immutable artifact acquisition, merged #1487 runtime identity, existing generation-cutover and Realm work, and Oracle #24 stale-session fencing.
• State where this specification supersedes the older Dock/application plan, package-level supersedes, RFC #25, and retired frontend branches while preserving their still-valid kernel and schema primitives.
• Maintain one forbidden-design list; later implementation issues may narrow this epic but may not silently redefine it.

Exit gate: reviewers can map every new durable identity, live handle, epoch, generation, and provider to the locked resource model.

1. Boot standalone Astrid and establish machine authority

• Freeze a canonical, bounded firmware/loader handoff and one experimental x86-64 UEFI QEMU/KVM machine contract.
• Build a deterministic image in which the loader separately authenticates the Astrid kernel/bootstrap closure and the distribution-supplied immutable System Generation containing plan-bounded services and initial provider artifacts.
• Authenticate exact executed bytes, enforce W^X, bind both closures at kernel handoff, and reject stale, revoked, incompatible, or below-floor inputs.
• Bring up memory protection, domains, scheduling, IPC, timers, fault delivery, capability tables, interrupt routing, IOMMU/DMA mediation, reset/quarantine, and resources reserved for recovery.
• Keep ring 0 free of device-specific protocols and a conventional driver subsystem. Hardware becomes typed services through isolated provider Capsules or dedicated compatibility/device Realms receiving exact claims.
• Start a provider from the preloaded bundle, then prove crash, IRQ storm, DMA revocation, reset failure, stale generation, and provider replacement containment.
• Implement independently authenticated recovery, A/B System Generation selection, boot-attempt/watchdog state, monotonic revocation/security floors, and state-preserving rollback rules.
• Define one-shot first-owner provisioning without a root account, default credential, console-is-admin rule, or first-network-caller ownership.

Exit gate: an Astrid image boots without a host operating system, realizes only its signed boot plan, contains a hostile native domain and provider failure, and reaches authenticated recovery independently of the active slot.

2. Land and certify Astrid-owned storage

Dependencies: #1535 first. Rebase and forward-port only the still-required #1562 WAL/cache work afterward; do not merge or cherry-pick #1562 wholesale. The branches overlap heavily and #1535 already contains the conditional KV batch surface. Any transaction WAL must target an AstridVolume/media-backed region rather than restore PathBuf/host-file authority.

• Make AstridVolume and owner roots authoritative; host directories remain adapters/import-export sources only.
• Certify released-layout migration, crash recovery, physical reclamation, quota/accounting, and exact owner binding.
• Bind mount/storage leases to acting PrincipalUid, authority epoch, provider/lifecycle generation, immutable owner, prefix, access mode, expiry, parent lifetime, and callback authentication.
• Revoke and drain all owner leases before principal deletion/root purge; revalidate liveness at consequential commit boundaries.
• Make read-handle registration atomic/GC-visible and add durable owner-scoped retention roots for promised checkpoint/export/rollback generations.
• Extract or establish a no_std + alloc storage model/format/media contract; keep FSKit/FUSE/WinFsp, PathBuf, hosted files, Tokio, and process control in adapters.
• Provide a Realm storage provider with the semantics advertised to the guest. Hermes SQLite/WAL state must not run on a projection that cannot prove required POSIX durability and locking.
• Keep destructive compaction disabled until open-read lease registration and the GC fact snapshot share an atomic fence or a registry generation is revalidated inside the mutation fence.
• Certify enforced storage accounting and operator ceilings; user/fleet allocation policy remains #1539 and is a separate pre-production multi-tenant gate.

Exit gate: authoritative state is host-layout independent, owner deletion cannot resurrect state through a stale mount, advertised retained generations survive restart/GC, and crash suites pass on the exact merge head.

3. Add portable resource and admitted-instance contracts

Current anchors include:

• crates/astrid-core/src/identity/

• crates/astrid-capsule/src/registry/runtime_id.rs

• crates/astrid-capabilities/

• crates/astrid-storage/src/resources/

• crates/astrid-capsule-types/src/{fuel_ledger,memory_ledger}.rs

• Introduce small portable value types for resource type, rights, object generation, derivation, authority epoch, application/lifecycle/provider/root/placement generations, budget account, causal request, and error/state codes.

• Add UntrustedEnvelope -> StampedInvocation | StampedServiceInvocation conversion exactly once at authenticated ingress.

• Normalize manifest/profile/token/device/provider requests into an immutable admission decision that binds principal UID, exact manifest digest, approved-grant snapshot, provider/application generations, epochs, budgets, expiry, and causal request.

• Implement the admitted resource-table contract with opaque typed handles, attenuation, epoch validation, bounded transfer, charge/release, revocation, and reclamation. The native implementation is authoritative; the hosted implementation is a differential adapter.

• Add a transactional effect journal for receipt-required effects: durably record intent before the external effect, use provider idempotency keys where available, commit completion/failure afterward, and record explicit outcome_unknown with no replay unless status reconciliation proves safety. #1562 may store these records efficiently, but #1562 is not itself the receipt-consistency boundary.

• Preserve existing management-capability and signed-runtime-token namespaces behind shared semantics; do not perform a flag-day universal-token rewrite.

• Version future durable tokens and receipts around PrincipalUid; retain verifiability of existing alias-bearing signed bytes without reinterpretation.

Exit gate: a negative/property suite proves no amplification, no cross-principal/session replay, stale-generation rejection, scoped revocation, crash cleanup, and conservative budget ancestry.

4. Define the host-neutral state and action contract

Current reusable anchors:

• crates/astrid-capsule/src/schema_catalog.rs
• crates/astrid-build/src/wit_schema.rs
• completed issue #643

These describe payload shapes but are not yet principal-scoped object identity or action authority.

• Define internal Rust domain types first: SemanticObjectId, ProjectionRevision, typed properties/relationships, ActionDescriptor, opaque admitted ActionHandle, pending-operation/challenge data, validation errors, and bounded snapshot/update messages. Do not define components or layout.
• An ActionHandle resolves only to a host-table entry binding the canonical digest of its ActionDescriptor, projection revision, target SemanticObjectIds, typed arguments, authority delta, confirmation policy, and expiry. Host labels, icons, ordering, layout, and component state never participate in resolution.
• Bind every projection snapshot to principal UID, application/attachment identity, application/lifecycle/provider generation, authority epoch, authenticated host session or service lease, and expiry.
• Define deterministic snapshot/update ordering, reconnect/resume, stale revision rejection, bounded payloads, backpressure, and cancellation.
• Construct pending-operation data exclusively from typed host-owned state. Bind a single-use expiring challenge to the exact principal UID, authenticated device/session, request nonce, operation, typed arguments/resource, authority delta/duration, application/provider/lifecycle generations, attachment epoch, and expiry. Reject replay or drift and report success only after Astrid validation, consumption, and receipt.
• Split global type metadata from principal/application/generation-filtered projection snapshots. Reject schema collisions rather than overwrite them; knowing a topic/schema never permits enumeration, subscription, publication, invocation, or data access. Treat WIT docs as untrusted descriptive content and resource/handle-like schema fields as non-actionable.
• Prove the boundary through two structurally different non-authoritative consumers or fixtures before freezing public WIT. A graphical renderer may be one external consumer, but is not implemented or owned here.
• Only after conformance, propose capsule-visible contracts through astrid-rfcs and canonical wit.

Exit gate: independent consumers observe the same eligible objects/state and dispatch the same admitted actions; consumer-controlled presentation cannot widen authority.

5. Enable host-owned experiences

• Document how a consuming host can compose Spaces, ongoing work, collaborators/agents, owned things, and change history from the typed boundary.
• Expose proposed copy/move/share/delegation operations for direct-manipulation hosts without treating drag/drop as authority.
• Expose receipts, storage, grants, usage, health, generations, and recovery as typed inspectable state.
• Keep host preferences, component grammar, layout, accessibility implementation, and graphical shell code outside Astr