[BUG] A retained lazy-bound App prevents sequential runtime restart and reuses closed lifespan resources

Describe the bug

An App bound to the default LazyEnvironment caches (Environment, core.App) after its first operation. LazyEnvironment.stop() clears its own environment reference and exits the lifespan, but does not invalidate the app's cache.

Two symptoms are verified on main at 3b4e54c3, with operations completed before stopping:

1. Restarting with coco.start() or coco.runtime() at the same db_path raises:

   复制

   RuntimeError: environment already open in this program; close it to be able to open it again with different options

   The retained app keeps the old heed/LMDB environment alive. Identical settings also fail: heed rejects an already-open canonical path, despite the error wording about different options. The attempted restart enters the lifespan, fails while opening storage, and unwinds cleanup.

2. Updating the retained app after coco.stop(), without explicitly restarting, skips lifespan setup and uses the old context provider. Resources closed by lifespan cleanup remain accessible through that provider, so the update can fail with errors such as pool is closed.

Module scope is not required. Any app retained across lifecycle cycles is affected; a module-level app is simply a common example. Existing default-environment tests cover initial startup and one lifecycle per app, not reuse of the same app across cycles.

To reproduce

import asyncio
import pathlib
import tempfile

import cocoindex as coco

DB = pathlib.Path(tempfile.mkdtemp()) / "cocoindex.db"


@coco.lifespan
async def _lifespan(builder):
    builder.settings.db_path = DB
    yield


@coco.fn
async def main_fn() -> str:
    return "hi"


app = coco.App(coco.AppConfig(name="svc"), main_fn)


async def run():
    async with coco.runtime():
        print("cycle1:", await app.update())
    async with coco.runtime():
        print("cycle2:", await app.update())


asyncio.run(run())

Output:

cycle1: hi
Traceback (most recent call last):
  ...
  File "python/cocoindex/_internal/environment.py", line 239, in __init__
    self._core_env = core.Environment(
RuntimeError: environment already open in this program; close it to be able to open it again with different options

The sync form (with coco.runtime(): app.update_blocking() twice) fails identically. In an isolated control, deleting the old app and collecting garbage, then constructing a replacement app, allows the second cycle to succeed.

A separate probe using a non-memoized function and a lifespan-managed resource with a closed-state marker produced:

update #1: generation 1 | lifespan enters=1 exits=0
after stop:               lifespan enters=1 exits=1
update #2: RuntimeError: review resource is closed
                          lifespan enters=1 exits=1

This verifies stale resource access without requiring a database connector.

Expected behavior and scope

After app operations have completed and coco.stop() has finished, retaining apps bound to that lazy environment should not by itself prevent a subsequent lifecycle at the same database path. A following explicit start, or an update/drop that implicitly starts the environment, should use a fresh lifespan and context provider.

This issue proposes support for sequential lifecycle cycles. It does not promise unconditional storage release while other objects retain the concrete environment, or safe shutdown concurrent with running operations.

The app docs describe automatic initial setup and explicit setup/cleanup, but do not explicitly define restart under retained references or concurrent operations. Document the supported sequential restart contract as part of the fix.

Root cause and execution path

References below are against 3b4e54c3:

• python/cocoindex/_internal/api.py:700-724: runtime() delegates entry/exit to start/stop.
• python/cocoindex/_internal/app.py:248-273: both _get_core_env_app() and _get_core_env_app_sync() return the cached tuple without consulting the lazy environment. _ensure_core_env_app() installs the tuple without checking that the environment is still current.
• python/cocoindex/_internal/app.py:300-318, :350-370, and :376-407: update and drop paths obtain the cached environment and pass its context provider into the core.
• python/cocoindex/_internal/environment.py:431-441: LazyEnvironment.stop() clears _env and _exit_stack, then awaits cleanup outside the lifecycle lock. It does not consult EnvironmentInfo.get_apps() (:158-161).
• python/cocoindex/_internal/context_keys.py:254-272: ContextProvider.get() returns stored values; aclose() closes resources without clearing those values or marking the provider closed.
• rust/py/src/app.rs:232-248, rust/core/src/engine/app.rs:92-105, and rust/core/src/engine/context.rs:35-38: the cached core.App also retains the Rust environment through its component's AppContext.
• rust/core/src/engine/environment.rs:17-39 and rust/core/src/state_store/storage.rs:122-125: the Rust environment owns storage containing the heed environment. Storage::new() opens <db_path>/mdb (:239-265).

The app cache assumes a concrete environment lasts as long as the app, while LazyEnvironment permits its lifespan to end earlier. No other layer reconciles those lifetimes. Both elements of the cached tuple need to be released.

Proposed direction

Keep the change private to Python lifecycle/cache management:

1. Add a lock-protected app cache release operation in app.py that clears the entire _core_env_app tuple.
2. In LazyEnvironment.stop(), snapshot registered apps and invalidate apps actually bound to that lazy environment as part of detaching the current environment. Keep app registration intact.
3. Coordinate cache installation with invalidation in both async and sync acquisition paths so an obsolete environment cannot be installed after invalidation. A separate unchecked identity comparison is insufficient. Release stale local environment references before retrying startup, otherwise the retry can prevent its own LMDB reopen. Cover the sync/threaded interleaving explicitly.
4. Clarify the supported lifecycle contract in docs/src/content/docs/programming_guide/app.mdx.

Do not defer all invalidation until the next app operation: an explicit start can fail before any app operation gets that opportunity. Do not call app.drop() or delete storage to release runtime references. Preserve persisted memoization, target state, app names, arguments, and configuration; reconstruct only transient core objects. No schema or cache-format bump is expected.

Registry membership alone is not sufficient to select apps. A concrete environment created by LazyEnvironment receives the lazy environment's _info (environment.py:403-410). Therefore an app explicitly bound to await coco.default_env() shares that registry too. Restricting invalidation to apps whose configured _environment is self preserves explicit bindings; the lifecycle contract for those explicit bindings needs clarification below.

The cache-release direction addresses the verified sequential bug. Its complete synchronization and regression behavior remain to be implemented and tested; the review did not implement a fix.

Tests

Existing tests in test_default_env.py, test_default_env_async.py, test_lazy_environment_lock.py, and test_app_drop.py all pass (19 tests). None deliberately asserts the buggy behavior. Despite its name, test_lazy_environment_lock.py currently constructs explicit environments rather than exercising default-environment restart or lock contention.

Add or extend coverage in python/tests/core/:

• test_default_env.py and test_default_env_async.py: retain the same app across three consecutive runtime cycles at the same path; assert one lifespan entry/exit per cycle. No explicit garbage collection should be needed between successful cycles.
• Start → update → stop → update, in both sync and async forms: a non-memoized function must observe a new resource generation that is still open.
• Multiple registered lazy-bound apps, including an app never initialized: all initialized caches release, and uninitialized apps remain usable. Also cover no apps, stop before start, and repeated stop.
• Drop after stop, sync and async: declare real target state and verify removal using fresh lifespan resources. Preserve the existing explicit-environment drop tests.
• Independent explicit-environment apps remain unchanged by default stop. Separately cover explicit binding to await coco.default_env() once its contract is decided.
• Restart preserves persisted memoization and target ownership: unchanged work remains reusable, and existing targets are reconciled without duplication. Changing the database path between cycles must not leave a lazy-bound app using the old database.
• Failure recovery: teardown raises, or startup fails with missing/invalid settings; caches must not retain the stopped environment, and a corrected subsequent startup must work. Preserve existing validation errors.
• Deterministic cache-install/stop interleaving, including the sync/threaded path. Use synchronization barriers rather than timing-based sleeps.
• Registration compatibility: retained apps remain registered, and a second live app with the same name in the same environment remains rejected.

Risks / open questions

• Other retained references: retaining await coco.default_env() or just its context provider prevents restart even when no app has run; both were reproduced. The provider holds core.Environment via set_core_env() (context_keys.py:160-175). App cache invalidation does not address these owners. Unconditional storage release would require a broader lifecycle design.
• Explicit binding to the default concrete environment: decide whether such an app remains bound to the old environment, becomes invalid after stop, or participates in restart. Independent explicit environments should remain unaffected. The proposed binding check preserves current explicit ownership rather than silently rebinding it.
• Concurrent operations: ordinary in-flight updates and drops, as well as live updates, can retain core component/environment state. Stop currently does not drain them. This issue's sequential restart guarantee requires operations to finish first; broader shutdown/cancellation semantics are separate work.
• Overlapping cleanup/startup: stop releases its lifecycle lock before awaiting cleanup. Another start can overlap teardown even after app caches are invalidated. Fixing cache installation does not by itself serialize the complete lifecycle. Do not claim concurrent restart support without resolving this ordering.
• Memoized results: restarting should rebuild lifespan resources without indiscriminately invalidating stored results. Resource-freshness tests must execute non-memoized code so a memo hit cannot hide stale-provider access.

Related and verification baseline

• #1963's local commit (a623ddf2) records the CLI __main__ loading problem and its environment-already-open error. Loading under the file's module name avoids that trigger; it does not fix app cache lifetime.
• #1495 introduced LazyEnvironment and the registry; #1708 introduced the app cache.
• python/tests/core/test_context_tracked_state_validation.py:18-23 explains why its tests reuse a single environment while it remains alive.
• Reviewed against main at 3b4e54c3. The relevant cache/stop behavior remains present, although these files have changed since the original issue baseline (459c9156).

Classification: confirmed bug. Implement the sequential lazy-bound-app fix with the scope above; resolve any stronger ownership or concurrent-shutdown guarantees separately before expanding it.

I'd like to work on this if the direction sounds right.