RFC: cross-worker propagation for runtime configuration (`addon_params` and role LLM config)

Summary

LightRAG has two runtime-mutable configuration surfaces — addon_params and the role LLM runtime driven by update_llm_role_config() / aupdate_llm_role_config(). Both are process-local instance state.

Under lightrag-gunicorn --workers N the app is preloaded (gunicorn_config.py:32, preload_app = True), so the LightRAG object is constructed once in the master and each worker inherits a copy-on-write copy at fork. A runtime change applied in one worker is invisible to the other N-1 workers, and no mechanism exists to propagate it.

This RFC proposes a versioned cross-worker runtime-configuration channel so that a change made in any worker becomes the effective configuration in every worker, with bounded staleness, atomic per-worker application, and without moving credentials into shared storage.

Current behavior

addon_params

• addon_params is an InitVar; the live store is _addon_params: ObservableAddonParams (lightrag/lightrag.py:939), exposed through a property attached after class creation (lightrag/lightrag.py:6984).
• Any top-level mutation calls _on_addon_params_changed (lightrag/lightrag.py:1024) → sets _addon_params_dirty → the derived caches (_resolved_summary_language, _entity_extraction_prompt_profile) are recomputed by _ensure_addon_params_cache() on the next _build_global_config() (lightrag/lightrag.py:1217, :1225).
• Only top-level keys are observed. Nested in-place mutation is corrected opportunistically at the next enqueue by resolve_chunk_options() (lightrag/parser/routing.py:679-693).
• There is no REST write path. The server only seeds {"language": args.summary_language} at construction (lightrag/api/lightrag_server.py:2444).
• Per-document chunk_options are frozen into full_docs at enqueue, so document processing is already independent of which worker picks the document up. That invariant is what keeps today's divergence from corrupting ingestion.

Role LLM config

• The role_llm_configs constructor field is read exactly once, in __post_init__ (lightrag/lightrag.py:1566); the live state is _role_llm_states (lightrag/lightrag.py:1581). Mutating rag.role_llm_configs at runtime silently does nothing.
• update_llm_role_config() / aupdate_llm_role_config() (lightrag/llm_roles.py:360, :395) delegate to _apply_llm_role_config_update() (lightrag/llm_roles.py:276), which snapshots state, applies the change, rebuilds the role wrapper, and rolls back on any failure. The retired wrapper's queue is drained in the background (sync) or awaited (async).
• The API server already registers a role builder (lightrag/api/lightrag_server.py:2530), so binding/model/host hot-swap works in-process — but no route calls the update methods.

Shared storage

• _shared_dicts, _update_flags, the ingress hub and the keyed-lock tables are Manager-backed and genuinely cross-process (lightrag/kg/shared_storage.py:1599-1620). Neither configuration surface is in that set.
• _global_concurrency_limits is derived from CLI args before fork (lightrag/api/run_with_gunicorn.py:32, :286-292) and stored as a plain module dict (lightrag/kg/shared_storage.py:1579), read locally by _acquire_global_slot() (lightrag/kg/shared_storage.py:4218).

Problems

1. Divergence is silent. Nothing detects or reports that workers disagree about the active model, language, or chunker defaults.
2. The LLM response cache splits. binding / model / host participate in the cache key (lightrag/lightrag.py:1263, lightrag/utils.py:839) and llm_response_cache is shared across workers. A model change in one worker makes that worker write and read a different key space than its peers. Not corruption — the partitioning is deliberate — but hit rate collapses with no visible cause.
3. max_async changes are one-way. The cross-worker gate reads the pre-fork read-only limit, so raising max_async at runtime only enlarges the local queue while the gate still throttles at the boot value; lowering it takes effect because the local limit is the stricter of the two.
4. Observability contradicts itself. /health reports rag.get_llm_role_config() from whichever worker served the request (lightrag/api/lightrag_server.py:2986), while get_llm_queue_status() aggregates published snapshots across all workers (lightrag/llm_roles.py:547-551). After a divergence, two refreshes of the same page can disagree.
5. It blocks the runtime-configuration REST surface that the already-registered role builder anticipates. Adding such an endpoint today would produce a control plane that reconfigures one worker at random.

Goals

• A change published from any worker becomes effective in every worker within a bounded, documented staleness window.
• Concurrent publishes from different workers are serialized and totally ordered; a slow worker can never apply an older configuration over a newer one.
• Application in each worker is atomic: it either fully succeeds or leaves that worker on its previous configuration, with a loud error.
• Credentials never enter shared storage; get_llm_role_config()'s "no escape hatch" property for secrets is preserved.
• max_async becomes meaningful at runtime in multi-worker mode, i.e. the cross-worker gate follows the published value.
• Single-process and SDK semantics are unchanged; the shared path activates only when workers > 1.
• Already-enqueued documents keep their frozen chunk_options and process_options. Configuration propagation must not retroactively change in-flight work.

Non-goals

• Persisting runtime configuration across restarts. Environment variables remain the boot authority; published runtime state dies with the Manager.
• Cross-host / multi-node synchronization. The multiprocessing.Manager is single-host by construction.
• Runtime reconfiguration of embedding_func / rerank_model_func. They use the same concurrency_group gate and would follow the same mechanism, but they have no runtime-update API today; deliberately deferred.
• Designing the REST control-plane endpoint itself (see Open questions for the phasing).

Proposed design

1. A versioned runtime-configuration namespace

Introduce a shared namespace, e.g. get_namespace_data("runtime_config", workspace=...) (lightrag/kg/shared_storage.py:3669), holding:

{
  "version": <monotonic int>,
  "updated_by_pid": <int>,
  "updated_at": <float>,
  "addon_params": { ...publishable keys... },
  "role_llm": { "<role>": { ...non-secret fields... } },
}

Writes go through get_storage_keyed_lock(...) on that namespace: read → validate → bump version → write → set_all_update_flags("runtime_config") (lightrag/kg/shared_storage.py:3565). Each worker holds its own flag from get_update_flag (:3530) plus a locally cached applied_version, and never applies a payload whose version is not greater than the one it already holds.

The precedent to follow is the storage_updated flag pattern in lightrag/kg/json_kv_impl.py, not a bespoke mechanism.

2. Explicit publish, never implicit mutation

Publishing must be an explicit call, not a side effect of mutating addon_params. Two reasons:

• resolve_chunk_options() corrects the live chunker config in place on the enqueue path; if mutation auto-published, ordinary ingestion would emit config broadcasts.
• Nested mutation is not observable at all, so an implicit contract would be honest for some writes and not others.

Proposed surface:

• await rag.apublish_runtime_config() — publish the current local addon_params (publishable subset) and role LLM config as a new version.
• aupdate_llm_role_config(..., publish=True) — opt-in publish as part of the update, so the common case is one call.
• Workers pull and apply; they never push implicitly.

3. Bounded-staleness apply points

Checking the shared flag inside _build_global_config() is the tempting choice — it is the existing convergence point for addon_params — but it runs per operation and a Manager proxy read is an IPC round trip. Proposed instead:

• Primary: at the HTTP request boundary (a FastAPI dependency) and at the per-document boundary in the pipeline. Both are coarse and already do IPC.
• Safety net: inside _ensure_addon_params_cache(), guarded by a short debounce (e.g. check at most once per N ms), so a long-running SDK loop still converges.
• The resulting staleness window must be documented as a contract, not left implicit.

4. Secrets stay out of shared state

Only non-secret fields are published: binding, model, host, max_async, timeout, and a scrubbed provider_options. api_key and any field matching _SECRET_MARKERS (lightrag/llm_roles.py:100-118) is never written to the shared namespace.

Each worker resolves the credential locally through its registered role builder from its own environment, keyed by the published binding / host. A worker that cannot resolve a credential for the published binding must fail the apply loudly and stay on its previous configuration rather than fall back to an unauthenticated call.

Publishing a credential reference (env var name or secret id) is an acceptable alternative; publishing the value is not.

5. Make the cross-worker concurrency gate dynamic

_global_concurrency_limits must move from a pre-fork read-only module dict to shared state (or be re-read from the runtime-config namespace by get_global_concurrency_limit()), otherwise a published max_async increase remains inert. Two details:

• _acquire_global_slot() reads the limit per acquisition (lightrag/kg/shared_storage.py:4218), so a changed number takes effect naturally once the read is shared.
• use_global_limit is resolved once per wrapper and cached (lightrag/utils.py:1183-1199). Today every role update rebuilds the wrapper, which re-resolves it; if a config-only path is ever added that does not rebuild, a group transitioning unlimited → limited would be missed. This needs an explicit test.

6. Atomic per-worker apply

Applying a published version in a worker reuses _apply_llm_role_config_update()'s snapshot/rollback discipline, extended to cover the whole batch: if any role in the payload fails to build, the worker rolls every role in that payload back and keeps applied_version unchanged, so the next flag check retries. Retired wrappers are drained through the existing _schedule_retired_llm_queue_cleanup path; N workers draining concurrently must not strand global slot leases (the lease TTL and reconcile pass should already cover this — it needs a test, not new code).

For addon_params, the apply path is _replace_addon_params(..., mark_dirty=True) followed by _apply_chunk_size_overlay() — i.e. the existing setter, not a new code path. entity_type_prompt_file is published as a path; each worker re-resolves and re-validates it, and a worker whose validation fails must report the failure rather than silently continue serving the previous profile.

Acceptance criteria

• A addon_params publish in one worker becomes effective in every other worker within the documented staleness window, for language, entity_type_prompt_file, entity_types_guidance, and chunker.
• A role LLM publish (binding / model / host / max_async / timeout / model_kwargs) becomes effective in every other worker within the same window.
• Two concurrent publishes from different workers are serialized; the final state in every worker equals the higher-versioned payload, and no worker ends up on the lower one.
• A worker that fails to apply a payload (unresolvable binding, invalid prompt profile, builder raising) keeps its previous configuration in full, logs an actionable error, and retries on the next check instead of silently diverging.
• No secret-marked field ever appears in the shared namespace. A test asserts the published payload against _SECRET_MARKERS, mirroring test_get_llm_role_config_has_no_secret_escape_hatch.
• A published max_async increase actually raises achieved concurrency in multi-worker mode (the gate follows), and a decrease still tightens.
• A group transitioning from unlimited to limited is honoured by wrappers created before the transition, or the limitation is documented and asserted.
• Documents enqueued before a publish still process with their frozen chunk_options; only subsequently enqueued documents see the new chunker defaults.
• /health and get_llm_queue_status() agree after convergence; while a publish is in flight, the response identifies which version each worker is on rather than presenting a single worker's view as global.
• Single-worker / SDK behaviour is byte-for-byte unchanged when workers == 1 — no Manager access, no new IPC, no behavioural change to rag.addon_params[...] = ....
• rag.role_llm_configs assignment at runtime either works or raises; it must stop being a silent no-op.
• A design contract document is added under docs/design/ (alongside PipelineConcurrencyContract.md and PurgeRecoveryContract.md) and referenced from AGENTS.md.
• English and Chinese documentation are updated together, including the addon_params["chunker"] runtime-mutation sections of docs/FileProcessingPipeline{,-zh}.md, which currently state that server deployments need a restart.

Test plan

Real forking is not required. The established pattern is initialize_share_data(2) in-process (see tests/kg/test_shared_storage_rpc_counts.py:237, tests/kg/test_keyed_lock_dead_worker_recovery.py), which starts the Manager and exercises the multiprocess code paths. Two LightRAG instances constructed in one test process against that shared state model two workers and can prove publish → flag → apply end to end, plus the version-ordering and rollback cases.

Placement per the repo layout: tests/llm/ for the role-config propagation, tests/kg/ for the shared-namespace and gate mechanics, tests/chunker/ for the chunk_options freeze invariant.

Open questions

• Phasing. Should this land as propagation-only (SDK-triggered publish, no endpoint), with the REST control plane as a follow-up issue? A propagation layer with no trigger is untestable in production; an endpoint without propagation is actively harmful. Suggested order: propagation + SDK publish first, endpoint second, gated on this being merged.
• Staleness window. What is the right bound — per request, per document, or a fixed TTL? What is the acceptable IPC budget per request for the flag check?
• Should addon_params and role LLM config share one version counter, or carry independent ones? One counter is simpler to reason about; two avoid waking role wrappers for a language change.
• Workspace scoping. Is runtime config per workspace or per process? Role LLM config is arguably process-wide, while addon_params is instance-scoped. This interacts with #3631 (Server-side Multi-Workspace Phase 1) and should be settled against that design rather than in isolation.
• Should a role model change interact with the LLM cache at all? The identity already partitions keys, so nothing needs invalidating — but should the old key space be reported as garbage for the maintenance GC in #3833?
• entity_type_prompt_file divergence. Should the file's content hash be published alongside its path, so a worker reading a different on-disk version is detected rather than silently serving a different prompt profile?