RFC: Add C strategy for Server-injected custom chunking callbacks

Summary

Add an explicit C process-options chunking strategy as a compatibility bridge for the existing instance-level LightRAG.chunking_func extension point.

The current Server does not inject a user-defined chunking_func, while API ingestion normally selects F/R/V/P and therefore bypasses the legacy callback. This RFC defines a safe, observable route for trusted Server-side customization without changing the semantics of F.

Related to #3603. The immediate observability fix is #3662.

Status (updated 2026-09-16)

• §1 C selector and §3 observable fallback — implemented in #3665.
• §2 Server registry injection — spun out to #3868 and implemented in #3869: one instance-wide callback, selected at startup by CUSTOM_CHUNKER from an allowlisted registry. Registry discussion belongs on #3868, not here.
• §4 context-aware callback contract — still open. It is the main piece of work this issue now tracks.
• C(name=...) — closed, will not be implemented. #3869 settled that a deployment has exactly one active custom chunker, so a per-document name has no range of values to select from: it would either restate CUSTOM_CHUNKER or name something that can only fail to resolve. Reviving it means revisiting #3869's injection model, not adding a hint parameter.
• Filename hints need no further work. Bare C, [engine-C…], combination with i/t/e/!, C(chunk_ts=…,chunk_ol=…) parameter blocks and LIGHTRAG_PARSER rules all select C today, and both the English and Chinese pipeline documents describe it.
• Why this issue stays open after §4 lands: the strategy-agnostic transform that composes with R/V/P — the part of #3603 that C and the registry do not deliver.

Goals

• Preserve the existing chunking_func mechanism for deployments that need it before a full named-chunker contract is available.
• Make custom chunking an explicit per-document choice: process_options=C.
• Let LightRAG Server resolve a trusted registered callback during startup and inject it into the constructed LightRAG instance.
• Give a context-aware callback access to the complete parser sidecar through full_docs.sidecar_location.
• Keep existing six-argument callbacks working unchanged.
• Make a missing Server injection visible rather than silently behaving as a custom chunker.

Proposed behavior

1. C selector

Extend the chunking-selector set from F/R/V/P to F/R/V/P/C.

• F/R/V/P retain their present meanings and dispatch behavior.
• C means “use the configured instance chunking_func.”
• C uses the fixed-token chunk-options snapshot for compatibility with the old callback arguments: separator, separator-only flag, overlap, and token size.
• C is accepted consistently by SDK enqueue, filename hints / parser rules, scan/upload routes, and the text-insert API (as a custom chunking strategy).
• The persisted selector remains authoritative on retry and resume.

2. Server registry injection

At trusted Server startup, resolve an operator-selected callback from a chunker registry and pass it as chunking_func= when constructing LightRAG.

• Registry keys are allowlisted, pre-registered names; HTTP request input never supplies Python code, module paths, or import paths.
• Registry lookup and callback construction happen only at startup, in the trusted server process.
• The selected callback should have a stable identifier/version recorded with the document's processing snapshot, so retries and post-restart processing can detect an unavailable or incompatible deployment rather than silently changing behavior.
• Superseded — see Status above. This section moved to #3868 and landed in #3869 as a single instance-wide injection. A named form such as C(name=...) will not be added: with one active custom chunker per deployment the name has nothing to select.

3. Required fallback when no callback was injected

If a document selects C but the Server has not injected a non-default chunking_func, and the document was admitted through a path with no caller present to receive a rejection (see acceptance criterion 1):

1. dispatch exactly as explicit F (the built-in fixed-token file chunker);
2. emit a WARNING that includes the document ID and states that C was unavailable and fell back to F;
3. persist observability metadata that distinguishes custom_chunking_fallback_fixed_token from successful custom chunking.

The warning is emitted once per document processing attempt, not once per output chunk. Retrying or reprocessing the document emits it again.

This fallback is intentionally explicit and observable. It must never be a silent conversion of C to F.

4. Context-aware callback contract

Do not break the existing six-argument callback signature. Add an opt-in context form, for example:

@dataclass(frozen=True, kw_only=True)
class ChunkingContext:
    doc_id: str
    file_path: str
    sidecar_location: str | None
    parse_format: str
    parse_engine: str | None
    process_options: str


@accepts_chunking_context
def chunker(
    tokenizer, content, split_by_character,
    split_by_character_only, overlap, size, *,
    context: ChunkingContext,
):
    ...

• Untagged legacy callbacks receive exactly the existing six arguments.
• Tagged / explicitly declared callbacks also receive the keyword-only context. Since #3869, a registry plugin declares this through its ChunkerSpec, so the declaration is resolved at startup rather than at document time; a decorator remains the declaration for constructor-supplied SDK callbacks.
• Do not implement compatibility by calling with context=, catching TypeError, then retrying: an internal callback bug could be misclassified and the callback could run twice.
• sidecar_location, rather than blocks_path, is the durable full_docs authority for the entire *.parsed/ directory. Custom chunkers should resolve it through a sidecar resolver, not cast a file:// URI directly to Path; future remote URI schemes must fail clearly or be handled by their resolver.
• Do not add document-level source_file to this contract. RFC LR2 deprecates it. file_path is the canonical logical display/citation name; protected internal source locators remain outside ordinary callback metadata.
• document_metadata is deliberately absent from the first implementation. Its only producer is the durable per-document metadata channel proposed in #3706, which is still open, and doc_status.metadata is internal pipeline bookkeeping that must not become a public callback surface. The field is added — together with a document as-of date — when #3706 lands; ChunkingContext is declared kw_only=True so fields may be added without breaking existing callbacks.

Compatibility and safety constraints

• Existing no-selector SDK behavior continues to call chunking_func via its legacy path.
• A custom callback must not receive private built-in-only arguments such as _emit_source_span.
• The custom path remains ineligible for built-in sidecar backfill unless the callback contract explicitly establishes compatible source spans.
• Callback exceptions fail the document with actionable context; they do not trigger an automatic fallback to F.
• sidecar_location may be absent, unknown, or non-local. The callback must tolerate None; it cannot rely on arbitrary physical paths.
• No callback code is accepted from API payloads or environment import strings.

Acceptance criteria

• C acceptance is split by path, because a deployment with no injected callback must not accept a document it will silently chunk with the wrong strategy:
  • Request paths with a caller present to see the error (/documents/text, upload with explicit options): C is parsed and validated like F/R/V/P, but rejected with a 4xx when the instance's chunking_func is still the built-in default. The document is never enqueued.
  • Paths with no caller present (already-persisted rows, scan-driven filename hints, post-restart reprocessing): C is accepted and persisted exactly like F/R/V/P, and an unavailable callback takes the observable fallback in section 3.
  • The persisted selector is written and read identically on both paths — the split governs admission only, never storage shape.
• With a Server-injected custom callback, C invokes it while F/R/V/P never do.
• Existing six-argument callbacks work without changes.
• Context-aware callbacks receive the correct doc_id, logical file_path, durable sidecar_location, parse fields, and validated document metadata.
• A C document that reached processing with no injected non-default callback — i.e. admitted through a no-caller path — executes the explicit-F algorithm, logs exactly one fallback warning per processing attempt, records the fallback method, and remains eligible for sidecar backfill because the built-in fixed-token chunker is what actually ran.
• No-selector legacy behavior remains unchanged.
• Retry/restart behavior uses the persisted selector and detects a changed/missing configured callback identity (recorded as an explicitly non-authoritative observation; it warns, it never gates).
• Async callback behavior, callback failures, and absent/non-local sidecars are covered. (Async and failure paths landed with #3665; the sidecar cases arrive with §4.)
• Server configuration/docs, API schemas, filename-hint documentation, and English/Chinese pipeline documentation are updated together. (Done for §1–§3; §4 repeats it.)

Open questions

All four are now answered; kept here as a decision record.

• What trusted Server configuration selects the registered callback key? One key naming an allowlisted registered name, never an import path — specified on #3868, implemented in #3869.
• What is the minimal stable callback identifier/version format to persist for restart safety? Recorded as {name, version, authoritative: false}, explicitly non-authoritative; it never gates a decision.
• Should C later mean the default configured callback only, while C(name=...) selects a named registered callback? No. C means the single configured callback. C(name=...) is closed — see Status.
• Should the initial fallback warning be rate-limited at the logger layer while preserving the per-attempt event/metadata record? No. Per-attempt warnings are never collapsed across a batch.