[FEAT]: General extension hooks for custom document processing

What would you like to see?

I would like to propose a general plugin/extension mechanism for selected AnythingLLM processing points, starting with document processing before embedding.

This proposal grew out of #6364, but intentionally proposes a different solution.

In #6364 I proposed structure-aware / per-heading chunking for SOP-style documents. That issue was closed as not planned, and I think that actually highlights the broader problem more clearly: specialized processing strategies like that probably should not all become first-class AnythingLLM core features.

The requirement itself still exists for deployments with different RAG constraints, though.

Rather than upstreaming another chunking algorithm, I would like to explore whether AnythingLLM could expose a supported extension seam so users can implement specialized processing behavior outside core.

In other words:

#6364 proposed the behavior. This issue proposes the seam that would allow deployments to own behavior like that themselves.

Conceptually, the ingestion pipeline could expose an optional hook somewhere before documents are embedded:

Collector
   ↓
Parsed document
   ↓
[ optional extension hook ]
   ↓
Normalized documents / chunks
   ↓
Embedding
   ↓
Vector DB

The default path would remain exactly as it is today.

If no extension is configured, AnythingLLM would continue using its existing document-processing and splitting behavior.

If an extension is configured, it could receive the parsed document and return the normalized units that should continue through the existing embedding pipeline.

For example, the interface could conceptually look something like:

processDocument({
  document,
  context,
  options,
}) => Promise<{
  documents: Array<{
    pageContent: string,
    metadata?: object,
  }>
}>

The exact API is not important to me. The important part is that core defines a stable input/output contract and lifecycle, while the processing algorithm remains outside core.

A plugin could then implement things such as:

• structure-aware or topic-aware chunking;
• semantic chunking;
• one-chunk-per-Q&A-pair processing;
• table-aware processing;
• metadata enrichment;
• filtering or normalization;
• custom handling for specific document formats;
• other deployment-specific preprocessing strategies.

None of those behaviors would need to become a core AnythingLLM feature.

Why this matters for our use case

Our AnythingLLM deployment is used as the RAG layer behind a voice-call assistant rather than a traditional chat interface.

That creates somewhat different retrieval constraints.

In chat, additional context is often relatively cheap. The user can also inspect previous messages, ask follow-up questions, or recover from a weak answer more naturally.

In a voice interaction, unnecessary retrieved context has a higher operational cost:

• more context can increase end-to-end response latency;
• irrelevant neighbouring content can make the generated answer less precise;
• the caller cannot simply scroll back and inspect the source of an incorrect spoken response;
• incomplete retrieval units can be problematic when procedures, disclaimers, or SOP steps depend on surrounding context.

Because of that, we have been experimenting with smaller but semantically complete retrieval units.

One of our POCs was structure/topic-aware splitting, where a complete SOP section is kept together and labeled with its section hierarchy instead of being split primarily by character count.

That approach is described in #6364.

It worked well enough for our use case to justify continuing the experiment, but I also understand why that specific strategy does not necessarily belong in AnythingLLM core.

It is only one strategy.

Another deployment may prefer semantic chunks, Q&A pairs, table rows, code symbols, or something completely different.

That is what led us to the extension approach instead.

My specific requirement

My immediate requirement is to run a custom document-splitting strategy that groups content by topic or logical section rather than primarily by fixed chunk size.

This comes from our current RAG POC, where AnythingLLM is used behind a voice agent for customer-service calls. In that environment, retrieval quality and latency are especially sensitive: larger chunks may introduce unrelated neighbouring content, while smaller size-based chunks can break a procedure, SOP, disclaimer, or multi-step instruction into incomplete pieces.

What we want to experiment with is keeping each logical topic or section together as one retrieval unit, while still keeping the resulting context small enough for a low-latency voice interaction.

I previously proposed that behavior directly in #6364. Since that specific strategy is too specialized to belong in core, the requirement now is not for AnythingLLM to implement topic-based splitting itself.

The requirement is for AnythingLLM to provide a supported extension point where we can implement that behavior ourselves.

Why not just preprocess outside AnythingLLM?

We also tried pre-processing the source externally and uploading the resulting sections individually through the API.

Technically, that works.

The problem is that the chunks then become separate AnythingLLM documents, even though logically they belong to one source document.

For one 361-page PDF, our preprocessing produced 601 sections.

That meant one logical file became 601 AnythingLLM documents.

In our test:

• attaching those documents to a workspace took approximately 356 seconds;
• removing them took approximately 20 seconds;
• the document picker contained 601 entries representing one original PDF.

So the bottleneck was no longer the splitting algorithm itself. It became the per-document lifecycle inside AnythingLLM.

For this use case, the original PDF should remain one logical document from the user's perspective. Only the transformation between the parsed document and the vectors needs to be customizable.

Why an extension/plugin mechanism instead of another core feature?

Without an extension point, specialized requirements currently tend toward one of three options:

1. add each processing strategy to AnythingLLM core;
2. preprocess externally and model every resulting unit as a separate document;
3. maintain a downstream fork and patch the ingestion pipeline.

None of those seems ideal for deployment-specific behavior.

I am completely fine with AnythingLLM not owning or maintaining our custom processing strategy.

What I am looking for is a supported place to plug that behavior in.

A general extension mechanism would let core remain opinionated and stable while users and the community own specialized behavior themselves.

AnythingLLM already has a precedent for community-distributed executable extensions through Agent Skills.

I am not suggesting that document-processing extensions must necessarily use the exact same runtime, permissions, or implementation details as Agent Skills.

However, the existing plugin/Hub model suggests that community-owned functionality outside core is already a concept that fits the project.

General extension model

I would prefer this not to become a dedicated text-splitter plugin type.

Text splitting is only the immediate use case that exposed the limitation.

A more flexible model would be a general extension/plugin type whose manifest declares which supported hook or capability it implements.

For example, conceptually:

{
  "type": "extension",
  "hooks": ["document:pre-embed"]
}

The first supported hook could be limited to document processing before embedding.

Later, if there are real use cases for additional extension points, AnythingLLM could expose more hooks without introducing a new plugin type for every category of customization.

For example:

document:post-parse
document:pre-embed
retrieval:post-search
...

I am not proposing that all of those hooks be added now.

The initial scope could remain deliberately small: one well-defined document-processing hook.

The important architectural point is that plugins would be defined by the capabilities/hooks they implement rather than by hard-coded algorithm categories such as text-splitter.

Expected behavior

The existing AnythingLLM behavior should remain the default and require no changes for existing users.

Conceptually:

No extension configured
        ↓
Current AnythingLLM processing
        ↓
Embedding

With an extension:

Extension configured
        ↓
Parsed document
        ↓
Extension processing
        ↓
Normalized documents/chunks
        ↓
Existing embedding pipeline

The eight vector database integrations, retrieval logic, and other downstream components should not need to care whether the normalized input came from the default processor or an extension.

Ideally, the extension would operate before that boundary and return data in the same normalized shape expected by the rest of the pipeline.

Structural metadata

One related question is how much of the collected document should be exposed to the extension.

For format-aware processors, pageContent alone may not be enough.

For example, a PDF processor may eventually want to use information such as heading hierarchy, page numbers, font sizes, table boundaries, or other structural hints extracted by the collector.

My preference would be for the extension to receive the document metadata object as well, with collectors free to attach optional structural information.

Core would not need to understand or depend on those fields.

Extensions that understand them could use them, while others could ignore them.

That would keep the contract generic without forcing format-specific processing rules into core.

Scope

For an initial implementation, I think the scope could stay relatively small:

• a general extension/plugin type;
• one document-processing hook before embedding;
• a stable input/output contract;
• a setting for selecting an installed processor/extension if applicable;
• the current processing path as the default;
• no new chunking algorithms in core;
• no requirement for per-workspace configuration initially;
• no changes required by vector DB providers.

Per-workspace configuration or additional hooks could be considered separately later if there is demand.

Contribution

If this architectural direction is something the project would accept, I would be interested in contributing the implementation.

I am also happy to adapt the design to fit the existing plugin, Community Hub, or ingestion abstractions rather than requiring the exact API shape suggested above.

The main thing I am trying to avoid is putting deployment-specific chunking/processing logic into AnythingLLM core while still having a supported way to customize that part of the pipeline.