#6142 [FEAT]: Token Estimation Improvements for better chunking

How are you running AnythingLLM?

AnythingLLM desktop app

AnythingLLM Version

v1.16.0

LLM Provider

Generic OpenAI (local llama.cpp server, Qwen3.5-4B-UD-Q4_K_XL.gguf)

Model context window: 32768 (matches -c 32768 on the llama-server side)
Max Tokens: 20480

Embedder

Generic OpenAI (local llama.cpp server, bge-m3-q4_k_m.gguf, --embedding)

What happened?

When attaching a document to the chat window (via drag-and-drop or the + icon, per the "Attach" flow described in the docs: https://docs.useanything.com/chatting-with-documents/introduction), the model only received approximately the first 3.6k tokens of the document, regardless of the document's actual size or the configured context window (32768). The remainder of the document was silently dropped — no warning, no truncation notice, and no context-overflow prompt was shown.

The token counter next to the + icon confirmed this: it consistently showed ~3.6-3.7k tokens used, even though the source file was ~78,000 characters / 853 lines / well within the configured 32768-token context window.

As a direct consequence, any question about content located past this cutoff point returned "not found in the document" even though the content was clearly present in the file.

I also observed a related, less consistent symptom with the workspace "Pin" feature (embed a document, then pin it to bypass RAG) — see "Note on Pin feature" below.

I identified the root cause: half-width katakana characters (Unicode range U+FF61–U+FF9F), commonly used in Japanese government/procurement documents for list markers like (ｱ), (ｲ), (ｳ). See "Root cause identified" below for the confirming test.

Expected behavior

Per the documentation, "By default, AnythingLLM will insert the full text of your documents into the chat window" when it fits within the model's context window. Since my document (~20-30k tokens estimated) is smaller than the configured 32768-token context window, I expected the full text to be included.

Steps to Reproduce

Start a local llama.cpp server (llama-server) with -c 32768 and a Generic OpenAI-compatible endpoint.
Configure AnythingLLM's LLM Preference: Generic OpenAI, BaseURL pointing to the local server, Model context window = 32768.
Start a new chat thread in a workspace.
Attach a Japanese-language .md or .txt file that contains half-width katakana characters (e.g. (ｱ), (ｲ), (ｳ)) via the + icon (not via workspace embedding — this is the chat-attach flow, separate from RAG/Pin).
Check the token count shown near the + icon before sending — it shows a token count far smaller than the actual document size.
Send a prompt asking to extract information located at various points throughout the document.
Observe that only content located before the first half-width katakana characters is available to the model; content past that point is treated as absent.

Root cause identified

I normalized all half-width katakana characters to full-width in the source document (using Python's jaconv.h2z(), equivalent to NFKC-style width normalization) and re-attached the exact same document (same content otherwise, same file size class, same encoding, same line endings).

Result: the Attach feature correctly ingested the full document, and the LLM correctly answered questions about content throughout the entire file, including content well past the previous ~3.6k token cutoff point.

This strongly suggests the Attach ingestion/extraction pipeline has a bug related to parsing or encoding-detection when it encounters half-width katakana characters, causing it to silently stop reading the file at that point.

Note on Pin feature

I also tested the workspace "Pin" feature (embedding a document into the workspace, then pinning it to bypass RAG and insert full text into context) using the same original file (containing half-width katakana). Across multiple attempts, results were inconsistent — different sections of the document appeared to be missing each time, and the number of tokens/chunks actually reaching the model varied between attempts.

After normalizing half-width katakana to full-width in the source document, the Pin feature worked reliably and consistently across repeated tests, correctly retrieving the full document content every time.

This suggests the half-width katakana issue may affect the Pin/embedding ingestion pipeline as well, not only the Attach flow — though I did not isolate this as rigorously (e.g. did not separately rule out encoding/line-ending factors for the Pin case the way I did for Attach below).

Ruled-out factors (tested before finding the actual root cause)

Before isolating half-width katakana as the trigger, I tested and ruled out the following — none of these affected the ~3.6-3.7k token cutoff in the Attach flow:

File format: Tested with both .md and .txt — identical truncation point.
Text encoding: Verified the source file is valid UTF-8 (confirmed via Notepad++), re-saved explicitly as UTF-8/UTF-8-BOM — no change.
Line endings: Original file used LF (Unix). Converted to CRLF (Windows) via Notepad++ EOL conversion — token count moved marginally (3.6k → 3.7k), essentially no meaningful change.
File integrity: Verified via Notepad++ that the file was complete and correct (853 lines / ~78,315 characters).
Embedder context window: Increased the embedder's (bge-m3 via llama.cpp) context window from 8192 to 32768 (capped back down to 8192 by llama.cpp since that's the model's native training context) — no change. Also confirmed via the embedder server's own logs that no embedding requests are triggered at all during the Attach flow, ruling out RAG/vector-search interference for this specific test.
Model's own context handling: Confirmed the underlying LLM and its context window setting work correctly — when I pasted the exact same full document text directly into the chat input (instead of using Attach), the model correctly received and processed the entire ~20-30k token document, including content well past the ~3.6k token cutoff point seen with Attach.

The last point confirmed the issue is specific to the document ingestion pipeline(s), not the LLM provider, context window configuration, embedder, or the source file's general validity — which led me to test individual character-level differences and find the half-width katakana trigger.

Additional context

Document is in Japanese, ~78,315 characters / 853 lines, contains standard Markdown headers, nested numbered/lettered list markers (including the half-width katakana parenthetical markers described above — a legacy Shift-JIS-era convention still used in Japanese government/procurement documents), and one large HTML <table> block embedded in the Markdown.
The Attach cutoff point (~3.6k tokens / roughly line 130-150 of 853) occurs before the HTML table appears in the document (around line 427), but around where the half-width katakana list markers begin appearing densely — consistent with the root cause identified above.
The document was originally generated by MinerU (PDF-to-Markdown conversion), which preserves half-width katakana as-is from the source PDF. As a workaround, I now run a normalization step (jaconv.h2z()) on MinerU output before attaching/embedding documents in AnythingLLM.
This may be related to #5334 (Pinned/Watched documents without RAG), which also describes inconsistent behavior between the intended "full text in context" behavior and what's actually observed — possibly the same underlying cause.
I am not able to share the source document itself (contains business-sensitive procurement details), but the trigger can likely be reproduced with any Japanese text file containing a sufficient density of half-width katakana characters (U+FF61–U+FF9F), e.g. (ｱ), (ｲ), (ｳ) used as list markers.

Are there known steps to reproduce?

Yes, see above.

Note: This investigation was conducted collaboratively with Claude (Anthropic), which helped isolate variables, interpret llama.cpp/embedder server logs, and draft this report. The original debugging conversation was in Japanese; this report has been translated into English for submission.