Web SDK RAG chunker makes a whitespace-free CJK document one unbounded chunk

Author: sanchitmonga22Created Sep 11, 2026Updated Sep 11, 2026
Labelsbugweb-sdk

Follow-up from #847 by @ayaangazali — thanks again for that PR!

What

splitRAGText in bindings/web/packages/core/src/Public/Extensions/RunAnywhere+RAG.ts:1292 tokenizes with [...text.matchAll(/\S+/g)] and sizes chunks by token count. A whitespace-free CJK paragraph is a single regex match, so ragIngestDocument (around line 698) turns the whole document into one unbounded chunk — the fix in #847 only reaches the C++ core, not this TypeScript path.

Why it matters

Web SDK users ingesting CJK (or any script without spaces) get chunking behavior that doesn't match the native core, and lose the point of chunking (bounded embedding/retrieval units) entirely for those documents.

Suggested approach

Have splitRAGText enforce a character budget in addition to the token-count budget, splitting on code-point boundaries when a single token (or run of non-whitespace text) exceeds chunkSize.

Done when

  • A whitespace-free CJK document produces multiple bounded chunks, sized consistently with the C++ core's behavior.
  • A regression test covers a long CJK document with no whitespace.

Not blocking #847. @ayaangazali, you know this code well now — you're welcome to take this one if you're interested.

Opened with help from Claude Code and Codex.

Reviewed with help from Claude Code and Codex.

Source: RunanywhereAI/runanywhere-sdks