RAG chunker's raw-byte splitter can cut CJK text mid-character
Follow-up from #845 by @ayaangazali -- thanks again for that PR!
What
The document chunker's last-resort split (core/src/features/rag/rag_chunker.cpp:63-67, and the same pattern at rag_chunker.cpp:126-128) splits text every chunk_size_chars bytes with no UTF-8 boundary check. It's chosen whenever none of the real separators (\n\n, \n, . , ? , ! , ; , , , ) are found. A long space-free CJK paragraph (roughly 240+ characters at the default chunk size) is therefore split mid-character.
Why it matters
Those chunks flow into RAGSearchResult.text (rac_rag_proto_abi.cpp:579, chunk->set_text(result.text)) and into rerank snippet heads. Strict proto decoders (SwiftProtobuf on iOS) reject invalid UTF-8 outright -- the same failure mode already documented in the comment at rag_backend.cpp:307-310 for the source-preview field, which #845 fixed for the rerank path but this chunker path is upstream of both and still broken.
Suggested approach
In both hard-split loops in rag_chunker.cpp, back each split boundary off to a UTF-8 lead byte using the same (byte & 0xC0) == 0x80 continuation-byte walk already used at rag_backend.cpp:311-315 and, as of #845, in rag_rerank.cpp. Add a chunker test that feeds a long space-free CJK string and asserts every produced chunk is valid UTF-8.
Done when
- Both raw-byte split loops in
rag_chunker.cppback off to a UTF-8 character boundary - A chunker test covers a long space-free CJK paragraph and asserts every chunk is valid UTF-8
Not blocking #845. @ayaangazali, you know this code well now -- you're welcome to take this one if you're interested.
Opened with help from Claude Code and Codex.
Reviewed with help from Claude Code and Codex.
Source: RunanywhereAI/runanywhere-sdks