rag_rerank truncates chunk text mid-UTF-8-character for CJK content
Follow-up from #847 by @ayaangazali — thanks again for that PR!
What
flatten_and_truncate in core/src/features/rag/rag_rerank.cpp:23-33 truncates each chunk's text to kMaxChunkChars (360) with a raw byte loop (for (char c : text) { if (out.size() >= max_chars) break; out.push_back(...); }) — the same bug class #847 just fixed in the chunker. Called from rag_rerank.cpp:112, a CJK chunk gets cut mid-character before it reaches the LLM rerank prompt.
Why it matters
Same failure mode as the chunker bug: invalid UTF-8 lands in a prompt built for an LLM call, which can reject it outright or feed the model a garbled trailing character.
Suggested approach
After truncating to kMaxChunkChars, back off any trailing continuation bytes (while the last byte is 10xxxxxx, drop it) — rag_backend.cpp:311-314 already does exactly this for its source preview and can be reused as the pattern.
Done when
-
flatten_and_truncatenever returns a string ending mid-character for CJK (or any multi-byte) input. - A regression test covers a CJK chunk long enough to hit the 360-char truncation.
Not blocking #847. @ayaangazali, you know this code well now — you're welcome to take this one if you're interested.
Opened with help from Claude Code and Codex.
Reviewed with help from Claude Code and Codex.
Source: RunanywhereAI/runanywhere-sdks