rag_rerank truncates chunk text mid-UTF-8-character for CJK content

Author: sanchitmonga22Created Sep 11, 2026Updated Sep 11, 2026
LabelsbugP2core

Follow-up from #847 by @ayaangazali — thanks again for that PR!

What

flatten_and_truncate in core/src/features/rag/rag_rerank.cpp:23-33 truncates each chunk's text to kMaxChunkChars (360) with a raw byte loop (for (char c : text) { if (out.size() >= max_chars) break; out.push_back(...); }) — the same bug class #847 just fixed in the chunker. Called from rag_rerank.cpp:112, a CJK chunk gets cut mid-character before it reaches the LLM rerank prompt.

Why it matters

Same failure mode as the chunker bug: invalid UTF-8 lands in a prompt built for an LLM call, which can reject it outright or feed the model a garbled trailing character.

Suggested approach

After truncating to kMaxChunkChars, back off any trailing continuation bytes (while the last byte is 10xxxxxx, drop it) — rag_backend.cpp:311-314 already does exactly this for its source preview and can be reused as the pattern.

Done when

  • flatten_and_truncate never returns a string ending mid-character for CJK (or any multi-byte) input.
  • A regression test covers a CJK chunk long enough to hit the 360-char truncation.

Not blocking #847. @ayaangazali, you know this code well now — you're welcome to take this one if you're interested.

Opened with help from Claude Code and Codex.

Reviewed with help from Claude Code and Codex.

Source: RunanywhereAI/runanywhere-sdks