extractSnippet + 6 echo paths slice UTF-16 code units without surrogate-pair awareness → lone surrogates → deterministic HTTP 500 on UTF-8 re-encoding gateways

Summary

extractSnippet() in src/server.ts slices content windows by UTF-16 code units with no surrogate-pair awareness. When a window boundary falls inside an astral character (e.g. an emoji), the echoed snippet contains a lone surrogate. Once such a snippet enters the conversation context, any model provider whose gateway re-encodes the request body to UTF-8 fails deterministically:

UnicodeEncodeError: 'utf-8' codec can't encode character '\udd16' in position 334437: surrogates not allowed

with HTTP 500 and an infinite retry loop — the poisoned message is re-sent verbatim on every retry, so the session never heals. (Apologies for writing the escape in indirection below: the literal sequence backslash-u-d-d-1-6 in this paragraph is the Python error text; rest of this report uses U+XXXX notation.)

Real-world incident (v1.0.169, 2026-09-15)

A ctx_batch_execute query echoed a snippet whose window start landed exactly between the two code units of (U+1F916 = U+D83E U+DD16). The snippet segment began with the lone low half U+DD16. Verified end-to-end:

• FTS match position for a query term: chunk code unit 1429
• Window start = Math.max(0, 1429 - 300) = 1129 — exactly the low-half offset of in the stored chunk
• The KB chunk itself is intact (7 complete , zero lone surrogates) — the chunker is code-point safe; only the echo splits the pair
• The lone surrogate survived JSON.stringify (it serializes back to an escape) and was persisted by the client into its session store, bricking that session against Python-based gateways until the store was surgically repaired

The same day, an independent ctx_search echo produced the window-end variant: a segment ending with a lone high half (U+D83D) followed by the … suffix marker.

Affected sites (src/server.ts @ v1.0.169)

All of these are plain String.slice/substring on code units.

Minimal repro

// extractSnippet semantics (WINDOW=300)
const q = 'scope';
const filler = 'a'.repeat(299);
const content = 'x'.repeat(800) + '\\\u{1F916}' + filler + q + ' y'.repeat(50);
// choose pos such that pos-300 lands exactly after the high half of the emoji:
const pos = content.indexOf(q);
const start = Math.max(0, pos - 300);        // == indexOf('\\\u{1F916}') + 1  → mid-pair
const seg = (start > 0 ? '…' : '') + content.slice(start, start + 600) + '…';
JSON.stringify(seg);  // contains a lone U+DD16 escape — will 500 any UTF-8 re-encoding gateway

Suggested fix

1. Make every cut surrogate-aware: if content.charCodeAt(cut - 1) is a high surrogate (0xD800–0xDBFF), move the cut back by one code unit. (For window starts, move forward past the low half or backward to include both halves.)
2. src/truncate.ts already ships code-point-safe helpers (charSafePrefix/byteSafePrefix) — extractSnippet and the echo paths listed above simply don't use them.
3. Add a regression test: for random emoji-dense content and random cut offsets, assert no code unit in 0xD800–0xDFFF appears unpaired in any echoed snippet / preview.

Impact

Any consumer whose provider gateway re-encodes request bodies to UTF-8 (notably Python-based gateways) gets a deterministic 500 per request, with retry loops that can never succeed — the failure is silent from the plugin's perspective (it only produces a "weird" string) and catastrophic downstream.