Embedding inputs at/over the context's physical batch boundary (512 tokens) silently store deterministic garbage vectors

Summary

When an embedding input's tokenized length reaches the embedding context's physical batch boundary (512 with the default embedding context), getEmbeddingFor silently returns deterministic garbage that is not an embedding of the input — no error, no warning. Exactly at the boundary the output is a constant independent of content: any 511-token input produces the same vector.

qmd's guard truncateToContextSize clips at contextSize (EMBED_CONTEXT_SIZE, default 2048), never at the batch boundary — so any chunk that tokenizes past ~510 is stored with a wrong vector. The chunker targets ~2,000 chars, which for plain prose stays under 510 tokens, but token-dense text (kebab-case wiki-links, code-adjacent markdown, CJK) crosses it routinely. In my ~1,560-doc index, ~70% of chunks exceeded the boundary once I started clamping — all of them had been embedding degraded or garbage vectors.

Minimal repro (node-llama-cpp level, content-free)

const model = await llama.loadModel({ modelPath: EMBEDDINGGEMMA_Q8 }); // BF16 identical
const ctx = await model.createEmbeddingContext({ contextSize: 2048 });
let text = "hello ";
while (model.tokenize(text.repeat(n)).length !== 511) n++;  // hit exactly 511 tokens
const e = await ctx.getEmbeddingFor(text.repeat(n).trim());
// -> BYTE-IDENTICAL to the vector returned for ANY other 511-token input.
//    Measured cosine 1.0000 against embeddings of 17 unrelated real documents
//    that also tokenized to the boundary; norm ~105 vs ~10 for healthy outputs.

Above the boundary the output is progressively corrupted rather than constant: tokens 512–522 measured norms 78 → 22 (healthy ≈ 10–14 for this model/precision) with cosine 0.67–0.82 to the garbage direction.

Observed damage in a real index

• 47 chunks across 44 documents stored byte-identical vectors in 5 groups (5 distinct boundary-hitting token totals). Documents whose every chunk was affected became invisible to qmd vsearch for their own literal text; unrelated documents scored 0.8+ doc-similarity through shared garbage chunks.
• Every re-embed reproduced the identical wrong bytes — CPU (Linux x86) and GPU (Apple Metal), fresh stores, any batch composition — because the trigger is the token count, which re-embedding doesn't change. Exit codes, pending counts, and stderr all looked healthy throughout.
• Character-level bisection of one affected document: prefix at N tokens healthy, N+1 lands attractor A, N+2 lands attractor B — one-token steps hop between constants.
• Identical group membership under ggml-org/embeddinggemma-300M-Q8_0 and a BF16 conversion of the same model — it's not quantization; only the attractor values differ per weights.

Environment

• qmd 2.5.3 (npm @tobilu/qmd), node v24.13.1
• macOS Apple silicon (Metal) and Linux x86 CPU-only — both reproduce identically
• Embed model: default hf:ggml-org/embeddinggemma-300M-GGUF/embeddinggemma-300M-Q8_0.gguf

Suggested fixes

1. truncateToContextSize should clamp to the effective batch limit rather than contextSize (or the embedding context should be created with batchSize: contextSize).
2. Upstream in node-llama-cpp: getEmbeddingFor should error rather than silently return unpooled/stale output when the sequence cannot be processed within one physical batch for pooled embeddings.
3. A cheap post-embed invariant would have caught this class early: distinct inputs must not produce byte-identical vectors, and output norm should sit in the model's expected band.

Workaround

QMD_EMBED_CONTEXT_SIZE=506 — qmd's own tokenizer-accurate truncation then keeps every input under the boundary (at the cost of truncating long chunks' tails). Full re-embed after setting it: zero identical-vector groups, affected documents findable again.