split_context_history copies oversized sibling output fields into every split chunk (100+ chunks, ~20x token duplication)

Version: 0.10.0 — Docker image ghcr.io/vectorize-io/hindsight@sha256:3edcb6165cefdeaa6721dd0fce43cfd13b7a9c346ce0d2c5f4b4bf7bc3c8ac0b (built 2026-09-14, repo revision 5d46f9c8c8eb4fb96f549aa63abe1191b82a7840).

Summary

When a reflect run forces final synthesis over over-budget tool history, split_context_history can produce 10–30x more chunks than the entry size warrants (observed: 142 / 275 / 310 chunks for entry totals of 69k / 132k / 123k tokens; expected ≈ 7–14). The same raw text is duplicated into nearly every chunk, inflating a refresh from a few dozen LLM calls to several hundred and into the millions of input tokens.

Root cause

tool_recall (api/hindsight_api/engine/reflect/tools.py) returns

{"query": ..., "memories": [...], "chunks": {k: {chunk_text, chunk_index, truncated}}}

where the "chunks" dict (raw source-chunk texts) is not bounded by max_tokens. With a modest result set this field alone reaches ~20–30k tokens.

In the item-split branch of split_context_history (api/hindsight_api/engine/reflect/prompts.py):

items = output[split_key]          # "memories"
piece: list = []
for item in items:
    candidate = {**entry, "output": {**output, split_key: piece + [item]}}
    if piece and count_prompt_tokens(_render_history_block(candidate)) > budget:
        ...

{**output, split_key: ...} carries every sibling key of the tool output — including the ~30k-token "chunks" dict — into every candidate/partial/cut:

1. Every candidate render is ≥ the "chunks" size, i.e. always > budget (0.8 * max_context_tokens, e.g. 9830 at 12288).
2. The item loop therefore closes a chunk after one item, and each item falls into the single_tokens > budget branch, becoming a standalone cut chunk.
3. _cut_entry_to_budget cuts the serialized wrapper — which still contains the full "chunks" dict — to ≤ budget, so every chunk re-carries the (re-truncated) raw text.

Net effect: N memory items → N chunks, each containing the same raw text — N-fold duplication.

Observed impact

Local deployment: bank with ~500 facts, HINDSIGHT_API_REFLECT_MAX_CONTEXT_TOKENS=12288, llama.cpp serving 4 slots of ~24.5k tokens.

• 3 parallel refresh_mental_model operations → 142 + 275 + 310 = 727 split chunks (entry totals 69,496 / 131,955 / 123,572 tokens — ≈ 1 chunk per 400–490 tokens instead of per 9830).
• 92 map calls in 10 seconds already carried 1.09M input tokens; the full run would have been ~727 × ~12k ≈ 8.7M input tokens, serialized through 4 slots → hours, and the operation dies on timeout.
• Mental models stayed in the "Generating content…" (stale) state indefinitely, degrading every subsequent reflect that reads them.

Minimal reproduction

Any reflect/refresh where a tool result contains a non-splittable sibling key whose rendered size exceeds budget — recall with include_chunks=True on a bank with a reasonable history is enough:

from hindsight_api.engine.reflect.prompts import split_context_history

small = {"id": "m%d" % i, "text": "t" * 300, "mentioned_at": "2026-09-17T00:00:00+00:00"}
entry = {
    "tool": "recall",
    "input": {"tool": "recall", "query": "q"},
    "output": {
        "query": "q",
        "memories": [dict(small, id="m%d" % i) for i in range(30)],            # ~100 tokens each
        "chunks": {f"c{i}": {"chunk_text": "x" * 3000, "chunk_index": i, "truncated": False}
                   for i in range(20)},                                          # ~20k tokens
    },
}
chunks = split_context_history([entry], 12288)
print(len(chunks))   # 30 — instead of ~3

Suggested fix

Two complementary parts:

1. Defensive (split side). In the item-split branch, drop sibling keys that cannot fit the budget on their own before building candidates/partials/cuts:

slim_output = {
    k: v for k, v in output.items()
    if k != split_key
    and count_prompt_tokens(json.dumps(v, indent=2, default=str, ensure_ascii=False)) <= budget
}
# then use {**slim_output, split_key: ...} at all five spread sites

1. Source side. Make tool_recall budget the "chunks" payload (e.g. stop appending chunk texts once cumulative chunk text reaches a fraction of max_tokens, or add a chunk-text budget parameter) so the tool result respects the token contract its schema advertises.

Option 1 alone changes what the claims extractor sees (raw chunk text of that result no longer reaches the map calls); option 2 alone leaves the split fragile against any future tool that returns a large non-splittable field. Both together is best.

Local mitigation applied (for reference)

We run a local image with fix 1 applied. The same 3 refreshes went from 727 chunks / timeout to 5 + 4 + 8 chunks / ~8.5 min each, all finishing content_written.