_cut_entry_to_budget budgets raw text, but cut chunks render a few % over budget after wrapper re-escaping

Version: 0.10.0 — Docker image ghcr.io/vectorize-io/hindsight@sha256:3edcb6165cefdeaa6721dd0fce43cfd13b7a9c346ce0d2c5f4b4bf7bc3c8ac0b (built 2026-09-14, repo revision 5d46f9c8c8eb4fb96f549aa63abe1191b82a7840).

Summary

_cut_entry_to_budget (api/hindsight_api/engine/reflect/prompts.py) shrinks the raw serialized output until count_prompt_tokens(output_str) <= token_budget, then wraps it:

return {**entry, "output": {"truncated": True, "content": output_str}}

_render_history_block re-serializes that wrapper with json.dumps(..., indent=2). The cut text is dense JSON, so every quote and newline inside it is re-escaped (" → \", newline → \n). In characters the wrapper adds 30–60%; in tokens (what the budget is measured in) escape sequences tokenize cheaply, so the rendered chunk comes out a few percent over budget — measured +9.4% on a realistic profile (103 short-field observation dicts, one over-budget item), typically 1–10% depending on field density.

So the documented invariant ("cut text … so the entry renders like any other block", i.e. ≤ per-chunk budget) is not actually what the cut produces: the loop measures the raw string, the consumer (split_context_history → _append_block → chunk packing, and the map prompt) sees the re-escaped rendered size.

Impact

Minor and systematic: up to ~10% of each cut chunk's budget is consumed by escaping scaffolding instead of evidence. On deployments with small per-request context (e.g. local llama.cpp slots of ~24.5k tokens) this forces a few more chunks than the history warrants. Note it compounds with the model-vs-budget tokenizer gap: a 9.83k-token chunk measured under o200k_base can render ~11% larger under a different production tokenizer, so a cut chunk's nominal headroom (20% to the 1.0× window) is partly eaten by both effects.

Reproduction

from hindsight_api.engine.reflect.prompts import _cut_entry_to_budget, _render_history_block
from hindsight_api.engine.reflect.tokenization import count_prompt_tokens

# realistic profile: short-field dicts (dense quotes/newlines), one over-budget item
items = [
    {"id": "m%d" % i, "text": "t" * 90, "fact_type": "observation",
     "entities": [{"canonical": "X", "type": "product", "mentions": ["x"]}],
     "mentioned_at": "2026-09-17T00:00:00+00:00", "tags": ["a", "b"],
     "source_fact_ids": ["s%d" % i]}
    for i in range(103)
]
items[0]["text"] = items[0]["text"] * 40   # single item > budget

entry = {"tool": "recall", "input": {}, "output": {"query": "q", "memories": items}}
cut = _cut_entry_to_budget(entry, 9830)
tokens = count_prompt_tokens(_render_history_block(cut))
print(tokens)   # ~10750 — over the 9830 budget, though the loop stopped at <= 9830 raw

Fix

Budget the cut against the rendered wrapper — loop on

count_prompt_tokens(
    _render_history_block({**entry, "output": {"truncated": True, "content": output_str}})
)

instead of on the raw output_str (both the initial count and the in-loop recount). The existing proportional-shrink loop converges unchanged. Measured with this change, on the same profiles: the +9.4% case becomes −5.6% (9281 vs 9830), the dense-field case 9717 → 8901.