Make Reflect search_observations token and entity budgets configurable

Use Case

I use Hindsight knowledge pages / mental models as persistent project memory for a coding agent.

Some of these pages need to synthesize durable decisions from a fairly large bank. During Reflect, search_observations can become the dominant contributor to the LLM context even when the source query itself is small.

I want to be able to tune how much observation evidence and entity metadata Reflect retrieves, preferably per bank and optionally per mental model, so I can balance evidence coverage against LLM context size, latency, and provider cost.

Problem Statement

Reflect currently exposes no operator-configurable default budget specifically for search_observations.

On current main, DEFAULT_OBSERVATIONS_TOOL_MAX_TOKENS is still fixed at 5000. #4239 added important protections around model-supplied tool budgets and remaining-context ceilings, but the default search_observations budget itself is still not configurable through env/bank settings.

search_observations also requests entities unconditionally. In my workload, entities are useful semantic data, but they can represent more than half of the serialized observation tool payload.

This means operators currently cannot express a policy such as:

• use a smaller default observation text budget for this bank/page;
• preserve entities but cap their contribution;
• or disable entity enrichment for a page where observation text alone is sufficient.

The only practical alternatives are soft prompt instructions, changing the global Reflect context budget, or maintaining a source patch.

How This Feature Would Help

This would let operators reduce Reflect context growth without globally reducing the Reflect context window or patching Hindsight.

In a controlled read-only test on Hindsight 0.10.0, using the same bank and the same observation query:

5000-token observation budget:

• 24 results
• 8,591 text chars
• 30,943 cleaned serialized chars

3000-token observation budget:

• 16 results
• 5,145 text chars
• 18,272 cleaned serialized chars

The 3000-token result retained:

• 100% of the 5000 Top 5
• 100% of the Top 10
• 93.33% of the Top 15
• 75% of the Top 20

All results dropped by the 3000-token run were in ranks 15–24.

The cleaned serialized payload was reduced by 40.95%.

This suggests that, at least for this workload, a smaller configurable observation budget can materially reduce Reflect context while preserving the highest-ranked evidence.

Making this configurable would allow each deployment to choose the appropriate tradeoff instead of changing Hindsight's global default for everyone.

Proposed Solution

Expose Reflect observation retrieval controls through the normal hierarchical configuration system.

For example:

reflect_observations_max_tokens

with the usual hierarchy:

environment default → tenant/bank config → optional mental-model trigger override

I would also like an entity-specific control for search_observations.

A token budget would be preferable to only a boolean, for example:

reflect_observations_entities_max_tokens

because canonical entity names can provide useful semantic information even when the observation surface text uses an alias.

A simpler first implementation could expose:

reflect_observations_include_entities

if an entity token budget is not practical.

The goal is not to change the shipped 5000-token default. The request is to make the default operator-configurable while preserving current behavior for users who do not configure it.

This would complement #4239 rather than replace it:

• #4239 bounds model-supplied tool token requests and propagates configured recall budgets correctly.
• This feature would let operators choose the default observation/entity budget appropriate for their workload.

Alternatives Considered

1. Put "use max_tokens=3000" in the mental-model source query.

This is only a soft model instruction. The model still controls the tool argument, so it is not a reliable operational limit.

1. Lower HINDSIGHT_API_REFLECT_MAX_CONTEXT_TOKENS.

That controls the accumulated Reflect context, not the search_observations retrieval budget specifically. Lowering it can force earlier synthesis or otherwise change the whole Reflect execution path.

1. Disable entities with a local source patch.

I do not want to do this globally because entities can contain canonical names that are semantically useful. It also creates an upgrade maintenance burden.

1. Change the global default from 5000 to 3000.

I do not think one workload is sufficient evidence to justify changing the default for every user. A configurable budget is safer.

1. Increase the Reflect wall/context budget.

That allows the larger payload to continue through the system but increases latency and token cost rather than controlling the source of the context growth.

Priority

Important - affects my workflow

Additional Context

Environment:

• Hindsight 0.10.0
• self-hosted
• observation retrieval tested through the read-only Recall API
• no LLM or Provider calls were involved in the comparison
• no database/config/source changes were made

Retrieval stability:

5000 × 3:

• identical 24/24 ID sets
• identical order
• pairwise Jaccard = 1.0

3000 × 3:

• identical 16/16 ID sets
• identical order
• pairwise Jaccard = 1.0

Entity payload measurement:

With entities enabled:

5000:

• cleaned payload: 30,943 chars
• entity fields: 17,579 chars
• entity share: 56.81%

3000:

• cleaned payload: 18,272 chars
• entity fields: 10,002 chars
• entity share: 54.74%

Diagnostic recalls with entities disabled:

5000: 30,943 → 13,340 cleaned chars (-56.89%)

3000: 18,272 → 8,254 cleaned chars (-54.83%)

Disabling entities did not change the observation IDs, rank order, or text content in these diagnostic runs.

I am not suggesting that entities should be disabled by default. The measurement only shows that entity enrichment is a significant part of the Reflect observation payload and may benefit from its own configurable budget.

Related issue:

#4239 fixed configured recall-budget propagation and added bounds for model-supplied retrieval token requests. This request is narrower: make the default search_observations budget, and ideally its entity enrichment budget, operator-configurable.

I can provide the anonymized per-rank measurements if they would be useful.

Checklist

• I would be willing to contribute this feature