#4457·hindsight

Required non-null extraction fields can cause local models to invent dates or owners

Author: stelio-KCreated Sep 16, 2026Updated Sep 16, 2026

Bug description

Hindsight's retain extractor requires when, where, who, and why as non-null strings:

python
what: str
when: str   # "N/A" if unknown
where: str  # "N/A" if none
who: str    # "N/A" if general
why: str    # "N/A" if obvious

OpenAIStrictSchemaGenerator then marks every model property required. For smaller/local structured-output models, this creates pressure to fill a descriptive field even when the source does not state a value. In a deterministic test, the model copied a nearby project date into an unrelated fact's when field while keeping the core what text faithful.

This is a correctness problem: a fabricated date/owner can become durable memory and be acted on later.

Environment

  • Hindsight: v0.9.2 source revision ebad478240d3171bb88201ececda5e8d9883d22d
  • Provider: Ollama 0.32.15
  • Model: qwen3.5:9b
  • Strict structured output enabled
  • Context: 16384
  • Temperature: 0.1
  • Seed: 42

Minimal reproduction

Representative source shape:

The project has a target start of September 14.
Customer notifications must be completed before fieldwork begins.

With the stock schema, the extracted fact was equivalent to:

json
{
  "what": "Customer notifications must be completed before fieldwork begins.",
  "when": "before 2026-09-14"
}

The date is not stated for that fact; it was copied from nearby context.

Prompt-only mitigation was insufficient. Re-running the same corpus/model/seed after strengthening the extraction mission still produced the same fabricated date.

Confirmed mitigation

Change the four descriptive fields to nullable while keeping the keys required under strict JSON output:

python
when: str | None = Field(
    default=None,
    description="Date/time explicitly stated for THIS fact; null if not explicitly stated. Never infer from nearby dates.",
)
where: str | None = Field(default=None, description="Location explicitly stated for THIS fact; null otherwise.")
who: str | None = Field(default=None, description="People/organizations explicitly stated for THIS fact; null otherwise.")
why: str | None = Field(default=None, description="Rationale/causal relationship explicitly stated for THIS fact; null otherwise.")

The strict schema still lists every property in required, but these four properties become string | null.

With that schema and otherwise identical model, corpus, temperature, seed, and context:

json
{
  "what": "Customer notifications must be completed before fieldwork begins.",
  "when": "N/A",
  "occurred_start": null,
  "occurred_end": null
}

No nearby calendar date was invented. A full three-document test produced zero invented owners, dates, statuses, deadlines, or causal links.

Downstream compatibility

The extraction parser already treats these values as optional in practice:

python
if when:
    combined_parts.append(f"When: {when}")
if who:
    combined_parts.append(f"Involving: {who}")
if why:
    combined_parts.append(why)

So None is already handled safely by the storage conversion path.

Proposed regression test

  1. Generate the retain strict schema.
  2. Assert when, where, who, and why are required keys whose value schema accepts both string and null.
  3. Run an extraction fixture containing an unrelated nearby date and assert the date is not copied into the fact with no explicit date.

This preserves strict JSON enforcement while removing pressure to fabricate descriptive metadata.