Required non-null extraction fields can cause local models to invent dates or owners
Bug description
Hindsight's retain extractor requires when, where, who, and why as non-null strings:
what: str
when: str # "N/A" if unknown
where: str # "N/A" if none
who: str # "N/A" if general
why: str # "N/A" if obviousOpenAIStrictSchemaGenerator then marks every model property required. For smaller/local structured-output models, this creates pressure to fill a descriptive field even when the source does not state a value. In a deterministic test, the model copied a nearby project date into an unrelated fact's when field while keeping the core what text faithful.
This is a correctness problem: a fabricated date/owner can become durable memory and be acted on later.
Environment
- Hindsight: v0.9.2 source revision
ebad478240d3171bb88201ececda5e8d9883d22d - Provider: Ollama 0.32.15
- Model:
qwen3.5:9b - Strict structured output enabled
- Context: 16384
- Temperature: 0.1
- Seed: 42
Minimal reproduction
Representative source shape:
The project has a target start of September 14.
Customer notifications must be completed before fieldwork begins.With the stock schema, the extracted fact was equivalent to:
{
"what": "Customer notifications must be completed before fieldwork begins.",
"when": "before 2026-09-14"
}The date is not stated for that fact; it was copied from nearby context.
Prompt-only mitigation was insufficient. Re-running the same corpus/model/seed after strengthening the extraction mission still produced the same fabricated date.
Confirmed mitigation
Change the four descriptive fields to nullable while keeping the keys required under strict JSON output:
when: str | None = Field(
default=None,
description="Date/time explicitly stated for THIS fact; null if not explicitly stated. Never infer from nearby dates.",
)
where: str | None = Field(default=None, description="Location explicitly stated for THIS fact; null otherwise.")
who: str | None = Field(default=None, description="People/organizations explicitly stated for THIS fact; null otherwise.")
why: str | None = Field(default=None, description="Rationale/causal relationship explicitly stated for THIS fact; null otherwise.")The strict schema still lists every property in required, but these four properties become string | null.
With that schema and otherwise identical model, corpus, temperature, seed, and context:
{
"what": "Customer notifications must be completed before fieldwork begins.",
"when": "N/A",
"occurred_start": null,
"occurred_end": null
}No nearby calendar date was invented. A full three-document test produced zero invented owners, dates, statuses, deadlines, or causal links.
Downstream compatibility
The extraction parser already treats these values as optional in practice:
if when:
combined_parts.append(f"When: {when}")
if who:
combined_parts.append(f"Involving: {who}")
if why:
combined_parts.append(why)So None is already handled safely by the storage conversion path.
Proposed regression test
- Generate the retain strict schema.
- Assert
when,where,who, andwhyare required keys whose value schema accepts bothstringandnull. - Run an extraction fixture containing an unrelated nearby date and assert the date is not copied into the fact with no explicit date.
This preserves strict JSON enforcement while removing pressure to fabricate descriptive metadata.
Source: vectorize-io/hindsight