Truncated extraction output (max_tokens hit mid-JSON) drops all facts, including the ones already written

Component: Core / Python SDK (memory extraction)

Description

When fact extraction produces a long response and the model hits max_tokens mid-output, it returns valid-but-incomplete JSON - an unterminated {"memory": [...]} array. In _add_to_vector_store, json.loads fails, the extract_json fallback also fails (the structure never closes), and the except block sets extracted_memories = []. Every fact is silently dropped, including the memories the model fully wrote before the cut. The only signal is an ERROR-level log.

This is a distinct failure mode from two adjacent ones:

• #4054 is the model returning prose with no JSON object at all (Expecting value: line 1 column 1 (char 0)).
• #3918 is Gemini leaking reasoning tokens into the JSON, malforming it near the start (char 1 / char 4). Its title says "Unterminated JSON," which sounds like this, but its reported errors land at low char indices from reasoning-token leakage; truncation here fails at a high index near the end because the output simply stopped.

Truncation is different: the JSON is well-formed up to a point and then simply stops, so the parse failure is Unterminated string / Expecting ',' delimiter at a high character index (near the end). It shows up on long conversations and lower max_tokens settings.

Two things are recoverable that are currently thrown away:

1. The complete memory objects the model finished writing before the cut (they parse fine on their own).
2. The cut-off remainder, if the extraction is retried with a higher max_tokens.

Minimal synthetic repro (no private data):

# Extraction hits max_tokens mid-array: memories 0 and 1 are complete,
# memory 2 is cut off mid-string.
truncated = (
    '{"memory": ['
    '{"id": "0", "text": "User likes hiking in the Laurel Highlands"}, '
    '{"id": "1", "text": "User was promoted to Senior Engineer"}, '
    '{"id": "2", "text": "User has a dog nam'
)
# Today: json.loads fails, extract_json fails, add() returns [] - all three
# facts lost, including the two that were fully written.

Happy to open a PR: salvage the complete objects from the truncated array (always, no extra LLM call), and optionally retry once at a raised max_tokens to recover the cut-off remainder (off by default, since it overrides the configured max_tokens and costs an extra call).