Side-car JSON does not scale: 5.5x larger than the index at 100k docs, and save peaks at 2x the payload

Split out of #350, whose four enumerated defects are all fixed (items 1 and 2 in b101343, items 3 and 4 in #423). This is the one observation from that issue that remains open, and #350 itself labels it "Not a bug at current targets, but it's the first thing that bites at 10M" — recording it separately so #350 can close without losing it.

Measured at 100k docs, 4-field metadata, dim 64:

artifact	size
JSON side-car	24.0 MB
.tvim index	4.4 MB

The side-car is 5.5x larger than the index it accompanies. Load takes 0.3 s at 472 MB peak RSS.

Extrapolating to 1M docs: ~240 MB side-car, ~3 s load, multi-GB peak.

The peak is the sharper end of it. json.dumps materialises the entire payload as a second in-memory string before anything is written, so save() peaks at roughly 2x the payload — the dict and its serialised form are both resident at once. json.load on the read side has the same shape.

Nothing here is wrong at the sizes the integrations target today, and the format is deliberately plain inspectable JSON (documented as "plain JSON, never pickle"), which is worth keeping. Possible directions, none obviously right:

• json.dump(payload, f) instead of f.write(json.dumps(payload)) — streams the encoder's chunks straight to the file and removes the 2x save peak without changing the format at all. Cheapest change by a wide margin. Note it interacts with the atomic-write path: the temp file is currently written from a single string, and atomic_save deliberately serialises before touching any file so a bad payload fails clean (see _check_json_faithful) — streaming would need to keep that ordering property, e.g. by validating first (which it already does) and accepting that a mid-write failure leaves a temp file, which the finally already unlinks.
• A streaming reader for load, or a line-delimited payload section, to avoid materialising the whole dict.
• Splitting bulk metadata out of the side-car entirely.

The handle/keyset consistency checks (check_persisted_handles, check_sidecar_keysets) must keep working under whatever shape this takes.

Generated with Claude Code