[BUG] Opening .mv2 file created with 2.0.152 in 2.0.159 hangs indefinitely on low-memory systems

Description

Opening a .mv2 file created with memvid-sdk==2.0.152 using memvid-sdk==2.0.159 causes the use() call to hang indefinitely (tested up to 3+ hours) on resource-constrained hardware. The process runs at 100% CPU with no progress. The same file opens instantly with memvid-sdk==2.0.152.

Use Case

We use memvid-sdk as the core memory engine in lolabot, an open-source Personal Assistant framework for Claude Code. Our PA stores facts, events, learnings, decisions, and personal knowledge as memories in a .mv2 file — semantic search over a person's life context. The index had ~145 frames accumulated over months of daily use.

We hit this bug while upgrading to leverage the new correct(), remove(), and enrich() APIs in 2.0.159.

Reproduction

Environment:

• OS: Ubuntu (Linux 6.8.0-106-generic)
• Hardware: Mac Mini, 4GB RAM, ARM64
• Python: 3.12
• memvid-sdk: upgraded from 2.0.152 → 2.0.159

Steps:

1. Create and populate a .mv2 file using memvid-sdk==2.0.152 with enable_lex=True, enable_vec=True
2. Add ~145 text frames over time (total file size ~42MB)
3. Upgrade to memvid-sdk==2.0.159
4. Attempt to open the file:

import memvid_sdk as m
mem = m.use('basic', 'memories.mv2')  # Hangs here forever

Observed behavior:

• use() never returns
• Process runs at 99.9% CPU, ~56MB RSS
• Tested with enable_vec=False, enable_lex=False, and both disabled — still hangs
• Ran for 3+ hours before being killed
• verify_single_file() returns None (no corruption detected)
• Fresh files created with 2.0.159 open instantly — the issue is specifically cross-version file opening

Expected behavior:

• File should open in seconds, possibly with an automatic index migration

Workaround

Export all frames using 2.0.152, then import into a fresh file created with 2.0.159:

import memvid_sdk as m
import json

# Step 1: Export with old SDK (pip install memvid-sdk==2.0.152)
mem = m.use('basic', 'old-file.mv2')
all_hits = {}
for query in ['Memory', 'fact', 'event', 'learning', 'decision', 'person', 'goal']:
    for h in mem.find(query, k=500).get('hits', []):
        fid = h.get('frame_id', h.get('id', id(h)))
        all_hits[fid] = h
mem.close()

with open('export.json', 'w') as f:
    json.dump(list(all_hits.values()), f)

# Step 2: Import with new SDK (pip install memvid-sdk==2.0.159)
memories = json.load(open('export.json'))
mem = m.use('basic', 'new-file.mv2', mode='create', enable_lex=True)
for memory in memories:
    mem.put(
        title=memory.get('title', ''),
        label=memory.get('label', 'note'),
        text=memory.get('text', ''),
        tags=memory.get('tags', []),
    )
mem.close()
# new-file.mv2 opens instantly in 2.0.159

The fresh file was 20x smaller (2.1MB vs 42MB for 145 frames) and opens instantly.

Additional Notes

• Related to #194 — the lex index also gets disabled when re-opening files created with 2.0.159 itself, requiring enable_lex() after each use() call.
• The 42MB → 2.1MB size reduction after export/import suggests the old file format carried significant overhead that the new version can't efficiently parse during migration.
• lolabot repo: https://github.com/23blocks-OS/lolabot