OME: SkillClusterUpdated persists a 1024-dim vector, inflating run_record ~18x
What
SkillClusterUpdated.case_vector (added in #393) carries the triggering case's 1024-dim embedding so extract_agent_skill avoids a second embedding call. The event payload is persisted verbatim, so the vector lands in two stores:
run_record.event_payload(_stores/run_record.py, column isTEXT NOT NULL)- the APScheduler jobstore, while a job is queued (
engine.py:686serializes the event intoargs)
Measured with a realistic payload:
with case_vector : 14,074 bytes
without : 765 bytes -> 18.4x, +13.3 KB/recordAt the default max_records_per_strategy = 1000, the skill_cluster_updated ring buffer is ~14 MB instead of ~0.8 MB. It has no audit value — nobody reads 1024 floats out of a run record.
The vector is only consumed when a cluster holds more skills than MAX_SKILLS_IN_PROMPT (extract_agent_skill.py:265-273), which for a highly-aggregated artifact like a skill should be the minority of runs. Most of the time the 13 KB rides along unused.
Why it wasn't fixed in #393
The obvious fix — strip large fields when persisting to run_record — is not local, and this is the part worth carrying forward:
Crash recovery replays the persisted payload to rebuild the event (_background/crash_recovery.py:63 -> engine.py:749 model_validate_json). A trimmed persisted copy deserializes with case_vector is None, which is a legal value that silently routes the recovered run down the md-ordering fallback branch while the original run took the LanceDB-ranked branch. Same event id, different selection, no error. So trimming needs either a payload/audit split with recovery reading the untrimmed copy, or an explicit "vector unavailable after recovery" signal.
Options
- Audit projection. Give
BaseEventa hook (or passmodel_dump_json(exclude=...)) used only for therun_recordwrite, keeping the recovery path on the full payload. Fixes the audit bloat, leaves the jobstore copy. - Don't carry the vector; re-embed in the
> MAX_SKILLS_IN_PROMPTbranch. Costs one embedding call on the minority path. Note this does not reintroduce the eventual-consistency dependency #393 removed: that branch already reads LanceDB (find_topk_relevant_in_cluster) and already degrades to md ordering when the index is stale. - Keep as-is and document the sizing (done in CHANGELOG 1.2.3 + #393 body).
Option 2 is the smaller diff and removes the field entirely; option 1 preserves the saved call. Either needs the recovery-divergence question answered first.
Not urgent
Bounded by the ring buffer, so it does not grow without limit. Filed so ome.db sizing is traceable to a decision rather than an accident.
Found during post-merge-review of #393.
Source: EverMind-AI/EverOS