BM25 keyword search and entity extraction are hardcoded to English

Component

Core / Python SDK

Use Case

The v3 hybrid search pipeline (merged via #4805, #4836, #4858) combines semantic similarity, BM25 keyword matching, and entity-based boosting. The BM25 and entity paths both depend on spaCy's English model en_core_web_sm, which is hardcoded in two places:

• mem0/utils/spacy_models.py:60 — spacy.load("en_core_web_sm") for entity extraction
• mem0/utils/spacy_models.py:85 — spacy.load("en_core_web_sm", disable=["ner", "parser"]) for lemmatization
• mem0/utils/lemmatization.py:30-50 runs this English lemmatizer on every query and on every stored memory's text_lemmatized field

For non-English deployments (Chinese, Japanese, Korean, Arabic, Thai, Hindi, etc.) the practical consequences are:

1. BM25 silently becomes a no-op. spaCy's English pipeline does not tokenize CJK / Arabic / Thai scripts into meaningful units, so query-side lemmas don't overlap with stored-side lemmas. bm25_scores comes back empty at main.py:1359-1366. scoring.py:85 then sets has_bm25 = False and the combined-score divisor stays at 1.0 — results aren't mis-ranked, but the BM25 signal is entirely lost.
2. Entity boost also disappears. extract_entities uses the same English NER model, so non-English proper nouns, IDs, and numbers receive no boost at main.py:1369-1371.

Net effect: for most of the world, the v3 hybrid pipeline degrades to a semantic-only pipeline. Users see no error — they just silently miss the precision improvements the feature is supposed to deliver (e.g. exact-match on a Chinese employee ID, a Japanese name, or a Hindi place name).

This is separate from the storage-side issue where extracted memories are translated into English (covered by #3206, #3707, and PR #4883). Even with memories correctly stored in the source language, the retrieval side still lemmatizes them with en_core_web_sm.

Proposed Solution

Raising this for maintainer awareness and discussion — I don't want to prescribe a specific implementation (language dispatch, character n-grams, multi-model spaCy, vector-store delegation, etc. all have trade-offs worth debating separately). The goal of this issue is to surface the gap so it can be prioritized and scoped.

Alternatives Considered

Users can currently work around parts of this by:

• Disabling BM25/entity contributions and relying on semantic similarity alone.
• Overriding lemmatize_for_bm25 and extract_entities locally.

Neither is documented, and both leave the default experience broken for non-English users.

Additional Context

• v3 hybrid pipeline PRs: #4805, #4836, #4858 (merged April 2026)
• Storage-side companion: PR #4883, issues #3206, #3707
• Key file references:
  • mem0/utils/spacy_models.py
  • mem0/utils/lemmatization.py
  • mem0/memory/main.py:1341, 1591, 1667, 2210, 2751, 3088