[Bug]: HUGGINGFACE_TOKENIZER override is ignored on the LiteLLM and OpenAI-compatible embedding paths

Summary

HUGGINGFACE_TOKENIZER is silently ignored on the LiteLLM and OpenAI-compatible embedding paths. The value is loaded into config correctly and threaded down to create_embedding_engine, but only OllamaEmbeddingEngine is actually handed it. The other two engines neither accept nor forward it, so resolve_embedding_tokenizer always receives huggingface_tokenizer=None and falls back to TikToken — while printing a warning that tells you to set the very variable it just ignored.

This appears to be an oversight in #3762 rather than intended behaviour. That PR describes its own design as:

ollama / openai-compatible / custom / other -> an explicit HUGGINGFACE_TOKENIZER override if set, otherwise the embedding model's own HuggingFace repo

and, under Backward compatibility:

an explicit HUGGINGFACE_TOKENIZER override still wins

resolver.py implements exactly that (target = huggingface_tokenizer or model), but two of the four categories it was written for can never reach it with an override set.

Note the docs currently state the opposite of the PR — "HUGGINGFACE_TOKENIZER is only used by the Ollama embedding engine. It is not needed for OpenAI, Fastembed, openai_compatible, or other providers." If Ollama-only truly is the intent, then the fallback warning should stop advising it on paths that ignore it; either way the current combination is misleading.

Status per engine (verified against main)

Version

• Reproduced on cognee/cognee:1.4.0 (Docker), Python 3.12.13
• Code paths re-verified against current main via the GitHub API — still present
• EMBEDDING_PROVIDER=voyage, EMBEDDING_MODEL=voyage/voyage-4-large, EMBEDDING_DIMENSIONS=1024

Reproduction

1. Configure Voyage embeddings as above (any LiteLLM-routed provider will do).
2. Start Cognee:

warning  Could not load a matching tokenizer for embedding model 'voyage/voyage-4-large'
(voyage/voyage-4-large is not a local folder and is not a valid model identifier listed on
'https://huggingface.co/models' ...). Falling back to TikToken, so token counts are
approximate. Token counts drive chunk sizing and the --dry-run estimate, so a tokenizer that
does not match the embedding model will mis-size chunks. Set HUGGINGFACE_TOKENIZER to a
tokenizer matching your embedding model to fix this.   [tokenizer_resolver]

1. Do what the warning says — set HUGGINGFACE_TOKENIZER="voyageai/voyage-4-large", a real, public, tokenizer-only repo that matches the embedding model exactly.
2. Recreate the container. The warning is byte-for-byte identical, and still names the embedding model rather than the override, because the override was never consulted.

Config loading is not the problem — the value is present at runtime:

>>> from cognee.infrastructure.databases.vector.embeddings.config import get_embedding_config
>>> c = get_embedding_config()
>>> c.embedding_provider, c.embedding_model, c.huggingface_tokenizer
('voyage', 'voyage/voyage-4-large', 'voyageai/voyage-4-large')

Where it is dropped

create_embedding_engine() receives huggingface_tokenizer and forwards it to Ollama only. Both other constructions omit it:

    if embedding_provider == "ollama":
        return OllamaEmbeddingEngine(
            ...
            huggingface_tokenizer=huggingface_tokenizer,     # forwarded
            ...
        )

    if embedding_provider == "openai_compatible":
        return OpenAICompatibleEmbeddingEngine(
            ...
            batch_size=embedding_batch_size,                 # omitted
        )

    from .LiteLLMEmbeddingEngine import LiteLLMEmbeddingEngine
    return LiteLLMEmbeddingEngine(
        ...
        batch_size=embedding_batch_size,                     # omitted
    )

Neither LiteLLMEmbeddingEngine.__init__ nor OpenAICompatibleEmbeddingEngine.__init__ declares a huggingface_tokenizer parameter, and neither get_tokenizer() passes one:

# LiteLLMEmbeddingEngine.get_tokenizer()
tokenizer = resolve_embedding_tokenizer(
    provider=self.provider,
    model=model,
    max_completion_tokens=self.max_completion_tokens,
)                                                    # no huggingface_tokenizer=

# OpenAICompatibleEmbeddingEngine.get_tokenizer()
return resolve_embedding_tokenizer(
    provider="openai_compatible",
    model=self.model,
    max_completion_tokens=self.max_completion_tokens,
)                                                    # no huggingface_tokenizer=

# OllamaEmbeddingEngine.get_tokenizer() — the working one, for contrast
tokenizer = resolve_embedding_tokenizer(
    provider="ollama",
    model=self.model,
    max_completion_tokens=self.max_completion_tokens,
    huggingface_tokenizer=self.huggingface_tokenizer_name,
)

Measured impact

Real but more modest than the warning's tone implies. Across 293 ingested documents, counting with cl100k_base (what actually happens) vs voyageai/voyage-4-large (what should happen):

• Corpus-wide: +3.50%
• Per-document delta: median +7.69%, max +27.1%
• 184 of 293 documents differ by >5%

The skew is systematic in one direction (Voyage counts >= TikToken on this corpus), so chunks come out uniformly ~7% larger than the configured target rather than erratically sized. With voyage-4-large's 32k input limit and a max observed chunk of ~1,000 tokens, nothing is truncated. So the practical effect is coarser-than-configured chunk granularity, not data loss.

Suggested fix

Mirror the Ollama wiring in the two engines that are missing it:

1. Add huggingface_tokenizer: Optional[str] = None to LiteLLMEmbeddingEngine.__init__ and OpenAICompatibleEmbeddingEngine.__init__, storing it on the instance.
2. Forward huggingface_tokenizer=huggingface_tokenizer from create_embedding_engine in both constructions.
3. Pass huggingface_tokenizer=self.huggingface_tokenizer in both get_tokenizer() calls.

resolver.py needs no change — it already honours the override, and already emits the "plausibly differs from the model" advisory for genuine mismatches.

Worth noting this makes exact token counting reachable for Voyage and Cohere, both of which publish tokenizer-only HF repos (e.g. voyageai/voyage-4-large).

A test gap that would have caught it: test_embedding_tokenizer_delegation.py asserts each engine delegates to the resolver, but not that a configured override survives the trip from create_embedding_engine through to resolve_embedding_tokenizer.

Minor, related

voyage and cohere route through LiteLLM and work, but aren't listed among the documented embedding providers, which makes their tokenizer story hard to piece together.