LLM cache keys can return cross-model responses
[x] I checked the documentation and current issues/PRs and did not find this collision reported.
Describe the bug
Cached LLM methods can return one model's response for a different model when both wrappers share a cache backend. _generate_cache_key() includes the function qualified name plus call arguments, but not the bound wrapper's model/provider configuration. Ragas binds the cache decorator to LLM methods at construction time, while the actual output also depends on instance state such as provider, model, model arguments, system prompt, and endpoint configuration.
This is especially risky with DiskCacheBackend or a shared backend across requests/workers: the second model can receive a plausible cached answer and never execute.
Ragas version: current main at 298b68274234c060deacab3cf5fb52aa3a20e885
Python version: 3.12
Code to reproduce
This no-network reproduction loads only the current cache module and mirrors the cached bound-method construction used by the LLM wrappers:
import importlib.util
from pathlib import Path
spec = importlib.util.spec_from_file_location("ragas_cache", Path("src/ragas/cache.py"))
cache_module = importlib.util.module_from_spec(spec)
assert spec.loader is not None
spec.loader.exec_module(cache_module)
class MemoryCache:
def __init__(self):
self.data = {}
def get(self, key):
return self.data[key]
def set(self, key, value):
self.data[key] = value
def has_key(self, key):
return key in self.data
class FakeLLM:
def __init__(self, name, cache):
self.name = name
self.calls = 0
self.generate_text = cache_module.cacher(cache_backend=cache)(self.generate_text)
def generate_text(self, prompt):
self.calls += 1
return f"{self.name}:{prompt}"
backend = MemoryCache()
model_a = FakeLLM("model-a", backend)
model_b = FakeLLM("model-b", backend)
print(model_a.generate_text("same prompt"))
print(model_b.generate_text("same prompt"))
print({"calls": (model_a.calls, model_b.calls), "entries": len(backend.data)})Actual:
model-a:same prompt
model-a:same prompt
{'calls': (1, 0), 'entries': 1}Expected: the second result is model-b:same prompt, both models are called once, and the two configurations do not share an entry.
Contract question
I do not think globally hashing self is safe or compatible. tests/unit/test_cache.py deliberately expects equal keys for the same bound method across different objects, and arbitrary client representations can be unstable or contain credentials.
Would the intended fix be an explicit, hashed LLM cache identity/namespace supplied by the wrappers? A useful identity appears to need at least the wrapper class, provider, model, generation arguments, and system prompt. The unresolved part is how endpoint/deployment identity should be normalized across OpenAI-compatible clients without incorporating API keys, userinfo, or raw client representations.
Once that identity contract is confirmed, I can add regressions for different-model misses, same-config cross-instance hits, provider/argument/system-prompt misses, and secret exclusion, then implement the narrow wrapper-level change.
Source: vibrantlabsai/ragas