[Feature Request]: per-call provider context — no supported way to attach request-scoped data to a query's LLM, embedding, or rerank calls
Do you need to file a feature request?
- I have searched the existing feature request and this feature request is not already filed.
- I believe this is a legitimate feature request, not just a question or bug.
Feature Request Description
There's no supported way to attach request-scoped data to the outbound provider calls a single query triggers.
A common need: forwarding a caller-scoped identity (e.g. a header value) onto those calls, so a downstream proxy/observability layer can attribute them to the right end user. A query can make three kinds of outbound provider call — an LLM completion, an embedding (to vector-search for the query text itself and, in local/global/hybrid/mix mode, for keywords), and, when enabled, a rerank call. The natural approach — set a contextvars.ContextVar right before calling aquery/aquery_llm, read it inside the custom llm_model_func/embedding_func/rerank_model_func — doesn't work with how LightRAG dispatches any of these.
Each is wrapped once by priority_limit_async_func_call (lightrag/utils.py) in a small, fixed pool of persistent asyncio.Task workers, created lazily on first use and never recreated per call. A call is handed to a worker as a plain tuple on an asyncio.PriorityQueue; the worker executes it inside its own task context, captured when the worker task was first created — long before any particular request existed. A ContextVar.set() made by the caller's own task is invisible inside that worker task. This is correct, standard asyncio behavior (contextvars propagate through a task/coroutine chain, or are copied at asyncio.create_task() time — never through an independent queue hand-off), not a bug in the queue itself, but it does mean there's currently no clean way to carry per-call data through to any of these calls without reimplementing the queue's own concurrency limiting and timeout handling on the caller's side.
Proposal: add fields to QueryParam that let a caller attach extra keyword arguments to that one query's provider calls — merged in as plain data at each call site, so they ride through the queue exactly like every other kwarg already does (system_prompt, stream, etc.), without bypassing the queue's concurrency limiting or timeout handling. One field per kind of call (LLM / embedding / rerank), since these are different callables with different signatures, often different providers entirely — forcing one caller-supplied dict onto all three risks a key valid for one colliding with, or being silently rejected by, another.
Alternatives considered: extending RoleLLMConfig (lightrag/llm_roles.py) instead. Rejected — that config is deliberately per-role and static across every call a role handles, while this needs a fresh value on every single call, which only a per-query field can express.
I have a working patch for this and will open a PR shortly.
Additional Context
Same general shape of problem as a few other open items, though each is scoped slightly differently, and none has landed a fix for the "per-call data through the shared dispatch queue" case specifically:
- #2904 — request-scoped state (a header meant to select a per-request workspace) is ignored because of how a shared instance is queried.
- #2133 — multi-tenant per-request state more generally.
- #3847 — an RFC acknowledging the same class of problem for
addon_params/ role LLM config.
Source: HKUDS/LightRAG