#4003·LightRAG

[Feature Request]: per-call provider context — no supported way to attach request-scoped data to a query's LLM, embedding, or rerank calls

Author: pennycodersCreated Sep 17, 2026Updated Sep 17, 2026

Do you need to file a feature request?

  • I have searched the existing feature request and this feature request is not already filed.
  • I believe this is a legitimate feature request, not just a question or bug.

Feature Request Description

There's no supported way to attach request-scoped data to the outbound provider calls a single query triggers.

A common need: forwarding a caller-scoped identity (e.g. a header value) onto those calls, so a downstream proxy/observability layer can attribute them to the right end user. A query can make three kinds of outbound provider call — an LLM completion, an embedding (to vector-search for the query text itself and, in local/global/hybrid/mix mode, for keywords), and, when enabled, a rerank call. The natural approach — set a contextvars.ContextVar right before calling aquery/aquery_llm, read it inside the custom llm_model_func/embedding_func/rerank_model_func — doesn't work with how LightRAG dispatches any of these.

Each is wrapped once by priority_limit_async_func_call (lightrag/utils.py) in a small, fixed pool of persistent asyncio.Task workers, created lazily on first use and never recreated per call. A call is handed to a worker as a plain tuple on an asyncio.PriorityQueue; the worker executes it inside its own task context, captured when the worker task was first created — long before any particular request existed. A ContextVar.set() made by the caller's own task is invisible inside that worker task. This is correct, standard asyncio behavior (contextvars propagate through a task/coroutine chain, or are copied at asyncio.create_task() time — never through an independent queue hand-off), not a bug in the queue itself, but it does mean there's currently no clean way to carry per-call data through to any of these calls without reimplementing the queue's own concurrency limiting and timeout handling on the caller's side.

Proposal: add fields to QueryParam that let a caller attach extra keyword arguments to that one query's provider calls — merged in as plain data at each call site, so they ride through the queue exactly like every other kwarg already does (system_prompt, stream, etc.), without bypassing the queue's concurrency limiting or timeout handling. One field per kind of call (LLM / embedding / rerank), since these are different callables with different signatures, often different providers entirely — forcing one caller-supplied dict onto all three risks a key valid for one colliding with, or being silently rejected by, another.

Alternatives considered: extending RoleLLMConfig (lightrag/llm_roles.py) instead. Rejected — that config is deliberately per-role and static across every call a role handles, while this needs a fresh value on every single call, which only a per-query field can express.

I have a working patch for this and will open a PR shortly.

Additional Context

Same general shape of problem as a few other open items, though each is scoped slightly differently, and none has landed a fix for the "per-call data through the shared dispatch queue" case specifically:

  • #2904 — request-scoped state (a header meant to select a per-request workspace) is ignored because of how a shared instance is queried.
  • #2133 — multi-tenant per-request state more generally.
  • #3847 — an RFC acknowledging the same class of problem for addon_params / role LLM config.