bug: streaming output rails reuse stale `$bot_message` / `$user_message` action params across chunks
Summary
When rails.output.streaming.enabled=True, output-rail actions that receive substituted params like text=$bot_message or bot_response=$bot_message can reuse stale values across streaming chunks and even across later requests on the same LLMRails instance.
This looks like a shared-mutation bug in the streaming output-rails path, not a general streaming failure.
Scope
This seems to affect only the combination of:
stream_async()- output rails configured
rails.output.streaming.enabled=True- output-rail actions that rely on direct substituted kwargs from
$bot_message/$user_message
Actions that read context["bot_message"] instead of the substituted kwargs appear to be unaffected.
Why this happens
In the streaming path:
get_action_details_from_flow_id()returnselement["action_params"]directly_prepare_params()then mutates that dict in place while replacing$bot_message/$user_message
Relevant spots:
nemoguardrails/rails/llm/utils.pyreturns the originalaction_paramsobject from the parsed flow confignemoguardrails/rails/llm/llmrails.pymutatesaction_paramsin_prepare_params()before executing the rail action
Because the dict comes straight from the parsed flow config, the first processed chunk can permanently replace the placeholder value. Later chunks then reuse the first chunk's text instead of the current chunk, and later requests on the same LLMRails instance can also inherit the stale value.
Minimal shape of the problem
A flow like this is enough to trigger it:
define flow capture output
execute capture_output(text=$bot_message)If capture_output records the text kwarg during streaming and the stream yields multiple chunks like A, B, C:
Expected:
- first call sees
A - second call sees
B - third call sees
C
Actual:
- first call sees
A - later calls can keep seeing
A
The same stale value can also survive into a later streamed request on the same LLMRails instance.
Affected built-in rails
This does not look limited to custom flows. There are built-in output rails that pass $bot_message directly into action params, for example:
prompt_security(protect response)privateaioutput PII detection/maskingglineroutput PII detection/maskingpolicyaioutput moderationtrend_microoutput guardactivefenceoutput moderationai_defenseresponse inspectionregexoutput check
Test coverage gap
The existing streaming output-rails tests seem to use actions that read context["bot_message"], which is probably why this path is not being caught today.
Possible fix
Copy action_params before placeholder substitution, for example in _prepare_params() and/or when returning from get_action_details_from_flow_id().
Something along the lines of:
action_params = dict(action_params or {})would avoid mutating the parsed flow config object in place.
Source: NVIDIA-NeMo/Guardrails