Sequential and async validator services disagree: the default path silently discards fixes when several validators use on_fail=fix
The same Guard with the same validators returns a different validated output depending on GUARDRAILS_RUN_SYNC, and the default path silently discards fixes.
Reproduction
Two validators, both on_fail="fix", each appending one character:
from guardrails import Guard
from guardrails.classes.validation.validation_result import FailResult
from guardrails.validator_base import Validator, register_validator
@register_validator(name="probe/append-a", data_type="string")
class AppendA(Validator):
def validate(self, value, metadata):
return FailResult(error_message="needs A", fix_value=f"{value}A")
@register_validator(name="probe/append-b", data_type="string")
class AppendB(Validator):
def validate(self, value, metadata):
return FailResult(error_message="needs B", fix_value=f"{value}B")
guard = Guard().use(AppendA(on_fail="fix"), AppendB(on_fail="fix"))
print(guard.validate("x").validated_output)GUARDRAILS_RUN_SYNC=true SequentialValidatorService 'xAB'
GUARDRAILS_RUN_SYNC=false AsyncValidatorService (default) 'xA'Adding a third validator widens it rather than shifting it:
GUARDRAILS_RUN_SYNC=true 'xABC'
GUARDRAILS_RUN_SYNC=false 'xA'Nothing is raised or warned in either case. The same divergence reproduces one level down, calling run_validators on the two services directly, so it is not an artifact of the Guard layer.
Why
The services have different models of what "several fixing validators" means.
SequentialValidatorService.run_validators threads the value through the loop, so each validator sees the previous one's correction and the fixes compose.
AsyncValidatorService.run_validators dispatches every validator against the original value with asyncio.gather, then reconciles:
fix_values = [res.value for res in results if ... on_fail_action == OnFailAction.FIX ...]
if len(fix_values) > 0:
value = self.merge_results(value, fix_values)merge_results folds the fixes pairwise through guardrails.merge.merge, which is a three-way text merge against the original. Two validators appending at the same position is a merge conflict, and the implementation resolves it by keeping one side. With three, two sides are dropped.
So this is not an oversight in the async path; merging is clearly the intent. The gap is that the merge resolves conflicts silently, and that the two services then disagree.
Worth noting the source already carries two # QUESTION: comments in exactly this area, one on whether reconciliation belongs in run_validators or in the merge, and one on what to do when deserializing a merged value fails.
The decision I did not want to presume
- Async should chain like sequential. The two services agree afterwards, and
on_fail="fix"means the same thing everywhere. It gives up concurrency between validators on one property, which may be exactly what the async service exists for. - Merging stays, but a conflict stops being silent. Keep the current behaviour for validators editing disjoint regions, where the merge is genuinely useful, and warn or fail when two fixes touch the same region instead of picking one.
- Merging stays and this is documented. If discarding a fix on conflict is acceptable, saying so next to
on_fail="fix"would be enough, since today the two services differ with nothing written down.
My own preference is 2, because 1 removes a capability and 3 leaves a user's validator silently ineffective, but this is a design call rather than a bug fix and it is yours to make.
Happy to send the PR for whichever you pick, with a test that pins the two services against each other so they cannot drift again.
Environment
guardrails-ai 0.11.0, main at 39714d6, Python 3.12.10. Nothing here is platform specific.
Source: guardrails-ai/guardrails