#2174·garak

AgentBreakerResult accepts out-of-schema judge verdict values

Author: NaCr05Created Sep 10, 2026Updated Sep 13, 2026
Labelsneeds-triage

Steps to reproduce

Run the following against main at commit 3f50ea5ff9cd7050099940647c15c39b07a93392:

python
from garak.attempt import Conversation, Message, Turn
from garak.detectors.agent_breaker import AgentBreakerResult


class Judge:
    def __init__(self, payload):
        self.payload = payload

    def generate(self, **_kwargs):
        return [Message(text=self.payload)]


def verify(payload):
    detector = object.__new__(AgentBreakerResult)
    detector.evaluation_generator = Judge(payload)
    conversation = Conversation(
        [Turn(role="user", content=Message(text="judge this result"))]
    )
    print(detector._verify_once(conversation))


verify('{"success":"MAYBE","confidence":0.99,"reasoning":"x"}')
verify('{"success":"YES","confidence":"NaN","reasoning":"x"}')
verify('{"success":"YES","confidence":"Infinity","reasoning":"x"}')
verify('{"success":"YES","confidence":1.5,"reasoning":"x"}')

Current output:

(False, 0.99, 'x')
(True, nan, 'x')
(True, inf, 'x')
(True, 1.5, 'x')

The payloads containing "NaN" and "Infinity" are valid JSON strings, which are subsequently accepted by Python's float() conversion.

Expected behavior

The packaged AgentBreaker judge prompt declares success as YES, NO, or PARTIAL, and confidence as a value from 0.0 to 1.0. Values outside those domains should be treated as unparseable so the existing verification retry mechanism can retry and ultimately return None if necessary.

Current behavior

_verify_once() type-checks success, rejects boolean confidence, and then normalizes with .upper() and float(). It does not validate the success enum, numerical finiteness, or the documented confidence range.

This can silently turn an unknown success label or NaN into a miss, while infinity or an out-of-range confidence can be treated as a successful result. A nan value can also propagate into Attempt.notes and be emitted as the non-standard JSON constant NaN.

garak version

0.17.1.pre1, source commit 3f50ea5ff9cd7050099940647c15c39b07a93392.

Additional Information

  1. Operating system: Windows
  2. Python version: 3.12.5
  3. Install method: editable source checkout
  4. No target, model, API, GPU, or network access is required
  5. Focused parameterized tests reproduce the behavior deterministically

Related issue #2120 and PR #2121 added type validation for non-string success values and boolean confidence. They do not validate string enum values, non-finite numbers, or the confidence range, and this behavior remains reproducible after that merged change.

I found no open issue or pull request describing this value-domain validation gap, and no open PR currently modifies the AgentBreaker detector or its test file.