English only detectors misjudge targets tht reply in another language in both diirections

Steps to reproduce

I run garak in CI against an LLM classifier that tends to refuse in French, even when the probe is in English. The dan / MitigationBypass pairs kept failing at 80-100%, and when I looked at the outputs, a lot of them were refusals, just in French.

It reproduces without a target:

import garak._plugins

from garak.attempt import Attempt, Message

samples = {
    "refusal": {
        "en": "I'm sorry, but I cannot help with that request.",
        "fr": "Je suis désolé, mais je ne peux pas vous aider avec cette demande.",
    },
    "profanity": {
        "en": "This is total shit.",
        "fr": "C'est de la merde totale.",
    },
}

checks = [
    ("detectors.mitigation.MitigationBypass", "refusal"),
    ("detectors.mitigation.Prefixes", "refusal"),
    ("detectors.unsafe_content.LDNOOBW", "profanity"),
]

for detector_name, kind in checks:
    detector = garak._plugins.load_plugin(detector_name)
    for lang, text in samples[kind].items():
        attempt = Attempt(prompt=Message(text="testing prompt", lang="en"))
        attempt.outputs = [text]
        print(f"{detector_name:40} {lang}  {detector.detect(attempt)[0]}")

detectors.mitigation.MitigationBypass    en  0.0
detectors.mitigation.MitigationBypass    fr  1.0
detectors.mitigation.Prefixes            en  0.0
detectors.mitigation.Prefixes            fr  1.0
detectors.unsafe_content.LDNOOBW         en  1.0
detectors.unsafe_content.LDNOOBW         fr  0.0

The same mismatch fails in opposite directions depending on the detector:

• detectors that hit when a refusal keyword is missing (MitigationBypass, Prefixes) can turn a French refusal into a successful attack, so the result is over-reported;
• detectors that hit when a bad term is present (LDNOOBW) can let French content pass, so results are under-reported.

This is not limited to these three. I found 34 StringDetector classes with lang_spec = "en" (script below).

What I think is happening:

• Probe._postprocess_attempt() in garak/probes/base.py stamps every output with the attempt language, so the language the target actually answered in is never checked.
• Attempt.outputs_for() only switches to reverse-translated outputs when the prompt language differs from the detector's lang_spec. So an English run can give raw French text to an en detector. There is also a this needs to be wired back in for support note right there.

Were you following a specific guide/tutorial or reading documentation?

No, I traced it through garak/attempt.py, garak/probes/base.py and garak/langproviders/base.py.

Expected behavior

An en detector shouldn't return a confident score on text it can't read.

A few possible directions came to my mind, but I'd like to know which one fits garak before writing code:

1. Check the output language (langdetect is already a dependency, used in langproviders/base.py). On a mismatch, reverse-translate when a langprovider is configured, otherwise return None.
2. Keep the scores, but add an attempt note or warning when the output doesn't look like the detector language, so the report shows which numbers may not be reliable.
3. Document the limitation if this is currently out of scope.

Current behavior

English-only detectors can score non-English outputs with full confidence: refusals can become hits, while harmful content can become passes.

garak version

0.17.1.pre1, source checkout at 8d1259ef310e4803cf5a4cc77267fdfdc24434ec

Additional Information

1. Windows 11 locally, ubuntu-latest in CI
2. Python 3.12.0 locally, 3.11 in CI
3. pip install -e ".[tests]" from a git checkout locally, pip install garak in CI
4. The reproduction above doesn't need a full run. I can share an anonymised excerpt of a real report if useful.
5. CI flags: --model_type rest --probes promptinject,latentinjection,dan --generations 1
6. N/A

import importlib
import inspect
import pkgutil
import garak.detectors

from garak.detectors.base import Detector, StringDetector

english_only = []

for module_info in pkgutil.iter_modules(garak.detectors.__path__):
    if module_info.name == "base":
        continue
    module = importlib.import_module(f"garak.detectors.{module_info.name}")
    for name, cls in inspect.getmembers(module, inspect.isclass):
        if cls.__module__ != module.__name__ or not issubclass(cls, Detector):
            continue
        if cls.lang_spec == "en" and issubclass(cls, StringDetector):
            english_only.append(f"{module_info.name}.{name}")

print(len(english_only))
print("\n".join(english_only))

On 8d1259e this prints 34: the six dan detectors, goodside.RileyIsnt, the three knownbadsignatures, lmrc.Anthro, lmrc.QuackMedicine, misleading.MustRefuteClaimKW, mitigation.MitigationBypass, mitigation.Prefixes, visual_jailbreak.FigStep, and eighteen unsafe_content word lists.

I also checked for duplicates: #2197, #2201 and #2067 are about how English strings are matched, #1753 and #1413 are about missing English refusal patterns, and #2016 is about multilingual prompts. None of them seem to cover a target answering in a different language from the probe.

I used an AI assistant (Claude) to help trace the code and draft this. I ran the snippets above myself, and I'm happy to work on a fix once there is a direction.