#2364·Guardrails

错误: ONNX 模型检测到 jailbreak,将良性提示分类为 jailbreak

作者: AliMoradiVartouni创建于 2026年9月6日更新于 2026年9月13日
标签bugstatus: needs triage

Describe the bug The jailbreak detection model rail produces incorrect classifications when using the published nvidia/NemoGuard-JailbreakDetect ONNX artifact. The same 768-dimensional embeddings produce different classifications when passed to the published snowflake.pkl and snowflake.onnx classifier artifacts. The NeMo SnowflakeEmbed implementation itself does not appear to be the source of the discrepancy: its output was verified to be exactly identical to the embedding produced directly using AutoTokenizer and AutoModel from the same Snowflake/snowflake-arctic-embed-m-long model snapshot. However, the ONNX classifier returns jailbreak=True for ordinary benign prompts such as Hello, What is Python?, Explain Docker, and Fix Python syntax. In contrast, the published sklearn PKL classifier classifies the same embeddings as class 0 (non-jailbreak). This makes the ONNX-based jailbreak detection path unusable as currently observed, since benign prompts are incorrectly detected as jailbreaks.

内容来源: NVIDIA-NeMo/Guardrails