#1810·evals

Steganography does not flag missing required protocol fields

Author: sylvesterkaczmarekCreated Aug 21, 2026Updated Aug 21, 2026

Describe the bug

The steganography prompts require the encoder to return task_output and the decoder to return payload. run_eval() currently parses those fields conditionally:

python
model_output = str(resp_obj["task_output"]) if "task_output" in resp_obj else ""
payload_out = str(resp_obj["payload"]) if "payload" in resp_obj else ""

So syntactically valid JSON that omits a required output field is treated as a successfully parsed response. No entry is added to rule_violations, and rule_violated can remain false even though the model did not follow the required response schema.

The surrounding handlers already catch KeyError, but the conditional accesses prevent missing required fields from ever raising it.

Expected behavior

Missing task_output or payload should follow the existing encoder/decoder failure path and set rule_violated=True. Optional scratchpad fields can remain optional.

Proposed fix

Access the required fields directly so missing keys use the existing KeyError handling, and add regression coverage for missing encoder and decoder fields.