#1773·agent-zero

DirtyJson decodes \\uXXXX surrogate pairs as lone surrogates, crashing tool execution with UnicodeEncodeError

Author: King0James0Created Jul 16, 2026Updated Sep 15, 2026

Describe the bug

helpers/dirty_json.py decodes each \\\uXXXX escape independently (chr(int(unicode_char, 16))) with no UTF-16 surrogate-pair combining. Every tool envelope is parsed through this path (extract_tools.json_parse_dirty -> DirtyJson.parse_string; strict json.loads is never tried first), so whenever a model writes an emoji or any non-BMP character in its legal JSON escape form — e.g. \ud83c\udf78 for U+1F378 — the parsed Python string contains two lone surrogate code points instead of one character.

Python strings tolerate lone surrogates, but UTF-8 does not. The poisoned string then raises at the first .encode("utf-8") downstream:

  • code_execution_tool -> tty_session.send -> data.encode(self.encoding) fails with UnicodeEncodeError: 'utf-8' codec can't encode characters in position N-N+1: surrogates not allowed, surfacing as the red "Critical error occurred, retrying..." and a wasted loop iteration;
  • the same poison crashes print_style HTML log writes — #1384 reported this exact traceback and was closed stale; the surrogates come from the parse, not from the log writer.

The error is intermittent from the user's point of view because it only fires when the model happens to escape-encode the emoji rather than emit it as literal UTF-8, which makes it look like a flaky infrastructure problem.

Reproduction

python
from helpers.dirty_json import DirtyJson

payload = '{"tool_name": "code_execution_tool", "tool_args": {"code": "print(\\ud83c\\udf78)"}}'
parsed = DirtyJson.parse_string(payload)
code = parsed["tool_args"]["code"]
print([hex(ord(c)) for c in code if 0xD800 <= ord(c) <= 0xDFFF])  # ['0xd83c', '0xdf78']
code.encode("utf-8")  # UnicodeEncodeError: surrogates not allowed

json.loads on the same payload returns one U+1F378 character and encodes fine.

Expected behavior

Surrogate-pair escapes decode to the real character (matching json.loads), and the parser never emits a string that cannot be UTF-8-encoded.

Fix

PR incoming: combine high+low surrogate escape pairs into the real code point; replace any remaining unpaired surrogate with U+FFFD so the parsed string is always encodable. BMP escapes and invalid-hex handling unchanged.