DirtyJson decodes \\uXXXX surrogate pairs as lone surrogates, crashing tool execution with UnicodeEncodeError
Describe the bug
helpers/dirty_json.py decodes each \\\uXXXX escape independently (chr(int(unicode_char, 16))) with no UTF-16 surrogate-pair combining. Every tool envelope is parsed through this path (extract_tools.json_parse_dirty -> DirtyJson.parse_string; strict json.loads is never tried first), so whenever a model writes an emoji or any non-BMP character in its legal JSON escape form — e.g. \ud83c\udf78 for U+1F378 — the parsed Python string contains two lone surrogate code points instead of one character.
Python strings tolerate lone surrogates, but UTF-8 does not. The poisoned string then raises at the first .encode("utf-8") downstream:
code_execution_tool->tty_session.send->data.encode(self.encoding)fails withUnicodeEncodeError: 'utf-8' codec can't encode characters in position N-N+1: surrogates not allowed, surfacing as the red "Critical error occurred, retrying..." and a wasted loop iteration;- the same poison crashes
print_styleHTML log writes — #1384 reported this exact traceback and was closed stale; the surrogates come from the parse, not from the log writer.
The error is intermittent from the user's point of view because it only fires when the model happens to escape-encode the emoji rather than emit it as literal UTF-8, which makes it look like a flaky infrastructure problem.
Reproduction
from helpers.dirty_json import DirtyJson
payload = '{"tool_name": "code_execution_tool", "tool_args": {"code": "print(\\ud83c\\udf78)"}}'
parsed = DirtyJson.parse_string(payload)
code = parsed["tool_args"]["code"]
print([hex(ord(c)) for c in code if 0xD800 <= ord(c) <= 0xDFFF]) # ['0xd83c', '0xdf78']
code.encode("utf-8") # UnicodeEncodeError: surrogates not allowedjson.loads on the same payload returns one U+1F378 character and encodes fine.
Expected behavior
Surrogate-pair escapes decode to the real character (matching json.loads), and the parser never emits a string that cannot be UTF-8-encoded.
Fix
PR incoming: combine high+low surrogate escape pairs into the real code point; replace any remaining unpaired surrogate with U+FFFD so the parsed string is always encodable. BMP escapes and invalid-hex handling unchanged.
Source: agent0ai/agent-zero