[Bug]: search_specs.py dies with UnicodeEncodeError on a non-UTF-8 console, blocking the mandatory local-spec-search step

Author: atishfadteCreated Aug 27, 2026Updated Aug 27, 2026
Labelstriage: needs-review

Before submitting

  • I searched existing issues and did not find a duplicate.
  • I confirmed this is a reproducible behavior problem, not a request for help using the project.

What happened?

forge/stage1_intake/search_specs.py prints its matches with ensure_ascii=False, and the vocabulary packs it searches are bilingual — docs/specs/vocabulary/core_3d.jsonl and cs2.jsonl carry Vietnamese and Chinese recognition cues. When the console cannot encode those characters, print() raises UnicodeEncodeError mid-write and the process exits non-zero with a partially written stream.

On a default Windows console (cp1252) that is every non-trivial query. local-spec-search is a mandatory checklist step in forge/state.py, so this stops the pipeline at stage 1 rather than degrading to a worse-but-working search.

I expected the search to print its bilingual snippets and exit 0 regardless of the host console's codec.

This is not Windows-only. PYTHONIOENCODING=cp1252 reproduces it on Linux and in CI, which is why CI has never caught it.

Reproduction steps

bash
# 1. From the skill root, on any OS:
PYTHONIOENCODING=cp1252 python3 forge/stage1_intake/search_specs.py \
  --collection core_3d --limit 3 hammer steel handle

# 2. Same failure via --json:
PYTHONIOENCODING=cp1252 python3 forge/stage1_intake/search_specs.py \
  --collection core_3d --limit 3 --json hammer steel handle

# 3. On a default Windows console, no env var is needed at all:
python3 forge/stage1_intake/search_specs.py --collection core_3d --limit 3 hammer steel handle

# 4. The test suite shows it too (2 failures + 1 related error before the fix):
python3 -m unittest discover -s forge/tests -p 'test_*.py'

Reference image or input characteristics

Not image-dependent. Any query whose top matches include a bilingual record reproduces it; the shipped core_3d pack is full of them.

Relevant output and evidence

Query: hammer steel handle
Collection: core_3d
Index: hit (current) ac4f0bc9028e34083dbb5d5531ae3df7699c8e7641d5f8e6d42e7492ea1be1d0
1. core.shader-mapping-torusgeometry-rings-loops  score=3.760685
   source: docs/raw/img2threejs-skill-dataset.json :: categories.rendering_shader_mapping[8]
Traceback (most recent call last):
  File "forge/stage1_intake/search_specs.py", line 258, in <module>
    raise SystemExit(main(sys.argv[1:]))
  File "forge/stage1_intake/search_specs.py", line 253, in main
    _print_human(payload)
  File "forge/stage1_intake/search_specs.py", line 158, in _print_human
    print(f"   {match['snippets'][0]}")
UnicodeEncodeError: 'charmap' codec can't encode character '\u1ea1' in position 31: character maps to <undefined>

Failing tests on a cp1252 host before the fix:

FAIL: test_human_output_is_nonempty_and_stable_on_cache_hits (test_search_specs.CliOutputTest)
FAIL: test_untruncated_snippet_has_no_ellipses (test_search_specs.SnippetTest)
ERROR: test_cs2_assessment_embeds_local_spec_search_results (test_pipeline.PipelineTest)

The third is the same root cause in the other direction: the assessment JSON is written as UTF-8 but read back with Path.read_text(), which decodes using the locale codec — UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d.

Two related observations while tracing this:

  • Argparse help mangles the same way — python3 forge/stage4_review/diagnose_render.py --help prints diagnostics � run BEFORE on a cp1252 console.
  • The pattern is broader than the one script: ~14 non-test sites under forge/ and scripts/ call read_text() / write_text() with no explicit encoding=, so they all decode through the host locale. Only the ones that actually carry non-ASCII fail today, but the rest are latent.

Environment

  • Windows 11, Python 3.13.14, default console codepage cp1252
  • Also reproduced deterministically on any OS via PYTHONIOENCODING=cp1252
  • Commit: 441af85

I have a fix and regression tests ready and will open a PR referencing this issue.