[Bug]: search_specs.py dies with UnicodeEncodeError on a non-UTF-8 console, blocking the mandatory local-spec-search step
Before submitting
- I searched existing issues and did not find a duplicate.
- I confirmed this is a reproducible behavior problem, not a request for help using the project.
What happened?
forge/stage1_intake/search_specs.py prints its matches with ensure_ascii=False, and the vocabulary packs it searches are bilingual — docs/specs/vocabulary/core_3d.jsonl and cs2.jsonl carry Vietnamese and Chinese recognition cues. When the console cannot encode those characters, print() raises UnicodeEncodeError mid-write and the process exits non-zero with a partially written stream.
On a default Windows console (cp1252) that is every non-trivial query. local-spec-search is a mandatory checklist step in forge/state.py, so this stops the pipeline at stage 1 rather than degrading to a worse-but-working search.
I expected the search to print its bilingual snippets and exit 0 regardless of the host console's codec.
This is not Windows-only. PYTHONIOENCODING=cp1252 reproduces it on Linux and in CI, which is why CI has never caught it.
Reproduction steps
# 1. From the skill root, on any OS:
PYTHONIOENCODING=cp1252 python3 forge/stage1_intake/search_specs.py \
--collection core_3d --limit 3 hammer steel handle
# 2. Same failure via --json:
PYTHONIOENCODING=cp1252 python3 forge/stage1_intake/search_specs.py \
--collection core_3d --limit 3 --json hammer steel handle
# 3. On a default Windows console, no env var is needed at all:
python3 forge/stage1_intake/search_specs.py --collection core_3d --limit 3 hammer steel handle
# 4. The test suite shows it too (2 failures + 1 related error before the fix):
python3 -m unittest discover -s forge/tests -p 'test_*.py'Reference image or input characteristics
Not image-dependent. Any query whose top matches include a bilingual record reproduces it; the shipped core_3d pack is full of them.
Relevant output and evidence
Query: hammer steel handle
Collection: core_3d
Index: hit (current) ac4f0bc9028e34083dbb5d5531ae3df7699c8e7641d5f8e6d42e7492ea1be1d0
1. core.shader-mapping-torusgeometry-rings-loops score=3.760685
source: docs/raw/img2threejs-skill-dataset.json :: categories.rendering_shader_mapping[8]
Traceback (most recent call last):
File "forge/stage1_intake/search_specs.py", line 258, in <module>
raise SystemExit(main(sys.argv[1:]))
File "forge/stage1_intake/search_specs.py", line 253, in main
_print_human(payload)
File "forge/stage1_intake/search_specs.py", line 158, in _print_human
print(f" {match['snippets'][0]}")
UnicodeEncodeError: 'charmap' codec can't encode character '\u1ea1' in position 31: character maps to <undefined>Failing tests on a cp1252 host before the fix:
FAIL: test_human_output_is_nonempty_and_stable_on_cache_hits (test_search_specs.CliOutputTest)
FAIL: test_untruncated_snippet_has_no_ellipses (test_search_specs.SnippetTest)
ERROR: test_cs2_assessment_embeds_local_spec_search_results (test_pipeline.PipelineTest)The third is the same root cause in the other direction: the assessment JSON is written as UTF-8 but read back with Path.read_text(), which decodes using the locale codec — UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d.
Two related observations while tracing this:
- Argparse help mangles the same way —
python3 forge/stage4_review/diagnose_render.py --helpprintsdiagnostics � run BEFOREon a cp1252 console. - The pattern is broader than the one script: ~14 non-test sites under
forge/andscripts/callread_text()/write_text()with no explicitencoding=, so they all decode through the host locale. Only the ones that actually carry non-ASCII fail today, but the rest are latent.
Environment
- Windows 11, Python 3.13.14, default console codepage cp1252
- Also reproduced deterministically on any OS via
PYTHONIOENCODING=cp1252 - Commit: 441af85
I have a fix and regression tests ready and will open a PR referencing this issue.
Source: img2threejs/img2threejs