poetry-core editable .pth is written as UTF-8, which Python ≤3.12 cannot read on non-UTF-8 Windows locales (venv interpreter fails to start)

Note: python-poetry/poetry-core has its issue tracker disabled and its README points here, so I am filing this in the main repo. The affected code is poetry-core's editable build.

Summary

The editable build writes the project's absolute path into <module>.pth inside the editable wheel, encoded as UTF-8 — _write_to_zip() does sio.getvalue().encode("utf-8"), hard-coded, with no way to influence it.

Python reads .pth files while initializing site. On Python ≤ 3.12 that read uses the locale encoding (io.TextIOWrapper(io.open_code(fullname), encoding="locale")). So on a Windows machine whose ANSI code page is not UTF-8 (cp936 / cp1252 / cp1251 / …), a project path containing non-ASCII characters yields a .pth the interpreter cannot decode — and a .pth failure is fatal, because it happens during site initialization:

Fatal Python error: init_import_site: Failed to import the site module

The venv's interpreter cannot start at all. Every invocation of it — including any console script — dies before user code runs, with no indication that the .pth is the cause.

Three conditions must hold together:

1. Python ≤ 3.12 for the venv (3.13+ tries UTF-8 first, see below)
2. Windows with a non-UTF-8 ANSI code page (any CJK/Cyrillic/etc. system locale)
3. The project path contains non-ASCII characters

Environment

• poetry-core 2.5.0 (also the newest on PyPI at the time of writing)
• uv 0.12.13 — the installer that invoked the build backend; it is not the writer of the file
• Windows 11 Pro 10.0.26200, ANSI code page 936 (Chinese Simplified)
• Venv interpreter: CPython 3.11.15, locale.getencoding() == 'cp936', sys.flags.utf8_mode == 0
• A project whose pyproject.toml declares [build-system] requires = ["poetry-core"]

To reproduce

1. Place a poetry-core project at a path containing non-ASCII characters, e.g. D:\文档\新建文件夹\my-project
2. Install it editable: uv sync (or pip install -e .)
3. Run the venv interpreter: .venv\Scripts\python.exe -c "print('hello')"

Actual result

Fatal Python error: init_import_site: Failed to import the site module
Python runtime state: initialized
Traceback (most recent call last):
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 980, in exec_module
  File "<frozen site>", line 626, in <module>
  File "<frozen site>", line 609, in main
  File "<frozen site>", line 541, in venv
  File "<frozen site>", line 394, in addsitepackages
  File "<frozen site>", line 236, in addsitedir
  File "<frozen site>", line 188, in addpackage
UnicodeDecodeError: 'gbk' codec can't decode byte 0xb9 in position 24: illegal multibyte sequence

The .pth poetry-core produced (.venv/Lib/site-packages/x64dbg_automate.pth, 52 bytes) — the first bytes are:

44 3A 2F E6 96 87 E6 A1 A3 2F E6 96 B0 E5 BB BA E6 96 87 E4 BB B6 E5 A4 B9 2F 78 36 34 ...
 D  :  /   文        档        /   新        建        文        件        夹        /  x  6  4 ...

i.e. the path D:/文档/新建文件夹/x64dbg-automate-pyclient, stored as UTF-8. Decoding those bytes as cp936 fails at byte 24 (the third byte of 夹).

Expected result

The venv interpreter starts and import <project> resolves to the source tree.

Root cause

poetry-core, src/poetry/core/masonry/builders/wheel.py:

@contextlib.contextmanager
def _write_to_zip(self, wheel: zipfile.ZipFile, rel_path: str):
    sio = StringIO()
    yield sio
    ...
    b = sio.getvalue().encode("utf-8")          # <-- hard-coded UTF-8
    wheel.writestr(zi, b, compress_type=zipfile.ZIP_DEFLATED)

and in the editable path:

content = ""
for path in paths:
    content += path + os.linesep
pth_file = Path(self._module.name).with_suffix(".pth")
with self._write_to_zip(wheel, str(pth_file)) as f:
    f.write(content)

The consumer side, CPython 3.11's Lib/site.py:184:

f = io.TextIOWrapper(io.open_code(fullname), encoding="locale")

The encoding written by the backend and the encoding used by site only agree if the path happens to be representable in both. Unicode paths are not.

Why this is Python-version dependent (important for reproducing)

CPython 3.13+ changed the .pth reader to try UTF-8 first and only fall back to the locale codec:

try:
    # Accept BOM markers in .pth files as we do in source files
    # (Windows PowerShell 5.1 makes it hard to emit UTF-8 files without a BOM)
    pth_content = pth_content.decode("utf-8-sig")
except UnicodeDecodeError:
    ...

so 3.13+ reads poetry-core's .pth fine, and the bug does not reproduce there. Verified on the same machine, same project, same cp936 locale:

This also means the failure depends on which Python the installer picks for a project. A requires-python range that allows 3.11/3.12 (or an installer defaulting to it) hits this; a project pinned to ≥ 3.13 never will.

Comparison: setuptools does not have this problem

On the same machine, an editable install produced by setuptools wrote the same kind of non-ASCII path in the ANSI code page:

__editable__.ida_pro_mcp-2.0.0.pth (28 bytes)
44 3A 5C 43 54 46 | B9 A4 | BE DF | 5C 69 64 61 ...
 D  :  \  C  T  F |  工   |  具   |  \  i  d  a ...

B9 A4 / BE DF are 工 / 具 in GBK. The locale-encoding read accepts it, so it round-trips. The difference is the encoding choice, not the presence of non-ASCII.

Suggested fix

Any of these would resolve it:

1. Write the .pth in the locale encoding (locale.getencoding(), i.e. the ANSI code page on Windows) — this matches what site reads, and is what setuptools effectively does.
2. Keep the .pth pure ASCII — write an import <helper> line and ship a small ASCII-named helper module that performs the path insertion (this is what setuptools does in its default "lenient" editable mode). This is encoding-agnostic and additionally covers paths that are not representable in the ANSI code page at all, which option 1 does not.
3. If UTF-8 output is intentional, it can only work on Python ≥ 3.13 — in that case it would be worth noting the constraint, since it silently breaks requires-python ranges that allow ≤ 3.12 on non-English Windows.

Option 2 is the only one that works for every path on every supported Python.

Workarounds for affected users

• Install non-editable (uv sync --no-editable, pip install .) — the .pth is then not used
• Use Python ≥ 3.13 for the venv (verified)
• Rewrite the installed .pth with the locale encoding afterwards (must be redone whenever the backend re-runs)

Impact

Not cosmetic: the interpreter is unusable, and installers that merely report the command failing give no hint about the cause. Diagnosing it required hex-dumping the .pth and reading site.py.