[RFC] Opt-in system python for Linux builds (LY_PYTHON_USE_SYSTEM)

Summary: an opt-in, default-off cmake option for Linux that builds the engine against the host's python instead of the downloaded 3rdParty python package, with pybind11 (the C++ to Python binding library the editor embeds), PySide6 (Qt's Python bindings) and shiboken6 (the generator that produces them), plus their tooling, resolved from the distribution as well. Off, nothing changes on any platform; that is the option's contract. This is the python-sized instance of the standardized system-library override proposed in #19815, and it exists as a working prototype: the spike/system-python branch at https://github.com/nickschuetz/o3de/tree/spike/system-python, eight commits on top of development commit 305b855e12 (see the note on its current shape below).

Why: Linux distributions and their packaging ecosystems (Fedora, Debian, Flatpak/Snap runtimes) cannot ship engines that bundle their own python, and the gap between the bundled interpreter and current distributions is widening: the bundled package is Python 3.10 (2021) while current Fedora ships 3.14. That gap is what forces the engine to also pin PySide6, shiboken6 and pybind11 to that interpreter (pybind11 at 2.10, which predates support for current Python), plus the python halves of other packages, because nothing distro-provided can load into a 3.10 interpreter. Building against the system python collapses that entire chain: PySide6, shiboken6 and pybind11 come from the distribution, stay consistent at the application binary interface (ABI) level with each other and with Qt by distro policy, and receive security updates without engine involvement. This is also the direction the 3rdParty system has already been moving on its own, from prebuilt packages toward source and FetchContent acquisition (pybind11 in #19566, GoogleBenchmark in #20019); a system-python option is the same trajectory applied to the python stack on Linux.

What the prototype does (all behind the option, Linux only):

• The per-engine venv (Python virtual environment) is created from the system interpreter with access to the distro site-packages; the venv layout, python.sh entry point and requirements flow are unchanged in shape.
• find_package(Python), find_package(pybind11) and find_package(pyside6) are served by system find modules that mirror the packaged find modules' target and variable contracts, so consumer cmake changes are minimal (two small guarded additions in the prototype: a Project Manager package-name fallback and a build dependency on the pyside6 target for the QtForPython module).
• The pyside6 module includes the add_shiboken_project machinery pointed at the system shiboken6 generator and typesystems; a wrapped-type test module generates, compiles, imports and passes isinstance checks against the distro PySide6.
• A reduced requirements pin set covers interpreters newer than the packaged one (the handful of C-extension pins in requirements.txt predate current python releases and have no wheels there).

Validation (Fedora 44, system Python 3.14.7, and the distribution's Qt 6 and PySide6/shiboken6; the Editor check below was re-confirmed on 2026-09-02 against the current packages, Qt and PySide6/shiboken6 all 6.11.2, running the same binaries): full profile builds of Editor, ProjectManager, AssetProcessor, AssetBuilder and the editor gem set; the Editor runs with a project and a --runpython script verifies azlmbr, the o3de CLI module and PySide6 attached to the live QApplication; AssetProcessorBatch runs the AutomatedTesting project with the PythonAssetBuilder and SceneAPI script-driven assets (the ones that drive the embedded interpreter) all processing successfully; the Project Manager's python bindings work.

One gap to state plainly: the EditorPythonBindings unit tests do not pass under the prototype. They make heavy direct use of pybind11 from the test body, which depends on the initializing thread still holding the global interpreter lock (GIL) after startup, and the prototype releases it so that other threads (asset builder job threads, for example) can enter python at all. Two honesty notes about the prototype's current shape: that GIL release is not yet gated behind the option in the prototype branch, and neither is the system-Qt find module it uses for the Qt pairing; both would be gated or split out in a real implementation so that the off-by-default contract above holds literally, and the branch should be read as a proof of the capability rather than as the final diff. The GIL question is the same one raised separately in #20047, and it needs a decision on where the contract should sit before the option can claim a clean test run. Everything above was exercised on the built engine rather than through that suite.

Groundwork this RFC stands on is already upstream and merged, as independent fixes that each stand on their own regardless of this proposal: the PyConfig initialization port (#20040; the deprecated pre-init API fails the build against Python 3.13 or newer headers, so this was a prerequisite for any interpreter past 3.12), the PEP 660 editable-install resolution fix (#20041; PEP 660 is the Python Enhancement Proposal that defines the modern editable-install format, and current pip's editables silently stop resolving without this), and wiring the Linux python preload library names from the build instead of hardcoded literals (#20044). One further piece turned out to need a decision rather than a patch: the GIL handling that the asset-builder path surfaced is open as #20047, after a first attempt (#20043) was withdrawn when review showed it was wrong for the bundled configuration. Two honest couplings to name: a system-python build wants a matching system Qt on Linux (the distro PySide6 carries versioned symbol needs against the distro Qt), and it wants a single in-process OpenSSL (see #20045; the bundled static 1.1.1t exports capture the OpenSSL 3 calls made by the system _hashlib, Python's OpenSSL-backed hashing module, and crash it).

Non-goals: no change to Windows, Mac, Android or iOS; no change to the default Linux build; no removal of the bundled python path. Distributions opt in; everyone else is untouched.

Open questions for the sig: where the option should live long-term relative to #19815's generalized mechanism; whether the reduced pin set should become the default requirements posture (pins with wheels for a range of interpreters); and what platform matrix the sig would want exercised before accepting the option (we can offer Fedora and CentOS Stream coverage from our packaging CI).