Onefile cached mode checksums the whole payload on every launch

Thank you for Nuitka. lilbee ships as a Nuitka onefile on Linux, macOS, and Windows.

Bug Report

Bug Description

--onefile-cache-mode=cached verifies the cached extraction on every launch. The bootstrap reads each cached file and compares its CRC32 with the payload entry. This runs before Python starts.

The cost scales with the payload. A 1 GiB payload costs 0.4 s per warm launch on Apple silicon. A 5 GB payload costs about 7 s on a Linux desktop. The process shows full CPU and no disk I/O for that time, and the terminal stays blank.

A stamp written after a complete extraction can identify the cache. A launch whose stamp matches the payload can skip the CRC pass. A patch to OnefileBootstrap.c that does this is linked below, with measurements.

️ Environment

1. Nuitka Version, Python Version, OS, and Platform

4.2
Update status: up to date with stable release '4.2'.
Commercial: None
Python: 3.13.3 (main, Apr  9 2025, 03:47:57) [Clang 20.1.0 ]
Flavor: Python Build Standalone
GIL: yes
Executable: ~/.cache/uv/archive-v0/IEl5hSwfo5RE_eIwWsvF2/bin/python
OS: Darwin
Arch: arm64
macOSRelease: 14.6.1
Version C compiler: clang (clang 16.0.0).

The same pass runs on Linux x86_64 (Ubuntu 22.04, gcc) and Windows Server 2025 (MSVC).

2. How Nuitka and Python were Installed

uv pip install 'nuitka[onefile]==4.2' into a virtualenv. Python is python-build-standalone 3.13.3, installed by uv.

3. Relevant PyPI Packages and Versions

Package   Version Location                                     Installer
Nuitka    4.2     .../lib/python3.13/site-packages             uv
zstandard 0.25.0  .../lib/python3.13/site-packages             uv

️ To Reproduce

1. "Hello World" Test (if applicable)

print("hello") compiles and runs. The issue is launch time, not compilation.

2. Short, Self-Contained, Correct, Eligible (SSCCE) Example

hello.py:

print("hello")

A 1 GiB data file makes the payload large enough to time. Random bytes keep zstd from shrinking it:

head -c 1073741824 /dev/urandom > blob.bin

3. Nuitka Command Line Options

python -m nuitka --mode=onefile --onefile-cache-mode=cached --onefile-as-archive \
  --product-version=1.0.0 --onefile-tempdir-spec='{CACHE_DIR}/hello-sscce/{VERSION}' \
  --include-data-files=blob.bin=blob.bin --output-filename=hello.bin hello.py
./hello.bin   # cold: extracts the payload
./hello.bin   # warm: the cache is complete
./hello.bin

--onefile-as-archive removes decompression from a warm launch, so the remaining cost is the CRC pass. The issue is specific to onefile. Standalone mode has no cache.

Expected Behavior

A warm launch with a complete cache starts Python at once. The bootstrap verifies a cache once, at extraction, and records that.

Actual Behavior & Output

Every launch reads and checksums the whole cached payload before the first line of output. Wall clock from /usr/bin/time -p on Apple silicon, cache on the internal SSD:

Both columns use the command above with the same hello.py and blob.bin. The stamped column is Nuitka 4.2 with the patch applied to OnefileBootstrap.c.

Code path: the extraction loop under _NUITKA_ONEFILE_TEMP_BOOL == 0 calls getFileCRC32(target_path) for each entry and rewrites the file when the checksum differs. getFileCRC32 maps the file, so the process shows CPU with no block I/O.

↩️ Regression (if applicable)

Not a regression. The pass is present in 4.0.x through 4.2.

Additional Context (Optional)

Proposed change. A third --onefile-cache-mode value, for example trusted:

• After a complete extraction, write a stamp that records the payload size and the entry point, and a manifest that lists each payload entry with its size.
• Write the manifest before the stamp. The stamp is the commit point. A crash between the two leaves no stamp, and the next launch extracts in full.
• On launch, when the stamp matches the payload, stat each manifest entry instead of checksumming it. A missing or resized file, or a stamp mismatch, falls back to the full extraction.

The stat sweep does not detect a same-size content change inside a cached file. Only the CRC pass detects that, and its per-launch cost is what this mode removes. The default mode keeps the current behavior.

Workaround. Until Nuitka has a cache mode like this, others with a large payload can apply the same change at build time. lilbee patches OnefileBootstrap.c in the installed Nuitka before it compiles. The patch adds the stamp, the manifest, and the fallback. It also draws a progress bar during cold extraction, which is separate from this proposal:

https://github.com/tobocop2/lilbee/blob/main/tools/wheel-build/onefile-bootstrap-lilbee.patch

I would be happy to open a pull request against develop with the stamp and manifest as a new --onefile-cache-mode value, with tests, if that is of interest.