Scalene occasionally hangs at startup on macOS hosted CI runners
Summary
On GitHub Actions macos-latest hosted runners, Scalene occasionally hangs at startup when launched as a subprocess with memory profiling enabled. The subprocess produces no stdout/stderr before it is SIGKILL'd by the parent's timeout. The hang has not been reproducible locally on macOS 3.12 (passes 10/10 in <3s each), only on hosted runners.
Symptom
A subprocess invocation like:
python -m scalene run --json --outfile out.json some_script.pynever returns. After the parent's timeout (we use 60s in tests), subprocess.TimeoutExpired fires and the runner reports returncode: -9 (SIGKILL). stdout_seq is empty in the captured exception, indicating Scalene wrote nothing before hanging.
Where it has been observed
- PR #1029 CI run,
run-tests (macos-latest, 3.12), intest/test_tracer.py::TestTracerModes::test_legacy_tracerandTestFunctionCallHandling::test_function_call_attribution. - Master CI run 24045404322 (Apr 6),
run-tests (macos-latest, 3.12), same file (test/test_tracer.py::TestFunctionCallHandling::test_function_call_attribution) — confirms pre-existing, not introduced by #1029.
What we ruled out
- Not pipe-buffer deadlock.
subprocess.run(..., capture_output=True, timeout=...)drains pipes viaselectinternally; pipe-fill cannot be the cause. Also, the captured stdout was empty — Scalene wrote nothing before the hang. - Not the test script being slow. The script is three
[0] * 10_000_000allocations and aprint(); runs in milliseconds without Scalene. - Not a regression from PR #1029. Same hang appeared on master before PR #1029 existed.
- Not Python-version-wide. Only macOS 3.12 has been observed hanging in the runs we examined; other macOS Pythons in the same matrix passed.
Plausible root causes (none verified)
- A race during
DYLD_INSERT_LIBRARIESinjection oflibscalene.dylib. - A
sys.monitoringsetup race on Python 3.12 specifically. - A native heap hook ↔ GIL deadlock under Python 3.12's adaptive specialization.
- macOS hosted runner kernel quirks around library injection / signal delivery.
Workaround currently in place
Commit 4e3b3180 in PR #1029 wraps subprocess.run calls in test/test_tracer.py with a retry helper that catches TimeoutExpired, retries up to 3× at 60s each, and skipTests rather than fails when every attempt hangs. This keeps CI green on transient hangs, but masks the underlying bug — a user actually running Scalene on macOS could hit the same hang.
Suggested next steps
- Add an opt-in
SCALENE_DEBUG=1env var that emits async-signal-safe stderr breadcrumbs at each major startup phase (preload init, pywhere import, sys.monitoring registration, signal handler install). Next time CI hangs, the logs will pinpoint the stuck phase. - Or: set up a
tmate-enabled CI job that pauses on failure so we can SSH into a hung runner and attachlldb/sampleto the Scalene process. - If reproducible, the fix likely lives in
src/source/libscalene.cpporscalene/scalene_tracer.py.
Source: plasma-umass/scalene