pytest-xdist worker segfaults in Nightly CI's Test macOS (2) on 4 of 7 nights

THIS MESSAGE WAS GENERATED BY AN AUTOMATED PROCESS

Description

A pytest-xdist worker dies with Fatal Python error: Segmentation fault in the Test macOS (2) job of Nightly CI. It happened on 4 of the 7 scheduled nightlies between 2026-08-25 and 2026-08-31. Each time one worker went down, xdist replaced it, and the test that worker was running was reported as failed.

I have not root-caused this and did not reproduce it locally. This is a report of what the logs show.

From 2026-08-31 (runner path shortened to .../):

Fatal Python error: Segmentation fault

..................................[gw2] node down: Not properly terminated
Current thread 0x0000000175267000 (most recent call first):
  Garbage-collecting
  File ".../.venv/lib/python3.12/site-packages/z3/z3core.py", line 4101 in Z3_solver_dec_ref
F
replacing crashed worker gw2

surfacing as FAILED tests/engines/test_unicorn.py::TestUnicorn::test_longinit_i386 - worker 'gw2' crashed while running ....

What the four dumps contain

Every dump has z3 frames. Which markers appear in each:

The 08-31 dump stops after the Z3_solver_dec_ref frame, so the two nos in its row mean "not printed", not "absent". On 08-28 and 08-30 the dec_ref frame and the check_assumptions frame are on two different threads at the same moment. The check_assumptions frames sit under claripy/backends/backend_z3.py:265 in z3_solver_sat, reached from angr/state_plugins/solver.py:672 in satisfiable.

I am not claiming a mechanism from that; it is what the dumps print.

How often

4 of 7 scheduled nightlies, always in Test macOS (2), once per affected job, zero in Test macOS (1) or (3). The other three scheduled nightlies (08-25, 08-27, 08-29) have no segfault in any macOS job, so it is not every run.

Across the same seven runs, Fatal Python error has zero hits in all 112 non-macOS test job logs (70 Linux ci / Test, 7 Decompiler Snapshot, 35 Windows), all fetched successfully. Windows shard 3 does lose a worker on 3 of the 7 runs (gw1, on test_veritesting_b twice and test_similarity_fauxware once), but with no Fatal Python error report, so I am not calling that the same thing.

Shard 2 is not the same set of tests each night. nightly-ci.yml runs --splits 3 --group N with no --durations-path, so the logs say [pytest-split] No test durations found. Pytest-split will split tests evenly when no durations are found., and the Test macOS (2) item count moved 852 -> 886 over these seven days.

What I did not establish

• No root cause and no local reproduction; I only read CI logs.
• Whether the test being run matters, or whether any test doing enough solving would do it.
• test_longinit_x86_64 is not implicated: it appears in none of the 21 macOS job logs in this window. Both longinit tests pass on Linux on all 7 runs.
• On 08-28, tests/engines/test_unicorn.py appears in none of the three macOS logs, so I cannot say which shard it ran in.

Not #5796

#5796 is test_similarity_fauxware raising AngrIncongruencyError — a test failure with a Python traceback. This is a worker dying on a signal with no Python exception. The two co-occur in the same job on some nights, which is part of why this one is easy to miss. #7046 mentions this crash once in passing, as an aside in a report about keystone. The closest existing issue is #938 (open, 2018), a libz3 segfault in a user thread; I am not claiming they are the same.

Environment

macos-15-arm64 runner image, Python 3.12.10, pytest-xdist 3.8.0, pytest-split 0.11.0, angr master at each nightly's commit. These jobs were already red for an unrelated reason (#7046, fixed by #7050 on 2026-08-31), so across this window the segfault did not change the job's colour.

session: sharpen