[BUG] First run on an empty model cache: every camera worker downloads and converts the same model at once, one worker fails, and the posthoc pipeline waits forever (2.0.0-alpha.23)

Hi all, second report from the same RTMPose evaluation (see #886 for the first). We processed a six-camera recording through alpha.23 on a fresh install, and the very first run stalled. Worth a report because anyone processing six or more cameras on a new machine will hit it.

Describe the bug

The posthoc pipeline starts one VideoNode per camera at the same time. On a cold cache each of them downloads and converts the same model files into the shared cache. They race each other, one of them reads a half-written file and dies while building its session, and the aggregator then waits for a worker that will never report. The pipeline stays alive, so a headless caller never returns and the UI never shows completion.

We hit it on both tracker paths:

• RTMPose (ONNX models in ~/.cache/skellytracker/models/), six workers, one dies:

Unhandled exception in ManagedThread VideoNode-Camera_5: Failed to load model 'yolox-m' with provider='coreml': [ONNXRuntimeError] : 1 : FAIL : Load model from /Users/.../.cache/skellytracker/models/yolox_m_8xb8-300e_humanart-c2c7a14a.onnx.prenms.onnx failed: ... onnxruntime::Model::Model(...) ModelProto does not have a graph.
skellytracker.core.sessions.session_errors.SessionCreationError: Failed to load model 'yolox-m' with provider='coreml' ...

The other five reported "session ready" and kept detecting. "ModelProto does not have a graph" is what a partially written .prenms.onnx looks like while another worker is still converting it.

• MediaPipe (.task files in ~/.freemocap/skellytracker-models/), six workers, the temp-file rename collides:

FileNotFoundError: [Errno 2] No such file or directory: '/Users/.../.freemocap/skellytracker-models/00b93401_pose_landmarker_full.tmp' -> '/Users/.../.freemocap/skellytracker-models/00b93401_pose_landmarker_full.task'

In that case the run reported completion with no output files at all.

• MediaPipe with the default HEAVY model, six workers, cold cache: the race surfaced as a model-build error rather than a file error, because one worker read the .task while another was still writing it:

RuntimeError: Could not build model from the provided pre-loaded flatbuffer: Input tensor 286 to op 187 (DENSIFY) is not produced

We first read this as an Apple Silicon incompatibility with the heavy model. It is not: on a warm cache the same HEAVY model loads and runs on one camera and on six (300 frames x 6 cameras in 23 s). So the race can also masquerade as a platform or model bug, which may be worth knowing when triaging reports.

To Reproduce

1. Empty (or start without) ~/.cache/skellytracker/models/ and ~/.freemocap/skellytracker-models/.
2. Process a six-camera recording with the posthoc pipeline (either detector).
3. Watch the per-worker logs: several "downloading" lines for the same model, then one SessionCreationError (RTMPose) or FileNotFoundError on the .tmp rename (MediaPipe).

Expected: one download and conversion per model, then all workers load it. Actual: as above, and the pipeline never finishes.

What works around it here

Build one skeleton session in a single process before starting the pipeline (build_skeleton_onnx_session(batch_size=1, model_name=...) for RTMPose, or a one-camera MediaPipe run first). Once the cache is warm, six-camera runs are clean every time (4.9 minutes for 2700 frames x 6 cameras on an M4 with CoreML).

Suggested fix

Any of these would do, in increasing order of thoroughness: download and convert the models in the pipeline manager before spawning the VideoNodes; take a per-model file lock around download-and-convert with a "skip if the final file already exists" check (write to a unique temp name, then rename); and treat a dead VideoNode as a pipeline failure so the aggregator errors out instead of waiting forever.

Log

Excerpts above; the full logs are available on request.

Environment Information:

• OS: macOS 26.6.2, Apple Silicon (M4, 16 GB)
• Python Version: 3.12.14
• Freemocap Version: v2.0.0-alpha.23 from the GitHub tag with uv sync --frozen (skellytracker cb739aa, onnxruntime 1.28.0 with CoreML, mediapipe 0.10.33)
• Are you running freemocap in a virtual environment?: yes (uv .venv)
• Blender Version (If applicable): n/a

Additional context

A smaller thing we noticed in the same runs, mentioned here rather than as its own issue: with WorkerMode.THREAD, the interpreter stays alive after "Posthoc mocap complete" and manager.shutdown(); something in the worker threads is non-daemon. That may only matter for headless callers like us.