[BUG] RTMPose whole-body 3D output is split into aspects in the wrong order: the saved hand files hold face points and the face file holds both hands (2.0.0-alpha.23)

Hi @aaroncherian, @jonmatthis and team, this one comes out of the RTMPose evaluation we talked about on the call. This week we re-tracked one of our six-camera classroom recordings through alpha.23 with the RTMPose whole-body detector to compare it with our 1.8.2 results. The body came out level with 1.8.2 at about four times the speed on an M4 Mac, which is great news on our end. The hands and face did not, and the cause looks like a one-line mismatch between two packages, so here it is with what we measured and a fix that works.

Describe the bug

With detector_type="rtmpose" (the default), the four aspect files in output_data/ are cut from the 133-point whole-body array in an order that does not match the order the tracker writes:

• rtmpose_face_3d_xyz.npy (68 points) actually contains the right hand (points 0 to 20), the left hand (21 to 41) and face points 0 to 25 (42 to 67).
• rtmpose_left_hand_3d_xyz.npy (21 points) actually contains face points 26 to 46.
• rtmpose_right_hand_3d_xyz.npy (21 points) actually contains face points 47 to 67.
• rtmpose_body_3d_xyz.npy is correct.

Measured on our recording (2700 frames, six cameras, anipose calibration from 1.8.2): the point labeled hand_root in the left hand file sits a median 761 mm from the body's left_wrist and 71 mm from the body's nose; the right hand file, 766 mm from right_wrist and 41 mm from nose. Face point 30 (the nose tip in the 68-point layout) sits 817 mm from the body's nose. Loaded into a rig, the hands are 75 cm long and the fingers are a knot.

Cause (as far as we can read it)

Three pieces disagree on the order of the four blocks:

1. skellytracker/core/detectors/keypoint_detectors/rtmpose/wholebody/rtmpose_wholebody.yaml (skellytracker cb739aa, also the head of development) documents its output as body(0-22) + right_hand(23-43) + left_hand(44-64) + face(65-132), and rtmpose_wholebody_detector.py deliberately permutes the model's native COCO-WholeBody order into that layout (_MODEL_TO_SCHEMA_PERM, lines 65 to 74). The tracker_schema.json written next to our recording confirms it: index 23 is right_hand_root, 44 is left_hand_root, 65 is face_0000.
2. skellyforge/skellymodels/tracker_info/rtmpose_model_info.yaml (skellyforge ddf24f3, today's main) says order: body, face, left_hand, right_hand. ModelInfo._build_slices walks that list cumulatively, so face becomes [23:91], left_hand [91:112], right_hand [112:133].
3. freemocap/core/tasks/mocap/mocap_helpers/skeleton_from_mediapipe_observations.py hands the filtered flat array straight to Human.from_tracked_points_numpy_array(model_info=RTMPoseModelInfo(), ...), with no name-based reordering in between, so the skellyforge slices win.

The MediaPipe path is not affected: mediapipe_model_info.yaml already uses body, right_hand, left_hand, face, which is the same layout the skellytracker whole-body schema was written to match. It looks like rtmpose_model_info.yaml (October 2025) simply predates the July 2026 skellytracker rewrite that introduced the permuted whole-body schema.

Why it is easy to miss

The two steps that would have flagged it both skip on RTMPose and only log a warning: Human.fix_hands_to_wrist looks for a marker containing wrist in the hand aspects (RTMPose names its hand point 0 hand_root), and Human.put_skeleton_on_ground looks for left_foot_index / right_foot_index (RTMPose has left_big_toe and friends). From our log:

Missing foot markers necessary to put skeleton on ground.   skellyforge.skellymodels.managers.human.put_skeleton_on_ground():191
Wrist markers missing from ['left_hand', 'right_hand']. Cannot fix hands to wrist   ...human.fix_hands_to_wrist():158

And since the Blender export does not handle RTMPose recordings yet (#848), nobody has had a reason to look at these files in 3D.

To Reproduce

1. Process any multi-camera recording with detector_type="rtmpose" (default model rtmw-x-l_256x192).
2. Load output_data/rtmpose_left_hand_3d_xyz.npy and output_data/rtmpose_body_3d_xyz.npy.
3. Compare hand point 0 against body point 9 (left_wrist) and body point 0 (nose).

Expected: hand point 0 near the wrist. Actual: hand point 0 near the nose, and rtmpose_face_3d_xyz.npy points 0 and 21 near the right and left wrists.

Fix that works here

Change the aspect order in skellyforge/skellymodels/tracker_info/rtmpose_model_info.yaml to match the tracker (and the MediaPipe file):

order:
  - body
  - right_hand
  - left_hand
  - face

We verified it with skellyforge itself: rebuild the flat 133-point array from the saved aspect files, feed it to Human.from_tracked_points_numpy_array with the installed YAML and with the reordered one, and measure.

INSTALLED yaml (body,face,left,right) | left_hand root -> left_wrist 761 mm (-> nose 71 mm) | right_hand root -> right_wrist 766 mm (-> nose 41 mm) | face point 30 (nose tip) -> body nose 817 mm
FIXED yaml     (body,right,left,face) | left_hand root -> left_wrist 21 mm (-> nose 756 mm) | right_hand root -> right_wrist 20 mm (-> nose 749 mm) | face point 30 (nose tip) -> body nose 7 mm

The alternative is to stop permuting in skellytracker and emit the model's native order, but then the MediaPipe and RTMPose layouts would differ, so the YAML change seems the smaller move. Happy to open the PR against skellyforge if that is welcome, or to leave it with you if the order belongs on the tracker side.

Two small follow-ons once the order is right, so the two skipped steps can run for RTMPose: accept hand_root as the hand wrist in fix_hands_to_wrist, and accept left_big_toe / right_big_toe as the toe markers in put_skeleton_on_ground. With the corrected order, RTMPose hands here land 20 to 21 mm from the body wrist, so the wrist snap would be a small correction rather than a rescue.

Log

Relevant lines are quoted above; the full run log is 27k lines and mostly per-frame progress. We can attach it, and a ZIP of the output_data folder (about 100 MB) or the whole session (about 600 MB, six 720p60 cameras, 45 s), on request.

Environment Information:

• OS: macOS 26.6.2, Apple Silicon (M4, 16 GB)
• Python Version: 3.12.14
• Freemocap Version: v2.0.0-alpha.23 installed from the GitHub tag with uv sync --frozen (uv.lock pins skellytracker cb739aa, skellyforge ddf24f3, onnxruntime 1.28.0 with CoreML, mediapipe 0.10.33)
• Are you running freemocap in a virtual environment?: yes (uv .venv)
• Blender Version (If applicable): 5.2.0 LTS, our own loader (the RTMPose export path was not used)

Additional context

Positive data from the same run, in case it is useful for the RTMPose switch: on our take, 2.0 + RTMPose whole-body matched 1.8.2 + MediaPipe on body, leg and foot jitter and on segment-length consistency, with zero left/right swaps on either side, and processed the six cameras in 4.9 minutes against 19.6 for 1.8.2 on the same laptop. Once unscrambled, the RTMPose hands are the right size and place, with finger segment lengths about 2.5 times less stable than MediaPipe's dedicated hand model at our camera distance (whole-body crop, hands about 15 px). Fuller notes alongside everything else we have learned running FreeMoCap in a classroom: https://github.com/davidbayus/hiho-software/blob/main/HIHO_MOCAP/UPSTREAM_NOTES_FOR_FREEMOCAP_2026-08-05.md