SAM3DBody-cpp

SAM3DBody-cpp

Standalone C++ inference engine for SAM-3D-Body — zero Python dependency at runtime.

Takes a BGR image and produces per-person MHR body pose parameters, camera translation, and optionally full 3D mesh vertices + 70 body/hand keypoints, all via ONNX Runtime + ggml.

Also includes Python frontends that call the compiled shared library via ctypes, and a CSV exporter for the 70 MHR keypoints.

Multi-person BVH motion-capture export

--bvh PATH writes a standard BVH motion-capture file per detected person (p_0.bvh, p_1.bvh, …). Identities are kept stable across frames by a built-in 2D-bbox IoU tracker, each file's joint OFFSETs are auto-resized to the actor's measured bone lengths, and the output drops straight into Blender / BVHTester / any DCC. A bundled blender/blender_bvh_plugin.py drives a MakeHuman-rigged character from the result. See BVH export for details.

./scripts/video.sh --from clip.mp4 --bvh ./p.bvh --headless
# → p_0.bvh, p_1.bvh, …

MPEG ARF avatar export

--arf PATH writes a MPEG Avatar Representation Format (.arfz) container per detected person — a self-contained base avatar (skeleton + skin weights + personalized rest mesh, plus facial blendshapes under --dev-face) animated by a per-frame Avatar Animation Unit stream. Unlike --bvh, it carries the mesh and skinning data too, not just the skeleton. Independent of --bvh — pass either, neither, or both. See knowledge/ARF.md for the container layout, the JSON schema subset implemented, and how it maps onto the spec (ISO/IEC 23090-39).

./scripts/offline_video.sh --from clip.mp4 --arf ./p.arfz
# → p_0.arfz, p_1.arfz, …

If you want to see the repo in action, there is a Youtube video here

Models

Pre-built ONNX / GGUF / LBS model files are hosted on HuggingFace:

https://huggingface.co/AmmarkoV/SAM3DBody-cpp-onnx-models

With a CUDA GPU (recommended)

Fetch the models into onnx/ at the repo root:

bash tools/fetch_model.sh shared cuda    # ~5.2 GB

scripts/setup.sh does this for you, and the binaries fetch them on first run if onnx/ is missing — so this is only needed for a manual install. Use shared cpu (~3.6 GB) for a CPU-only machine, shared trt (~1.9 GB) for TensorRT, or all for every variant. Re-running only fetches what is absent, and an interrupted download resumes.

--refined-pose additionally needs the refined profile (~607 MB of iterative decoder graphs + pipeline_refined.gguf — fetched automatically on first run with the flag, or by hand with bash tools/fetch_model.sh cuda refined; see "Refined pose" under Running).

Without a CUDA GPU (CPU-only)

The standard backbone.onnx and decoder.onnx are both exported in BFloat16 and require a CUDA GPU — the ORT CPU execution provider has no BF16 kernels, so it refuses to load either one (Could not find an implementation for Expand(13) … / MatMul(13) …). CPU-runnable replacements for both are separate downloads from the same HuggingFace repo.

Download these four files and place them alongside the rest of the models in onnx/:

Once they are in onnx/, --cuda -1 picks both up by itself — no extra flags:

./scripts/video.sh --from your_video.mp4 --cuda -1

Performance expectations for CPU inference: DINOv3-ViT-H has ~630 M parameters. On a modern laptop CPU, one backbone forward pass takes 5–15 seconds, making video processing impractical. Single-image or low-frequency use cases are feasible. For anything approaching real-time, a CUDA-capable GPU is required.

WSL2 users: if nvidia-smi works inside WSL, install the CUDA toolkit for WSL2 (do not install the full Linux driver — only the WSL2 toolkit) and use the standard backbone.onnx instead. See WSL.md for a full step-by-step WSL2 setup guide.

CMake will warn at configure time if neither onnx/ nor the zip is found.

Pipeline

This is not 2D-to-3D lifting. The network directly regresses 3D body model parameters from image features — no depth sensor, no floor plane, no stereo.

Starting from a raw image the pipeline does:

1. YOLO detects person bounding boxes.
2. Each crop is fed to a DINOv2-ViT-H backbone producing a [1280, 32, 32] spatial feature map.
3. A transformer decoder (conditioned on the crop's ray directions and focal length) compresses that feature map into a 1024-dim pose token.
4. Two small FFN heads (run on CPU via ggml) decode the token into:
   • 519 pose parameters — global orientation (6D continuous rotation), per-joint Euler angles for 127 joints, SMPL-like shape betas (45), hand pose (108), and face expression (72).
   • 3 camera parameters [scale, tx, ty] → world-space translation [tx, ty, tz].
5. Those parameters drive linear blend skinning (LBS) over 18 439 vertices to produce the full body mesh and 70 keypoints.

The focal length is estimated from the image diagonal and baked into the decoder's conditioning input, so the network learns to associate apparent body size in the crop with metric depth — the same monocular depth-from-body-proportions approach used by SMPL/HMR-family models. The body is placed in camera space via the predicted root translation; floor-plane recovery is a downstream step if needed.

BGR image
  │
  ▼  yolo.onnx            ONNX Runtime (CUDA EP)   person bboxes + 17 COCO keypoints
  │
  ▼  backbone.onnx        ONNX Runtime (CUDA EP)   feature map  [B, 1280, 32, 32]
  │   DINOv3-ViT-H/14+
  │
  ▼  decoder.onnx         ONNX Runtime (CUDA EP)   pose token   [B, 1024]
  │   6-layer PromptableDecoder
  │
  ▼  pipeline.gguf        CPU matmul (ggml)         MHR params [B, 519] + camera [B, 3]
  │   MHR head + camera head weights
  │
  ▼  body_model.lbs       native C LBS (optional)   vertices [18439, 3] in metres
      extracted once by tools/extract_lbs_data.py

Note: body_model.onnx export is blocked on PyTorch ≥ 2.x (torch.export rejects TorchScript modules). The native C LBS path reads body_model.lbs directly and produces identical output to Python mhr_forward (body model stores data in cm; mhr_lbs_compute applies ×0.01 to match Python's /100 conversion).

Per-person output (MHRResult / FsbResult):

See OUTPUT.md for the full reference on every 2D/3D point set and its labels — the COCO-17, MHR-70 and MHR-127 keypoint name tables, coordinate spaces, and the .joints / .obj / .csv / .bvh file formats.

Directory layout

…

Developer tools (ONNX/GGUF export, LBS extraction, debug scripts, Python training env) live in the parent project: https://github.com/AmmarkoV/Fast-SAM-3D-Body

Setup

One-command setup (recommended)

scripts/setup.sh handles everything: downloads models from HuggingFace, clones required libraries, builds the C++ binaries, and creates a Python venv.

git clone https://github.com/AmmarkoV/SAM3DBody-cpp
cd SAM3DBody-cpp
bash scripts/setup.sh

For a machine without a CUDA GPU, also pass --cpu-backbone to fetch the CPU-runnable models (fp32 backbone + fp16 decoder — the stock ones are BF16 and will not load on the CPU):

bash scripts/setup.sh --cpu-only --cpu-backbone

Common flags:

To build models from source (requires the original Python training environment), see Fast-SAM-3D-Body.

Manual setup

1. Download models

See the Models section above. After extracting the zip, onnx/ should be at the repo root.

2. Build

Requirements: CMake ≥ 3.18, C++17 compiler, OpenCV (core/imgproc/videoio/highgui/dnn), optional CUDA Toolkit.

cd SAM3DBody-cpp
mkdir -p build && cd build

cmake .. -DCMAKE_BUILD_TYPE=Release
make -j$(nproc)

CMake handles dependencies automatically:

• ONNX Runtime 1.20.1 – downloaded from GitHub releases if not found; point to an existing install with -DONNX_RUNTIME_DIR=/path/to/onnxruntime
• ggml – fetched via FetchContent from GitHub
• CUDA – auto-detected; set -DCMAKE_CUDA_ARCHITECTURES=86 (or 75, 89, etc.) for your GPU; falls back to CPU-only if not found

Builds fine but fails at runtime? The ORT 1.20.1 CUDA provider needs CUDA 12.x + cuDNN 9.x. If you see Failed to load library libonnxruntime_providers_cuda.so or Could not find an implementation for Expand(13), see DEPENDENCIES.md for the cause and fix.

Windows (headless build)

Windows is supported as a headless build only (MSVC + CMake; OpenCV via vcpkg). CMake automatica