SAM3DBody-cpp

> 编程语言

开源

从单个摄像头实时生成全身 3D 重建,多人 BVH 输出,纯 C++ 运行时,ONNX + ggml,带手臂的 70 关节骨架。

610 stars0 点赞3 次浏览

访问官网 GitHub

工具介绍

从单个摄像头实时生成全身 3D 重建,多人 BVH 输出,纯 C++ 运行时,ONNX + ggml,带手臂的 70 关节骨架。

SAM3DBody-cpp

Standalone C++ inference engine for SAM-3D-Body — zero Python dependency at runtime.

Takes a BGR image and produces per-person MHR body pose parameters, camera translation, and optionally full 3D mesh vertices + 70 body/hand keypoints, all via ONNX Runtime + ggml.

Also includes Python frontends that call the compiled shared library via ctypes, and a CSV exporter for the 70 MHR keypoints.

Multi-person BVH motion-capture export

--bvh PATH writes a standard BVH motion-capture file per detected person (p_0.bvh, p_1.bvh, …). Identities are kept stable across frames by a built-in 2D-bbox IoU tracker, each file's joint OFFSETs are auto-resized to the actor's measured bone lengths, and the output drops straight into Blender / BVHTester / any DCC. A bundled blender/blender_bvh_plugin.py drives a MakeHuman-rigged character from the result. See BVH export for details.

bash

./scripts/video.sh --from clip.mp4 --bvh ./p.bvh --headless
# → p_0.bvh, p_1.bvh, …

MPEG ARF avatar export

--arf PATH writes a MPEG Avatar Representation Format (.arfz) container per detected person — a self-contained base avatar (skeleton + skin weights + personalized rest mesh, plus facial blendshapes under --dev-face) animated by a per-frame Avatar Animation Unit stream. Unlike --bvh, it carries the mesh and skinning data too, not just the skeleton. Independent of --bvh — pass either, neither, or both. See knowledge/ARF.md for the container layout, the JSON schema subset implemented, and how it maps onto the spec (ISO/IEC 23090-39).

bash

./scripts/offline_video.sh --from clip.mp4 --arf ./p.arfz
# → p_0.arfz, p_1.arfz, …

If you want to see the repo in action, there is a Youtube video here

Models

Pre-built ONNX / GGUF / LBS model files are hosted on HuggingFace:

https://huggingface.co/AmmarkoV/SAM3DBody-cpp-onnx-models

With a CUDA GPU (recommended)

Fetch the models into onnx/ at the repo root:

bash

bash tools/fetch_model.sh shared cuda    # ~5.2 GB

scripts/setup.sh does this for you, and the binaries fetch them on first run if onnx/ is missing — so this is only needed for a manual install. Use shared cpu (~3.6 GB) for a CPU-only machine, shared trt (~1.9 GB) for TensorRT, or all for every variant. Re-running only fetches what is absent, and an interrupted download resumes.

--refined-pose additionally needs the refined profile (~607 MB of iterative decoder graphs + pipeline_refined.gguf — fetched automatically on first run with the flag, or by hand with bash tools/fetch_model.sh cuda refined; see "Refined pose" under Running).

File	Size	Description
`onnx/backbone.onnx` + `.data`	~4.8 GB	DINOv3-ViT-H/14+ encoder (BF16, CUDA EP only)
`onnx/decoder.onnx`	~93 MB	6-layer PromptableDecoder
`onnx/yolo.onnx`	~81 MB	YOLO11m-pose person detector
`onnx/pipeline.gguf`	~5 MB	MHR + camera projection heads
`onnx/body_model.lbs`	~27 MB	Native C LBS data (joints, weights, shape)
`onnx/correctives.bin`	~33 MB	Pose corrective blend shapes
`onnx/keypoint_mapping.bin`	~8 KB	MHR-70 keypoint index map
`onnx/libreyolo9.onnx`	~8 MB	Optional LibreYOLO bbox detector — fetched automatically when explicitly selected (`--detector libreyolo`, or `--yolo` pinned to a `libreyolo*` file)

Without a CUDA GPU (CPU-only)

The standard backbone.onnx and decoder.onnx are both exported in BFloat16 and require a CUDA GPU — the ORT CPU execution provider has no BF16 kernels, so it refuses to load either one (Could not find an implementation for Expand(13) … / MatMul(13) …). CPU-runnable replacements for both are separate downloads from the same HuggingFace repo.

Download these four files and place them alongside the rest of the models in onnx/:

File	Size	Description	Links
`backbone_fp32.onnx`	~1 MB	Graph (references external data)	Link
`backbone_fp32.onnx.data`	~3.2 GB	Float32 weights — no BF16, CPU EP compatible	Link
`decoder_fp16.onnx`	~275 KB	Graph (references external data)	Link
`decoder_fp16.onnx.data`	~97 MB	Float16 weights — no BF16, CPU EP compatible	Link

Once they are in onnx/, --cuda -1 picks both up by itself — no extra flags:

bash

./scripts/video.sh --from your_video.mp4 --cuda -1

Performance expectations for CPU inference: DINOv3-ViT-H has ~630 M parameters. On a modern laptop CPU, one backbone forward pass takes 5–15 seconds, making video processing impractical. Single-image or low-frequency use cases are feasible. For anything approaching real-time, a CUDA-capable GPU is required.

WSL2 users: if nvidia-smi works inside WSL, install the CUDA toolkit for WSL2 (do not install the full Linux driver — only the WSL2 toolkit) and use the standard backbone.onnx instead. See WSL.md for a full step-by-step WSL2 setup guide.

CMake will warn at configure time if neither onnx/ nor the zip is found.

Pipeline

This is not 2D-to-3D lifting. The network directly regresses 3D body model parameters from image features — no depth sensor, no floor plane, no stereo.

Starting from a raw image the pipeline does:

YOLO detects person bounding boxes.
Each crop is fed to a DINOv2-ViT-H backbone producing a [1280, 32, 32] spatial feature map.
A transformer decoder (conditioned on the crop's ray directions and focal length) compresses that feature map into a 1024-dim pose token.
Two small FFN heads (run on CPU via ggml) decode the token into:
- 519 pose parameters — global orientation (6D continuous rotation), per-joint Euler angles for 127 joints, SMPL-like shape betas (45), hand pose (108), and face expression (72).
- 3 camera parameters [scale, tx, ty] → world-space translation [tx, ty, tz].
Those parameters drive linear blend skinning (LBS) over 18 439 vertices to produce the full body mesh and 70 keypoints.

The focal length is estimated from the image diagonal and baked into the decoder's conditioning input, so the network learns to associate apparent body size in the crop with metric depth — the same monocular depth-from-body-proportions approach used by SMPL/HMR-family models. The body is placed in camera space via the predicted root translation; floor-plane recovery is a downstream step if needed.

BGR image
  │
  ▼  yolo.onnx            ONNX Runtime (CUDA EP)   person bboxes + 17 COCO keypoints
  │
  ▼  backbone.onnx        ONNX Runtime (CUDA EP)   feature map  [B, 1280, 32, 32]
  │   DINOv3-ViT-H/14+
  │
  ▼  decoder.onnx         ONNX Runtime (CUDA EP)   pose token   [B, 1024]
  │   6-layer PromptableDecoder
  │
  ▼  pipeline.gguf        CPU matmul (ggml)         MHR params [B, 519] + camera [B, 3]
  │   MHR head + camera head weights
  │
  ▼  body_model.lbs       native C LBS (optional)   vertices [18439, 3] in metres
      extracted once by tools/extract_lbs_data.py

Note: body_model.onnx export is blocked on PyTorch ≥ 2.x (torch.export rejects TorchScript modules). The native C LBS path reads body_model.lbs directly and produces identical output to Python mhr_forward (body model stores data in cm; mhr_lbs_compute applies ×0.01 to match Python's /100 conversion).

Per-person output (MHRResult / FsbResult):

Field	Shape	Description
`bbox`	[4]	x1 y1 x2 y2 in original image pixels
`focal_length`	scalar	Estimated focal length (pixels)
`pred_cam_t`	[3]	Raw camera head output: [s, tx, ty]
`global_rot`	[3]	Global orientation – Euler ZYX (radians)
`body_pose`	[133]	Body joint angles – Euler
`shape`	[45]	SMPL-like identity blend shape betas
`scale`	[28]	Scale PCA components
`hand_pose`	[108]	Hand joints: left [54] + right [54]
`face_params`	[72]	Facial expression parameters
`mhr_model_params`	[204]	Assembled LBS parameter vector (passed to `mhr_lbs_compute`)
`yolo_kps`	[51]	COCO 17 keypoints × [x, y, confidence]
`pred_vertices`	[55317]	18439 verts × 3, metres (when native C LBS runs)
`kps_3d`	[210]	70 joints × 3, metres (when native C LBS runs)
`kps_2d`	[140]	70 joints × 2 projected (when native C LBS runs)

See OUTPUT.md for the full reference on every 2D/3D point set and its labels — the COCO-17, MHR-70 and MHR-127 keypoint name tables, coordinate spaces, and the .joints / .obj / .csv / .bvh file formats.

Directory layout

…

Developer tools (ONNX/GGUF export, LBS extraction, debug scripts, Python training env) live in the parent project: https://github.com/AmmarkoV/Fast-SAM-3D-Body

Setup

One-command setup (recommended)

scripts/setup.sh handles everything: downloads models from HuggingFace, clones required libraries, builds the C++ binaries, and creates a Python venv.

bash

git clone https://github.com/AmmarkoV/SAM3DBody-cpp
cd SAM3DBody-cpp
bash scripts/setup.sh

For a machine without a CUDA GPU, also pass --cpu-backbone to fetch the CPU-runnable models (fp32 backbone + fp16 decoder — the stock ones are BF16 and will not load on the CPU):

bash

bash scripts/setup.sh --cpu-only --cpu-backbone

Common flags:

Flag	Effect
`--cuda-arch 89`	Set CUDA architecture (RTX 4090 = 89, RTX 3090 = 86, RTX 4070 = 89)
`--cpu-only`	Install CPU-only PyTorch in the venv
`--cpu-backbone`	Also download the CPU-runnable models (fp32 backbone + fp16 decoder)
`--skip-models`	Skip model download (models already in `onnx/`)
`--skip-build`	Skip C++ build (Python venv only)
`--skip-venv`	Skip Python venv (build only)
`-j N`	Limit `make` to N parallel jobs

To build models from source (requires the original Python training environment), see Fast-SAM-3D-Body.

Manual setup

1. Download models

See the Models section above. After extracting the zip, onnx/ should be at the repo root.

2. Build

Requirements: CMake ≥ 3.18, C++17 compiler, OpenCV (core/imgproc/videoio/highgui/dnn), optional CUDA Toolkit.

bash

cd SAM3DBody-cpp
mkdir -p build && cd build

cmake .. -DCMAKE_BUILD_TYPE=Release
make -j$(nproc)

CMake handles dependencies automatically:

ONNX Runtime 1.20.1 – downloaded from GitHub releases if not found; point to an existing install with -DONNX_RUNTIME_DIR=/path/to/onnxruntime
ggml – fetched via FetchContent from GitHub
CUDA – auto-detected; set -DCMAKE_CUDA_ARCHITECTURES=86 (or 75, 89, etc.) for your GPU; falls back to CPU-only if not found

Builds fine but fails at runtime? The ORT 1.20.1 CUDA provider needs CUDA 12.x + cuDNN 9.x. If you see Failed to load library libonnxruntime_providers_cuda.so or Could not find an implementation for Expand(13), see DEPENDENCIES.md for the cause and fix.