pose2sim

Visit the website!

Please visit the website! (after leaving a ⭐star⭐! )

https://perfanalytics.github.io/pose2sim

\

Pose2Sim

https://github.com/user-attachments/assets/51a9c5a1-a168-4747-9f99-b0670927df95

Pose2Sim provides a workflow for 3D markerless kinematics (human or animal), as an alternative to traditional marker-based MoCap methods.

Pose2Sim is free and open-source, requiring low-cost hardware but with research-grade accuracy and production-grade robustness. It gives maximum control over clearly explained parameters. It also respects data privacy and GDPR rules since it runs locally. Any combination of phones, webcams, or GoPros can be used with fully clothed subjects, so it is particularly adapted to the sports field, the doctor's office, or for outdoor 3D animation capture.

[!TIP] For real-time analysis with a single camera, please consider Sports2D (note that the motion must lie in the sagittal or frontal plane).

[!NOTE] Features:

• Research-Grade Accuracy (2-6° joint angle error, validated by peer-reviewed studies)
• Any Cameras (phones, webcams, GoPros...)
• Multi-Person Support (track multiple people simultaneously)
• Full 3D kinematics (complete OpenSim skeletal analysis with joint angles)

Use cases:

• Sports Analysis (field-based 3D motion capture)
• Clinical Assessment (gait analysis in doctor's office)
• Animation (outdoor 3D capture with fully clothed subjects)
• Research (biomechanics studies with multiple participants)

[!NOTE] Fun fact: Pose2Sim stands for "OpenPose to OpenSim", as it originally used OpenPose inputs (2D keypoints coordinates) and led to an OpenSim result (full-body 3D joint angles). Pose estimation is now performed with more recent models from RTMPose, and custom models (from DeepLabCut, for example) can also be used.

[!IMPORTANT] If you like it, ⭐ please leave a star ⭐ on the Pose2Sim GitHub repository!
This project is completely free: this is your chance to support the project and make it more visible to the community.

[!IMPORTANT] If you want to contribute to Sports2D or Pose2Sim, please see How to contribute or join the Discord community!

Contents

1. Installation and Demonstration
   1. Installation
   2. Demonstration Part-1: End to end video to 3D joint angle computation
   3. Demonstration Part-2: Visualize your results with OpenSim or Blender
   4. Demonstration Part-3: Try multi-person and batch analyses
   5. Demonstration Part-4: Go further
   6. Too slow for you?
2. Use on your own data
   1. Start filming
   2. Set up your project
   3. Estimate 2D pose
      1. With RTMPose (default)
      2. With MMPose (coming soon)
      3. With DeepLabCut
      4. With OpenPose (legacy)
      5. With Mediapipe BlazePose (legacy)
      6. With AlphaPose (legacy)
   4. Calibrate your cameras
      1. Convert from Caliscope, AniPose, FreeMocap, Qualisys, Optitrack, Vicon, OpenCap, EasyMocap, or bioCV
      2. Calculate from scratch
   5. Synchronize your cameras
   6. Associate persons across cameras
   7. Triangulate keypoints
   8. Filter 3D coordinates
   9. Augment markers
   10. OpenSim kinematics
   11. Within Pose2Sim
   12. Within OpenSim GUI
   13. Command line
3. All the parameters
4. Utilities
5. How to cite and how to contribute
   1. How to cite
   2. How to contribute and to-do list

Installation and Demonstration

Installation

Let's get started!

[!NOTE] If you'd rather use conda, you can still use the old installation procedure. Still works fine but not recommended, since uv is faster, lighter, better at handling dependencies, and generally more modern.

1. Set up a uv environment:

Open a terminal (conda, powershell, bash, or zsh).

On Windows:

  # Install uv
  powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"
  # Create uv environment
  uv venv "$env:USERPROFILE\.venv\pose2sim" --python 3.13 # or 3.11, or 3.12 
  # Activate the uv environment
  & "$env:USERPROFILE\.venv\pose2sim\Scripts\Activate.ps1"

On Linux or MacOS:

  # Install uv
  curl -LsSf https://astral.sh/uv/install.sh | sh
  # Create uv environment
  uv venv ~/.venv/pose2sim --python 3.13 # or 3.11, or 3.12 
  # Activate the uv environment
  source ~/.venv/pose2sim/bin/activate

[!TIP]

Activate your environment

Remembering the command for activating the uv environment can be a pain. Just type Ctrl+R in your terminal and start typing activate to find it.

2. Install Pose2Sim:

Open a terminal (conda, powershell, bash, or zsh).
Activate your environment.

• OPTION 1: Latest stable version:

  cmd复制

    uv pip install pose2sim --upgrade

• OPTION 2: For developers who want to test and edit the bleeding edge version:

  cmd复制

    git clone --depth 1 https://github.com/perfanalytics/pose2sim.git
    cd pose2sim
    uv pip install -e .

3. Optional:

For faster inference, you can run on the GPU.
Be aware that GPU support takes almost 5 GB more on disk.

Open a terminal, activate your environment, and run nvidia-smi. If this results in an error, your GPU is probably not compatible with CUDA. If not, note the "CUDA version": it is the latest version your driver is compatible with (more information on this post).

Then go to the ONNXruntime requirement page, note the latest compatible CUDA and cuDNN requirements. Next, go to the pyTorch website and install the latest version that satisfies these requirements (beware that torch 2.4 ships with cuDNN 9, while torch 2.3 installs cuDNN 8). For example:

  uv pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu124

Finally, install ONNX Runtime with GPU support:

  uv pip uninstall onnxruntime
  uv pip install onnxruntime-gpu

Check that everything went well within Python with these commands:

  import torch; import onnxruntime as ort
  print(torch.cuda.is_available(), ort.get_available_providers())
  # Should print "True ['CUDAExecutionProvider', ...]"

[!TIP]

Note on storage use:

A full installation takes up to 14 GB of storage space. HoweveGPU and GUI supports are not mandatory and take about 5 GB. Tcache can be cleared to save space if you don't care for futuinstallations to be instantaneous. A minimal installation with carefully chosen pose models and without GPU support would take less than 2 GB. \

Demonstration Part-1: End to end video to 3D joint angle computation

Test the full pipeline!

• Open a terminal (conda, powershell, bash, or zsh) and activate your environment.

• Find the Demo folder under \Pose2Sim\Demo_SinglePerson:

  powershell复制

  uv pip show pose2sim # to find 

• Copy-paste the Demo folder wherever you like, and rename it as you wish (manually or using the cp command).
  Go to the Demo_SinglePerson folder and start python:

  powershell复制

  cd 
  ipython

• Try the following code:

  python复制

    from Pose2Sim import Pose2Sim
    Pose2Sim.calibration()
    Pose2Sim.poseEstimation()
    Pose2Sim.synchronization()
    Pose2Sim.personAssociation()
    Pose2Sim.triangulation()
    Pose2Sim.filtering()
    Pose2Sim.markerAugmentation()
    Pose2Sim.kinematics()

[!NOTE] 3D marker trajectories are stored as .trc files in each trial folder in the pose-3d directory.
3D joint angles are stored as .mot files in the kinematics directory. Scaled .osim models are also stored in the same directory.
Processing details and statistics are stored in logs.txt.

Demonstration Part-2: Visualize your results with OpenSim or Blender

Visualize your results and look in detail for potential areas of improvement.

Basic visualization with the OpenSim GUI

• Install OpenSim GUI:
  Download the executable there.
• Visualize results:
  • File > Open Model: Open your scaled .osim model in the kinematics folder.
  • File > Load Motion: Open your joint angle .mot file in the kinematics folder.
  • File > Preview Experimental Data: Open your 3D marker .trc file in the pose-3d folder.

Further investigation with the Pose2Sim Blender add-on

• Install the add-on:
  Follow instructions on the Pose2Sim_Blender add-on page.

• Visualize results:
  Just play with the buttons!
  Visualize camera positions, videos, triangulated keypoints, OpenSim skeleton, video overlay your results on videos, ... or let your creativity flow and create your own animations!

https://github.com/user-attachments/assets/53bfd9e2-f542-410f-8e9f-7b70229d598b

[!NOTE] The OpenSim skeleton is not rigged yet. Feel free to contribute!

Demonstration Part-3: Try multi-person and batch analyses

Try multi-person and batch analyses

Open a terminal (conda, powershell, bash, or zsh) and activate your environment.

Multi person analysis

Discover another person, hidden all along!

Similarly to Part-1, find the Multi-Person Demo folder under \Pose2Sim\Demo_MultiPerson, and move it and rename it if you like. Make sure you set multi_person = true in your Config.toml file. Go to the folder and start python:

  cd 
  ipython
  from Pose2Sim import Pose2Sim
  Pose2Sim.runAll()

[!NOTE] Pose2Sim uses sophisticated algorithms to Match people across views, Track IDs over time, Handle occlusions.

Batch processing

Scale from research studies to production pipelines!
Run numerous analyses with different parameters and minimal friction.

Find the Batch Demo folder under \Pose2Sim\Demo_Batch, and move it and rename it if you like.