VisionProTeleop

VisionProTeleop

A complete ecosystem for using Apple Vision Pro in robotics research — from real-world teleoperation to simulation teleoperation to egocentric dataset recording. Stream hand/head tracking from Vision Pro, send video/audio/simulation back, and record everything to the cloud.

For a more detailed explanation, check out this short paper.

The recently updated App Store version of Tracking Streamer requires python library avp_stream over 2.50.0. It will show a warning message on the VisionOS side if the python library is outdated. You can upgrade the library by running pip install --upgrade avp_stream.

Table of Contents

• Overview
• Installations
• External Network (Remote) Mode
  • How It Works
  • Usage
  • Notes
• Use Case 1: Real-World Teleoperation
  • Video & Audio Streaming
  • Video Configuration Examples
  • Audio Configuration Examples
• Use Case 2: Simulation Teleoperation
  • MuJoCo Streaming
  • Isaac Lab Streaming
  • Positioning Your Simulation in AR
  • Hand Tracking Coordinate Frame
• Use Case 3: Egocentric Video Dataset Recording
  • USB Camera Mounting on AVP
  • Camera Calibration
• Recording & Cloud Storage
  • Automatic Cloud Sync
  • Companion iOS App: Tracking Manager
  • Public Dataset Sharing
• App Settings & Customization
• API Reference
  • Hand Tracking Data
    • Predictive Hand Tracking
  • Marker & Image Tracking
  • Stylus Tracking
  • Configuration Reference
  • Axis Convention
  • Hand Skeleton
• Performance
• Examples
• Appendix
  • Citation
  • Acknowledgements

Overview

This project provides:

1. Tracking Streamer: A VisionOS app that
   • streams hand/head tracking data to Python client
   • receive stereo/mono video/audio streams from Python client
   • present simulation scenes (MuJoCo and Isaac Lab) and its updates with native AR rendering using RealityKit
   • record egocentric video with hand tracking with arbitrary UVC camera connected to Vision Pro
   • (optionally) record every sessions to user's personal cloud storage
2. avp_stream: A Python library for
   • receiving tracking data from Vision Pro
   • streaming video/audio/simulation back to Vision Pro
3. Tracking Manager: A companion iOS app for
   • managing and viewing recordings on their personal cloud storage
   • configuring app settings for VisionOS app
   • calibrating camera mounted on Vision Pro
   • sharing recorded datasets with others
   • viewing publicly shared datasets

Together, they enable three major workflows for robotics research:

Use Case
Description
Primary Tools




Real-World Teleoperation
Control physical robots with hand tracking while viewing robot camera feeds
avp_stream + WebRTC streaming configure_video() (or direct USB connection) of Physical Camera




Simulation Teleoperation
2D Renderings
Control simulated robots with hand tracking while viewing 2D renderings from simulation
avp_stream + WebRTC streaming of 2D simulation rendering configure_video() 




AR
Control simulated robots with MuJoCo/Isaac Lab with scenes directly presented in AR
avp_stream + MuJoCo/Isaac Lab streaming configure_mujoco() configure_isaac()




Egocentric Human Video Recording
Record first-person manipulation videos with synchronized tracking
UVC camera + Developer Strap

Installations

Installing is easy: install it from the App Store and PyPI.

No other network configurations are required. Everything should work out of the box after installation. An easy way to get onboarded is to go through the examples folder. All examples should work out of the box without any extra configurations required.

Note: Some examples demonstrate teleoperating things within IsaacLab world; since IsaacLab is an extremely heavy dependency, I did not include that as a dependency for avp_stream. If you're trying to run examples including IsaacLab as a simulation backend, you should install things according to their official installation guide.

External Network (Remote) Mode

So far, Vision Pro and your Python client had to be on the same local network (e.g., same WiFi) for them to communicate. With External Network Mode with v2.5 release, you can make bilateral connection from anywhere over the internet! It's extremely useful when your robot is in a lab (likely behind school/company network's firewall) and you're working remotely using your home WiFi outside the school network.

How It Works

External Network Mode uses WebRTC with TURN relay servers for NAT traversal:

1. Vision Pro generates a room code and connects to a signaling server
2. Python client connects using the same room code
3. Signaling server facilitates the initial handshake (SDP offer/answer, ICE candidates)
4. TURN servers relay media when direct peer-to-peer connection isn't possible
5. Once connected, all streaming works the same as local mode

Usage

from avp_stream import VisionProStreamer

# Instead of IP address, use the room code shown on Vision Pro
s = VisionProStreamer(ip="ABC-1234")

# Everything else works exactly the same
s.configure_video(device="/dev/video0", format="v4l2", size="1280x720", fps=30)
s.start_webrtc()

while True:
    r = s.get_latest()
    # ...

Notes

• Latency: Expect slightly higher latency compared to local network due to relay routing
• Signaling/TURN server: We provide a Cloudflare-hosted signaling and TURN server for now by default. If we detect extreme usage or abuse, we may introduce usage limits or require a paid tier in the future.

Use Case 1: Real-World Teleoperation

Stream your robot's camera feed to Vision Pro while receiving hand/head tracking data for control. Perfect for teleoperating physical robots with visual feedback.

Video & Audio Streaming

from avp_stream import VisionProStreamer

avp_ip = "10.31.181.201"  # Vision Pro IP (shown in the app)
s = VisionProStreamer(ip=avp_ip)

# Configure video streaming from robot camera
s.configure_video(device="/dev/video0", format="v4l2", size="1280x720", fps=30)
s.start_webrtc()

while True:
    r = s.get_latest()
    # Use tracking data to control your robot
    head_pose = r['head']
    right_wrist = r['right_wrist']
    right_fingers = r['right_fingers']

Video Configuration Examples

Camera with overlay processing:

def add_overlay(frame):
    return cv2.putText(frame, "Robot View", (50, 50), 
                       cv2.FONT_HERSHEY_SIMPLEX, 1, (0, 255, 0), 2)

s = VisionProStreamer(ip=avp_ip)
s.register_frame_callback(add_overlay)
s.configure_video(device="/dev/video0", format="v4l2", size="640x480", fps=30)
s.start_webrtc()

Stereo camera (side-by-side 3D):

s = VisionProStreamer(ip=avp_ip)
s.configure_video(device="/dev/video0", format="v4l2", size="1920x1080", fps=30, stereo=True)
s.start_webrtc()

Synthetic video (generated frames):

s = VisionProStreamer(ip=avp_ip)
s.register_frame_callback(render_visualization)  # Your rendering function
s.configure_video(size="1280x720", fps=60)  # No device = synthetic mode
s.start_webrtc()

Audio Configuration Examples

With microphone input:

s = VisionProStreamer(ip=avp_ip)
s.configure_video(device="/dev/video0", format="v4l2", size="1280x720", fps=30)
s.configure_audio(device=":0", stereo=True)  # Default mic
s.start_webrtc()

With synthetic audio (feedback sounds):

def beep_on_pinch(audio_frame):
    # Generate audio based on hand tracking state
    return audio_frame

s = VisionProStreamer(ip=avp_ip)
s.register_audio_callback(beep_on_pinch)
s.configure_video(size="1280x720", fps=60)
s.configure_audio(sample_rate=48000, stereo=True)
s.start_webrtc()

Use Case 2: Simulation Teleoperation

Render MuJoCo/Isaac Lab physics simulations directly in AR on Vision Pro. Consider it as a 3D Lifted version of your simulation renderings; rather than rendering your simulation environments on a 2D flat screen (either mono/stereo), you can view them in a 3D space in AR with super realistic rendering provided by Apple RealityKit.

The simulation environment (both for MuJoCo/Isaac Lab) is automatically converted to USD and rendered natively using RealityKit, with real-time pose updates streamed via WebRTC. Note that pose updates are way more compact than the full rendered frames in terms of network communication, enabling low-latency and reliable teleoperation experience.

Control simulated robots with your hands in a mixed-reality environment.

https://github.com/user-attachments/assets/7e6a3b6a-34f8-472a-ac6f-0f032fc0eae5

MuJoCo Streaming

import mujoco
from avp_stream import VisionProStreamer

model = mujoco.MjModel.from_xml_path("robot.xml")
data = mujoco.MjData(model)

s = VisionProStreamer(ip=avp_ip)
s.configure_mujoco("robot.xml", model, data, relative_to=[0, 0, 0.8, 90])
s.start_webrtc()

while True:
    # Your control logic using hand tracking
    r = s.get_latest()
    # ... update robot based on hand positions ...
    
    mujoco.mj_step(model, data)
    s.update_sim()  # Stream updated poses to Vision Pro

Isaac Lab Streaming

from avp_stream import VisionProStreamer

# After creating your Isaac Lab environment...
streamer = VisionProStreamer(ip=avp_ip)
streamer.configure_isaac(
    scene=env.scene,
    relative_to=[0, 0, 0.8, 90],
    include_ground=False,
    env_indices=[0],  # Stream only first environment
)
streamer.start_webrtc()

while simulation_app.is_running():
    env.step(action)
    streamer.update_sim()  # Stream updated poses to Vision Pro

Positioning Your Simulation in AR

Since AR blends your simulation with the real world, you need to decide where the simulation's world frame should be placed in your physical space. Use relative_to parameter:

• 4-dim: [x, y, z, yaw°] — translation + rotation around z-axis (degrees)
• 7-dim: [x, y, z, qw, qx, qy, qz] — full quaternion orientation

# Place world frame 0.8m above ground, rotated 90° around z-axis
s.configure_mujoco("robot.xml", model, data, relative_to=[0, 0, 0.8, 90])

Default Behavior: VisionOS automatically detects the physical ground