VisionClaw

VisionClaw

A real-time AI assistant for Meta Ray-Ban smart glasses. See what you see, hear what you say, and take actions on your behalf -- all through voice.

Built on Meta Wearables DAT SDK (iOS) / DAT Android SDK (Android) + Gemini Live API + OpenClaw (optional).

Supported platforms: iOS (iPhone) and Android (Pixel, Samsung, etc.)

What It Does

Put on your glasses, tap the AI button, and talk:

• "What am I looking at?" -- Gemini sees through your glasses camera and describes the scene
• "Add milk to my shopping list" -- delegates to OpenClaw, which adds it via your connected apps
• "Send a message to John saying I'll be late" -- routes through OpenClaw to WhatsApp/Telegram/iMessage
• "Search for the best coffee shops nearby" -- web search via OpenClaw, results spoken back

The glasses camera streams at ~1fps to Gemini for visual context, while audio flows bidirectionally in real-time.

How It Works

…

Key pieces:

• Gemini Live -- real-time voice + vision AI over WebSocket (native audio, not STT-first)
• OpenClaw (optional) -- local gateway that gives Gemini access to 56+ tools and all your connected apps
• Phone mode -- test the full pipeline using your phone camera instead of glasses
• WebRTC streaming -- share your glasses POV live to a browser viewer

The app asks for an access code

The first screen asks for an access code. The code is not something the author issues -- it is a token from whichever gateway the app points at, and the gate verifies it against that gateway before unlocking.

• The default gateway (api.visionagents.app) is a small private pilot instance for a user study, not a public service, so its codes are not handed out.
• Self-hosting: run the gateway in gateway/ and set GATEWAY_TOKENS="<any-secret>:<your-name>" in its .env -- that secret is your access code. Point the app at your gateway either in Secrets.swift / Secrets.kt (gatewayBaseUrl) before building, or at runtime with "Using your own gateway?" on the access-code screen.
• Voice needs the whole stack: the gateway only mints room tickets and runs tasks. Calls also need the agent/ worker running against a LiveKit Cloud project (free tier works), with the same LiveKit credentials on both the gateway and the worker -- otherwise the app connects and waits on "Waiting for agent" forever.
• Ray-Ban Meta glasses need Developer Mode enabled in the Meta AI app on the phone; without it the glasses link is rejected ("Error opening link").

Details for the gateway, tokens and app connections: gateway/README.md.

Quick Start (iOS)

1. Clone and open

git clone https://github.com/sseanliu/VisionClaw.git
cd VisionClaw/samples/CameraAccess
open CameraAccess.xcodeproj

2. Add your secrets

Copy the example file and fill in your values:

cp CameraAccess/Secrets.swift.example CameraAccess/Secrets.swift

Edit Secrets.swift with your Gemini API key (required) and optional OpenClaw/WebRTC config.

3. Build and run

Select your iPhone as the target device and hit Run (Cmd+R).

4. Try it out

Without glasses (iPhone mode):

1. Tap "Start on iPhone" -- uses your iPhone's back camera
2. Tap the AI button to start a Gemini Live session
3. Talk to the AI -- it can see through your iPhone camera

With Meta Ray-Ban glasses:

First, enable Developer Mode in the Meta AI app:

1. Open the Meta AI app on your iPhone
2. Go to Settings (gear icon, bottom left)
3. Tap App Info
4. Tap the App version number 5 times -- this unlocks Developer Mode
5. Go back to Settings -- you'll now see a Developer Mode toggle. Turn it on.

Then in VisionClaw:

1. Tap "Start Streaming" in the app
2. Tap the AI button for voice + vision conversation

Quick Start (Android)

1. Clone and open

git clone https://github.com/sseanliu/VisionClaw.git

Open samples/CameraAccessAndroid/ in Android Studio.

2. Configure GitHub Packages (DAT SDK)

The Meta DAT Android SDK is distributed via GitHub Packages. You need a GitHub Personal Access Token with read:packages scope.

1. Go to GitHub > Settings > Developer Settings > Personal Access Tokens and create a classic token with read:packages scope
2. In samples/CameraAccessAndroid/local.properties, add:

github_token=YOUR_GITHUB_TOKEN

Tip: If you have the gh CLI installed, you can run gh auth token to get a valid token. Make sure it has read:packages scope -- if not, run gh auth refresh -s read:packages.

Note: GitHub Packages requires authentication even for public repositories. The 401 error means your token is missing or invalid.

3. Add your secrets

cd samples/CameraAccessAndroid/app/src/main/java/com/meta/wearable/dat/externalsampleapps/cameraaccess/
cp Secrets.kt.example Secrets.kt

Edit Secrets.kt with your Gemini API key (required) and optional OpenClaw/WebRTC config.

4. Build and run

1. Let Gradle sync in Android Studio (it will download the DAT SDK from GitHub Packages)
2. Select your Android phone as the target device
3. Click Run (Shift+F10)

Wireless debugging: You can also install via ADB wirelessly. Enable Wireless debugging in your phone's Developer Options, then pair with adb pair <ip>:<port>.

5. Try it out

Without glasses (Phone mode):

1. Tap "Start on Phone" -- uses your phone's back camera
2. Tap the AI button (sparkle icon) to start a Gemini Live session
3. Talk to the AI -- it can see through your phone camera

With Meta Ray-Ban glasses:

Enable Developer Mode in the Meta AI app (same steps as iOS above), then:

1. Tap "Start Streaming" in the app
2. Tap the AI button for voice + vision conversation

Setup: OpenClaw (Optional)

OpenClaw gives Gemini the ability to take real-world actions: send messages, search the web, manage lists, control smart home devices, and more. Without it, Gemini is voice + vision only.

1. Install and configure OpenClaw

Follow the OpenClaw setup guide. Make sure the gateway is enabled:

In ~/.openclaw/openclaw.json:

{
  "gateway": {
    "port": 18789,
    "bind": "lan",
    "auth": {
      "mode": "token",
      "token": "your-gateway-token-here"
    },
    "http": {
      "endpoints": {
        "chatCompletions": { "enabled": true }
      }
    }
  }
}

Key settings:

• bind: "lan" -- exposes the gateway on your local network so your phone can reach it
• chatCompletions.enabled: true -- enables the /v1/chat/completions endpoint (off by default)
• auth.token -- the token your app will use to authenticate

2. Configure the app

iOS -- In Secrets.swift:

static let openClawHost = "http://Your-Mac.local"
static let openClawPort = 18789
static let openClawGatewayToken = "your-gateway-token-here"

Android -- In Secrets.kt:

const val openClawHost = "http://Your-Mac.local"
const val openClawPort = 18789
const val openClawGatewayToken = "your-gateway-token-here"

To find your Mac's Bonjour hostname: System Settings > General > Sharing -- it's shown at the top (e.g., Johns-MacBook-Pro.local).

Both iOS and Android also have an in-app Settings screen where you can change these values at runtime without editing source code.

3. Start the gateway

openclaw gateway restart

Verify it's running:

curl http://localhost:18789/health

Now when you talk to the AI, it can execute tasks through OpenClaw.

Architecture

Key Files (iOS)

All source code is in samples/CameraAccess/CameraAccess/:

Key Files (Android)

All source code is in samples/CameraAccessAndroid/app/src/main/java/.../cameraaccess/:

Audio Pipeline

• Input: Phone mic -> AudioManager (PCM Int16, 16kHz mono, 100ms chunks) -> Gemini WebSocket
• Output: Gemini WebSocket -> AudioManager playback queue -> Phone speaker
• iOS iPhone mode: Uses .voiceChat audio session for echo cancellation + mic gating during AI speech
• iOS Glasses mode: Uses .videoChat audio session (mic is on glasses, speaker is on phone -- no echo)
• Android: Uses VOICE_COMMUNICATION audio source for built-in acoustic echo cancellation

Video Pipeline

• Glasses: DAT SDK video stream (24fps) -> throttle to ~1fps -> JPEG (50% quality) -> Gemini
• Phone: Camera capture (30fps) -> throttle to ~1fps -> JPEG -> Gemini

Tool Calling

Gemini Live supports function calling. Both apps declare a single execute tool that routes everything through OpenClaw:

1. User says "Add eggs to my shopping list"
2. Gemini speaks "Sure, adding that now" (verbal acknowledgment before tool call)
3. Gemini sends toolCall with execute(task: "Add eggs to the shopping list")
4. ToolCallRouter sends HTTP POST to OpenClaw gateway
5. OpenClaw executes the task using its 56+ connected skills
6. Result returns to Gemini via toolResponse
7. Gemini speaks the confirmation

WebRTC Live Streaming

Share your glasses POV in real-time to a browser viewer with bidirectional audio and video.

1. Tap the Live button in the app
2. The app connects to a signaling server and gets a 6-character room code
3. Share the code -- the viewer opens the server URL in a browser and enters it
4. WebRTC peer connection is established (SDP + ICE via the signaling server)
5. Media flows peer-to-peer: glasses video to browser, browser camera back to iOS PiP

Key details:

• Signaling server: Node.js + WebSocket, located at samples/CameraAccess/server/ -- serves the browser viewer and relays SDP/ICE
• NAT traversal: Google STUN servers + ExpressTURN relay (fetched from /api/turn)
• Video: 24 fps, 2.5 Mbps max bitrate
• Background handling: 60-second grace period for iOS app backgrounding -- room stays al