realtime-ai

> 编程语言

开源

一个用于音频和视频的实时代理框架。

182 stars0 点赞0 次浏览

访问官网 GitHub

工具介绍

一个用于音频和视频的实时代理框架。

Realtime AI

Overview

Realtime AI is a WebRTC-based framework for building low-latency AI applications with audio and video. It features a modular pipeline architecture inspired by GStreamer, enabling you to compose processing elements for speech recognition, LLM interactions, and text-to-speech.

Architecture:

Client (Browser) → WebRTC Gateway → AI Pipeline
                                    (Decode → STT → LLM → TTS → Encode)

Features

Low Latency - WebRTC for real-time audio/video streaming
Modular Pipelines - Composable processing elements
AI Integrations - Gemini, OpenAI Realtime API, Azure STT/TTS
⚡ Interruption Support - Natural conversation flow

Quick Start

Installation

macOS:

bash

brew install opus ffmpeg go

Ubuntu/Debian (推荐使用安装脚本):

bash

# 使用预编译 FFmpeg (更稳定)
./scripts/setup-ffmpeg.sh
eval "$(./scripts/setup-ffmpeg.sh --env)"

# 安装其他依赖
apt-get install pkg-config libopus-dev

Ubuntu/Debian (手动安装):

bash

apt-get install pkg-config libopus-dev libavcodec-dev libavformat-dev libavutil-dev libswresample-dev

Setup:

bash

git clone https://github.com/realtime-ai/realtime-ai.git
cd realtime-ai
go mod download

Run Example

bash

# Set API key
export GOOGLE_API_KEY="your_api_key"

# Run Gemini assistant
go run examples/gemini-assis/main.go

# Open browser
open http://localhost:8080

Basic Usage

// Create pipeline
pipeline := pipeline.NewPipeline("assistant")

// Add and link elements
resample := elements.NewAudioResampleElement("resample")
gemini := elements.NewGeminiElement("gemini", apiKey)
audioPacer := elements.NewAudioPacerSinkElement("audioPacer")

pipeline.Link(resample, gemini)
pipeline.Link(gemini, audioPacer)

// Start processing
pipeline.Start(ctx)

Documentation

CLAUDE.md - Development guide and architecture details
WebRTC Protocol - Signaling protocol specification

Project Structure

pkg/
├── pipeline/      # Core pipeline system
├── elements/      # AI, codecs, and processing elements
├── connection/    # WebRTC abstractions
├── server/        # HTTP/WebRTC server
└── audio/         # Audio utilities

examples/
├── gemini-assis/  # Gemini multimodal assistant
├── local-assis/   # Local connection example
└── openai-realtime/ # OpenAI Realtime API