ChatdollKit

ChatdollKit

3D virtual assistant SDK that enables you to make your 3D model into a voice-enabled chatbot. 日本語のREADMEはこちら

• Live demo A WebGL demo. Say "Hello" to start conversation. She’s multilingual, so you can ask her something like "Let's talk in Japanese" when you want to switch languages.
• iOS App: OshaberiAI A Virtual Agent App made with ChatdollKit: a perfect fusion of character creation by AI prompt engineering, customizable 3D VRM models, and your favorite voices by VOICEVOX.

✨ Features

• Generative AI Native: Supports multiple LLMs like ChatGPT, Anthropic Claude, Google Gemini Pro, Dify, and others, with function calling (ChatGPT/Gemini) and multimodal capabilities.
• 3D model expression: Synchronizes speech and motion, controls facial expressions and animations autonomously, supports blinking and lip-sync.
• Dialog control: Integrates Speech-to-Text and Text-to-Speech (OpenAI, Azure, Google, VOICEVOX / AivisSpeech, Aivis Cloud API, Style-Bert-VITS2 etc.), manages dialog state (context), extracts intents and routes topics, supports wakeword detection.
• Multi platforms: Compatible with Windows, Mac, Linux, iOS, Android, and other Unity-supported platforms, including VR, AR, and WebGL.

What's New in Version 0.8.16

• ️ WebSocket Streaming STT: WebSocket-based streaming speech recognition offloads VAD to the server and completes recognition during turn-end detection, reducing overall response latency by several hundred milliseconds.
• ️ Barge-in Support: Users can now interrupt AI speech mid-sentence with their voice, making conversations feel more natural and responsive.
• ** ModelController Refactoring**: Extracted speech handling into SpeechController and face expressions into FaceController, improving maintainability and extensibility.

️ Previous Updates (click to expand)

0.8.15

• ** WebGL Enhancements**: Add Silero VAD support, camera switching (front/rear) with correct aspect ratio handling, file upload for images, optimized microphone data transfer, and fixes for lip-sync when muted.
• ✨ UI Control Improvements: Sleeker and more streamlined UI controls that work out-of-the-box with zero configuration—just drop them onto your scene’s Canvas.
• ** Stronger Noise Resistance**: Combine multiple voice activity detection methods (e.g., Silero VAD + built-in energy-based VAD) to better capture user speech even in noisy environments like event venues.

0.8.14

• ️ Echo Cancelling Support: Add native microphone support for Android, iOS, and macOSX that support AEC, noise cancelling and other features for voice conversation.
• ️ Conversation Improvement: Prevent conversation breakdown caused by turn-end misrecognition and improve conversation experience with features like automatic volume control when users interrupt during AI speech
• ** Platform Expansion**: Support for Aivis Cloud API TTS, AIAvatarKit TTS/STT, and GPT-5 reasoning_effort parameter

0.8.13

• ** Silero VAD Support**: ML-based voice-activity detection vastly improves turn-end accuracy in noisy settings, enabling smooth conversations outdoors or at events.
• ** TTS Pre-processing**: Optional text pre-processing lets you fine-tune pronunciation (e.g., convert “OpenAI” to katakana) before synthesis.
• ** Grok & Gemini Compatibility**: Removes OpenAI-specific params from the OpenAI-style endpoint, so Grok, Gemini, and other API-compatible models work out of the box.

0.8.11 and 0.8.12

• ** AIAvatarKit Backend**: Offloads AI agent logic to the server—boosting front-end maintainability—while letting you plug in frameworks like AutoGen (and any other agent SDK) for unlimited capability expansion.
• ** WebGL Improvements**: Upgraded mic capture to modern AudioWorkletNode for lower latency and reliability; stabilized mute/unmute handling; improved error handling to immediately surface HTTP errors and prevent hangs; fixed API-key authorization in WebGL builds.

0.8.10

• ** Dynamic Multi-Language**: The system can now autonomously switch languages for both speaking and listening during conversations.
• ** Long-Term Memory**: Past conversation history can now be stored and searched. Components are provided for ChatMemory, but you can also integrate with services like mem0 or Zep.

0.8.8 and 0.8.9

• ✨ Support NijiVoice as a Speech Synthesizer: Now support NijiVoice, an AI-Powered Expressive Speech Generation Service.
• ** Support Multiple AITuber Dialogue**: AITubers can now chat with each other, bringing dynamic and engaging interactions to life like never before!
• ** Support Dify as a backend for AITuber**: Seamlessly integrate with any LLM while empowering AITubers with agentic capabilities, blending advanced knowledge and functionality for highly efficient and scalable operations!

0.8.7

• ✨ Update AITuber demo: Support more APIs, bulk configuration, UI and mode!. (v0.8.7)

0.8.6

• ️ Support VOICEVOX and AivisSpeech inline style: Enables dynamic and autonomous switching of voice styles to enrich character expression and adapt to emotional nuances.
• ** Improve VRM runtime loading**: Allows seamless and error-free switching of 3D models at runtime, ensuring a smoother user experience.

0.8.5

• ** Chain of Thought Prompting**: Say hello to Chain of Thought (CoT) Prompting! Your AI character just got a major boost in IQ and EQ!

0.8.4

• ** Modularized for Better Reusability and Maintainability**: We’ve reorganized key components, focusing on modularity to improve customizability and reusability. Check out the demos for more details!
• ** Removed Legacy Components**: Outdated components have been removed, simplifying the toolkit and ensuring compatibility with the latest features. Refer to Migration from 0.7.x if you're updating from v0.7.x.

0.8.3

• ** Stream Speech Listener**: We’ve added AzureStreamSpeechListener for smoother conversations by recognizing speech as it’s spoken.
• ️ Improved Conversation: Interrupt characters to take your turn, and enjoy more expressive conversations with natural pauses—enhancing the overall experience.
• ** Easier Animation Registration**: We’ve simplified the process of registering animations for your character, making your code cleaner and easier to manage.

0.8.2

• ** Control WebGL Character from JavaScript**: We’ve added the ability to control the ChatdollKit Unity application from JavaScript when running in WebGL builds. This allows for more seamless interactions between the Unity app and web-based systems.
• ️ Speech Synthesizer: A new SpeechSynthesizer component has been introduced to streamline text-to-speech (TTS) operations. This component is reusable across projects without Model package, simplifying maintenance and reusability.

0.8.1

• ️ User-Defined Tags Support: You can now include custom tags in AI responses, enabling dynamic actions. For instance, embed language codes in replies to switch between multiple languages on the fly during conversations.
• ** External Control via Socket**: Now supports external commands through Socket communication. Direct conversation flow, trigger specific phrases, or control expressions and gestures, unlocking new use cases like AI Vtubers and remote customer service. Check out the client-side demo here: https://gist.github.com/uezo/9e56a828bb5ea0387f90cc07f82b4c15

0.8 Beta

• ⚡ Optimized AI Dialog Processing: We've boosted response speed with parallel processing and made it easier for you to customize behavior with your own code. Enjoy faster, more flexible AI conversations!
• ** Emotionally Rich Speech**: Adjusts vocal tone dynamically to match the conversation, delivering more engaging and natural interactions.
• ** Enhanced Microphone Control**: Microphone control is now more flexible than ever! Easily start/stop devices, mute/unmute, and adjust voice recognition thresholds independently.

Quick Start

You can learn how to setup ChatdollKit by watching this video that runs the demo scene(including chat with ChatGPT): https://www.youtube.com/watch?v=rRtm18QSJtc To run the demo for version 0.8, please follow the steps below after importing the dependencies:

• Open scene Demo/Demo08.
• Select AIAvatarVRM object in scene.
• Set OpenAI API key to following components on inspector:
  • ChatGPTService
  • OpenSpeechSynthesizer
  • OpenAISpeechListener
• Run on Unity Editor.
• Say "こんにちは" or word longer than 3 characters.

Table of Contents

• Setup New Project
  • Import dependencies
  • Resource preparation
  • AIAvatarVRM prefab
  • ModelController
  • Animator
  • AIAvatar
  • LLM Service
  • Speech Service
  • Microphone Controller
  • Run
• LLM Service
  • Basic Settings
  • Facial Expressions
  • Animations
  • Pause in Speech
  • User Defined Tag
  • Multi Modal
  • Chain of Thought Prompting
  • Consecutive Request Merging
  • Timestamp Insertion
  • Edit Chat Completion Request
  • Long-Term Memory
• ️ Speech Synthesizer (Text-to-Speech)
  • Voice Prefetch Mode
  • Make custom SpeechSynthesizer
  • Performance and Quality Tuning
  • Preprocessing
• Speech Listener (Speech-to-Text)
  • Settings on AIAvatar Inspector
  • Downsampling
  • Using AzureStreamSpeechListener
  • Using Silero VAD
  • Using Multiple VADs Combination
  • Echo Cancelling
  • Custom Barge-in Condition
• ⏰ Wake Word Detection
  • Wake Words
  • Cancel Words
  • Interrupt Words
  • Ignore Words
  • Wake Length
• ⚡️ AI Agent (Tool Call)
• ️ Devices
  • Microphone
  • Camera
• 3D Model Control
  • Idle Animations
  • Control by Script
• ️ UI Components
• Control from External Programs
  • ChatdollKit Remote Client
• Run on WebGL
• Migration from 0.7.x
• ❤️ Thanks

Setup New Project

The steps for setting up with a VRM model are as follows. For instructions on using models for VRChat, refer to README v0.7.7.

⚠️CAUTION: Do not use the SRP (Scriptable Render Pipeline) project template in Unity. UniVRM, which ChatdollKit depends on, does not support SRP.

Import dependencies

Download the latest version of ChatdollKit.unitypackage and import it into your Unity project after import dependencies;

• Burst fro