Live2D VTuber 由 Unity 和 Cubism 打造。该项目基于 TCP 套接字、OpenCV 和 深度学习。
Live2D VTuber 由 Unity 和 Cubism 打造。该项目基于 TCP 套接字、OpenCV 和 深度学习。
Hello, I am Momose Hiyori, a Live2D VTuber from Japan. My parents are Cubsim and Unity, my teachers are OpenCV and Deep Learning. I am really good at mimicking faces including poses and expressions, it is a pleasure to meet you all and look forward to being good friends~ ❤️
| System | Windows 10 x64 |
| Language | Python 3.8 (algorithm), C# (Unity script) |
| IDE | Visual Studio Code 1.83.1 (algorithm), Live2D Cubism Editor 4.0 (Live2D model), Unity 2022.3.12f1 LTS (Engine) |
Install Python 3.8 (recommend to use Anaconda), Unity (recommend to use UnityHub).
Download the executable file from the latest release.
Environment configuration.
git clone https://github.com/KennardWang/VTuber-MomoseHiyori.git
cd VTuber-MomoseHiyori
conda env create -f environment.ymlActivate environment and install dependencies.
conda activate l2d-vtb
pip install -r requirements.txtThere are 2 types of running environments, please choose the env based on your condition.
CPU env
Install dlib, my version is v19.24.2.
conda install -c conda-forge dlibGPU env
Install scipy with version > 0.16, using conda is faster than pip.
conda install scipyWindows
Install CUDA v10.2 & CUDNN v8.3.1.
Install Pytorch.
pip install torch==1.10.2+cu102 torchvision==0.11.3+cu102 torchaudio===0.10.2+cu102 -f https://download.pytorch.org/whl/cu102/torch_stable.htmlMacOS (X86-64) with Intel
Install Pytorch.
conda install pytorch torchvision torchaudio -c pytorchMacOS (ARM64) with Apple M1
Install Pytorch.
conda install pytorch torchvision torchaudio -c pytorch-nightly(Optional) Download models in ckpts, unzip and place it at face_alignment/ckpts. If not, it will automatically download from the site.
Step 1 : Test camera
If it runs normally, you can see your face, and press q to quit.
CPU env
python main.py --debugGPU env
python main.py --debug --gpuCPU env (left) has higher FPS that may perform better in VTuber, but lower robustness that cannot recognize well if some parts of the face are covered.
GPU env (right) has higher robustness, better fluency and consecutiveness but lower FPS because of busier data IO.
Step 2 : Connect Unity
CPU env
python main.py --debug --connectGPU env
python main.py --debug --gpu --connectOptional : Set Parameters
There are some parameters you can change. The default port number is 14514, you could set another port number via --port=PORT_NUMBER, but keep it the same as that in the Unity client. Moreover, you can set the camera via --cam=CAM_INDEX if you have more than one camera, the default camera index is 0. Please check the guidance of all parameters by using python main.py --help.
Tips : Improve Performance
| File / Folder | Description |
|---|---|
| UnityAssets | Unity materials for those who want to make Live2D VTuber by themselves, see tutorial. |
| face_alignment | Module for face detection and landmarks by a powerful face alignment network (FAN), recommend to use under GPU env. |
| face_pose | Module for face pose detection, contains the pose estimator and stabilizer. |
| tests | Test folder for GitHub Actions. |
| main.py | Main program. |
| sock.py | File with Socket class. |
| utils.py | File with utility functions. |
Obviously, the position of landmarks is meaningless. In most cases, we focus on the ratio (or aspect ratio) which could describe the degree, because in Unity, we use a scrolling bar to control actions or events. However, the ratio is mostly not perfect correspondence with the bar value. An interesting question is how to calibrate and map the ratio to the bar value. There are some feasible methods I implement in this project:
…Feel free to open an issue or submit PRs. Be careful that the submitted code needs to pass format tests and pytests by GitHub Actions, please self-check before pulling requests.
flake8 format test (Windows, MacOS and Linux)
flake8 --exclude=face_alignment,face_pose --ignore=E501 .pytest (Windows, MacOS and Linux)
pytest tests/test_win.py
pytest tests/test_mac.py
pytest tests/test_linux.py© Kennard Wang ( 2020.6.27 )
暂无开放 Issues,或尚未同步最近议题。