百科.dev
全部条目在线工具AI 编程趋势榜开源项目技术资讯提交条目
登录
< 返回工具列表
V

VTuber-MomoseHiyori

> 编程语言
开源

Live2D VTuber 由 Unity 和 Cubism 打造。该项目基于 TCP 套接字、OpenCV 和 深度学习。

199 stars0 点赞1 次浏览
访问官网GitHub

工具介绍

Live2D VTuber 由 Unity 和 Cubism 打造。该项目基于 TCP 套接字、OpenCV 和 深度学习。

VTuber-MomoseHiyori

Hello, I am Momose Hiyori, a Live2D VTuber from Japan. My parents are Cubsim and Unity, my teachers are OpenCV and Deep Learning. I am really good at mimicking faces including poses and expressions, it is a pleasure to meet you all and look forward to being good friends~ ❤️

&nbsp;
  • Head: 6 degrees of freedom.
  • Eyes: opening, half-opening, glare, closing, blinking, eyeballs rotation.
  • Eyebrows: raising.
  • Mouth: opening and closing, grinning and pouting.

Table of Contents

  • Development Environment
  • Install
  • Usage
  • File/Folder Description
  • Calibration Algorithms
  • References
  • Releases
  • Maintainers
  • Contributing
  • License

Development Environment

System Windows 10 x64
Language Python 3.8 (algorithm), C# (Unity script)
IDE Visual Studio Code 1.83.1 (algorithm), Live2D Cubism Editor 4.0 (Live2D model), Unity 2022.3.12f1 LTS (Engine)

Install

  1. Install Python 3.8 (recommend to use Anaconda), Unity (recommend to use UnityHub).

  2. Download the executable file from the latest release.

  3. Environment configuration.

    bash
    git clone https://github.com/KennardWang/VTuber-MomoseHiyori.git
    cd VTuber-MomoseHiyori
    conda env create -f environment.yml
  4. Activate environment and install dependencies.

    bash
    conda activate l2d-vtb
    pip install -r requirements.txt
  5. There are 2 types of running environments, please choose the env based on your condition.

    • CPU env

      • Install dlib, my version is v19.24.2.

        conda install -c conda-forge dlib
    • GPU env

      • Install scipy with version > 0.16, using conda is faster than pip.

        conda install scipy
      • Windows

        • Install CUDA v10.2 & CUDNN v8.3.1.

        • Install Pytorch.

          bash
          pip install torch==1.10.2+cu102 torchvision==0.11.3+cu102 torchaudio===0.10.2+cu102 -f https://download.pytorch.org/whl/cu102/torch_stable.html
      • MacOS (X86-64) with Intel

        • Install Pytorch.

          conda install pytorch torchvision torchaudio -c pytorch
      • MacOS (ARM64) with Apple M1

        • Install Pytorch.

          conda install pytorch torchvision torchaudio -c pytorch-nightly
      • (Optional) Download models in ckpts, unzip and place it at face_alignment/ckpts. If not, it will automatically download from the site.

Usage

Step 1 : Test camera

If it runs normally, you can see your face, and press q to quit.

  • CPU env

    bash
    python main.py --debug
  • GPU env

    bash
    python main.py --debug --gpu

    CPU env (left) has higher FPS that may perform better in VTuber, but lower robustness that cannot recognize well if some parts of the face are covered.

    GPU env (right) has higher robustness, better fluency and consecutiveness but lower FPS because of busier data IO.

Step 2 : Connect Unity

  1. Run the Unity executable file.
  2. Run the command according to your env.
    • CPU env

      bash
      python main.py --debug --connect
    • GPU env

      bash
      python main.py --debug --gpu --connect

Optional : Set Parameters

There are some parameters you can change. The default port number is 14514, you could set another port number via --port=PORT_NUMBER, but keep it the same as that in the Unity client. Moreover, you can set the camera via --cam=CAM_INDEX if you have more than one camera, the default camera index is 0. Please check the guidance of all parameters by using python main.py --help.

Tips : Improve Performance

  • Use spotlight: Try to make your face look brighter, the spotlight probably shows a better effect than the natural light.
  • Adjust face position: The debug mode of the camera may help you to know the position of your face. Try to put your face in a central position, and let the frame extend but not reach the boundary.
  • No glasses: Wearing glasses may influence the accuracy of recognition on eyes.
  • Show forehead: If your hair is so long that covers your eyes, it may also cause side effects on the recognition of eyes.

File/Folder Description

File / Folder Description
UnityAssets Unity materials for those who want to make Live2D VTuber by themselves, see tutorial.
face_alignment Module for face detection and landmarks by a powerful face alignment network (FAN), recommend to use under GPU env.
face_pose Module for face pose detection, contains the pose estimator and stabilizer.
tests Test folder for GitHub Actions.
main.py Main program.
sock.py File with Socket class.
utils.py File with utility functions.

Calibration Algorithms

Obviously, the position of landmarks is meaningless. In most cases, we focus on the ratio (or aspect ratio) which could describe the degree, because in Unity, we use a scrolling bar to control actions or events. However, the ratio is mostly not perfect correspondence with the bar value. An interesting question is how to calibrate and map the ratio to the bar value. There are some feasible methods I implement in this project:

…

References

  • Head pose estimation by Yin Guobing.
  • Face alignment by Adrian Bulat.
  • Gaze tracking by Antoine Lamé.
  • VTuber Unity project by AI葵.
  • Live2D model by kani_biimu.
  • How to update parameters of Live2D model in Unity.
  • TCP socket in C#.
  • How to create a display window.

Releases

Maintainers

@KennardWang

Contributing

Feel free to open an issue or submit PRs. Be careful that the submitted code needs to pass format tests and pytests by GitHub Actions, please self-check before pulling requests.

  • flake8 format test (Windows, MacOS and Linux)

    flake8 --exclude=face_alignment,face_pose --ignore=E501 .
  • pytest (Windows, MacOS and Linux)

    pytest tests/test_win.py
    pytest tests/test_mac.py
    pytest tests/test_linux.py

License

© Kennard Wang ( 2020.6.27 )

Issues· 2 开放

查看全部 Issues在 GitHub 打开

暂无开放 Issues,或尚未同步最近议题。

> 标签

Pythoncsharpcubismdeep-learningdlib

暂无评论,来聊聊你的看法吧

> 工具信息

发布日期2026年8月1日
最后更新2026年9月18日
分类编程语言
定价开源

> 相关工具

T
TypeScript
JavaScript 的超集,为前端与全栈提供静态类型
P
Python
通用编程语言,广泛用于 Web、数据与 AI
G
Go
Google 推出的简洁高效系统语言