Rhubarb Lip Sync is a command-line tool that automatically creates 2D mouth animation from voice recordings. You can use it for characters in computer games, in
Rhubarb Lip Sync is a command-line tool that automatically creates 2D mouth animation from voice recordings. You can use it for characters in computer games, in
= Rhubarb Lip Sync :toc: :icons: font
:A: Ⓐ :B: Ⓑ :C: Ⓒ :D: Ⓓ :E: Ⓔ :F: Ⓕ :G: Ⓖ :H: Ⓗ :X: Ⓧ
image:https://img.shields.io/twitter/follow/RhubarbLipSync.svg?style=social&label=Follow["Twitter", link="https://twitter.com/RhubarbLipSync"] image:https://github.com/DanielSWolf/rhubarb-lip-sync/actions/workflows/ci.yml/badge.svg["Build status", link="https://github.com/DanielSWolf/rhubarb-lip-sync/actions/workflows/ci.yml"]
image:img/logo.png[align="center"]
Rhubarb Lip Sync allows you to quickly create 2D mouth animation from voice recordings. It analyzes your audio files, recognizes what is being said, then automatically generates lip sync information. You can use it for animating speech in computer games, animated cartoons, or any similar project.
Rhubarb Lip Sync integrates with the following applications:
In addition, you can use Rhubarb Lip Sync's command line interface (CLI) to generate files in various <<outputFormats,output formats>> (<<tsv,TSV>>/<<xml,XML>>/<<json,JSON>>).
== Demo video
Click the image for a demo video.
https://www.youtube.com/watch?v=zzdPSFJRlEo[image:http://img.youtube.com/vi/zzdPSFJRlEo/0.jpg[]]
== Integrations
[[afterEffects]] === Adobe After Effects
You can use Rhubarb Lip Sync to animate dialog right from Adobe After Effects. For more information, <<extras/AdobeAfterEffects/README.adoc#,follow this link>> or see the directory extras/AdobeAfterEffects.
image:img/after-effects.png[]
[[moho]] === Moho and OpenToonz
Rhubarb Lip Sync can create .dat switch data files, which are understood by Moho and OpenToonz. You can set the frame rate using the --datFrameRate option; to control the shape names, use the --datUsePrestonBlair flag. For more details, see <>.
image:img/moho.png[]
[[spine]] === Spine by Esoteric Software
Rhubarb Lip Sync for Spine is a graphical tool that allows you to import a Spine project, perform automatic lip sync, then re-import the result into Spine. For more information, <<extras/EsotericSoftwareSpine/README.adoc#,follow this link>> or see the directory extras/EsotericSoftwareSpine of the download.
image:img/spine.png[]
[[vegas]] === Vegas Pro by Magix
Rhubarb Lip Sync also comes with two plugin scripts for Vegas Pro (previously Sony Vegas). For more information, <<extras/MagixVegas/README.adoc#,follow this link>> or see the directory extras/MagixVegas of the download.
image:img/vegas.png[]
[[mouth-shapes]] == Mouth shapes
Rhubarb Lip Sync can use between six and nine different mouth positions. The first six mouth shapes ({A}-{F}) are the basic mouth shapes and the absolute minimum you have to draw for your character. These six mouth shapes were invented at the Hanna-Barbera studios for shows such as Scooby-Doo and The Flintstones. Since then, they have evolved into a de-facto standard for 2D animation, and have been widely used by studios like Disney and Warner Bros.
In addition to the six basic mouth shapes, there are three extended mouth shapes: {G}, {H}, and {X}. These are optional. You may choose to draw all three of them, pick just one or two, or leave them out entirely.
[cols="1h,2,6"] |===
| {A} | image:img/lisa-A.png[]
| Closed mouth for the "P", "B", and "M" sounds. This is almost identical to the {X} shape, but there is ever-so-slight pressure between the lips.
| {B} | image:img/lisa-B.png[]
| Slightly open mouth with clenched teeth. This mouth shape is used for most consonants ("K", "S", "T", etc.). It's also used for some vowels such as the "EE" sound in bee.
| {C} | image:img/lisa-C.png[]
| Open mouth. This mouth shape is used for vowels like "EH" as in men and "AE" as in bat. It's also used for some consonants, depending on context.
This shape is also used as an in-between when animating from {A} or {B} to {D}. So make sure the animations {A}{C}{D} and {B}{C}{D} look smooth!
| {D} | image:img/lisa-D.png[]
| Wide open mouth. This mouth shapes is used for vowels like "AA" as in father.
| {E} | image:img/lisa-E.png[]
| Slightly rounded mouth. This mouth shape is used for vowels like "AO" as in off and "ER" as in bird.
This shape is also used as an in-between when animating from {C} or {D} to {F}. Make sure the mouth isn't wider open than for {C}. Both {C}{E}{F} and {D}{E}{F} should result in smooth animation.
| {F} | image:img/lisa-F.png[]
| Puckered lips. This mouth shape is used for "UW" as in you, "OW" as in show, and "W" as in way.
| {G} | image:img/lisa-G.png[]
| Upper teeth touching the lower lip for "F" as in for and "V" as in very.
This extended mouth shape is optional. If your art style is detailed enough, it greatly improves the overall look of the animation. If you decide not to use it, you can specify so using the <<extendedShapes,extendedShapes>> option.
| {H} | image:img/lisa-H.png[]
| This shape is used for long "L" sounds, with the tongue raised behind the upper teeth. The mouth should be at least far open as in {C}, but not quite as far as in {D}.
This extended mouth shape is optional. Depending on your art style and the angle of the head, the tongue may not be visible at all. In this case, there is no point in drawing this extra shape. If you decide not to use it, you can specify so using the <<extendedShapes,extendedShapes>> option.
| {X} | image:img/lisa-X.png[] | Idle position. This mouth shape is used for pauses in speech. This should be the same mouth drawing you use when your character is walking around without talking. It is almost identical to {A}, but with slightly less pressure between the lips: For {X}, the lips should be closed but relaxed.
This extended mouth shape is optional. Whether there should be any visible difference between the rest position {X} and the closed talking mouth {A} depends on your art style and personal taste. If you decide not to use it, you can specify so using the <<extendedShapes,extendedShapes>> option.
|===
== How to run Rhubarb Lip Sync
=== General usage ===
Rhubarb Lip Sync is a command-line tool that is currently available for Windows, macOS, and Linux.
rhubarb, passing it an audio file as argument and telling it where to create the output file. In its simplest form, this might look like this: rhubarb -o output.txt my-recording.wav. There are additional <<options,command-line options>> you can specify in order to get better results.stderr and exit with a non-zero exit code.[[options]] === Command-line options ===
==== Basic command-line options ====
The following command-line options are the most common:
[cols="2,5a"] |=== | Option | Description
| | The audio file to be analyzed. This must be the last command-line argument. Supported file formats are WAVE (.wav) and Ogg Vorbis (.ogg).
| -r , --recognizer
| Specifies how Rhubarb Lip Sync recognizes speech within the recording. Options: pocketSphinx (use for English recordings), phonetic (use for non-English recordings). For details, see <>.
Default value: pocketSphinx
| -f , --exportFormat
| The export format. Options: tsv (tab-separated values, see <<tsv,details>>), xml (see <<xml,details>>), json (see <<json,details>>), dat (see <>).
Default value: tsv
| -d , --dialogFile
| With this option, you can provide Rhubarb Lip Sync with the dialog text to get more reliable results. Specify the path to a plain-text file (in ASCII or UTF-8 format) containing the dialog contained in the audio file. Rhubarb Lip Sync will still perform word recognition internally, but it will prefer words and phrases that occur in the dialog file. This leads to better recognition results and thus more reliable animation.
For instance, let's say you're recording dialog for a computer game. The script says: "That's all gobbledygook to me." But actually, the voice artist ends up saying "That's _just_ gobbledygook to me," deviating from the dialog. If you specify a dialog file with the original line ("That's all gobbledygook to me"), this will still allow Rhubarb Lip Sync to produce better results, because it will watch out for the uncommon word "gobbledygook". Rhubarb Lip Sync will ignore the dialog file where it audibly differs from the recording, and benefit from it where it matches.
It is always a good idea to specify the dialog text. This will usually lead to more reliable mouth animation, even if the text is not completely accurate.
[[extendedShapes]]
| --extendedShapes
| As described in <>, Rhubarb Lip Sync uses six basic mouth shapes and up to three extended mouth shapes, which are optional. Use this option to specify which extended mouth shapes should be used. For example, to use only the {G} and {X} extended mouth shapes, specify GX; to use only the six basic mouth shapes, specify an empty string: "".
Default value: GHX
| -o, --output
| The name of the output file to create. If the file already exists, it will be overwritten. If you don't specify an output file, the result will be written to stdout.
| --version
| Displays version information and exits.
| -h, --help
| Displays usage information and exits.
| --datFrameRate number
| Only valid when using the dat export format. Controls the frame rate for the output file.
Default value: 24
| --datUsePrestonBlair
| Only valid when using the dat export format. Uses Preston Blair mouth shapes names instead of the default alphabetical ones. This applies the following mapping:
!=== ! Alphabetic name ! Preston Blair name
! A ! MBP ! B ! etc ! C ! E ! D ! AI ! E ! O ! F ! U ! G ! FV ! H ! L ! X ! rest !===
Caution: This mapping is only applied when exporting, after the recording has been animated. To control which mouth shapes to use, use the <<extendedShapes,extendedShapes>> option with the alphabetic names.
Tip: For optimal results, make sure your mouth drawings follow the guidelines in the <> section. This is easier if you stick to the alphabetic names instead of the Preston Blair names. The only situation where you need to use the Preston Blair names is when you're using OpenToonz, because OpenToonz only supports the Preston Blair names.
|===
==== Advanced command-line options ====
The following command-line options can be helpful in special situations, especially when automating Rhubarb Lip Sync.
[cols="2,5"] |=== | Option | Description
[[quiet]]
| -q, --quiet
| By default, Rhubarb Lip Sync writes a number of progress messages to stderr. If you're using it as part of a batch process, this may clutter your console. If you specify the --quiet flag, there won't be any output to stderr unless an error occurred.
You can combine this option with the <<consoleLevel,consoleLevel>> option to change the minimum event level that is printed to stderr.
| --machineReadable
a| This option is useful if you want to integrate Rhubarb Lip Sync with another (possibly graphical) application. All status messages to stderr will be in structured JSON format, allowing your program to parse them and display
No open issues yet, or sync has not completed.