正式实现 AnimateDiff。
Model：ToonYou
Model：Realistic Vision V2.0
## Quick Start ***Note:*** AnimateDiff is also offically supported by Diffusers. Visit [AnimateDiff Diffusers Tutorial](https://huggingface.co/docs/diffusers/api/pipelines/animatediff) for more details. *Following instructions is for working with this repository*. ***Note:*** For all scripts, checkpoint downloading will be *automatically* handled, so the script running may take longer time when first executed. ### 1. Setup repository and environment ``` git clone https://github.com/guoyww/AnimateDiff.git cd AnimateDiff pip install -r requirements.txt ``` ### 2. Launch the sampling script! The generated samples can be found in `samples/` folder. #### 2.1 Generate animations with comunity models ``` … ``` #### 2.2 Generate animation with MotionLoRA control ``` python -m scripts.animate --config configs/prompts/2_motionlora/2_motionlora_RealisticVision.yaml ``` #### 2.3 More control with SparseCtrl RGB and sketch ``` python -m scripts.animate --config configs/prompts/3_sparsectrl/3_1_sparsectrl_i2v.yaml python -m scripts.animate --config configs/prompts/3_sparsectrl/3_2_sparsectrl_rgb_RealisticVision.yaml python -m scripts.animate --config configs/prompts/3_sparsectrl/3_3_sparsectrl_sketch_RealisticVision.yaml ``` #### 2.4 Gradio app We created a Gradio demo to make AnimateDiff easier to use. By default, the demo will run at `localhost:7860`. ``` python -u app.py ``` ## Technical Explanation Technical Explanation ### AnimateDiff **AnimateDiff aims to learn transferable motion priors that can be applied to other variants of Stable Diffusion family.** To this end, we design the following training pipeline consisting of three stages. - In **1. Alleviate Negative Effects** stage, we train the **domain adapter**, e.g., `v3_sd15_adapter.ckpt`, to fit defective visual aritfacts (e.g., watermarks) in the training dataset. This can also benefit the distangled learning of motion and spatial appearance. By default, the adapter can be removed at inference. It can also be integrated into the model and its effects can be adjusted by a lora scaler. - In **2. Learn Motion Priors** stage, we train the **motion module**, e.g., `v3_sd15_mm.ckpt`, to learn the real-world motion patterns from videos. - In **3. (optional) Adapt to New Patterns** stage, we train **MotionLoRA**, e.g., `v2_lora_ZoomIn.ckpt`, to efficiently adapt motion module for specific motion patterns (camera zooming, rolling, etc.). ### SparseCtrl **SparseCtrl aims to add more control to text-to-video models by adopting some sparse inputs (e.g., few RGB images or sketch inputs).** Its technicall details can be found in the following paper: **[SparseCtrl: Adding Sparse Controls to Text-to-Video Diffusion Models](https://arxiv.org/abs/2311.16933)** [Yuwei Guo](https://guoyww.github.io/), [Ceyuan Yang✝](https://ceyuan.me/), [Anyi Rao](https://anyirao.com/), [Maneesh Agrawala](https://graphics.stanford.edu/~maneesh/), [Dahua Lin](http://dahua.site), [Bo Dai](https://daibo.info) (✝Corresponding Author) ## Model Versions Model Versions ### AnimateDiff v3 and SparseCtrl (2023.12) In this version, we use **Domain Adapter LoRA** for image model finetuning, which provides more flexiblity at inference. We also implement two (RGB image/scribble) [SparseCtrl](https://arxiv.org/abs/2311.16933) encoders, which can take abitary number of condition maps to control the animation contents. AnimateDiff v3 Model Zoo | Name | HuggingFace | Type | Storage | Description | | - | - | - | - | - | | `v3_adapter_sd_v15.ckpt` | [Link](https://huggingface.co/guoyww/animatediff/blob/main/v3_sd15_adapter.ckpt) | Domain Adapter | 97.4 MB | | | `v3_sd15_mm.ckpt.ckpt` | [Link](https://huggingface.co/guoyww/animatediff/blob/main/v3_sd15_mm.ckpt) | Motion Module | 1.56 GB | | | `v3_sd15_sparsectrl_scribble.ckpt` | [Link](https://huggingface.co/guoyww/animatediff/blob/main/v3_sd15_sparsectrl_scribble.ckpt) | SparseCtrl Encoder | 1.86 GB | scribble condition | | `v3_sd15_sparsectrl_rgb.ckpt` | [Link](https://huggingface.co/guoyww/animatediff/blob/main/v3_sd15_sparsectrl_rgb.ckpt) | SparseCtrl Encoder | 1.85 GB | RGB image condition | #### Limitations 1. Small fickering is noticable; 2. To stay compatible with comunity models, there is no specific optimizations for general T2V, leading to limited visual quality under this setting; 3. **(Style Alignment) For usage such as image animation/interpolation, it's recommanded to use images generated by the same community model.** #### Demos| Input (by RealisticVision) | Animation | Input | Animation |
| Input Scribble | Output | Input Scribbles | Output |
| Original SDXL | Community SDXL | Community SDXL |
| Zoom In | Zoom Out | Zoom Pan Left | Zoom Pan Right | ||||
暂无开放 Issues，或尚未同步最近议题。