MagicTime

MagicTime: Time-lapse Video Generation Models

as Metamorphic Simulators

If you like our project, please give us a star ⭐ on GitHub for the latest update.
We also have other video generation projects that may interest you ✨.

Open-Sora Plan: Open-Source Large Video Generation Model

Bin Lin, Yunyang Ge and Xinhua Cheng etc.

Helios: Real Real-Time Long Video Generation Model

Shenghai Yuan, Jinfa Huang and Xianyi He etc.

OpenS2V-Nexus: A Detailed Benchmark and Million-Scale Dataset for Subject-to-Video Generation

Shenghai Yuan, Xianyi He and Yufan Deng etc.

ConsisID: Identity-Preserving Text-to-Video Generation by Frequency Decomposition

Shenghai Yuan, Jinfa Huang and Xianyi He etc.

ChronoMagic-Bench: A Benchmark for Metamorphic Evaluation of Text-to-Time-lapse Video Generation

Shenghai Yuan, Jinfa Huang and Yongqi Xu etc.

News

• ⏳⏳⏳ Training a stronger model with the support of Open-Sora Plan.
• ⏳⏳⏳ Release the training code of MagicTime.
• [2026.03.08] We introduce Helios, a breakthrough video generation model that achieves minute-scale, high-quality video synthesis at 19.5 FPS on a single H100 GPU — without relying on conventional long video anti-drifting strategies or standard video acceleration techniques. Welcome to check Technical Report!
• [2025.04.08] We have updated our technical report. Please click here to view it.
• [2025.03.28] MagicTime has been accepted by TPAMI, and we will update arXiv with more details soon, keep tuned!
• [2024.07.29] We add batch inference to inference_magictime.py for easier usage.
• [2024.06.27] Excited to share our latest ChronoMagic-Bench, a benchmark for metamorphic evaluation of text-to-time-lapse video generation, and is fully open source! Please check out the paper.
• [2024.05.27] Excited to share our latest Open-Sora Plan v1.1.0, which significantly improves video quality and length, and is fully open source! Please check out the report.
• [2024.04.14] Thanks @camenduru and @ModelsLab for providing Jupyter Notebook and Replicate Demo.
• [2024.04.13] We have compressed the size of repo with less than 1.0 MB, so that everyone can clone easier and faster. You can click here to download, or use git clone --depth=1 command to obtain this repo.
• [2024.04.12] Thanks @Kijai and @Baobao Wang for providing ComfyUI Extension ComfyUI-MagicTimeWrapper. If you find related work, please let us know.
• [2024.04.11] We release the Hugging Face Space of MagicTime, you can click here to have a try.
• [2024.04.10] We release the inference code and model weight of MagicTime.
• [2024.04.09] We release the arXiv paper for MagicTime, and you can click here to see more details.
• [2024.04.08] We release the subset of ChronoMagic dataset used to train MagicTime. The dataset includes 2,265 metamorphic video-text pairs and can be downloaded at HuggingFace Dataset or Google Drive.
• [2024.04.08] All codes & datasets are coming soon! Stay tuned !

Highlights

MagicTime shows excellent performance in metamorphic video generation.

Related Resources

• ChronoMagic: including 2265 time-lapse video-text pairs. (captioned by GPT-4V)
• ChronoMagic-Bench: including 1649 time-lapse video-text pairs. (captioned by GPT-4o)
• ChronoMagic-Bench-150: including 150 time-lapse video-text pairs. (captioned by GPT-4o)
• ChronoMagic-Pro: including 460K time-lapse video-text pairs. (captioned by ShareGPT4Video)
• ChronoMagic-ProH: including 150K time-lapse video-text pairs. (captioned by ShareGPT4Video)

Metamorphic Videos vs. General Videos

Compared to general videos, metamorphic videos contain physical knowledge, long persistence, and strong variation, making them difficult to generate. We show compressed .gif on github, which loses some quality. The general videos are generated by the Animatediff and MagicTime.

Type  
"Bean sprouts grow and mature from seeds"
"[...] construction in a Minecraft virtual environment"
"Cupcakes baking in an oven [...]"
"[...] transitioning from a tightly closed bud to a fully bloomed state [...]"




General Videos  








Metamorphic Videos  

Gallery

We showcase some metamorphic videos generated by MagicTime, MakeLongVideo, ModelScopeT2V, VideoCrafter, ZeroScope, LaVie, T2V-Zero, Latte and Animatediff below.

Method  
"cherry blossoms transitioning [...]"
"dough balls baking process [...]"
"an ice cube is melting [...]"
"a simple modern house's construction [...]"




MakeLongVideo  








ModelScopeT2V  








VideoCrafter  








ZeroScope  








LaVie  








T2V-Zero 








Latte








Animatediff








Ours  

We show more metamorphic videos generated by MagicTime with the help of Realistic, ToonYou and RcnzCartoon.

"[...] bean sprouts grow and mature from seeds"
"dough [...] swells and browns in the oven [...]"
"the construction [...] in Minecraft [...]"











"a bud transforms into a yellow flower"
"time-lapse of a plant germinating [...]"
"[...] a modern house being constructed in Minecraft [...]"











"an ice cube is melting"
"bean plant sprouts grow and mature from the soil"
"time-lapse of delicate pink plum blossoms [...]"

Prompts are trimmed for display, see here for full prompts.

Integrate into DiT-based Architecture

The mission of this project is to help reproduce Sora and provide high-quality video-text data and data annotation pipelines, to support Open-Sora-Plan or other DiT-based T2V models. To this end, we take an initial step to integrate our MagicTime scheme into the DiT-based Framework. Specifically, our method supports the Open-Sora-Plan v1.0.0 for fine-tuning. We first scale up with additional metamorphic landscape time-lapse videos in the same annotation framework to get the ChronoMagic-Landscape dataset. Then, we fine-tune the Open-Sora-Plan v1.0.0 with the ChronoMagic-Landscape dataset to get the MagicTime-DiT model. The results are as follows (257×512×512 (10s)):

"Time-lapse of a coastal landscape [...]"
"Display the serene beauty of twilight [...]"
"Sunrise Splendor: Capture the breathtaking moment [...]"
"Nightfall Elegance: Embrace the tranquil beauty [...]"





  


  


  


  





"The sun descending below the horizon [...]"
"[...] daylight fades into the embrace of the night [...]"
"Time-lapse of the dynamic formations of clouds [...]"
"Capture the dynamic formations of clouds [...]"

Prompts are trimmed for display, see here for full prompts.

Demo

Gradio Web UI

Highly recommend trying out our web demo by the following command, which incorporates all features currently supported by MagicTime. We also provide online demo in Hugging Face Spaces.

python app.py

CLI Inference

# For Realistic
python inference_magictime.py --config sample_configs/RealisticVision.yaml --human

# or you can directly run the .sh
sh inference_cli.sh

warning: It is worth noting that even if we use the same seed and prompt but we change a machine, the results will be different.

⚙️ Requirements and Installation

We recommend the requirements as follows.

Environment

git clone --depth=1 https://github.com/PKU-YuanGroup/MagicTime.git
cd MagicTime
conda create -n magictime python=3.10.13
conda activate magictime
pip install -r requirements.txt

Download MagicTime

The weights are available at HuggingFace and WiseModel, or you can download it with the following commands.

# way 1
# if you are in china mainland, run this first: export HF_ENDPOINT=https://hf-mirror.com
huggingface-cli download --repo-type model \
BestWishYsh/MagicTime \
--local-dir ckpts

# way 2
git lfs install
git clone https://www.wisemodel.cn/SHYuanBest/MagicTime.git

Once ready, the weights will be organized in this format:

 ckpts/
├──  Base_Model/
│   ├──  motion_module/
│   ├──  stable-diffusion-v1-5/
├──  DreamBooth/
├──  Magic_Weights/
│   ├──  magic_adapter_s/
│   ├──  magic_adapter_t/
│   ├──  magic_text_encoder/

️ Training & Inference

The