Emu3

We introduce Emu3, a new suite of state-of-the-art multimodal models trained solely with next-token prediction! By tokenizing images, text, and videos into a discrete space, we train a single transformer from scratch on a mixture of multimodal sequences.

Emu3 excels in both generation and perception

Emu3 outperforms several well-established task-specific models in both generation and perception tasks, surpassing flagship open models such as SDXL, LLaVA-1.6 and OpenSora-1.2, while eliminating the need for diffusion or compositional architectures.

Highlights

• Emu3 is capable of generating high-quality images following the text input, by simply predicting the next vision token. The model naturally supports flexible resolutions and styles.
• Emu3 shows strong vision-language understanding capabilities to see the physical world and provides coherent text responses. Notably, this capability is achieved without depending on a CLIP and a pretrained LLM.
• Emu3 simply generates a video causally by predicting the next token in a video sequence, unlike the video diffusion model as in Sora. With a video in context, Emu3 can also naturally extend the video and predict what will happen next.

News

• [2025.08] Emu3-Chat with Transformers backend has been supported by VLLM as Emu3ForConditionalGeneration.
• [2024.10] We release the image pretrained model Emu3-Stage1 and the sft scripts. The model supports image captioning and can generate images at a resolution of 512x512. You can use our training scripts for further instruction tuning for more image generation and perception tasks.
• [2024.09] We relase Emu3-Chat and Emu3-Gen which are post training models separately for vision-language understanding and vision generation.
• [2024.09] We introduce Emu3, a new suite of state-of-the-art multimodal models trained solely with next-token prediction.

TODO

• Release model weights of tokenizer, Emu3-Chat and Emu3-Gen
• Release the inference code.
• Release the evaluation code.
• Release training scripts for sft.
• Release training scripts for pretrain and dpo.

Setup

Clone this repository and install required packages:

git clone https://github.com/baaivision/Emu3
cd Emu3

pip install -r requirements.txt

Model Weights

Model name	HF Weight	Modelscope	Wisemodel
Emu3-Stage1	HF link	Modelscope link
Emu3-Chat	HF link	Modelscope link	Wisemodel link
Emu3-Gen	HF link	Modelscope link	Wisemodel link
Emu3-VisionTokenizer	HF link	Modelscope link	Wisemodel link

Quickstart

Use Transformers to run Emu3-Gen/Stage1 for image generation

…

Use Transformers to run Emu3-Chat/Stage1 for vision-language understanding

…

Use Transformers to run Emu3-VisionTokenzier for vision encoding and decoding

…

Acknowledgement

We thank the great work from Emu Series, QWen2-VL and MoVQGAN

This work is supported by the National Science and Technology Major Project (No. 2022ZD0116314).

本项目受新一代人工智能国家科技重大专项（No. 2022ZD0116314）支持。

Citation

If you find Emu3 useful for your research and applications, please consider starring this repository and citing:

@article{wang2024emu3,
  title={Emu3: Next-Token Prediction is All You Need},
  author={Wang, Xinlong and Zhang, Xiaosong and Luo, Zhengxiong and Sun, Quan and Cui, Yufeng and Wang, Jinsheng and Zhang, Fan and Wang, Yueze and Li, Zhen and Yu, Qiying and others},
  journal={arXiv preprint arXiv:2409.18869},
  year={2024}
}