InternLM-XComposer

InternLM-XComposer-2.5

Thanks the community for HuggingFace Demo | OpenXLab Demo of InternLM-XComposer-2.5.

join us on Discord and WeChat

## **InternLM-XComposer2.5-Reward** We release **InternLM-XComposer2.5-Reward** (IXC-2.5-Reward, ACL 2025 Findings), a simple yet effective multi-modal reward model, including training code, evaluation scripts, and parts of the traininig data. Please refer to the [project page](InternLM-XComposer-2.5-Reward) for details. ## **InternLM-XComposer2.5-OmniLive** We release **InternLM-XComposer2.5-OmniLive**, a comprehensive multimodal system for long-term streaming video and audio interactions. Please refer to the [project page](InternLM-XComposer-2.5-OmniLive) for details.
## Multimodal Projects of Our Team > [**InternLM-XComposer-2.5-Reward**](https://arxiv.org/pdf/2501.12368): **A Simple Yet Effective Multi-Modal Reward Model** > [**InternLM-XComposer-2.5-OmniLive**](): **A Specialized Generalist Multimodal System for Streaming Video and Audio Interactions** > [**InternLM-XComposer-2.5**](https://arxiv.org/pdf/2407.03320): **A Versatile Large Vision Language Model Supporting Long-Contextual Input and Output** > [**InternLM-XComposer2-**](https://github.com/InternLM/InternLM-XComposer): **A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HD** > [**InternLM-XComposer2**](https://github.com/InternLM/InternLM-XComposer): **Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Models** > [**InternLM-XComposer**](https://github.com/InternLM/InternLM-XComposer/tree/main/InternLM-XComposer-1.0): **A Vision-Language Large Model for Advanced Text-image Comprehension and Composition** > [**ShareGPT4Video:**](https://github.com/InternLM/InternLM-XComposer/tree/main/projects/ShareGPT4Video) **Improving Video Understanding and Generation with Better Captions** > [**ShareGPT4V:**](https://github.com/InternLM/InternLM-XComposer/tree/main/projects/ShareGPT4V) **Improving Large Multi-modal Models with Better Captions** > [**MMDU:**](https://liuziyu77.github.io/MMDU/) **A Multi-Turn Multi-Image Dialog Understanding Benchmark and Instruction-Tuning Dataset for LVLMs** > [**DualFocus**](https://github.com/InternLM/InternLM-XComposer/tree/main/projects/DualFocus): **Integrating Macro and Micro Perspectives in Multi-modal Large Language Models**
**InternLM-XComposer-2.5** excels in various text-image comprehension and composition applications, achieving GPT-4V level capabilities with merely 7B LLM backend. IXC-2.5 is trained with 24K interleaved image-text contexts, it can seamlessly extend to 96K long contexts via RoPE extrapolation. This long-context capability allows IXC-2.5 to perform exceptionally well in tasks requiring extensive input and output contexts. - **Ultra-High Resolution Understanding**: IXC-2.5 enhances the dynamic resolution solution proposed in IXC2-4KHD with a native 560 × 560 ViT vision encoder, supporting high-resolution images with any aspect ratio. - **Fine-Grained Video Understanding**: IXC-2.5 treats videos as a ultra-high-resolution composite picture consisting of tens to hundreds of frames, allowing it to capture fine details through dense sampling and higher resolution for each frame. - **Multi-Turn Multi-Image Dialogue**: IXC-2.5 supports free-form multi-turn multi-image dialogue, allowing it to naturally interact with humans in multi-round conversations. - **Webpage Crafting**: IXC-2.5 can be readily applied to create webpages by composing source code (HTML, CSS, and JavaScript) following text-image instructions. - **Composing High-Quality Text-Image Articles**: IXC-2.5 leverages specially designed Chain-of-Thought (CoT) and Direct Preference Optimization (DPO) techniques to significantly enhance the quality of its written content. - **Awesome performance**: IXC-2.5 has been evaluated on 28 benchmarks, outperforming existing open-source state-of-the-art models on 16 benchmarks. It also surpasses or competes closely with GPT-4V and Gemini Pro on 16 key tasks.

Please refer to [Technical Report](https://arxiv.org/abs/2407.03320) for more details.
## Demo Video For the best experience, please keep the audio on while enjoying the video. [https://github.com/InternLM/InternLM-XComposer/assets/147793160/8206f07f-3166-461e-a631-9cbcdec6ae75](https://github.com/InternLM/InternLM-XComposer/assets/147793160/8206f07f-3166-461e-a631-9cbcdec6ae75) [Youtube Video](https://youtu.be/8tYpiQNOJww) Please refer to [Chinese Demo](./README_CN.md#demo) for the demo of the Chinese version. ## News and Updates - `2024.12.12` [InternLM-XComposer2.5-7B-Reward](https://huggingface.co/internlm/internlm-xcomposer2d5-7b-reward) is publicly available. - `2024.12.12` [InternLM-XComposer2.5-OmniLive-7B](https://huggingface.co/internlm/internlm-xcomposer2d5-ol-7b) is publicly available. - `2024.07.15` [ModelScope Swift](https://github.com/InternLM/lmdeploy/blob/main/docs/en/multi_modal/xcomposer2d5.md) supports InternLM-XComposer2.5-7B for finetuning and inference. - `2024.07.15` [LMDepoly](https://github.com/InternLM/lmdeploy/blob/main/docs/en/multi_modal/xcomposer2d5.md) supports InternLM-XComposer2.5-7B for 4 bit quantization and inference. - `2024.07.15` [InternLM-XComposer2.5-7B-4bit](https://huggingface.co/internlm/internlm-xcomposer2d5-7b-4bit) is publicly available. - `2024.07.03` [InternLM-XComposer2.5-7B](https://huggingface.co/internlm/internlm-xcomposer2d5-7b) is publicly available. - `2024.07.01` [ShareGPT4V](https://github.com/InternLM/InternLM-XComposer/tree/main/projects/ShareGPT4V) is accepted by ECCV2024. - `2024.04.22` The [finetune code](./finetune/) of **InternLM-XComposer2-VL-7B-4KHD-7B** are publicly available. - `2024.04.09` [InternLM-XComposer2-4KHD-7B](https://huggingface.co/internlm/internlm-xcomposer2-4khd-7b) and [evaluation code](./evaluation/README.md) are publicly available. - `2024.04.09` [InternLM-XComposer2-VL-1.8B](https://huggingface.co/internlm/internlm-xcomposer2-vl-1_8b) is publicly available. - `2024.02.22` We release [DualFocus](https://github.com/InternLM/InternLM-XComposer/tree/main/projects/DualFocus), a framework for integrating macro and micro perspectives within MLLMs to enhance vision-language task performance. * `2024.02.06` [InternLM-XComposer2-7B-4bit](https://huggingface.co/internlm/internlm-xcomposer2-7b-4bit) and [InternLM-XComposer-VL2-7B-4bit](https://huggingface.co/internlm/internlm-xcomposer2-vl-7b-4bit) are publicly available on **Hugging Face** and **ModelScope**. - `2024.02.02` The [finetune code](./finetune/) of **InternLM-XComposer2-VL-7B** are publicly available. - `2024.01.26` The [evaluation code](./evaluation/README.md) of **InternLM-XComposer2-VL-7B** are publicly available. - `2024.01.26` [InternLM-XComposer2-7B](https://huggingface.co/internlm/internlm-xcomposer2-7b) and [InternLM-XComposer-VL2-7B](https://huggingface.co/internlm/internlm-xcomposer2-vl-7b) are publicly available on **Hugging Face** and **ModelScope**. - `2024.01.26` We release a [technical report](https://arxiv.org/abs/2401.16420) for more details of InternLM-XComposer2 series. - `2023.11.22` We release the [ShareGPT4V](https://github.com/InternLM/InternLM-XComposer/tree/main/projects/ShareGPT4V), a large-scale highly descriptive image-text dataset generated by GPT4-Vision and a superior large multimodal model, ShareGPT4V-7B. - `2023.10.30` InternLM-XComposer-VL achieved the top 1 ranking in both [Q-Bench](https://github.com/Q-Future/Q-Bench/tree/master/leaderboards#overall-leaderboards) and [Tiny LVLM](https://github.com/OpenGVLab/Multi-Modality-Arena/tree/main/tiny_lvlm_evaluation). - `2023.10.19` Support for inference on multiple GPUs. Two 4090 GPUs are sufficient for deploying our demo. - `2023.10.12` 4-bit demo is supported, model files are available in [Hugging Face](https://huggingface.co/internlm/internlm-xcomposer-7b-4bit) and [ModelScope](https://modelscope.cn/models/Shanghai_AI_Laboratory/internlm-xcomposer-7b-4bit). - `2023.10.8` [InternLM-XComposer-7B](https://modelscope.cn/models/Shanghai_AI_Laboratory/internlm-xcomposer-7b) and [InternLM-XComposer-VL-7B](https://modelscope.cn/models/Shanghai_AI_Laboratory/internlm-xcomposer-vl-7b) are publicly available on **ModelScope**. - `2023.9.27` The [evaluation code](./InternLM-XComposer-1.0/evaluation/) of **InternLM-XComposer-VL-7B** are publicly available. - `2023.9.27` [InternLM-XComposer-7B](https://huggingface.co/internlm/internlm-xcomposer-7b) and [InternLM-XComposer-VL-7B](https://huggingface.co/internlm/internlm-xcomposer-vl-7b) are publicly available on **Hugging Face**. - `2023.9.27` We release a [technical report](https://arxiv.org/pdf/2309.15112.pdf) for more details of our model series.
## Model Zoo | Model | Usage | Transformers(HF) | ModelScope(HF) | Release Date | | ------------------------------- | ----------------------------------------------- | -------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------ | | **InternLM-XComposer-2.5** | Video Understanding, Multi-image Multi-tune Dialog, 4K Resolution Understanding, Web Craft, Article creation, Benchmark | [internlm-xcomposer2.5](https://huggingface.co/internlm/internlm-xcomposer2d5-7b) | [ internlm-xcomposer2.5](https://modelscope.cn/models/Shanghai_AI_Laboratory/internlm-xcomposer2d5-7b/summary) | 2024-07-03 | | **InternLM-XComposer2-4KHD** | 4K Resolution Understanding, Benchmark, VL-Chat | [internlm-xcomposer2-4khd-7b](https://huggingface.co/internlm/internlm-xcomposer2-4khd-7b) | [ internlm-xcomposer2-4khd-7b](https://modelscope.cn/models/Shanghai_AI_Laboratory/internlm-xcomposer2-4khd-7b/summary) | 2024-04-09 | | **InternLM-XComposer2-VL-1.8B** | Benchmark, VL-Chat | [internlm-xcomposer2-vl-1_8b](https://huggingface.co/internlm/internlm-xcomposer2-vl-1_8b) | [ internlm-xcomposer2-vl-1_8b](https://modelscope.cn/models/Shanghai_AI_Laboratory/internlm-xcomposer2-vl-1_8b/summary) | 2024-04-09 | | **InternLM-XComposer2** | Text-Image Composition | [internlm-xcomposer2-7b](https://huggingface.co/internlm/internlm-xcomposer2-7b) | [ internlm-xcomposer2-7b](https://modelscope.cn/models/Shanghai_AI_Laboratory/internlm-xcomposer2-7b/summary) | 2024-01-26 | | **InternLM-XComposer2-VL** | Benchmark, VL-Chat | [internlm-xcomposer2-vl-7b](https://huggingface.co/internlm/internlm-xcomposer2-vl-7b) | [ internlm-xcomposer2-vl-7b](http