LLaMA-Adapter

# LLaMA-Adapter: Efficient Fine-tuning of LLaMA ## Announcement: We release **[LLaMA2-Accessory](https://github.com/Alpha-VLLM/LLaMA2-Accessory)**, an open-source toolkit for **pre-training**, **fine-tuning** and **deployment** of **LLMs** and **mutlimodal LLMs**. Official implementation of ['LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention'](https://arxiv.org/pdf/2303.16199.pdf) and ['LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Model'](https://arxiv.org/pdf/2304.15010.pdf).

This repo proposes **LLaMA-Adapter (V2)**, a lightweight adaption method for fine-tuning **Instruction-following** and **Multi-modal** [LLaMA](https://github.com/facebookresearch/llama) models . Try out the web demo of LLaMA-Adapter: [](https://huggingface.co/spaces/csuhan/LLaMA-Adapter), [LLaMA-Adapter V2](http://llama-adapter.opengvlab.com/) and [ImageBind-LLM](http://imagebind-llm.opengvlab.com/). Join us at [Wechat](https://github.com/Alpha-VLLM/LLaMA2-Accessory/blob/main/docs/wechat.md)! ## News - **[2024.01.18]** [LLaMA-Adapter](https://arxiv.org/pdf/2303.16199.pdf) is accepted by **ICLR 2024**! - **[2023.12.08]** We release [OneLLM](https://github.com/csuhan/OneLLM) which aligns **eight** modalities to language using a unified framework. - **[2023.11.11]** We release [SPHINX](https://github.com/Alpha-VLLM/LLaMA2-Accessory/tree/main/SPHINX), a new multi-modal LLM, which is a huge leap from LLaMa-Adapter-V2. - **[2023.10.11]** We realse **LLaMA-Adapter V2.1**, an improved version of LLaMA-Adapter V2 with stronger multi-modal reasoning performance. Check [llama_adapter_v2_multimodal7b](llama_adapter_v2_multimodal7b) for details. - **[2023.08.28]** We release quantized LLM with [OmniQuant](https://github.com/OpenGVLab/OmniQuant), which is an efficient, accurate, and omnibearing (even extremely low bit) quantization algorithm. Multimodal version is coming soon. - **[2023.07.24]** We release **[LLaMA2-Accessory](https://github.com/Alpha-VLLM/LLaMA2-Accessory)**, an open-source toolkit for **pre-training**, **fine-tuning** and **deployment** of **Large Language Models (LLMs)** and **mutlimodal LLMs**. Please check [Alpha-VLLM/LLaMA2-Accessory](https://github.com/Alpha-VLLM/LLaMA2-Accessory) for more details! - **[2023.07.05]** We release the pretrain/finetune code of [llama_adapter_v2_multimodal7b](https://github.com/OpenGVLab/LLaMA-Adapter/tree/main/llama_adapter_v2_multimodal7b). - **[2023.07.04]** We release the code for reproducing [Gorilla](https://github.com/ShishirPatil/gorilla) by both full finetune and LLaMA-Adapter, please see [gorilla/README.md](https://github.com/OpenGVLab/LLaMA-Adapter/blob/main/gorilla/README.md). - **[2023.06.08]** We release the [demo](http://imagebind-llm.opengvlab.com/) of ImageBind-LLM. - **[2023.06.06]** We release [Point-Bind](https://github.com/ZrrSkywalker/Point-Bind) to extend ImageBind with 3D point clouds, which achieves 3D instruction-following capacity for [imagebind_LLM](imagebind_LLM). - **[2023.06.05]** We support the integration of LLaMA-Adapter (both V1 and V2) and [LangChain](https://python.langchain.com/en/latest/index.html). Check out the [Notebook](/docs/langchain_LLaMA_AdapterV2_demo.ipynb). - **[2023.05.29]** We release the code of ImageBind-LLM at [imagebind_LLM](imagebind_LLM). - **[2023.05.23]** We release the [demos](http://llama-adapter.opengvlab.com/) and [multi-modal code](llama_adapter_v2_multimodal7b) of LLaMA-Adapter V2! - **[2023.05.05]** We release the paper and code of our new work [Personalize Segment Anything](https://github.com/ZrrSkywalker/Personalize-SAM), which efficiently fine-tunes Segment Anything with **10 seconds**, and improves DreamBooth for better **text-to-image generation**. - **[2023.04.30]** We noticed that GPT-4 evaluation has a strong positional bias in favor of the first response. We will soon update the paper to reveal the position bias. Great thanks to [Canwen Xu](https://scholar.google.com/citations?user=oopKCDMAAAAJ&hl=en). - **[2023.04.28]** We release **LLaMA-Adapter V2**, a multi-modal instruction model. Check out our [paper](https://arxiv.org/abs/2304.15010), [demos](#demos) and [code](llama_adapter_v2_chat65b)! - **[2023.03.28]** The [paper](https://arxiv.org/pdf/2303.16199.pdf) and [training code](alpaca_finetuning_v1) for **LLaMA-Adapter V1** are released. ## Released Models | Name | Approach | Data | Modality | Visual | Text | | ------------------------------------------------------------ | ------------------------------------------------------ | ------------------------------------------------------------ | ---------------------------------- | -------------- | --------------------- | | [LLaMA-Adapter V1](./alpaca_finetuning_v1) | prefix, gate | Alpaca | Text | × | LLaMA-7B | | [LLaMA-Adapter V2 dialog](./llama_adapter_v2_chat65b) | scale, bias, norm | ShareGPT | Text | × | LLaMA-65B | | [LLaMA-Adapter V2 multimodal](./llama_adapter_v2_multimodal7b) | [P] prefix, projection, gate
[F] bias, norm | [P] Image-Text-V1
[F] GPT4LLM, LLaVA | Image&Text | CLIP-ViT-L/14 | LLaMA-7B | | [LLaMA-Adapter V2.1 multimodal](./llama_adapter_v2_multimodal7b) | [P] prefix, projection, gate
[F] bias, norm, lora | [P] Image-Text-V1
[F] GPT4LLM, LLaVA, VQAv2 | Image&Text | CLIP-ViT-L/14 | LLaMA-7B | | [ImageBind-LLM](./imagebind_LLM) | [P] prefix, projection, gate
[F] bias, norm, lora | [P] Image-Text-V1
[F] [Instruction Following](https://github.com/OpenGVLab/LLaMA-Adapter/blob/main/imagebind_LLM/docs/train.md#data-1) | ImageBind Modalities + Point Cloud | imagebind_huge | Open-Chinese-LLaMA-7B | | ImageBind-dialog | [P] prefix, projection, gate
[F] bias, norm, lora | [P] Image-Text-V1
[F] LLaVA, ShareGPT | ImageBind Modalities + Point Cloud | imagebind_huge | Open-Chinese-LLaMA-7B | + [P] means **P**re-train and [F] means **F**ine-tune + **Image-Text-V1** is a concatenation of LAION400M, COYO, MMC4, SBU, Conceptual Captions, and COCO + **ImageBind Modalities** include image, video, text, audio, depth, thermal, IMU + **ImageBind-dialog** will be release soon ## Overview Efficiency Comparison: | Model | Parameters | Storage Space | Training Time | :-----: | :-----: |:-----:| :-----: | | [Alpaca](https://github.com/tatsu-lab/stanford_alpaca) | 7B |13G| 3 Hours| | LLaMA-Adapter | 1.2M |4.7M| 1 Hour| By inserting adapters into LLaMA's transformer, our method only introduces **1.2M** learnable parameters, and turns a LLaMA into an instruction-following model within **1 hour**. For stablizing training at early stages, we propose a novel **Zero-init Attention** with zero gating mechanism to adaptively incorporate the instructional signals. After fine-tuning, LLaMA-Adapter can generate high-quality instruction-following sentences, comparable to the fully fine-tuned [Stanford Alpaca](https://github.com/tatsu-lab/stanford_alpaca) and [Alpaca-Lora](https://github.com/tloen/alpaca-lora). Our approach can be simply extended to **Multi-modal Input Instructions**. The reasoning framework of image-conditioned LLaMA-Adapter for [ScienceQA](https://scienceqa.github.io/) is as follows, which is also shared by other modalities, such as audio and video. ## Setup Here is a from-scratch script for **LLaMA-Adapter V1**. ```bash conda create -n llama_adapter -y python=3.8 conda activate llama_adapter # install pytorch conda install pytorch cudatoolkit -c pytorch -y # install dependency and llama-adapter pip install -r requirements.txt pip install -e . ``` **Note**: **To setup other models**, please refer to [llama_adapter_v2_chat65b](llama_adapter_v2_chat65b), [llama_adapter_v2_multimodal7b](llama_adapter_v2_multimodal7b) and [imagebind_LLM](imagebind_LLM) for more details. ## Inference Please request access to the pre-trained LLaMA from [this form](https://forms.gle/jk851eBVbX1m5TAv5) (official) or download the LLaMA-7B from [Hugging Face](https://huggingface.co/nyanko7/LLaMA-7B/tree/main) (unofficial). Then, obtain the weights of our LLaMA-Adapter from [here](https://github.com/OpenGVLab/LLaMA-Adapter/releases/download/v.1.0.0/llama_adapter_len10_layer30_release.pth). We denote the path to the downloaded weights of LLaMA and adapters as `TARGET_FOLDER` and `ADAPTER_PATH`. Here is an example to generate instruction-following sentences with 7B LLaMA model and our LLaMA-Adapter: ```bash torchrun --nproc_per_node 1 example.py \ --ckpt_dir $TARGET_FOLDER/model_size\ --tokenizer_path $TARGET_FOLDER/tokenizer.model \ --adapter_path $ADAPTER_PATH ``` ## Training We release the simple fine-tuning code of LLaMA-Adapter on LLaMA-7B model at [here](alpaca_finetuning_v1), which is for effortless reproduction with minimal dependencies. We will soon release the fine-tuning code for LLaMA-65B and multi-model LLaMA-Adapter. Please download the 52K instruction-following training [data](https://github.com/tatsu-lab/stanford_alpaca/blob/main/alpaca_data.json) from Standford Alpaca, and put it under `DATA_PATH`. Then run: ```bash cd alpaca_finetuning_v1 torchrun --nproc_per_node 8 finetuning.py \ --model Llama7B_adapter \ --llama_model_path $TARGET_FOLDER/ \ --data_path $DATA_PATH/alpaca_data.json \ --adapter_layer 30 \ --adapter_len 10 \ --max_seq_len 512 \ --batch_size 4 \ --epochs 5 \ --warmup_epochs 2 \ --blr 9e-3 \ --weight_decay 0.02 \ --output_dir ./checkpoint/ ``` ## Comparison with Other Methods ### Instruction 1: How can I develop my critical thinking skills? Give me three suggestions. * **LLaMA-Adapter V2:** 1. Read widely: Reading books, articles, and other sources of information can help you develop your critical thinking skills. By exposing yourself to different perspectives and ideas, you can learn to analyze and evaluate information more effectively. 2. Practice mindfulness: Mindfulness is the practice of being present in the moment and paying attention to your thoughts and feelings. By being more aware of your thoughts and emotions, you can better understand and evaluate them. This can help you develop a more objective and rational approach to thinking. 3. Engage in open-minded discussions: Join a discussion group or debate club where you can engage in open-minded discussions with others. This can help you develop your critical thinking skills by challenging your assumptions and encouraging you to consider different perspectives. * **LLaMA-Adapter V1:** 1. Read widely and critically. 2. Engage in discussions with people who have different perspectives. 3. Ask questions and seek out answers. ### Instruction 2: As a superhero, how would you explain your origin story to a curious child? * **LLaMA-Adapter V2:** As a superhero, I have a very interesting and exciting origin story to share with you. Once upon a time, there was a young boy named John who lived in a small town. John was always curious and wanted to help others, so he spent his days exploring the