AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning (ICLR 2023).
AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning (ICLR 2023).
This pytorch package implements Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning (ICLR 2023).
The implementaion of AdaLoRA has been merged to the parameter-efficient fine-tuning repository (PEFT) supported by HuggingFace: PEFT: State-of-the-art Parameter-Efficient Fine-Tuning. Feel free to raise any issues when you using AdaLoRA in PEFT or our repository.
There are several directories in this repo:
loralib, which include our implementation of AdaLoRA (loralib/adalora.py) and needs to be installed to run the examples;loralib:pip install -e loralib/ # ===== Before =====
# layer = nn.Linear(in_features, out_features)
# ===== After ======
import loralib
# Add a SVD-based adaptation matrices with rank r=12
layer = loralib.SVDLinear(in_features, out_features, r=12)Also, before the training loop begins, mark only LoRA parameters as trainable.
model = BigModel()
# This sets requires_grad to False for all parameters without the string "lora_" in their names
loralib.mark_only_lora_as_trainable(model)from loralib import RankAllocator
from loralib import compute_orth_regu
# Initialize the RankAllocator
rankallocator = RankAllocator(
model, lora_r=12, target_rank=8,
init_warmup=500, final_warmup=1500, mask_interval=10,
total_step=3000, beta1=0.85, beta2=0.85,
)lora_r: The initial rank of each incremental matrix.
target_rank: The average target rank of final incremental matrices, i.e. the average number of singular values per matrix.
init_warmup: The steps of initial warmup for budget scheduler.
final_warmup: The steps of final warmup for budget scheduler.
mask_interval: The time internval between two budget allocations.
beta1 and beta2: The coefficient of exponentional moving average when updating importance scores.
At each step of back-propagation, we apply an additional regularization to enforce the orthongonality of SVDLinear modules by compute_orth_regu(model). After each step of optimizer.step(), we then call RankAllocator to update importance estimation and allocate the budget accordingly:
# ===== Before =====
# loss.backward()
# optimizer.step()
# global_step += 1
# ===== After ======
(loss+compute_orth_regu(model, regu_weight=0.1)).backward
optimizer.step()
rankallocator.update_and_mask(model, global_step)
global_step += 1Check the folder NLU for more details about reproducing the GLUE results.
An example of adapting DeBERTaV3-base on MNLI:
…Please see NLU/scripts for more examples of GLUE.
Check the folder NLG_QA for more details about reproducing the results of summarization and question-answering tasks.
An example of adapting DeBERTaV3-base on SQuADv2:
…@inproceedings{
zhang2023adaptive,
title={Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning },
author={Qingru Zhang and Minshuo Chen and Alexander Bukharin and Pengcheng He and Yu Cheng and Weizhu Chen and Tuo Zhao},
booktitle={The Eleventh International Conference on Learning Representations },
year={2023},
url={https://openreview.net/forum?id=lq62uWRJjiY}
}No open issues yet, or sync has not completed.