百科.dev
全部条目在线工具AI 编程趋势榜开源项目技术资讯提交条目
登录
< 返回工具列表
A

AdaLoRA

> 编程语言
开源

AdaLoRA: 适应性预算分配用于参数高效微调 (ICLR 2023)。

394 stars0 点赞0 次浏览
访问官网GitHub

工具介绍

AdaLoRA: 适应性预算分配用于参数高效微调 (ICLR 2023)。

AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning

This pytorch package implements Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning (ICLR 2023).

The implementaion of AdaLoRA has been merged to the parameter-efficient fine-tuning repository (PEFT) supported by HuggingFace: PEFT: State-of-the-art Parameter-Efficient Fine-Tuning. Feel free to raise any issues when you using AdaLoRA in PEFT or our repository.

Repository Overview

There are several directories in this repo:

  • loralib/ contains the source code of the updated package loralib, which include our implementation of AdaLoRA (loralib/adalora.py) and needs to be installed to run the examples;
  • NLU/ contains an example implementation of AdaLoRA in DeBERTaV3-base, which produces the results on the GLUE benchmark;
  • NLG_QA/ contains an example implementation of AdaLoRA in BART-large and DeBERTaV3-base, which can be used to reproduce the results of summarization and question-answering tasks.

Quickstart of AdaLoRA

  1. Install the updated loralib:
bash
pip install -e loralib/ 
  1. Then we apply SVD-based adaptation of AdaLoRA. Here is an example (For more examples, please see modeling_debertav2.py for how we adapte DeBERTa):
python
# ===== Before =====
# layer = nn.Linear(in_features, out_features)

# ===== After ======
import loralib 
# Add a SVD-based adaptation matrices with rank r=12
layer = loralib.SVDLinear(in_features, out_features, r=12)

Also, before the training loop begins, mark only LoRA parameters as trainable.

python
model = BigModel()
# This sets requires_grad to False for all parameters without the string "lora_" in their names
loralib.mark_only_lora_as_trainable(model)
  1. During the training loop, we apply RankAllocator of AdaLoRA to update importance scores of incremental matrices and allocate budget accordingly.
python
from loralib import RankAllocator
from loralib import compute_orth_regu 
# Initialize the RankAllocator 
rankallocator = RankAllocator(
    model, lora_r=12, target_rank=8,
    init_warmup=500, final_warmup=1500, mask_interval=10, 
    total_step=3000, beta1=0.85, beta2=0.85, 
)
  • lora_r: The initial rank of each incremental matrix.

  • target_rank: The average target rank of final incremental matrices, i.e. the average number of singular values per matrix.

  • init_warmup: The steps of initial warmup for budget scheduler.

  • final_warmup: The steps of final warmup for budget scheduler.

  • mask_interval: The time internval between two budget allocations.

  • beta1 and beta2: The coefficient of exponentional moving average when updating importance scores.

    At each step of back-propagation, we apply an additional regularization to enforce the orthongonality of SVDLinear modules by compute_orth_regu(model). After each step of optimizer.step(), we then call RankAllocator to update importance estimation and allocate the budget accordingly:

    python
    # ===== Before =====
    # loss.backward() 
    # optimizer.step() 
    # global_step += 1 
    
    # ===== After ======
    (loss+compute_orth_regu(model, regu_weight=0.1)).backward
    optimizer.step()
    rankallocator.update_and_mask(model, global_step)
    global_step += 1

GLUE benchmark

Check the folder NLU for more details about reproducing the GLUE results. An example of adapting DeBERTaV3-base on MNLI:

…

Please see NLU/scripts for more examples of GLUE.

Summarization and Question Answering Task

Check the folder NLG_QA for more details about reproducing the results of summarization and question-answering tasks.
An example of adapting DeBERTaV3-base on SQuADv2:

…

Citation

@inproceedings{
   zhang2023adaptive,
   title={Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning },
   author={Qingru Zhang and Minshuo Chen and Alexander Bukharin and Pengcheng He and Yu Cheng and Weizhu Chen and Tuo Zhao},
   booktitle={The Eleventh International Conference on Learning Representations },
   year={2023},
   url={https://openreview.net/forum?id=lq62uWRJjiY}
}

Issues· 20 开放

查看全部 Issues在 GitHub 打开

暂无开放 Issues,或尚未同步最近议题。

> 标签

Python

暂无评论,来聊聊你的看法吧

> 工具信息

发布日期2026年8月1日
最后更新2026年9月17日
分类编程语言
定价开源

> 相关工具

T
TypeScript
JavaScript 的超集,为前端与全栈提供静态类型
P
Python
通用编程语言,广泛用于 Web、数据与 AI
G
Go
Google 推出的简洁高效系统语言