Feature Request: Support MiCA (Minor Component Adaptation)
Feature Request: Support MiCA (Minor Component Adaptation)
Description
Would it be possible to add support for MiCA (Minor Component Adaptation) in FastLanguageModel.get_peft_model()?
MiCA has now been merged into Hugging Face PEFT and is designed as a LoRA-compatible initialization method via:
init_lora_weights="mica"
References
- Paper: https://arxiv.org/abs/2604.01694
- PEFT implementation: https://github.com/huggingface/peft/issues/3142
- Author announcement (MiCA merged into PEFT): https://www.linkedin.com/posts/sten-r%C3%BCdiger-373a52162_mica-is-now-part-of-hugging-face-peft-i-activity-7475983913727967232-yGif
Why This Matters
MiCA is intended for continued pretraining and domain adaptation while remaining compatible with the existing PEFT/LoRA workflow. Since it has now been integrated upstream into PEFT, it would be great if it could also be used through Unsloth's optimized training pipeline for testing and evals.
I was curious whether support is planned once MiCA is included in an official PEFT release, or if compatibility with the PEFT main branch is already feasible.
Thanks for all the work on Unsloth and if nothing else, reading the proposal!
Source: unslothai/unsloth