Bug: Hotswapping error with rank pattern
Author: BenjaminBossanCreated Sep 14, 2026Updated Sep 15, 2026
Labelsbugcontributions-welcome
System Info
I found an issue with hotswapping in combination with rank_pattern. I only did a quick investigation so far and could not determine the issue yet. What's clear is that after hotswapping, the scaling for lin1 in the example below is incorrectly set to 3.0, even though the rank is correctly set to 4 and the lora_alpha to 9, which should lead to a scaling of 9 / 4 == 2.25.
Who can help?
Contributions are welcome. If you figure out the cause, please report it here and how you plan to fix it. I will assign the issue on a first come first serve basis. If there are no takers, I'll tackle this myself once I have time (probably not this week).
Reproduction
from copy import deepcopy
import torch
from torch import nn
from peft import LoraConfig, PeftModel, get_peft_model
from peft.utils.hotswap import hotswap_adapter
def test_foobar(tmp_path):
class MLP(nn.Module):
def __init__(self):
super().__init__()
self.lin0 = nn.Linear(10, 20)
self.relu = nn.ReLU()
self.lin1 = nn.Linear(20, 2)
def forward(self, X):
X = self.lin0(X)
X = self.relu(X)
X = self.lin1(X)
return X
base_model = MLP()
inputs = torch.randn(2, 10)
configs = [
# FIXME test passes with rank_pattern={"lin1": 3} but fails with anything != the base rank
LoraConfig(r=3, lora_alpha=9, target_modules=["lin0", "lin1"], rank_pattern={"lin1": 4}),
LoraConfig(r=2, lora_alpha=4, target_modules=["lin0"]),
LoraConfig(r=1, lora_alpha=3, target_modules=["lin1"]),
]
expected = []
with torch.inference_mode():
base_output = base_model(inputs)
for index, config in enumerate(configs):
config.init_lora_weights = False
adapter = get_peft_model(deepcopy(base_model), deepcopy(config)).eval()
adapter.save_pretrained(tmp_path / str(index))
with torch.inference_mode():
output = adapter(inputs)
# sanity check
assert not torch.allclose(output, base_output)
expected.append(output)
# note: the first loaded adapter must be the one that targets all layers
model = PeftModel.from_pretrained(deepcopy(base_model), tmp_path / "0")
with torch.inference_mode():
for index in [0]:
hotswap_adapter(model, tmp_path / str(index), adapter_name="default")
torch.testing.assert_close(model(inputs), expected[index])Expected behavior
The test should pass even if the rank in rank_pattern differs from the global LoRA rank.
Source: huggingface/peft