Confusing logic for `reset_parameters()` in `Embedding` and `Linear` layer
Author: aeryskyBCreated Mar 30, 2025Updated Mar 30, 2025
Problem 1
Source:
LoRA/loralib/layers.py
Issue: lora_B (instead of lora_A) is sampled from normal distribution.
Problem 2
Source: LoRA/loralib/layers.py
Issue: The comment is misguiding because lora_A is supposed to get Kaiming initialization, whereas lora_B is supposed to be initialized as zero.
Source: microsoft/LoRA