#194·LoRA

Confusing logic for `reset_parameters()` in `Embedding` and `Linear` layer

Author: aeryskyBCreated Mar 30, 2025Updated Mar 30, 2025

Problem 1

Image Source: LoRA/loralib/layers.py
Issue: lora_B (instead of lora_A) is sampled from normal distribution.

Problem 2

Image
Source: LoRA/loralib/layers.py
Issue: The comment is misguiding because lora_A is supposed to get Kaiming initialization, whereas lora_B is supposed to be initialized as zero.