How to understand the relationship between mamba2 and GLA?
Hi, author, thank you for sharing your fantastic work.
When I was reading things about mamba, I found that in mamba-mini , it says that GLA is a special case of mamba and when the dimension of w changed from [batch, seqlen, dtstate, dim] to [batch, seqlen, dtstate], then they are equalivent.
The author of VMamba also suggested that in the arxiv paper:
Then I found that in the SSD, which is the core component of mamba2, the dimension of matrix A $\odot$ dt is also reduced to [..., nheads], which is may suggests that the matrix w has been reduced to [batch, seqlen]. So my question is, is mamba2 a special case of GLA?
Moreover, I did an experiment testing mamba2 and GLA, and found that they almost share the same result with each other, only with numerical differences (1e-5).
So, How to understand the relationship between mamba1, mamba2 and GLA?
Source: state-spaces/mamba