[Feature request] Add "generalized delta rule" to LinearAttention op

System information

No response

What is the problem that this feature solves?

RWKV-7 and GDN-2 cannot be implemented with LinearAttention-27.

Generalized delta rule proposed in RWKV-7 "Goose" with Expressive Dynamic State Evolution is a superset of linear attentions used in RWKV-7 and also GDN-2. So if this update rule is supported, RWKV-7 and GDN-2 can be implemented without using Scan op and ensure efficient kernel is used if runtime libraries support it.

Alternatives considered

1. Implementing such linear attentions by using Scan op.
2. Implementing RWKV-7 and GDN-2's linear attention separately.

For 1, it can be, however this cannot ensure RWKV-7 or GDN-2's linear attentions are calculated using efficient kernel even if runtime library support it.

For 2, in #8027,

I think traditionally we preferred a small set of operators over a larger set of them.

So this isn't a better option.

Describe the feature

By implementing generalized delta rule, this can be used for both RWKV-7 and GDN-2 implementation in ONNX.

If GDN-2's linear attention is supported, GDN-2 can, but RWKV-7 cannot be implemented with updated LinearAttention because RWKV-7's linear attention (wkv7) is a superset of GDN-2's one.

Will this influence the current api (Y/N)?

No response

Feature Area

No response

Are you willing to contribute it (Y/N)

Yes

Notes

No response