I am not able to understand the following conversation from Reddit about a RNN based LLM named RKVM.

NAME_1: "Looks awesome, congratulations. And it is not that important but I do not get how this is faster. You save time by removing matmul (i.e. using linear attention) but it is still sequential forward, thus slow? What am I missing?"

NAME_2: "When the ctxlen is short, FFN is the bottleneck.

However for longer ctxlens, The kv cache of GPT grows (consumes VRAM) and the SA matmul becomes increasingly expensive.

A RNN has other benefits. For example, a simple hidden state that does not grow over ctxlen means we can compress very long prompts."

NAME_1: "Thanks, this makes sense. So for longer ctxlens, RNN could also be slow but so are transformers. However, VRAM is a great argument/advantage for using RNNs esp. in edge devices (also other benefits like non fixed ctxlen)"

NAME_2: "RNN generates token at constant speed & VRAM, while GPT becomes slower (per token) and takes more VRAM over time."
