text
New preprint out with colleagues from MIT and IBM Research
New preprint out with colleagues from MIT and IBM Research
new_line
new_line
resource
https://huggingface.co/papers/2405.12981
paper
2405.12981
https://huggingface.co/papers/2405.12981
Reducing Transformer Key-Value Cache Size with Cross-Layer Attention (2405.12981)
new_line
new_line
text
We introduce a simple mechanism of sharing keys and values across layers, reducing the memory needed for KV cache during inference!!
We introduce a simple mechanism of sharing keys and values across layers, reducing the memory needed for KV cache during inference!!
New preprint out with colleagues from MIT and IBM Research

https://huggingface.co/papers/2405.12981

We introduce a simple mechanism of sharing keys and values across layers, reducing the memory needed for KV cache during inference!!
👍
Jaward DmitryRyumin osanseviero alielfilali01 whitebill
https://cdn-avatars.huggingface.co/v1/production/uploads/6032802e1f993496bc14d9e3/w6hr-DEQot4VVkoyRIBiy.png
Omar Sanseviero
osanseviero
user