text
More context for your Pascal GPU or older!
More context for your Pascal GPU or older!
new_line
new_line
text
Update: Now available in the official releases of KoboldCpp!
Update: Now available in the official releases of KoboldCpp!
new_line
text
[releases] 
[releases] 
link
https://github.com/LostRuins/koboldcpp/releases/latest
https://github.com/LostRuins/koboldcpp/releases/latest
new_line
new_line
text
These are great news for all the users with GTX 10XX, P40...
These are great news for all the users with GTX 10XX, P40...
new_line
new_line
text
Flash Attention implementation for older NVIDIA GPUs without requiring Tensor Cores has come to llama.cpp in the last few days, and should be merged in the next version of KoboldCpp, you can already try it with another fork or by building it.
Flash Attention implementation for older NVIDIA GPUs without requiring Tensor Cores has come to llama.cpp in the last few days, and should be merged in the next version of KoboldCpp, you can already try it with another fork or by building it.
new_line
new_line
text
[Mentioned KCPP fork] 
[Mentioned KCPP fork] 
link
https://github.com/Nexesenex/kobold.cpp/releases/latest
https://github.com/Nexesenex/kobold.cpp/releases/latest
new_line
new_line
text
[PR] 
[PR] 
link
https://github.com/ggerganov/llama.cpp/pull/7188
https://github.com/ggerganov/llama.cpp/pull/7188
new_line
new_line
text
You should expect less VRAM usage for the same context, allowing you to experience higher contexts with your current GPU.
You should expect less VRAM usage for the same context, allowing you to experience higher contexts with your current GPU.
new_line
new_line
text
There have also been reported final tokens/second speed improvements for inference, so that's also grand!
There have also been reported final tokens/second speed improvements for inference, so that's also grand!
new_line
new_line
text
If you have tried it, I'd like to hear your experiences with --flashattention so far, especially for this implementation and for the large number of Pascal (GTX 10XX, P40...) cards.
If you have tried it, I'd like to hear your experiences with --flashattention so far, especially for this implementation and for the large number of Pascal (GTX 10XX, P40...) cards.
new_line
new_line
text
Discussion linked bellow, with more links to relevant information:
Discussion linked bellow, with more links to relevant information:
new_line
new_line
link
https://huggingface.co/LWDCLS/LLM-Discussions/discussions/11
https://huggingface.co/LWDCLS/LLM-Discussions/discussions/11
new_line
new_line
text
Cheers!
Cheers!
More context for your Pascal GPU or older!

Update: Now available in the official releases of KoboldCpp!
[releases] https://github.com/LostRuins/koboldcpp/releases/latest

These are great news for all the users with GTX 10XX, P40...

Flash Attention implementation for older NVIDIA GPUs without requiring Tensor Cores has come to llama.cpp in the last few days, and should be merged in the next version of KoboldCpp, you can already try it with another fork or by building it.

[Mentioned KCPP fork] https://github.com/Nexesenex/kobold.cpp/releases/latest

[PR] https://github.com/ggerganov/llama.cpp/pull/7188

You should expect less VRAM usage for the same context, allowing you to experience higher contexts with your current GPU.

There have also been reported final tokens/second speed improvements for inference, so that's also grand!

If you have tried it, I'd like to hear your experiences with --flashattention so far, especially for this implementation and for the large number of Pascal (GTX 10XX, P40...) cards.

Discussion linked bellow, with more links to relevant information:

https://huggingface.co/LWDCLS/LLM-Discussions/discussions/11

Cheers!
🚀
Lewdiculous gate369 odyss3y fluffyone Ramikan-BR Bjorno
👍
malificar xpgx1 Ramikan-BR
❤️
fluffyone Ramikan-BR fjall
🔥
AlekseiPravdin Ramikan-BR svenard
👀
Ramikan-BR den0620
➕
Ardvark123
https://cdn-avatars.huggingface.co/v1/production/uploads/65f115216c162dbd552fb41d/-hfb5OFTL5Ex4lJziSgxg.png
Ardvark-san
Ardvark123
user
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/HOCJv7WE2C4eqqqppDFpA.png
Lewdiculous (AetherArchitectural)
Lewdiculous
user
/avatars/ac0c386e729cccdc8290cb5d28ddc4a4.svg
Virt
Virt-io
user