text
Up to 3x faster LLM generation with no extra resources/requirements - ngram speculation has landed in 🤗 transformers! 🏎️💨
Up to 3x faster LLM generation with no extra resources/requirements - ngram speculation has landed in 🤗 transformers! 🏎️💨
new_line
new_line
text
All you need to do is to add 
All you need to do is to add 
inline_code
`prompt_lookup_num_tokens=10`
prompt_lookup_num_tokens=10
text
 to your 
 to your 
inline_code
`generate`
generate
text
 call, and you'll get faster LLMs 🔥
 call, and you'll get faster LLMs 🔥
new_line
new_line
new_line
text
How does it work? 🤔
How does it work? 🤔
new_line
new_line
text
Start with assisted generation, where a smaller model generates candidate sequences. The net result is a significant speedup if the model agrees with the candidate sequences! However, we do require a smaller model trained similarly 😕
Start with assisted generation, where a smaller model generates candidate sequences. The net result is a significant speedup if the model agrees with the candidate sequences! However, we do require a smaller model trained similarly 😕
new_line
new_line
text
The idea introduced (and implemented) by Apoorv Saxena consists of gathering the candidate sequences from the input text itself. If the latest generated ngram is in the input, use the continuation therein as a candidate! No smaller model is required while still achieving significant speedups 🔥
The idea introduced (and implemented) by Apoorv Saxena consists of gathering the candidate sequences from the input text itself. If the latest generated ngram is in the input, use the continuation therein as a candidate! No smaller model is required while still achieving significant speedups 🔥
new_line
new_line
text
In fact, the penalty of gathering and testing the candidates is so small that you should use this technique whenever possible!
In fact, the penalty of gathering and testing the candidates is so small that you should use this technique whenever possible!
new_line
new_line
text
Here is the code example that produces the outputs shown in the video: 
Here is the code example that produces the outputs shown in the video: 
link
https://pastebin.com/bms6XtR4
https://pastebin.com/bms6XtR4
new_line
new_line
text
Have fun 🤗
Have fun 🤗
Up to 3x faster LLM generation with no extra resources/requirements - ngram speculation has landed in 🤗 transformers! 🏎️💨

All you need to do is to add `prompt_lookup_num_tokens=10` to your `generate` call, and you'll get faster LLMs 🔥


How does it work? 🤔

Start with assisted generation, where a smaller model generates candidate sequences. The net result is a significant speedup if the model agrees with the candidate sequences! However, we do require a smaller model trained similarly 😕

The idea introduced (and implemented) by Apoorv Saxena consists of gathering the candidate sequences from the input text itself. If the latest generated ngram is in the input, use the continuation therein as a candidate! No smaller model is required while still achieving significant speedups 🔥

In fact, the penalty of gathering and testing the candidates is so small that you should use this technique whenever possible!

Here is the code example that produces the outputs shown in the video: https://pastebin.com/bms6XtR4

Have fun 🤗
❤️
samusenps Dlbk akashicmarga nickandbro osanseviero abidlabs clem alielfilali01 gblazex DamarJati jaigouk Goekdeniz-Guelmez MaziyarPanahi nbroad abdullah m-ric clefourrier takeraparterer Felladrin
🤗
samusenps osanseviero clem alielfilali01 Svngoku takeraparterer
👍
MoritzLaurer osanseviero DRXD1000 manu clem takeraparterer
https://cdn-avatars.huggingface.co/v1/production/uploads/5fd5e18a90b6dc4633f6d292/gZXHW5dd9R86AV9LMZ--y.png
Maziyar Panahi
MaziyarPanahi
user
https://cdn-avatars.huggingface.co/v1/production/uploads/1641203017724-noauth.png
Joao Gante
joaogante
user