text
New sampling strategy dropped in 🤗 transformers -- Min P sampling 🔥
New sampling strategy dropped in 🤗 transformers -- Min P sampling 🔥
new_line
new_line
text
Are you tired of having 
Are you tired of having 
inline_code
`top_k`
top_k
text
 arbitrarily discarding high-quality continuations? Or 
 arbitrarily discarding high-quality continuations? Or 
inline_code
`top_p`
top_p
text
 forgetting to exclude low-probability tokens, derailing your generation? Try out the new 
 forgetting to exclude low-probability tokens, derailing your generation? Try out the new 
inline_code
`min_p`
min_p
text
 flag in 
 flag in 
inline_code
`generate`
generate
text
, fresh from a PR merged today! 🥬
, fresh from a PR merged today! 🥬
new_line
new_line
text
Min P consists of a dynamic token filter -- as opposed to Top K, which keeps the K most likely tokens, and Top P, which keeps the most likely tokens up to a fixed cumulative probability, both static filters. Min P takes a base probability (defined in the 
Min P consists of a dynamic token filter -- as opposed to Top K, which keeps the K most likely tokens, and Top P, which keeps the most likely tokens up to a fixed cumulative probability, both static filters. Min P takes a base probability (defined in the 
inline_code
`min_p`
min_p
text
 flag) and multiplies it by the probability of the most likely token in the distribution for the next token. All tokens less likely than the resulting value are filtered. What happens with this strategy?
 flag) and multiplies it by the probability of the most likely token in the distribution for the next token. All tokens less likely than the resulting value are filtered. What happens with this strategy?
new_line
text
👉 High probability token present -> aggressive filter (we don't want to miss on that high-probability case and risk derailing generation)
👉 High probability token present -> aggressive filter (we don't want to miss on that high-probability case and risk derailing generation)
new_line
text
👉 No high probability token present -> relaxed filter (there are many continuation possibilities that the model finds plausible)
👉 No high probability token present -> relaxed filter (there are many continuation possibilities that the model finds plausible)
new_line
new_line
text
You should set 
You should set 
inline_code
`min_p`
min_p
text
 to a low value, between 0.05 and 0.1. It behaves particularly well for creative text generation when paired up with temperature > 1.
 to a low value, between 0.05 and 0.1. It behaves particularly well for creative text generation when paired up with temperature > 1.
new_line
new_line
text
Kudos to 
Kudos to 
mention
@kalomaze
kalomaze
text
 and 
 and 
mention
@menhguin
menhguin
text
 for creating this technique 🔥 Read their discussion in the original issue for benchmarks (
 for creating this technique 🔥 Read their discussion in the original issue for benchmarks (
link
https://github.com/huggingface/transformers/issues/27670
https://github.com/huggingface/transformers/issues/27670
text
)
)
new_line
new_line
text
Copy-pasteable version of the example in the image below here: 
Copy-pasteable version of the example in the image below here: 
link
https://pastebin.com/VqXNtuxd
https://pastebin.com/VqXNtuxd
new_line
new_line
text
Have fun experimenting! 😎
Have fun experimenting! 😎
New sampling strategy dropped in 🤗 transformers -- Min P sampling 🔥

Are you tired of having `top_k` arbitrarily discarding high-quality continuations? Or `top_p` forgetting to exclude low-probability tokens, derailing your generation? Try out the new `min_p` flag in `generate`, fresh from a PR merged today! 🥬

Min P consists of a dynamic token filter -- as opposed to Top K, which keeps the K most likely tokens, and Top P, which keeps the most likely tokens up to a fixed cumulative probability, both static filters. Min P takes a base probability (defined in the `min_p` flag) and multiplies it by the probability of the most likely token in the distribution for the next token. All tokens less likely than the resulting value are filtered. What happens with this strategy?
👉 High probability token present -> aggressive filter (we don't want to miss on that high-probability case and risk derailing generation)
👉 No high probability token present -> relaxed filter (there are many continuation possibilities that the model finds plausible)

You should set `min_p` to a low value, between 0.05 and 0.1. It behaves particularly well for creative text generation when paired up with temperature > 1.

Kudos to @kalomaze and @menhguin for creating this technique 🔥 Read their discussion in the original issue for benchmarks (https://github.com/huggingface/transformers/issues/27670)

Copy-pasteable version of the example in the image below here: https://pastebin.com/VqXNtuxd

Have fun experimenting! 😎
🤗
jankke lunarflu osanseviero menhguin kristaller486 MoritzLaurer florentgbelidji banelingz abdullah ariG23498 John6666 Manel
👍
stzhao