

This guide illustrates the main parameters that enable various decoding strategies. More advanced parameters exist for the
[generate] method, which gives you even further control over the [generate] method's behavior.
For the complete list of the available parameters, refer to the API documentation.
Speculative Decoding
Speculative decoding (also known as assisted decoding) is a modification of the decoding strategies above, that uses an
assistant model (ideally a much smaller one) with the same tokenizer, to generate a few candidate tokens. The main
model then validates the candidate tokens in a single forward pass, which speeds up the decoding process. If
do_sample=True, then the token validation with resampling introduced in the
speculative decoding paper is used.
Currently, only greedy search and sampling are supported with assisted decoding, and assisted decoding doesn't support batched inputs.
To learn more about assisted decoding, check this blog post.
To enable assisted decoding, set the assistant_model argument with a model.
thon