Order of execution of Top-K, Top-P sampling along with temperature
Let’s say I use:sample_outputs = model.generate(**model_inputs,max_new_tokens=40,do_sample=True,
    top_k=3,top_p=0.51, temperature=0.6, num_return_sequences=3,)What is the order of execution in this one?Looked at the code forlabml.aisampling example and it doesn’t make sense because when using Temperature with top-K or Top-P, it first uses Softmax and selects and then use theSamplerIn theGoogle Cloud Document, it says that you use Top-K then Filter with Top-P along with Temperature.Let’s say yourPROBABILITIESas t0→0.4 , t1→0.2, t2→0.2, t3→0.15, t4→0.05`You useTop-K = 3and now you havet0,t1,t2. Now you have 2 choices:Whether to applyTop-P = 0.51and then you again NormalizeOr Normalize first then ApplyTop-P = 0.51and then again NormalizeOn top of it, if we usetemperature = 0.6do we apply it in beginning? If yeas, then it is different when we would have used on justTop-Pbecause the distribution has been shifted already.How does that work? Can someone please explain in terms of execution?
2023-10-31T17:34:00Z
From the sourcehere, functiontop_k_top_p_filtering, I believe it first does top-k sampling then top-p sampling.