text
Fun fact about evaluation!
Fun fact about evaluation!
new_line
new_line
text
Did you know that, if you evaluate the same model, with the same prompt formatting & the same fixed few-shot examples, only changing 
Did you know that, if you evaluate the same model, with the same prompt formatting & the same fixed few-shot examples, only changing 
new_line
text
♻️the order in which the few shot examples are added to the prompt ♻️
♻️the order in which the few shot examples are added to the prompt ♻️
new_line
text
you get a difference of up to 3 points in evaluation score? 
you get a difference of up to 3 points in evaluation score? 
new_line
new_line
text
I did a small experiment using some MMLU subsets on the best performing 7B and lower pretrained models from the leaderboard.
I did a small experiment using some MMLU subsets on the best performing 7B and lower pretrained models from the leaderboard.
new_line
new_line
text
I tried 8 different prompting methods (containing more or less information, such as just the question, or Question: question, or Question: question Choices: ..., see the x axis) that are commonly used in evaluation.
I tried 8 different prompting methods (containing more or less information, such as just the question, or Question: question, or Question: question Choices: ..., see the x axis) that are commonly used in evaluation.
new_line
new_line
text
I then compared the results for all these methods, in 5-shot, during 2 runs. The *only difference* between the first and second run being that the samples used in few-shot are not introduced in the same order.
I then compared the results for all these methods, in 5-shot, during 2 runs. The *only difference* between the first and second run being that the samples used in few-shot are not introduced in the same order.
new_line
text
For example, run one would have been "A B C D E Current sample", vs, in run 2, "D C E A B Current sample". 
For example, run one would have been "A B C D E Current sample", vs, in run 2, "D C E A B Current sample". 
new_line
text
All the other experiment parameters stayed exactly the same.
All the other experiment parameters stayed exactly the same.
new_line
new_line
text
As you can see on the attached picture, you get a difference of up to 3 points between the 2 few-shot samples shuffling.
As you can see on the attached picture, you get a difference of up to 3 points between the 2 few-shot samples shuffling.
new_line
new_line
text
So, when just changing *the order of the few shot samples* can change your results by several points, what is the impact of all other "minimal" and unreported prompting changes?
So, when just changing *the order of the few shot samples* can change your results by several points, what is the impact of all other "minimal" and unreported prompting changes?
new_line
new_line
text
-> Any kind of model score, provided without an evaluation script for reproducibility, is basically bullshit (or coms).
-> Any kind of model score, provided without an evaluation script for reproducibility, is basically bullshit (or coms).
new_line
text
-> This is why we need reproducible evaluation in a fair and exactly similar setup, using evaluation suites such as 
-> This is why we need reproducible evaluation in a fair and exactly similar setup, using evaluation suites such as 
inline_code
`lm_eval`
lm_eval
text
 from the Harness, 
 from the Harness, 
inline_code
`lighteval`
lighteval
text
 from HF, or the Open LLM Leaderboard.
 from HF, or the Open LLM Leaderboard.
Fun fact about evaluation!

Did you know that, if you evaluate the same model, with the same prompt formatting & the same fixed few-shot examples, only changing 
♻️the order in which the few shot examples are added to the prompt ♻️
you get a difference of up to 3 points in evaluation score? 

I did a small experiment using some MMLU subsets on the best performing 7B and lower pretrained models from the leaderboard.

I tried 8 different prompting methods (containing more or less information, such as just the question, or Question: question, or Question: question Choices: ..., see the x axis) that are commonly used in evaluation.

I then compared the results for all these methods, in 5-shot, during 2 runs. The *only difference* between the first and second run being that the samples used in few-shot are not introduced in the same order.
For example, run one would have been "A B C D E Current sample", vs, in run 2, "D C E A B Current sample". 
All the other experiment parameters stayed exactly the same.

As you can see on the attached picture, you get a difference of up to 3 points between the 2 few-shot samples shuffling.

So, when just changing *the order of the few shot samples* can change your results by several points, what is the impact of all other "minimal" and unreported prompting changes?

-> Any kind of model score, provided without an evaluation script for reproducibility, is basically bullshit (or coms).
-> This is why we need reproducible evaluation in a fair and exactly similar setup, using evaluation suites such as `lm_eval` from the Harness, `lighteval` from HF, or the Open LLM Leaderboard.
👍
shiv2050 ssml2050 InferenceIllusionist mmhamdy ricfergas giux78 clem lunarflu nikgr
🔥
osanseviero clem lunarflu
👀
osanseviero lunarflu Tonic
❤️
samusenps lunarflu eduagarcia
😔
SerialKicked lunarflu
https://cdn-avatars.huggingface.co/v1/production/uploads/63e80664e02ee67e8e570ec4/rGfRhywmjd_lbqfYzOEdd.png
EsKa
SerialKicked
user
https://cdn-avatars.huggingface.co/v1/production/uploads/1644340617257-noauth.png
Clémentine Fourrier
clefourrier
user
https://cdn-avatars.huggingface.co/v1/production/uploads/645cfe4603fc86c46b3e46d1/-72dah0aoELhfwYwNJ6Ig.jpeg
Lee Jackson
NeuralNovel
user