text
Fun fact about evaluation, part 2!
Fun fact about evaluation, part 2!
new_line
new_line
text
How much do scores change depending on prompt format choice?
How much do scores change depending on prompt format choice?
new_line
new_line
text
Using different prompts (all present in the literature, from 
Using different prompts (all present in the literature, from 
inline_code
`Prompt question?`
Prompt question?
text
 to 
 to 
inline_code
`Question: prompt question?\nChoices: enumeration of all choices\nAnswer: `
Question: prompt question?\nChoices: enumeration of all choices\nAnswer: 
text
), we get a score range of...
), we get a score range of...
new_line
new_line
text
10 points for a single model! 
10 points for a single model! 
new_line
text
Keep in mind that we only changed the prompt, not the evaluation subsets, etc.
Keep in mind that we only changed the prompt, not the evaluation subsets, etc.
new_line
text
Again, this confirms that evaluation results reported without their details are basically bullshit.
Again, this confirms that evaluation results reported without their details are basically bullshit.
new_line
new_line
text
Prompt format on the x axis, all these evals look at the logprob of either "choice A/choice B..." or "A/B...".
Prompt format on the x axis, all these evals look at the logprob of either "choice A/choice B..." or "A/B...".
new_line
new_line
text
Incidentally, it also changes model rankings - so a "best" model might only be best on one type of prompt... 
Incidentally, it also changes model rankings - so a "best" model might only be best on one type of prompt... 
Fun fact about evaluation, part 2!

How much do scores change depending on prompt format choice?

Using different prompts (all present in the literature, from `Prompt question?` to `Question: prompt question?\nChoices: enumeration of all choices\nAnswer: `), we get a score range of...

10 points for a single model! 
Keep in mind that we only changed the prompt, not the evaluation subsets, etc.
Again, this confirms that evaluation results reported without their details are basically bullshit.

Prompt format on the x axis, all these evals look at the logprob of either "choice A/choice B..." or "A/B...".

Incidentally, it also changes model rankings - so a "best" model might only be best on one type of prompt... 
❤️
eduagarcia SivilTaram andysalerno samusenps osanseviero EricNarro
🔥
ancs21 dark-pen
🤯
seth-zou jaahas