text
🔍 Today's pick in Interpretability & Analysis of LMs: The Calibration Gap between Model and Human Confidence in Large Language Models by 
🔍 Today's pick in Interpretability & Analysis of LMs: The Calibration Gap between Model and Human Confidence in Large Language Models by 
mention
@coolprof
coolprof
text
 H. Tejeda A. Kumar 
 H. Tejeda A. Kumar 
mention
@Cbelem
Cbelem
text
mention
@skarny
skarny
text
 et al.
 et al.
new_line
new_line
text
This work involves an experimental study to assess human confidence in LLM responses to multiple-choice MMLU questions based on explanations the LLM provides together with the selected answer. The authors experiment with altering the model prompt to reflect the actual prediction confidence in models’ explanations, showing improved calibration for users’ assessment of LLM’s reliability and a better ability to discriminate between correct and incorrect answers. These results suggest the importance of further research on the impact of automatic explanations on users’ perception.
This work involves an experimental study to assess human confidence in LLM responses to multiple-choice MMLU questions based on explanations the LLM provides together with the selected answer. The authors experiment with altering the model prompt to reflect the actual prediction confidence in models’ explanations, showing improved calibration for users’ assessment of LLM’s reliability and a better ability to discriminate between correct and incorrect answers. These results suggest the importance of further research on the impact of automatic explanations on users’ perception.
new_line
new_line
text
📄 Paper: 
📄 Paper: 
resource
https://huggingface.co/papers/2401.13835
paper
2401.13835
https://huggingface.co/papers/2401.13835
The Calibration Gap between Model and Human Confidence in Large Language
  Models (2401.13835)
🔍 Today's pick in Interpretability & Analysis of LMs: The Calibration Gap between Model and Human Confidence in Large Language Models by @coolprof H. Tejeda A. Kumar @Cbelem @skarny et al.

This work involves an experimental study to assess human confidence in LLM responses to multiple-choice MMLU questions based on explanations the LLM provides together with the selected answer. The authors experiment with altering the model prompt to reflect the actual prediction confidence in models’ explanations, showing improved calibration for users’ assessment of LLM’s reliability and a better ability to discriminate between correct and incorrect answers. These results suggest the importance of further research on the impact of automatic explanations on users’ perception.

📄 Paper: https://huggingface.co/papers/2401.13835
❤️
osanseviero clem victor clefourrier afrideva skarny samusenps aakritikumar adhisetiawan sbrandeis