text
🔍 Today's pick in Interpretability & Analysis of LMs: Universal Neurons in GPT2 Language Models by 
🔍 Today's pick in Interpretability & Analysis of LMs: Universal Neurons in GPT2 Language Models by 
mention
@wesg
wesg
text
mention
@thorsley
thorsley
text
mention
@luminolblue
luminolblue
text
 T.R. Kheirkhah Q. Sun, 
 T.R. Kheirkhah Q. Sun, 
mention
@willhath
willhath
text
mention
@NeelNanda
NeelNanda
text
 D. Bertsimas
 D. Bertsimas
new_line
new_line
text
This work investigates the universality of individual neurons across GPT2 models trained from different initial random seeds, starting from the assumption that such neurons are likely to exhibit interpretable patterns. The authors find 1-5% of neurons showing high correlation across five model seeds, i.e. consistently activating for the same inputs. In particular, those neurons can be grouped into families exhibiting similar functional roles, e.g. modulating the next token prediction entropy, controlling the output norm of an attention head, and promoting/suppressing vocabulary elements in the prediction. Finally, universal neurons are often observed to form antipodal pairs, conjectured to improve the robustness and calibration of model predictions via ensembling.
This work investigates the universality of individual neurons across GPT2 models trained from different initial random seeds, starting from the assumption that such neurons are likely to exhibit interpretable patterns. The authors find 1-5% of neurons showing high correlation across five model seeds, i.e. consistently activating for the same inputs. In particular, those neurons can be grouped into families exhibiting similar functional roles, e.g. modulating the next token prediction entropy, controlling the output norm of an attention head, and promoting/suppressing vocabulary elements in the prediction. Finally, universal neurons are often observed to form antipodal pairs, conjectured to improve the robustness and calibration of model predictions via ensembling.
new_line
new_line
text
📄 Paper: 
📄 Paper: 
resource
https://huggingface.co/papers/2401.12181
paper
2401.12181
https://huggingface.co/papers/2401.12181
Universal Neurons in GPT2 Language Models (2401.12181)
🔍 Today's pick in Interpretability & Analysis of LMs: Universal Neurons in GPT2 Language Models by @wesg @thorsley @luminolblue T.R. Kheirkhah Q. Sun, @willhath @NeelNanda D. Bertsimas

This work investigates the universality of individual neurons across GPT2 models trained from different initial random seeds, starting from the assumption that such neurons are likely to exhibit interpretable patterns. The authors find 1-5% of neurons showing high correlation across five model seeds, i.e. consistently activating for the same inputs. In particular, those neurons can be grouped into families exhibiting similar functional roles, e.g. modulating the next token prediction entropy, controlling the output norm of an attention head, and promoting/suppressing vocabulary elements in the prediction. Finally, universal neurons are often observed to form antipodal pairs, conjectured to improve the robustness and calibration of model predictions via ensembling.

📄 Paper: https://huggingface.co/papers/2401.12181
👍
merve osanseviero wesg lunarflu abidlabs afrideva LiraMirui Aryanne sbrandeis
🤗
s3nh merve osanseviero wesg clem abidlabs lunarflu