text
🔍 Today's pick in Interpretability & Analysis of LMs: AtP*: An efficient and scalable method for localizing LLM behaviour to components by J. Kramár T. Lieberum R. Shah 
🔍 Today's pick in Interpretability & Analysis of LMs: AtP*: An efficient and scalable method for localizing LLM behaviour to components by J. Kramár T. Lieberum R. Shah 
mention
@NeelNanda
NeelNanda
new_line
new_line
text
The attribution patching method (AtP) can provide fast and effective approximations of activation patching, requiring only two forward passes and one backward pass to estimate the contribution of all network components for a given prompt pair.
The attribution patching method (AtP) can provide fast and effective approximations of activation patching, requiring only two forward passes and one backward pass to estimate the contribution of all network components for a given prompt pair.
new_line
new_line
text
While previous work highlighted the effectiveness of attribution patching, authors identify two settings leading to false negatives using AtP:
While previous work highlighted the effectiveness of attribution patching, authors identify two settings leading to false negatives using AtP:
new_line
new_line
text
- When estimating the contribution of pre-activation components, if clean and noise inputs don’t lie in the same activation region, the first-order gradient approximation provided by the gradient leads to large errors (Fig 3).
- When estimating the contribution of pre-activation components, if clean and noise inputs don’t lie in the same activation region, the first-order gradient approximation provided by the gradient leads to large errors (Fig 3).
new_line
text
- When the sum of direct and indirect effects is close to 0, even small approximation errors introduced by nonlinearities can greatly  affect the estimated contribution.
- When the sum of direct and indirect effects is close to 0, even small approximation errors introduced by nonlinearities can greatly  affect the estimated contribution.
new_line
new_line
text
Authors propose two changes to the AtP method to mitigate such issues:
Authors propose two changes to the AtP method to mitigate such issues:
new_line
new_line
text
- recomputing the attention softmax for the selected component, and then taking a linear approximation to the remaining part of the model (QK Fix)
- recomputing the attention softmax for the selected component, and then taking a linear approximation to the remaining part of the model (QK Fix)
new_line
text
- Iteratively zeroing gradients at layers contributing to the indirect effects causing cancellation (GradDrop)
- Iteratively zeroing gradients at layers contributing to the indirect effects causing cancellation (GradDrop)
new_line
new_line
text
AtP and AtP* are compared across several patching settings for Pythia models, finding them effective while much less computationally expensive than other approaches. A new methodology is also proposed to estimate the magnitude of AtP* false negatives given a set of samples and desired confidence levels.
AtP and AtP* are compared across several patching settings for Pythia models, finding them effective while much less computationally expensive than other approaches. A new methodology is also proposed to estimate the magnitude of AtP* false negatives given a set of samples and desired confidence levels.
new_line
new_line
text
📄 Paper: 
📄 Paper: 
resource
https://huggingface.co/papers/2403.00745
paper
2403.00745
https://huggingface.co/papers/2403.00745
AtP*: An efficient and scalable method for localizing LLM behaviour to
  components (2403.00745)
new_line
new_line
text
🔍 All daily picks: 
🔍 All daily picks: 
resource
https://huggingface.co/collections/gsarti/daily-picks-in-interpretability-and-analysis-of-lms-65ae3339949c5675d25de2f9
collection
gsarti/daily-picks-in-interpretability-and-analysis-of-lms-65ae3339949c5675d25de2f9
https://huggingface.co/collections/gsarti/daily-picks-in-interpretability-and-analysis-of-lms-65ae3339949c5675d25de2f9
new_line
🔍 Today's pick in Interpretability & Analysis of LMs: AtP*: An efficient and scalable method for localizing LLM behaviour to components by J. Kramár T. Lieberum R. Shah @NeelNanda

The attribution patching method (AtP) can provide fast and effective approximations of activation patching, requiring only two forward passes and one backward pass to estimate the contribution of all network components for a given prompt pair.

While previous work highlighted the effectiveness of attribution patching, authors identify two settings leading to false negatives using AtP:

- When estimating the contribution of pre-activation components, if clean and noise inputs don’t lie in the same activation region, the first-order gradient approximation provided by the gradient leads to large errors (Fig 3).
- When the sum of direct and indirect effects is close to 0, even small approximation errors introduced by nonlinearities can greatly  affect the estimated contribution.

Authors propose two changes to the AtP method to mitigate such issues:

- recomputing the attention softmax for the selected component, and then taking a linear approximation to the remaining part of the model (QK Fix)
- Iteratively zeroing gradients at layers contributing to the indirect effects causing cancellation (GradDrop)

AtP and AtP* are compared across several patching settings for Pythia models, finding them effective while much less computationally expensive than other approaches. A new methodology is also proposed to estimate the magnitude of AtP* false negatives given a set of samples and desired confidence levels.

📄 Paper: https://huggingface.co/papers/2403.00745

🔍 All daily picks: https://huggingface.co/collections/gsarti/daily-picks-in-interpretability-and-analysis-of-lms-65ae3339949c5675d25de2f9

❤️
osanseviero NeelNanda clem Kukedlc julien-c nirmalendu01
🤯
Kukedlc sbrandeis
/avatars/d364246314535c44cacd87a8e773525a.svg
nirmalendu prakash
nirmalendu01
user
https://cdn-avatars.huggingface.co/v1/production/uploads/1670231290373-5e7749883d77a72421292d07.jpeg
Gabriele Sarti
gsarti
user