Bert - missing layer norm and resudual after attention block
I’m looking at the BERT implementation. According to the paper, BERT employs the same encoder block of theoriginal transformer.I’m may missing something, but I can’t find theLayerNorm + residual connectionafter the multihead attention in thehuggingface implementation. It looks like the residual is added only after the feed forward block. As a reference, I also point out thepytorch codefor the vanilla transformer, in which you can clearly see the normalization + residual (and dropout) after the multi head attention.Can you confirm that there exists such a difference concerning the BERT implementation and possibly provide me some explanation of that?
2023-08-28T19:58:19Z
Did you ever find the solution to this? I am also confused by this now.
2023-09-27T14:00:53Z
I agree that this seem to be the case - looking forLayerNormin the code clearly shows that, withinBertLayer(and alsoRobertaLayer) it’s only applied withinBertOutput, i.e. after the feed-forward layer, not also between attention and feedforward. Similarly, as far as I could see, there’s no residual connection skipping attention (although it’s harder to ctrl+F through all the usages of+).Has anybody found evidence in the literature, or even some blogs, that this doesn’t affect quality of the overall model? I’ve tried looking around for information, but nobody ever mentions training a transformer with only oneLayerNormper transformer block - let alone, omitting the residual connection. Yet HuggingFace’s implementations of BERT/RoBERTa are extremely popular and quite robust, which sounds like a weird combination?
2023-10-24T08:41:42Z
It is defined here:https://github.com/huggingface/transformers/blob/ede051f1b85a33d2e0576b48042a58dc5332ed70/src/transformers/models/bert/modeling_bert.py#L378C33-L378C33
2023-10-25T07:03:01Z
Thank you, to be honest I don’t know how I could miss that(or even how two different people could miss it independently!)Maybe what put me off is that, in the skip connection, there is one dense layer and one dropout, neither of which is actually there in the original transformer model. So, I guess, HuggingFace’s implementation of BERT/RoBERTa actually has a few layersmore, notlessthan the original transformer? There’s one more dense layer, followed by dropout, hence also a few more weights.The side effect is also that there is no “highway” across any transformer layer: the attention output is not summed to the unchanged input, but rather to the output of another linear layer. It probably doesn’t matter much, it just makes it a bit harder to pass forward the input unchanged.Or am I still missing something?Anyway, thanks for the pointer!