BertSelfAttention, BertSelfOutput implementation
Hi, I’m studying BERT and am curious about the implementation ofBertSelfAttentionandBertSelfOutput.According to the Transformer paper(Attention Is All You Need (2017, Vaswani et el.)), multi-head self attention requires 4 Linear layers: one for query, one for key, one for value, and the last for final output.However,BertSelfAttentiononly has three Linear layers; only for query, key, and value. The Linear layer for the final output is inBertSelfOutput.Is there any reason why they are implemented like this?
2022-08-10T09:51:41Z
I think this was a design decision by@thomwolf.
2022-08-10T12:33:17Z
It was done like this in the original tensorflow implementation of Bert and can probably be traced back to the tensor2tensor library
2022-08-10T16:33:03Z
Its model/original implementation dependent, DistilBERT does the 4th linear transformation for example.
2022-08-11T07:31:58Z
Thank you all for your reply:-)