Add SENet Blocks in Encoding Layers
Feature RequestI read the article “SesameBERT: Attention for Anywhere” and would like to add SENet blocks in the Huggingface implementation. The article’s authors made an implementation withTensorflow, but I would like to use the lib in pytorch.MotivationThe use of SENet Blocks has obtained state-of-the-art results. And they seem to be promising in NLP.Your contributionI know that it is possible to modify the [BertLayer() and [BertEnconder()] classesAny suggestions on how to modify the code so that you can apply the idea used in the article?Captura de Tela 2021-06-02 às 22.50.351060×572 94.3 KB