How to add gru layer in distilbert model?
Screenshot (363)1360×768 134 KB
2022-08-02T12:53:25Z
Hey@jarif! I don’t really know how to help on that, but just for curiosity, why would you add those layers to the model? Doesn’t the self-attention mechanism have the same intent? (Measuring and weighting how relevant past and further information is to the current embedding)
2022-08-02T15:20:24Z
maybe you can ask the question ondistilbert-base-uncased · Discussionsas this will notify the authors too?
2022-08-03T12:17:59Z
my superviser told me to do that