Implementation difference between Bert and Roberta ForSequenceClassification?
Hi everyone, I was looking into the source code forBertForSequenceClassificationandRobertaForSequenceClassificationand found that theBertForSequenceClassificationuses thepooled_outputproduced by theBertPoolerto generate the head, on the other handRobertaForSequenceClassificationuses thesequence_outputinstead of using thepooled_outputbut reimplements the same with an additional dropout layer to generate the head.If I unravel and compare the code, you can see an extra dropout layer.Why not simply use theRobertaPooler/pooled_outputlike Bert, What is the significance of this extra dropout layer?