Dropout before layer normalization
At first stage of BartDecoder, we computecompute token embeddingadd positional embeddinglayer normalizationdropout (optional)x = self.embed_tokens(input_ids)x += positionsx = self.layernorm(x)x = dropout(x, p, self.training)I am thinking of moving dropout right before adding positional embedding for making token embedding noisyx = self.embed_tokens(input_ids)x = dropout(x, p, self.training)x += positionsx = self.layernorm(x)Is there any belief that dropout needs to be placed after layer normalization?