How is CLS special token embedding initialized?
Hi! Does anyone know how CLS token is initialized in BERT? I mean, let’s say I would like to train a BERT model from scratch (which of course I’m not doing), how should I initialize CLS embedding? Just at random under some distribution such as uniform? How is this done in BERT?
2022-03-16T15:46:26Z
Just like other tokens, the CLS token is randomly initialized from a normal distribution. The only exception is the padding token, which is set to zero.github.comhuggingface/transformers/blob/204c54d411c2b4c7f31405203533a51632f46ab1/src/transformers/models/bert/modeling_bert.py#L731-L734elif isinstance(module, nn.Embedding):module.weight.data.normal_(mean=0.0, std=self.config.initializer_range)if module.padding_idx is not None:module.weight.data[module.padding_idx].zero_()