Why OPT's token embeddings are not scaled by sqrt(dim) as in the original OPT implementation?
In the original OPT implementation from metaseq, token embeddings are multiplied by sqrt(embedding_dim):github.comfacebookresearch/metaseq/blob/59403be56179e2ad4bad1be339b741a78e684cd1/metaseq/models/transformer_decoder.py#L327# see IncrementalDecoder for important information about# incremental stateif incremental_state:tokens = tokens[:, -1:]if positions is not None:positions = positions[:, -1:]if token_embedding is None:token_embedding = self.embed_tokens(tokens)x = embed = self.embed_scale * token_embeddingif positions is not None:x += positionsif self.dropout_module is not None:x = self.dropout_module(x)# Returning in T x B x C format as that makes integrating sequence parallelism easier.x = x.transpose(0, 1).contiguous()return x, embed, positionsHowever, I couldn’t find this scaling in HF’s implementation. Is there a reason for not having it?
2023-02-07T14:18:54Z
Hi,Theself.embed_scaleattribute is set to 1.0, hence no scaling is happening (args.no_scale_embedding is set toTrue). See:github.comfacebookresearch/metaseq/blob/59403be56179e2ad4bad1be339b741a78e684cd1/metaseq/models/transformer_decoder.py#LL83C50-L83C50self.embed_scale = 1.0 if args.no_scale_embedding else math.sqrt(self.embed_dim)
2023-02-07T16:54:13Z
It seems that no_scale_embeddings is set to False here:github.comfacebookresearch/metaseq/blob/51871bd73cd04c038f239ea2a26db1d7f6b37927/metaseq/models/transformer_lm.py#L76decoder_learned_pos: bool = field(default=False,metadata={"help": "use learned positional embeddings in the decoder"},)decoder_learned_sinusoidal: bool = field(default=False,metadata={"help": "use learned positional embeddings init with sinusoidal in the decoder"},)no_scale_embedding: bool = field(default=False, metadata={"help": "if True, dont scale embeddings"})checkpoint_activations: bool = field(default=False, metadata={"help": "checkpoint activations at each layer"})offload_activations: bool = field(default=False,metadata={"help": "move checkpointed activations to CPU after they are used."},)# config for Fully Sharded Data Parallel (FSDP) trainingand here:github.comfacebookresearch/metaseq/blob/51871bd73cd04c038f239ea2a26db1d7f6b37927/metaseq/models/transformer_lm.py#L348args.attention_dropout = getattr(args, "attention_dropout", 0.0)args.decoder_embed_dim = getattr(args, "decoder_embed_dim", 512)args.decoder_ffn_embed_dim = getattr(args, "decoder_ffn_embed_dim", 2048)args.decoder_layers = getattr(args, "decoder_layers", 6)args.decoder_attention_heads = getattr(args, "decoder_attention_heads", 8)args.share_decoder_input_output_embed = getattr(args, "share_decoder_input_output_embed", False)args.decoder_learned_pos = getattr(args, "decoder_learned_pos", False)args.decoder_learned_sinusoidal = getattr(args, "decoder_learned_sinusoidal", False)args.no_scale_embedding = getattr(args, "no_scale_embedding", False)args.add_bos_token = getattr(args, "add_bos_token", False)@register_model_architecture("model_parallel_transformer_lm", "transformer_lm_megatron")def transformer_lm_megatron(args):args.decoder_embed_dim = getattr(args, "decoder_embed_dim", 3072)args.decoder_ffn_embed_dim = getattr(args, "decoder_ffn_embed_dim", 3072 * 4)args.decoder_layers = getattr(args, "decoder_layers", 72)args.decoder_attention_heads = getattr(args, "decoder_attention_heads", 32)args.dropout = getattr(args, "dropout", 0.1)Or is it overwritten by a config file?
2023-02-08T09:41:36Z
Pinging@ybelkadahere