The confusing parts of phi-3's implementation
As far as I know, phi-3’s Phi3SuScaledRotaryEmbedding is an implementation of LongRoPE. However, several aspects of the huggingface transformers’s implementation leave me perplexed.Firstly, there’s the determination of the scale. Here, it calculates constants from two configurations, seemingly leading to only one branch being entered.github.comhuggingface/transformers/blob/7ae4fc271d2f0f3cc950b74cdbd9f2af810bea52/src/transformers/models/phi3/modeling_phi3.py#L153-L157scale = self.max_position_embeddings / self.original_max_position_embeddingsif scale <= 1.0:scaling_factor = 1.0else:scaling_factor = math.sqrt(1 + math.log(scale) / math.log(self.original_max_position_embeddings))If I’m not mistaken, this should implement the non-interpolated starting tokens part in LongRoPE.Additionally, this piece of code seems to clearly disregard coding standards. The uninitialized self.inv_freq is defined, and it appears that self.inv_freq could be initialized as self.long_inv_freq and self.short_inv_freq during the initialization phase.github.comhuggingface/transformers/blob/7ae4fc271d2f0f3cc950b74cdbd9f2af810bea52/src/transformers/models/phi3/modeling_phi3.py#L133-L140seq_len = torch.max(position_ids) + 1if seq_len > self.original_max_position_embeddings:ext_factors = torch.tensor(self.long_factor, dtype=torch.float32, device=x.device)else:ext_factors = torch.tensor(self.short_factor, dtype=torch.float32, device=x.device)inv_freq_shape = torch.arange(0, self.dim, 2, dtype=torch.int64, device=x.device).float() / self.dimself.inv_freq = 1.0 / (ext_factors * self.base**inv_freq_shape)Does anyone understand the deeper purpose of these implementations, or is this simply a bug?
2024-06-17T08:40:02Z
Source:As far as I know, phi-3’s Phi3SuScaledRotaryEmbedding is an implementation of LongRoPE(source).If I’m not mistaken, this should implement the non-interpolatedstarting tokens partin LongRoPE.