Question about RecurrentGemma intermediate_size
Hello,I have a question regarding theintermediate_sizeparameter in theRecurrentGemmaConfig. I noticed in the source code thatintermediate_sizeis divided by 2.Here’s the snippet of the code:github.comhuggingface/transformers/blob/a0e77a1f6bdfcceccdc5618e8a01ee32ef47bfa8/src/transformers/models/recurrent_gemma/modeling_recurrent_gemma.py#L470-L473self.intermediate_size = config.intermediate_size // 2self.gate_proj = nn.Linear(self.hidden_size, self.intermediate_size, bias=True)self.up_proj = nn.Linear(self.hidden_size, self.intermediate_size, bias=True)self.down_proj = nn.Linear(self.intermediate_size, self.hidden_size, bias=True)I observed thatintermediate_sizeseems to correspond tomlp_expanded_widthin RecurrentGemma’s official implementation. In the official 2B model, the model width is 2560 and the MLP expansion factor is 3, resulting in anmlp_expanded_widthof 7680. However, in the Hugging Face implementation, the defaultintermediate_sizeis 15360.Given that dividing by 2 yields 7680, which aligns with the official implementation, I’m curious why the division is necessary. Would it not be simpler and clearer to setintermediate_sizedirectly to 7680, thus avoiding the division?Thank you for your time and assistance.