Issues with Whisper Encoder: Positional Encoding
The positional embedding inhugging face implementationof whisper is a  nn.Embedding layer (learnable) as opposed toOpenAI’s implementationof a standard sinusoid positional embedding.The input to the model is assumed to be padded to 3000 frames (both in hugging face and OpenAI’s implementation). Isn’t it non-optimal? Rather, can we pad it to the largest sequence in a batch?@sanchit-gandhican you help here?
2022-11-15T11:56:27Z
Hey@sahuamrit,The models are equivalent - maybe@ArthurZcan shed light on why we use a learnable embed layer in Transformers?This is how OpenAI trained the model (pad/truncate audio inputs to 30s, then compute log-Mel filter bank features). You can read here why this works:Fine-Tune Whisper For Multilingual ASR with 🤗 Transformers. Training the model by padding/truncating to max length would require an attention mask, which is not how the model is trained. Hence, we pad/truncate to a fixed length.
2022-11-15T15:09:13Z
Hey!I think it was a choice of simplicity; it is equivalent at inference but might indeed be problematic for training and should be frozen. We can open an issue on this to useregister_buffer.Nice catch
2022-11-16T05:25:40Z
Hi@ArthurZ,I dont think it will be same even during inference. When you load official whisper weights, self.positional_encoding won’t have anything and will be random.I think since the demo had to do with finetuning, the nn.Embedding got learnt.
2022-11-16T11:06:20Z
Not really sure I follow, the weights are not random for the 2 different positional embeddings.In the audio encoder we haveself.register_buffer("positional_embedding", sinusoids(n_ctx, n_state)). Which uses thesinusoidsfunction that has no randomness.In the text encoder, thepositional_embeddingis learned.