Wav2Vec2 Hidden States
@sanchit-gandhi, when we set output_hidden_states=True for the Wav2Vec2 model, we get 13 tensors, where 12 correspond to the outputs from each Encoder Layer. What is the very first output tensor? In the BERT model this corresponds to the output from the embedding layer. In Wav2Vec2 is this the output of the feature extractor projected into some space in combination with positional information?Best,Raj
2023-03-03T16:19:12Z
Hey@RajSang! Great question! That’s exactly right: the first hidden state is the output of the CNN layers with an added positional embedding, i.e. the latent speech representations that we pass into the first transformer layer:wav2vec2897×465 80.2 KBIt’s this hidden state in the code:transformers/modeling_wav2vec2.py at 8c40ba73d8091ebe0bdc8da5b634bf7951d18f99 · huggingface/transformers · GitHubHope that answers your question!
2023-03-12T10:05:46Z
Thanks a lot Sanchit!