VisionEncoderDecoder X-Attn Question
@nielsr- I am trying to understand the X-attn code in VisionEncoderDecoder architecture. I can see the comment in the code" Cross-attention layers are automatically added to the decoder and should be fine-tuned on a downstream generative task, like image captioning."What is the best place to see the actual implementation of the X-attn between V Encoder and L Decoder?Please advice.ThanksPrithivi
2022-06-18T08:07:33Z
You can seeherethat theBertSelfAttentionclass can be used both for self-attention and cross-attention, depending on whether or notencoder_hidden_statesare provided.
2022-06-20T07:30:31Z
Thanks@nielsr. Just to close out the loading and fine-tuning models withVisionEcoderDecoderonly finetunes the X-Attn layers - is my assumption right? if not is there an explicit way to freeze the pretrained vision and language models?Please advice
2022-06-20T08:09:41Z
No, all parameters get updated. It’s just that the weights of the cross-attention layers are the only ones that start from scratch, all other weights are already pre-trained, and those get fine-tuned.
2022-06-20T09:15:16Z
Ok thank you