How to modify each decoding step in ProphetNet Transformer
Hi everyone, I am working on a project which needs modification on each decoding step of the ProphetNet model. At each decoding step, I want to concatenate an embedding representation with decoding output representation before it passes to the softmax layer. I am not sure which script/code needs modification for this purpose. I am new to the transformer libaray. Can you please suggest or provide some references? It will be helpful. Thank you!
2021-07-13T05:52:24Z
Hi@valhallaand@s4sarath, any suggestions for this.  It need not be a prophet-net model, it can t5 or BART model in general. Thank you!
2021-07-13T09:47:07Z
Hi Kaushal,I am also not too familiar with transformers library. It’s isn’t not achievable. But you might need little bit of effort.I suggest you can try it inside you model layer itself. Decoding can be untouched. Just make a condition in your model, that at inference pass the extra embedding for concatenation. It’s fairly straightforward in PyTorch. But not so in TF2 , if you want to serialize.I suggest try these in transformers PyTorch implementation. From my personal experience, their TF implementation is no where close to perfect ( which is no one’s fault ), TF requires effort.
2021-07-14T05:07:21Z
Thank you@s4sarathfor the directions. Let me try your suggestions.