Fine-tuning GPT2 for bidirectional training
Hallo,we all know, that GPT2 is widely used in text generation task as autoregressive model. May I ask, could I further train GPT2 in a bidirectional attention, like BERT as auto-encoding model? If yes, how could I flexible control the uni- or bidirectional training in GPT2? Thanks a lot!!
2021-11-22T21:23:02Z
I don’t think this is possible at all due to structural differences in the model’s design, but I may be wrong.Reference this if you want more clarity on what I am saying -Comparison between BERT, GPT-2 and ELMo | by Gaurav Ghati | Medium"Drawbacks:GPT is itsuni-directionalnature — the model is only trained to predict the futureleft-to-right context." - from the articleHope this answers your question
2021-11-24T10:04:46Z
Thanks for your info. Really appreciate!While, I know there are already some work to control the bidi- or uni-directional attention by using a matrix. like this:unilm/unilm-v1 at master · microsoft/unilm · GitHubso I am wondering if we could also realise flexible attention by using this in GPT2.
2021-11-24T10:21:43Z
Hi,The reason autoregressive models like GPT2 are trained using a causal attention mask is because otherwise you “leak” information from the future. These models are trained to predict the next token, but if you let the model pay attention to the next token (while it needs to predict this), then that defeats the purpose of training the model.
2021-11-24T11:09:08Z
yes, you make sense.While, in my case, I want to bidirectional process the context part, while generation part is still in unidirectional (thus, the future information won’t be leaked). does that make sense?
2021-11-24T11:10:05Z
Yes that makes sense, and can be done indeed (this is actually a nice research topic).Note thatBigScienceis investigating this, they call it “prefix-lm”:Models - Hugging FaceI assume that bidirectional attention is applied to the “prefix” (i.e. context), and unidirectional on the generation part.
2021-11-24T11:39:51Z
Thanks for the sharing. But there is no detailed infos for the model “prefix-lm”.And actually I wanna to do this with pre-trained GPT or DialoGPT. Could you tell me more infos about how to do that? Thanks a lot!!
2021-11-24T12:05:26Z
Hi,I just found BigScience’s implementation here:bigscience/inference at f5f883866af9d58871d9dc498646dcc133b01b3c · bigscience-workshop/bigscience · GitHubThey adapted GPT-2.
2021-11-24T12:25:49Z
thanks a lot! I will take a look at that. really appreciate!