Fine tuning GPT2 on persona chat dataset outputs gibberish
Hello allI’m trying to fine-tune GPT2 more or less using the code from that example:GitHubGitHub - huggingface/transfer-learning-conv-ai: 🦄 State-of-the-Art...🦄 State-of-the-Art Conversational AI with Transfer Learning - GitHub - huggingface/transfer-learning-conv-ai: 🦄 State-of-the-Art Conversational AI with Transfer LearningSome things seem slightly outdated and I adapted the code to train with Pytorch-Lightning in a Jupyter notebook. Still im using 99% unchanged code from Github and the same dataset.Fine-tuning GPT2-medium seems to work. After one epoch the loss is down to roughly 4.At inference the chatbot only outputs gibberish like for example:Hello. How are you?!hey therehow are youwoooowhat are you?wherew where are?do you knowwayokhow are u?tellwhat are uwhatoodoiokwhere dohowi i’mdowhat aredo you?okdo you areyou are ado.you arei doyou arewowi’m soI don’t understand that.are there are what?do you?yesdo you?do you?whati amwhat?i.do you have anydodo youokwhatare?yourwhat are what?i see?sohow are youdoisoi’ve anddotoareiidoi’m youidowhat areiokWhat do you want to say??doidowhatyou are udoi’mdo uaredo uiyou?dodo uiiok,doiokdoi do you aredoare there aredoyouhow arewhat aredodoiwhat uiithat aresodorightwhat?doido uI tried several settings at inference but it’s mostly similar.Where do you think it goes wrong? Is the training not working? Over- or underfittig? Or am I making a mistake at inference?I’m hesitating to post the code yet. Maybe someone of you can already tell if it’s rather about inference or training and I will only post those parts.
2021-04-14T09:28:16Z
Hi, did you ever manage to get this sorted? I’m coming across this problem myself, and was wondering if you could help. ThanksError when finetuning pretrained huggingface conv-ai chatbot model