Pretrain GPT-2 from scratch in Thai
GPT2 for ThaiThe goal is to create a strong language generation model for ThaiSince initial code and data are pretty much written by@patrickvonplatenand other huggingface members, it should not be so hard to get the first sense.ModelRandomly initialized GPT2 modelDatasetsWe can use OSCAR which is available throughdatasetsDatasetsA causal language modeling script for Flax is availablehere. It can be used pretty much without any required code changes.If there is time left, I’d love to try some private crawling and integrate it datasets.Expected OutcomeUnderstandable Thai text generation modelChallengesLack of data → The OSCAR dataset might be too small (it has < 20GB of data for Thai). It would be better to collect and accumulate to bigger dataset.LinksColab Notebook to train CLMOSCARflax codeTeam MembersSakares Saengkaew (@sakares)Wilson Wongso (@w11wo)