KoCLIP: Pretraining CLIP on Korean
KoCLIPBuilding on top of Korean Language models that are publicly available, we want to train a multimodal generative system. Specifically, we train CLIP on Korean datasets usingKoBERTand ViT as backbones.ModelViT, KoBERT (or any other Korean encoder LM)DatasetsKETI has released a Korean image captioning dataset, available onAI Hub. We can also utilize theWIT dataset, which is a multilingual dataset scraped from Wikipedia.Training ScriptThe training script is (almost) already available, seehere.ChallengesKoBERT likely performs slightly worse than the default English BERT. While this could be a bottleneck, we can always write the script in such a way that it is easy to plug-and-play different LMs. This way, when a better Korean LM is released, it can easily be used.There might be minor architectural adjustments we have to make (e.g. adding projection layers).Attaining fluency in JAX will take time and effort.Desired OutcomesThe final deliverable of this project will most likely be an open source repository, accompanying documentation, model weights, and potentially a demo Streamlit app.
2021-06-25T14:14:42Z
jaketae:. While this could be a bottleneck, we can always write the script in such a way that it is easy to plug-and-play different LMs. This way, when a better Korean LM is released, it can easily be usedGreat! Iet me join this fun project!
2021-06-25T14:42:07Z
Thanks everyone! Please feel free to share any thoughts you have regarding the direction or details of this project. Looking forward to the next couple of weeks.
2021-06-26T13:52:00Z
Agreed. I mean I don’t think timezone should ever prevent anyone from joining, but for the purposes of arranging logistics, it would certainly be helpful. I’m also on KST/GMT+9.
2021-06-27T05:31:57Z
Hello@jaketae& Team,I am interested to be a part of such an amazing project & team. I will try my best to contribute to the Korean version of the CLIP model. It would be nice if we could discuss some learning resources that would be useful for this project. I can work in any time zone that is comfortable for everyone in the team.
2021-06-28T16:34:53Z
Great! let’s officially define this projectPutting everybody in the official sheethere. More people can still join!  Leave a comment here or on the sheet if you want to change something.
2021-06-28T16:56:26Z
IMHO aligning Korean encoders with the pre-trained CLIP text encoder will probably suffice. It would be great if we could do better. Count me in(I’m on GMT+9)
2021-06-28T17:06:14Z
Added you to the team
2021-06-30T01:08:08Z
Hi@jaketae& Team,I am interested to be a part of KoCLIP team. I am Korean and will try my best to contribute to this project as possible as i can. It would be nice if i could participate in this project and discuss about it. I can work in any time zone that is comfortable for everyone in the team.
2021-06-30T09:11:48Z
Hi@amphora,@kyungeunadded you to the team