Pretrain T5 from scratch in Chinese
Pretrain T5 for ChineseCurrently there is No T5 pretrained model for Chinese language on the Huggingface hub. The goal is to train a T5-base model in Chinese and finetune it on Chinese QA dataset CMRC2018 to see its performance compare to other non-T5 model.ModelA randomly initialized T5 modelDatasetsCLUECorpus2020OSCARmC4CMRC2018 (CMRC 2018)Available training scriptsWill be using standard T5 training script from Huggingface(Optional) Desired project outcomeA ChineseT5 model that is competitive with other similar size/speed non-T5 model (Roberta, GPTs etc） on Chinese QA dataset(Optional) ChallengesChinese character is non-spaced, might need special treatment compare to spaced language like English.
2021-07-07T10:30:15Z
sounds good - let’s define it!