Sharing BERT formatted corpus
First of all, sorry if I am missing something because this is my first post on the forum.I have thought that users / teams that train and share BERT like models could share the formatted corpus in order to other users to be able to use it in new models such as Electra easily saving time and computation (as the corpus format is the same).Maybe HF could store and share formatted corpus or just the users/team can provide it.The idea comes because I have seen several BERT like models in the hub for different languages and I would like to bring it to Electra as it needs less resources for pretraining and I (and maybe many users) can deal with it.
2020-07-08T07:18:28Z
HuggingFace already provides ~100 NLP datasets in theirnlprepository. I think these are mostly evaluation datasets. However, you can addyour own dataset, too!
2020-07-08T07:22:06Z
Indeed, we provide theToronto Book CorpusandWikipedia(i.e. the datasets used to pretrain Bert) but we are happy to welcome more datasets. Currently we are in the process of adding a very large multilingual dataset based on CommonCrawl and called “OSCAR”. You can follow this process here:https://github.com/huggingface/nlp/pull/348
2020-07-08T11:18:20Z
Great! It would be so cool to have several languages big corpus on the nlp lib
2020-07-08T12:41:48Z
Hi@mrm8488.Second your point, I am also thinking for storage space’s sake, we canhost different preprocessing script. So when you sayload dataset(Electra), it actually doload_dataset(wikipedia)andload_dataset(bookcorpus)and go through electra preprocessing script, and for sure, cache it.Before that come into reality, You definitely would want to check my work that pretrain Electra from the scratch, and fine tune on GLUE ,with fastai and huggin face/nlp.repository:Pretrain-MLM-and-finetune-on-GLUE-with-fastaithe latest post of a series:[HuggingFace/nlp] Create fastai Dataloaders, show batch, and create dataset for LM, MLMI have successfully finetune using hf model and reproduce the Electra small++ result, I am pretraining model from the scratch. You can get updates from my twitterRichard Wang
2020-09-14T00:26:17Z
Is the Toronto Book Corpus you provide the original one (used to pretrain BERT, XLNet, GPT, etc.) or is it recreated? I thought that it is not provided by the authors (any more) but it would be great if they are publicly available
2020-09-14T07:37:52Z
Hi@anon7651424,The source of bookcorpus on the hub is based onthis.As it is w/o document boundary and is not official, may be we should add sth. like “at your own risk” to the introduction string.
2020-09-15T06:27:38Z
Indeed we should add a disclaimer that no-one really knows what was in the TBC used for training Bert.I’ll add one.