How much memory is needed for training ByteLevelBPETokenizer?
Hi, I’m trying to train LM for Japanese from scratch.To be honest, I copied almost everything fromhttps://github.com/huggingface/blog/blob/master/notebooks/01_how_to_train.ipynb.I just changed datasets from Esperanto datasets to Japanese wiki datasets.But when I tried to train tokenizer, my notebook crashed and restarted probably because of out of memory. My datasets is an entire wikipedia text which is 5.1 G. But my server had 64G memory.How much memory do I need to train tokenizer from scratch?or can I prevent out of memory error with some options?Thanks in advance.
2020-09-16T13:26:00Z
Pinging@anthony
2020-09-17T00:15:17Z
[UPDATE] I tried subset of entire dataset.66,101,891 lines, 5.1 G ( entire datasets ) : out of memory15,000,000 lines: out of memory3,000,000 lines, 231M: successCan I use subset to train tokenizer and use it to train LM with entire datasets?
2020-09-18T05:49:14Z
I could train a tokenizer withtrain_bert_wordpiece.pywithout errors.So it could be jupyter notebooks or something else which caused out of memory errors.