Question on language modeling preprocessing
I am trying running the language modeling script run_mlm.py script but I am facing some storage issues when running the preprocessing of the input text data. The main issue here is that the preprocess data by default gets saved in the .cache/huggingface/datasets folder. But my .cache folder is pretty small. Is it possible to redirect the preprocessing of the input text data to a different folder?Thanks a lot for your help.
2021-01-21T15:47:36Z
You can set an environment variable to control where the cache goes and change that default. For all HF libraries, the variable is"HF_HOME".
2021-01-21T22:03:52Z
Thanks for the quick reply. It works like a charm.