Training a language model from scratch with tensorflow (not pytorch)?
Hello there,I am interested in training a language model from scratch. Not fine tuning the usual distilbert ---- running the whole thing on my GPU instead!I found this interesting notebookHow to train a new language model from scratch using Transformers and Tokenizersand I would be interested to know if there is one that usestensorflowinstead. I cannot havepytorchon my machine unfortunately.Is there a huggingface example notebook that would help me do that?Thanks!
2021-08-05T19:05:00Z
Hi there!This might be what you’re after:transformers/examples/tensorflow/language-modeling at master · huggingface/transformers · GitHubYou’d use therun_clmscript for a GPT-2 like model, and therun_mlmscript for a BERT-like model.EDIT: If you’re able to use docker on your machine, you could also use ahuggingface imageto run that notebook you linked to.
2021-08-05T19:52:02Z
thanks@christopheraka aznavourbut these notebooks are much, much more complex than the code on the notebook I liked (or the blog post). Looking for something more streamlined…
2021-08-06T12:02:12Z
@christopherconceptually, I wonder if training my own language model and then fine-tune it for text-classification will work better than fine-tuning the same old distilbert model that everybody is using. The corpus I am working on is highly specialized (say, medicine for instance) so a dedicated language model makes sense.What do you think?
2021-08-09T08:55:24Z
I think it would depend on how much (and how different) specialized data you have (perhaps compare that to the size of the dataset the model was initially pretrained on). If it’s a considerable amount, it might make sense to continue pre-training from the checkpoint of the model you’re interested in.