LM from Scratch for Tensorflow
Hi,I was following this tutorial to train a LM from scratch:How to train a new language model from scratch using Transformers and TokenizersThe result is a pytorch model. Though I need one for Tensorflow. Is there an easy way to convert it?I tried to modify the training code  by using TFTrainer, TFBertForModelLM instead. but TFTrainer is causing trouble with the Data_collator and LineByLineTextDataset objects.When intitializing the trainer with the data collator i get the error:init() got an unexpected keyword argument ‘data_collator’When calling trainer.train() (without collator) I receive the error: LineByLineTextDataset object has no attribute ‘_variant_tensor’ .
2021-01-16T03:05:01Z
Hi! Here’s a nice example of custom TF MLM learning on XLM-Roberta with Kaggle TPUhttps://www.kaggle.com/riblidezso/finetune-xlm-roberta-on-jigsaw-test-data-with-mlm
2021-01-18T13:30:13Z
Thank you! Will check that link later. Sometimes I wonder why I don’t generally start my research on kaggleHas been a helpful source quite a few times.