@sgugger Progress Update Aug 4 -> Aug 19
Following@sshleiferexample, here is what I worked on the past few weeks and plan to work in the near future.).Model outputs:Finished cleaning up all model outputs and made sure PyTorch and TensorFlow have the same API.Documentation:Work on cleaning up and updating docstrings and documentation of the main classes:config,tokenizer,modelsandpipelines.Automatic conversion of the tutorials into notebooks and added the “open in colab” button.Trainer:A bit of clean-up to make sureTrainerandTFTrainerhave the same API.Exposed the customization points when the user wants to subclass and override.Initial work to add hyperparameter search (see#6576).Initial work to have an easy bridge betweennlpandTrainer(see#6449).Repository consistency:People love the fact each model file is self-contained and the code is not refactored since they can then quickly experiment, but it can be hard to maintain! Added a script that checks all models are tested (by the common tests) and documented.Funnel Transformer:Paper- Initial work to understand the implementation and port it toTransformers. PyTorch version is almost done.Plans:Continue the work onTrainerwith hyperparameter search andnlp interface.Work on refactoring all examples to useTrainerandnlpFinish porting Funnel Transformer.
2020-08-19T17:34:54Z
Very excited about Optuna integration and Funnel Transformer! Also absolutely love your code reviews and suggestions !
2020-08-19T18:01:44Z
+1 to that. Your code reviews have been super helpful and appreciated.
2020-08-19T20:10:49Z
Optuna integration would be ground-breaking, at least for my use case. Now I used to write my own config class that is an iterator over all possible hyperparameter combinations and just loops over them. Tedious to maintain. Being able to rely on a well tested, well supported package would be a load of my mind!
2020-08-20T02:08:20Z
Looking forward to Optuna integration.
2020-08-20T13:57:15Z
Great to see that you’re working on the Funnel Transformer implementationI’ve already trained a model from scratch and can’t wait to properly evaluate it with Transformers