Pre-train DistilmByT5Neo
Pre-train DistilmByT5NeoLet’s combine ByT5, mT5, GPT-Neo and DistilBert!ModelFor initial pre-training, a randomly initialized ByT5 model which we distil after pre-training is completed.Datasetsc4multilingualThe PileTraining ScriptsTraining scripts will be created as part of the projectExpected ResultA Distilled model that combines the power of T5 and GPT-Neo while removing the need for tokenization.UPDATE: We could just go for it and implement this with rotary embeddings as well.
2021-06-29T07:12:46Z
We could kick this up a notch further with rotary embeddings…DistilmRoByT5Neo?
2021-06-30T19:36:34Z
Finalizing since another team member will join@wolosonovichit would be great if you could post the hub name of your team member here once you have it
2021-06-30T19:44:03Z
I will do that, thank you very much@patrickvonplaten!
2021-07-01T18:27:23Z
@patrickvonplatencan you add@vmazelisto the spreadsheet. he is a part of our team as well for this project.
2021-07-01T22:41:20Z
should be done
2021-07-02T18:30:40Z
@patrickvonplatencan you update the spreadsheet and replace Brett with@bneb10when you have a chance? thanks so much!
2021-07-02T18:58:28Z
link to our discord serverFlax-HuggingFace-Community-Week