Mobilebert, training from scratch. Not seeing where loads the teacher
Hello, I am modifying mobilbert with some bells and whistles but I am not finding how to start the pertaining with the teacher-student way. Could you help me with that?
2020-07-13T22:02:04Z
The training script ishere. In general, everything linked to distillation is inthis folder.
2020-07-20T10:34:48Z
Thanks a lot!
2023-09-11T15:19:17Z
Hello@gaceladri! Hope You’re doing well!I’m in the same situation training “mobilebert” on a persian dataset and I couldn’t find a suitable way for pretraining using teacher-student method.I wanted to know if you’d found the answer and successfully trained your model.And also is there any other way to use in this task instead of the distillation?