Seq2Seq Distillation: Methodology Questions
This thread should be used to ask questions about howexamples/seq2seq/distillation.pyworks, and to ask questions about the associated paper after it gets released.
2020-09-28T09:37:08Z
What is the reasoning behind choosing alternating layers ?no teacher distillation scores for XSUM ?no teacher is working for non seq-2-seq task as well as we saw with MNLI, should we also see if it works other tasks as well ?
2020-10-05T13:24:05Z
Alternating layers seems to perform the best by a moderate amount.Definitely interested to see results for other tasks!
2020-12-17T05:26:42Z
relocated toexamples/research_projects/seq2seq-distillation/distillation.py?
2020-12-17T06:44:19Z
Yes, that project is now moved toresearch_projectsdir.
2021-06-25T06:59:05Z
Hey@sshleifer, I was trying to fine-tune thedistill-pegasus-cnn-16-4modelprovided by you but I am not sure of the hyper-parameters. Could you please share the hyper-parameters that you used to train this model (and achieve the results shown in Table 5 from yourpaper?Thanks a lot!Naman
2022-06-03T16:24:59Z
Hi! have a question regarding the article «Pre-trained Summarization Distillation» (https://arxiv.org/pdf/2010.13002.pdf). In section 6.2, it is said «Table 10 shows results from fine-tuningteachermodels…». However, throughout the paper it is stated that the experiments with pseudo-labeling only when fine-tuning thestudentmodel were performed. Is it a typo and the result from fine-tuningstudentmodels is indeed depicted?Thanks in advance!
2023-08-07T16:12:56Z
Hi@sshleifer. Any thoughts on if the T5 distillation would still be feasible with PEFT techniques such as  LORA? I have a fine tuned T5-11B using LORA and want to distill this model to something feasible like T5-base or even T5-large. But I’m not sure if the teacher model , which essentially has a LoRA adapter work on a similar way ? Any thoughts / ideas regarding this would be great help. Thanks