Distilbart paper
Good evening,Is there a paper about distilbart? I need documentation for my master thesis and I couldn’t find any. Thanks for your help!
2020-07-22T14:21:41Z
Hi@Hildweig, There is no paper for distilbart, the idea of distllbart came from@sshleifer’s great mindYou can find the details of the distillation processhereFor the CNN models, the distiiled model is created by copying the alternating layers frombart-large-cnn. This is no teacher distillation i.e you just copy layers from teacher model and then fine-tune the student model in stander way.And for XSUM it uses ombination of Distillbert’s ce_loss and the hidden states MSE loss used in the tinybert paper.DistilBERTpapertinybertpaper
2020-07-22T14:42:47Z
Thanks a lot! So you’re saying that we still can fine tine distilbart-cnn-12-6 on cnn dm?
2020-07-22T14:49:22Z
distilbart-cnn-12-6  is already fine-tuned on cnn dm. The way it is done is first a student model is created as described above and then its finetuned on cnn dm.You can create your own student(unfine-tuned) following the readme and then fine-tune it on cnn dm. Here are already available unfine-tunedstudents
2020-07-22T14:51:03Z
Thanks a lot!
2020-07-22T20:18:21Z
I’ll add that although there’s nothing out yet, there will likely be a paper at some point.
2020-07-30T12:47:10Z
For the alternated layers, did he choose for example odd layers/ even layers? Or how does it work?
2020-07-30T15:48:31Z
Pinging@sshleifer
2020-07-31T05:04:49Z
thx for the pingfor 12-6 it’s[0, 2, 4, 7, 9, 11]bc I arbitrarily decided I wanted to keep the first and last layer in.for 12-3 it’s[0,6,11]for 12-1 it’s[0]. 12-4[ 0, 4, 8, 11]working way better than 12-3 for mbart-large-enro in early experiments.Note: A full copy of the teacher would be 12-12.Code:https://github.com/huggingface/transformers/blob/09a2f40684f77e62d0fd8485fe9d2d610390453f/examples/seq2seq/distillation.py#L403layers_to_copy = {  # maps  num layers in student -> which teacher layers to copy
                1: [0],
                2: [0, 6],
                3: [0, 6, 11],
                4: [0, 4, 8, 11],
                6: [0, 2, 4, 7, 9, 11],
                9: [0, 1, 2, 4, 5, 7, 9, 10, 11],
                12: all_layers,
            }
2020-08-08T16:19:49Z
Also, what training algorithm was used to train /finetune ? And the learning rate value for which it was fine-tuned on cnn dailymail?
2020-08-09T08:46:17Z
Hi@Hildweig, it was fine-tuned the way you fine-tune/train any encoder-decoder  Transformer model.On a high level, the encoder takes the input sequence, creates a hidden representation of it.The deocoder then receives the encoder representation, and is trained to generate the output sequence auto-regressively using teacher forcing.You may have already read this, butThe Illustrated Transformerby Jay Alammar explains it really well.all the training hparams are in thisfile
2020-09-22T07:57:57Z
@sshleiferis there a reason that the R-L scores in the sheet(~30) differ a lot from R-L scores reported by Facebook in the paper (~40)? (specifically on the CNN/DM dataset)
2020-09-22T15:16:46Z
rouge package:https://github.com/huggingface/transformers/issues/6808
2020-11-10T08:49:47Z
If Anyone is still reading this:The Paper was released on 28.09.2020:Pre-trained Summarization Distillation
2021-03-25T19:50:12Z
@sshleiferHi, Could you please share the hyperparameters that you use for Distilbart-xsum-12-3? I am trying to reproduce the result with KD loss but getting a lower result (21.18 Rouge-2 vs. 21.63 reported in the paper). Thank you!
2021-03-26T13:36:02Z
They should be somewhere in theresearch_projects/directory, but things have moved around.If you show me a command, I can check for glaring errors.Before you retrain, make sure yourmax_lengthandlength_penalty(may have been renamed) match my xsum-12-3 config in the model hub.
2021-03-26T15:34:54Z
This is the command that I am usingtrain_distilbart_seq2seq --output_dir /tmp/path/to/dir/ \
                         --model_name facebook/bart-large-xsum \
                         --tokenizer_name facebook/bart-large-xsum \
                         --task summarization \
                         --dataset_name xsum \
                         --learning_rate 3e-4 \ (taken from the [example script](https://github.com/huggingface/transformers/tree/master/examples/research_projects/seq2seq-distillation))
                         --use_kd_loss True \
                         --alpha_data 1.0 \
                         --alpha_logits 0.8 \
                         --alpha_hidden 3.0 \
                         --max_source_length 1024 \
                         --max_target_length 256 \
                         --do_train \
                         --do_eval \
                         --do_predict \
                         --num_beams 6 \
                         --num_train_epochs 5 \
                         --evaluation_strategy steps \
                         --save_total_limit 5 \
                         --load_best_model_at_end \
                         --predict_with_generateI did not specifymin_length,max_length, andlength_penaltyas I let them take the values from the teacher model (min_length=11, max_length=62, which match the config in the model hub, I will need to double-checklength_penalty). Other than that, please let me know if there’s anything wrong with my command. Thank you!
2021-03-27T18:12:53Z
Command looks fine, but you may want differentlength_penaltythan teacher.