How can I do text Summarization using ProphetNet
The ProphetNet is now integrated,huggingface.coProphetNet — transformers 3.4.0 documentationAny idea how to make the model for a text summarization ?
2020-11-05T09:17:40Z
ProphetNet automatically shifts the tokens so you call compute the loss as follows:prophetnet = ProphetNetForConditionalGeneration.from_pretrained(...) 

loss = prophetnet(input_ids=tokenized_article, decoder_input_ids=tokenized_summary, labels=tokenized_summary)
2020-11-05T09:22:38Z
Also you could make use of this notebook - it should be applicable 1-to-1, when you change the tokenizer correctly and switchEncoderDecoderModelwithProphetNetForConditionalGeneration:github.compatrickvonplaten/notebooks/blob/master/BERT2BERT_for_CNN_Dailymail.ipynb{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "BERT2BERT for CNN/Dailymail",
      "provenance": [],
      "collapsed_sections": [],
      "authorship_tag": "ABX9TyMYchE5beUGXDKxJoG2JsLp",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU",
    "widgets": {
      "application/vnd.jupyter.widget-state+json": {
        "e13b8fdb8a564e8699c77890e29995bf": {
          "model_module": "@jupyter-widgets/controls",This file has been truncated.show original
2020-11-18T01:37:02Z
Hi,I tried applying ProphetNet to the Seq2Seq Trainer fine-tuning example script.Environment: Colaboratorytransformersversion: 4.0.0-devPlatform: Linux-4.19.112±x86_64-with-Ubuntu-18.04-bionicPython version: 3.6.9PyTorch version (GPU?): 1.7.0+cu101 (True)Tensorflow version (GPU?): 2.3.0 (True)GPU : Tesla T4Task: summarizationDataset: about one-tenth of XSum (considering the 12-hours limitation of Colab)Scriptpython finetune_trainer.py \
    --learning_rate=3e-5 \
    --fp16 \
    --do_train --do_eval --do_predict --evaluate_during_training \
    --predict_with_generate \
    --max_source_length 510 \
    --per_device_train_batch_size 2 \
    --per_device_eval_batch_size 2 \
    --n_train 20400 \
    --n_val 100 \
    --model_name_or_path microsoft/prophetnet-large-uncased \
    --data_dir $XSUM_DIR \
    --output_dir prophetnet_large_uncased_xsum \
    --save_steps 10000 \
    --save_total_limit 5 \
    --overwrite_output_dirIt took 6:28:28 for fine-tuning and 2:36:39 for prediction.Here is the result.11/17/2020 21:28:35 - INFO - root -   *** Test ***
[INFO|trainer.py:1387] 2020-11-17 21:28:35,447 >> ***** Running Prediction *****
[INFO|trainer.py:1388] 2020-11-17 21:28:35,448 >>   Num examples = 11332
[INFO|trainer.py:1389] 2020-11-17 21:28:35,448 >>   Batch size = 2
100% 5666/5666 [2:36:39<00:00,  1.67s/it]11/18/2020 00:06:13 - INFO - __main__ -   ***** Test results *****
11/18/2020 00:06:13 - INFO - __main__ -     test_loss = 2.11389422416687
11/18/2020 00:06:13 - INFO - __main__ -     test_rouge1 = 38.5525
11/18/2020 00:06:13 - INFO - __main__ -     test_rouge2 = 15.35
11/18/2020 00:06:13 - INFO - __main__ -     test_rougeL = 30.6488
11/18/2020 00:06:13 - INFO - __main__ -     test_rougeLsum = 30.7002
11/18/2020 00:06:13 - INFO - __main__ -     test_gen_len = 26.0
100% 5666/5666 [2:37:53<00:00,  1.67s/it]The prediction (generation) results are output astest_generation.txt.Here are some examples from it.n - dubz have revealed they are up for four prizes at this year’s mtv europe music awards.j craig venter, who won the nobel prize for his work in human genomics, is one of the world’s most celebrated scientists.Considering the time limitations of the execution environment, I used only about one tenth of the dataset for now, but I think we could get better results if we used the entire dataset.For more details, please check the .ipynb in the URL:https://github.com/forest1988/colaboratory/blob/main/prophetnet_seq2seqtrainer_finetuning_experiment.ipynbThank you.
2020-11-18T07:21:15Z
That’s super useful thank you !
2020-11-18T08:23:22Z
It’s my pleasure!I’m glad to hear that it is useful for you.