Use Pretrained T5 for Summarization

Beginners
1

Hello,

Is there any code snippet of how to use T5 pretrained model in order to do summarization?

2

I used the following code to do my task:

from transformers import T5ForConditionalGeneration

tokenizer = T5Tokenizer.from_pretrained('t5-small')

model = T5ForConditionalGeneration.from_pretrained('t5-small', return_dict=True)

input = "This  is a summarization example. This is a large  sentence."
input_ids = tokenizer("summarize: "+input, return_tensors="pt").input_ids  # Batch size 1

outputs = model.generate(input_ids)

decoded = tokenizer.decode(outputs[0], skip_special_tokens=True)

print(decoded)
3

You can take a look at the contributed notebooks here: https://github.com/huggingface/transformers/tree/master/notebooks#community-notebooks

For instance this one is about finetuning (and using) T5 for summarization: https://github.com/abhimishra91/transformers-tutorials/blob/master/transformers_summarization_wandb.ipynb

4

Hey there @thomwolf , I happened to observe that the ROUGE scores achieved on the tutorial notebook that you have posted are kind of far from the results presented on the paper (roughly 43 for rouge-1).

Are there any fine-tune implementations that have results closer to the original paper?

Thanks a lot in advance!