
# Multi-purpose Summarizer (Fine-tuned 11B google/flan-t5-xxl on several Summarization datasets)

 <a href="https://colab.research.google.com/drive/1fNOfy7oHYETI_KzJSz8JrhYohFBBl0HY">
  <img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/>
</a>

A fine-tuned version of [google/flan-t5-xxl](https://huggingface.co/google/flan-t5-xxl) on various summarization datasets (xsum, wikihow, cnn_dailymail/3.0.0, samsum, scitldr/AIC, billsum, TLDR, wikipedia-summary)

70% of the data was also filtered with the use of the [contriever](https://github.com/facebookresearch/contriever) with a cosine similarity between text and summary of 0.6 as threshold.

Goal: a model that can be used for a general-purpose summarizer for academic and general usage. Control over the type of summary can be given by varying the instruction prepended to the source document. The result works well on lots of text, although trained with a max source length of 512 tokens and 150 max summary length. 

---

## Usage 
Check the colab notebook for desired usage.
**The model expects a prompt prepended to the source document to indicate the type of summary**, this model was trained with a large (100s) variety of prompts:
```

. 
example_prompts = {
    "social": "Produce a short summary of the following social media post:",
    "ten": "Summarize the following article in 10-20 words:",
    "5": "Summarize the following article in 0-5 words:",
    "100": "Summarize the following article in about 100 words:",
    "summary": "Write a ~ 100 word summary of the following text:",
    "short": "Provide a short summary of the following article:",
}
```
The model has also learned for the length of the summary to be specified in words by a range "x-y words" or e.g. "~/approximately/about/ x words."

Prompts should be formatted with a colon at the end so that the input to the model is formatted as e.g. "Summarize the following: \n\n  {input_text}"

After `pip install transformers` run the following code:

This pipeline will run slower and not have some of the tokenization parameters as the colab.
```python
from transformers import pipeline

summarizer = pipeline("summarization", "jordiclive/flan-t5-11b-summarizer-filtered", torch_dtype=torch.bfloat16)

raw_document = 'You must be 18 years old to live or work in New York State...'
prompt = "Summarize the following article in 10-20 words:"
results = summarizer(
        f"{prompt} \n\n {raw_document}",
        num_beams=5,
        min_length=5,
        no_repeat_ngram_size=3,
        truncation=True,
        max_length=512,
    )
```

---

## Training procedure

- Training was done in BF16, deepspeed stage 2 with CPU offload for 1 epoch with val loss monitored.

## Hardware
- GPU count	8 NVIDIA A100-SXM4-80GB
- CPU count	48
### Training hyperparameters


The following hyperparameters were used during training:
- learning_rate: 3e-05
- train_batch_size: 4
- eval_batch_size: 4
- seed: 42
- distributed_type: multi-GPU
- gradient_accumulation_steps: 2
- effective_train_batch_size: 64
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- warmup_steps: 2000
- num_epochs: 4


### Framework versions

- Transformers 4.24.0
- Pytorch 1.9.1+cu111
- Deepspeed 0.7.4
- Pytorch-lightning 1.8.1
{"language": ["en"], "license": ["apache-2.0", "bsd-3-clause"], "tags": ["summarization", "extractive", "summary", "abstractive", "multi-task", "document summary"], "datasets": ["jordiclive/scored_summarization_datasets", "jordiclive/wikipedia-summary-dataset"], "metrics": ["rouge"]}
transformers pytorch t5 text2text-generation summarization extractive summary abstractive multi-task document summary en dataset:jordiclive/scored_summarization_datasets dataset:jordiclive/wikipedia-summary-dataset license:apache-2.0 license:bsd-3-clause autotrain_compatible endpoints_compatible has_space text-generation-inference region:us
TAGS
#transformers #pytorch #t5 #text2text-generation #summarization #extractive #summary #abstractive #multi-task #document summary #en #dataset-jordiclive/scored_summarization_datasets #dataset-jordiclive/wikipedia-summary-dataset #license-apache-2.0 #license-bsd-3-clause #autotrain_compatible #endpoints_compatible #has_space #text-generation-inference #region-us 


# Multi-purpose Summarizer (Fine-tuned 11B google/flan-t5-xxl on several Summarization datasets)

 <a href="URL
  <img src="URL alt="Open In Colab"/>
</a>

A fine-tuned version of google/flan-t5-xxl on various summarization datasets (xsum, wikihow, cnn_dailymail/3.0.0, samsum, scitldr/AIC, billsum, TLDR, wikipedia-summary)

70% of the data was also filtered with the use of the contriever with a cosine similarity between text and summary of 0.6 as threshold.

Goal: a model that can be used for a general-purpose summarizer for academic and general usage. Control over the type of summary can be given by varying the instruction prepended to the source document. The result works well on lots of text, although trained with a max source length of 512 tokens and 150 max summary length. 

---

## Usage 
Check the colab notebook for desired usage.
The model expects a prompt prepended to the source document to indicate the type of summary, this model was trained with a large (100s) variety of prompts:

The model has also learned for the length of the summary to be specified in words by a range "x-y words" or e.g. "~/approximately/about/ x words."

Prompts should be formatted with a colon at the end so that the input to the model is formatted as e.g. "Summarize the following: \n\n  {input_text}"

After 'pip install transformers' run the following code:

This pipeline will run slower and not have some of the tokenization parameters as the colab.


---

## Training procedure

- Training was done in BF16, deepspeed stage 2 with CPU offload for 1 epoch with val loss monitored.

## Hardware
- GPU count	8 NVIDIA A100-SXM4-80GB
- CPU count	48
### Training hyperparameters


The following hyperparameters were used during training:
- learning_rate: 3e-05
- train_batch_size: 4
- eval_batch_size: 4
- seed: 42
- distributed_type: multi-GPU
- gradient_accumulation_steps: 2
- effective_train_batch_size: 64
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- warmup_steps: 2000
- num_epochs: 4


### Framework versions

- Transformers 4.24.0
- Pytorch 1.9.1+cu111
- Deepspeed 0.7.4
- Pytorch-lightning 1.8.1
# Multi-purpose Summarizer (Fine-tuned 11B google/flan-t5-xxl on several Summarization datasets)

 <a href="URL
  <img src="URL alt="Open In Colab"/>
</a>

A fine-tuned version of google/flan-t5-xxl on various summarization datasets (xsum, wikihow, cnn_dailymail/3.0.0, samsum, scitldr/AIC, billsum, TLDR, wikipedia-summary)

70% of the data was also filtered with the use of the contriever with a cosine similarity between text and summary of 0.6 as threshold.

Goal: a model that can be used for a general-purpose summarizer for academic and general usage. Control over the type of summary can be given by varying the instruction prepended to the source document. The result works well on lots of text, although trained with a max source length of 512 tokens and 150 max summary length. 

--- ## Usage 
Check the colab notebook for desired usage.
The model expects a prompt prepended to the source document to indicate the type of summary, this model was trained with a large (100s) variety of prompts:

The model has also learned for the length of the summary to be specified in words by a range "x-y words" or e.g. "~/approximately/about/ x words."

Prompts should be formatted with a colon at the end so that the input to the model is formatted as e.g. "Summarize the following: \n\n  {input_text}"

After 'pip install transformers' run the following code:

This pipeline will run slower and not have some of the tokenization parameters as the colab.


--- ## Training procedure

- Training was done in BF16, deepspeed stage 2 with CPU offload for 1 epoch with val loss monitored. ## Hardware
- GPU count	8 NVIDIA A100-SXM4-80GB
- CPU count	48 ### Training hyperparameters


The following hyperparameters were used during training:
- learning_rate: 3e-05
- train_batch_size: 4
- eval_batch_size: 4
- seed: 42
- distributed_type: multi-GPU
- gradient_accumulation_steps: 2
- effective_train_batch_size: 64
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- warmup_steps: 2000
- num_epochs: 4 ### Framework versions

- Transformers 4.24.0
- Pytorch 1.9.1+cu111
- Deepspeed 0.7.4
- Pytorch-lightning 1.8.1
TAGS
#transformers #pytorch #t5 #text2text-generation #summarization #extractive #summary #abstractive #multi-task #document summary #en #dataset-jordiclive/scored_summarization_datasets #dataset-jordiclive/wikipedia-summary-dataset #license-apache-2.0 #license-bsd-3-clause #autotrain_compatible #endpoints_compatible #has_space #text-generation-inference #region-us 
 # Multi-purpose Summarizer (Fine-tuned 11B google/flan-t5-xxl on several Summarization datasets)

 <a href="URL
  <img src="URL alt="Open In Colab"/>
</a>

A fine-tuned version of google/flan-t5-xxl on various summarization datasets (xsum, wikihow, cnn_dailymail/3.0.0, samsum, scitldr/AIC, billsum, TLDR, wikipedia-summary)

70% of the data was also filtered with the use of the contriever with a cosine similarity between text and summary of 0.6 as threshold.

Goal: a model that can be used for a general-purpose summarizer for academic and general usage. Control over the type of summary can be given by varying the instruction prepended to the source document. The result works well on lots of text, although trained with a max source length of 512 tokens and 150 max summary length. 

--- ## Usage 
Check the colab notebook for desired usage.
The model expects a prompt prepended to the source document to indicate the type of summary, this model was trained with a large (100s) variety of prompts:

The model has also learned for the length of the summary to be specified in words by a range "x-y words" or e.g. "~/approximately/about/ x words."

Prompts should be formatted with a colon at the end so that the input to the model is formatted as e.g. "Summarize the following: \n\n  {input_text}"

After 'pip install transformers' run the following code:

This pipeline will run slower and not have some of the tokenization parameters as the colab.


--- ## Training procedure

- Training was done in BF16, deepspeed stage 2 with CPU offload for 1 epoch with val loss monitored. ## Hardware
- GPU count	8 NVIDIA A100-SXM4-80GB
- CPU count	48 ### Training hyperparameters


The following hyperparameters were used during training:
- learning_rate: 3e-05
- train_batch_size: 4
- eval_batch_size: 4
- seed: 42
- distributed_type: multi-GPU
- gradient_accumulation_steps: 2
- effective_train_batch_size: 64
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- warmup_steps: 2000
- num_epochs: 4 ### Framework versions

- Transformers 4.24.0
- Pytorch 1.9.1+cu111
- Deepspeed 0.7.4
- Pytorch-lightning 1.8.1
TAGS
#transformers #pytorch #t5 #text2text-generation #summarization #extractive #summary #abstractive #multi-task #document summary #en #dataset-jordiclive/scored_summarization_datasets #dataset-jordiclive/wikipedia-summary-dataset #license-apache-2.0 #license-bsd-3-clause #autotrain_compatible #endpoints_compatible #has_space #text-generation-inference #region-us 
# Multi-purpose Summarizer (Fine-tuned 11B google/flan-t5-xxl on several Summarization datasets)

 <a href="URL
  <img src="URL alt="Open In Colab"/>
</a>

A fine-tuned version of google/flan-t5-xxl on various summarization datasets (xsum, wikihow, cnn_dailymail/3.0.0, samsum, scitldr/AIC, billsum, TLDR, wikipedia-summary)

70% of the data was also filtered with the use of the contriever with a cosine similarity between text and summary of 0.6 as threshold.

Goal: a model that can be used for a general-purpose summarizer for academic and general usage. Control over the type of summary can be given by varying the instruction prepended to the source document. The result works well on lots of text, although trained with a max source length of 512 tokens and 150 max summary length. 

---## Usage 
Check the colab notebook for desired usage.
The model expects a prompt prepended to the source document to indicate the type of summary, this model was trained with a large (100s) variety of prompts:

The model has also learned for the length of the summary to be specified in words by a range "x-y words" or e.g. "~/approximately/about/ x words."

Prompts should be formatted with a colon at the end so that the input to the model is formatted as e.g. "Summarize the following: \n\n  {input_text}"

After 'pip install transformers' run the following code:

This pipeline will run slower and not have some of the tokenization parameters as the colab.


---## Training procedure

- Training was done in BF16, deepspeed stage 2 with CPU offload for 1 epoch with val loss monitored.## Hardware
- GPU count	8 NVIDIA A100-SXM4-80GB
- CPU count	48### Training hyperparameters


The following hyperparameters were used during training:
- learning_rate: 3e-05
- train_batch_size: 4
- eval_batch_size: 4
- seed: 42
- distributed_type: multi-GPU
- gradient_accumulation_steps: 2
- effective_train_batch_size: 64
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- warmup_steps: 2000
- num_epochs: 4### Framework versions

- Transformers 4.24.0
- Pytorch 1.9.1+cu111
- Deepspeed 0.7.4
- Pytorch-lightning 1.8.1