Pegasus Questions
Q: Max model input size varies between checkpoints, what is the max num input tokens that each model can process?A:max_model_length = {
    "xsum": 512,
    "cnn_dailymail": 1024,
    "newsroom": 512,
    "wikihow": 512,
    "multi_news": 1024,
    "reddit_tifu": 512,
    "big_patent": 1024,
    "arxiv": 1024,
    "pubmed": 1024,
    "gigaword": 128,
    "aeslc": 512,
    "billsum": 1024,
    "large": 1024,
}that constant is defined here:github.comsshleifer/transformers_fork/blob/f69cac3347641beaba5037b9a6fca1a46f423639/src/transformers/configuration_pegasus.py#L67-L67max_model_length = {
2020-08-24T14:59:48Z
So does that meanmax_position_embeddingis reduced for fine-tuned models (gigaword, wikihow). i.e if themax_position_embeddingfor the pre-trained model is 1024 then all fine-tuned models should also have same, right ?
2020-08-24T15:42:51Z
The positional embeddings are static, so the max model_input_length is more a reflection of how the model was finetuned than a fundamental limitation.I got the params from:https://github.com/google-research/pegasus/blob/master/pegasus/params/public_params.pyFor some of the savedstate_dicts on S3, I simply removed theembed_positions.weightto allow passingmax_position_embeddings=ANY_INTtofrom_pretrainedallocate more atinit.  Let me know if you try that and it fails and I can delete theembed_positions.weightas needed.
2020-08-24T15:47:49Z
Was suggested I re-direct my questions from this thread:Questions about Pegasus for SummarizationTwo questions: (1) does anyone have good examples of the google/pegasus models being used for summarization using the Automodal and Autotokenizer classes?(2) Using the google/pegasus models, how do you set the minimum and/or maximum length for the generated summary?Thanks!
2020-08-24T19:50:37Z
Thanks for redirectingDocshave a usage example, that I’ll put hereimport torch
from transformers import PegasusForConditionalGeneration, PegasusTokenizer
src_text = [
    """ PG&E stated it scheduled the blackouts in response to forecasts for high winds amid dry conditions. The aim is to reduce the risk of wildfires. Nearly 800 thousand customers were scheduled to be affected by the shutoffs which were expected to last through at least midday tomorrow."""
]

model_name = 'google/pegasus-xsum'
torch_device = 'cuda' if torch.cuda.is_available() else 'cpu'
tokenizer = PegasusTokenizer.from_pretrained(model_name)
model = PegasusForConditionalGeneration.from_pretrained(model_name).to(torch_device)
batch = tokenizer.prepare_seq2seq_batch(src_text, truncation=True, padding='longest').to(torch_device)
translated = model.generate(**batch)
tgt_text = tokenizer.batch_decode(translated, skip_special_tokens=True)
assert tgt_text[0] == "California's largest electricity provider has turned off power to tens of thousands of customers."For all seq2seq/ForConditionalGeneration models, you can control generated summary lengths with themin_lengthandmax_lengthkeyword arguments to the generate function.If you don’t pass them, the (good) defaults from config will be used. The defaults from config were set by the authors to get a good validation rouge on the finetuning dataset.
2020-08-25T01:46:52Z
In the original paper Section 6:We used sinusoidal positional encodingand Section 6.2:This would present a problem for position embeddings which would never be updated for longer input lengths, but we confirm the postulation that sinusoidal positional encodings (Vaswani et al., 2017) generalize well when fine-tuning PEGASUSLARGE beyond the input lengths observed in training up to Linput = 1024 tokens.So you can use longer input than training, but you have to observe if model still generalize with longer input.
2020-08-25T08:58:45Z
So if we want to use longer input we’ll need to resize the embeddings (as these are static)  and these newly added embeddings will be randomly initialised and need to be trained.Is this correct ?
2020-08-25T15:18:49Z
Ideally you won’t need to manually resize, and can just passmax_position_embeddingsas a kwarg tofrom_pretrained
2020-08-27T20:54:36Z
Something is wrong ?UPDATE : Normal behavior…as pubmed dataset(biomedical text) is used for finetuningText :The tower is 324 metres (1,063 ft) tall, about the same height as an 81-storey building, and the tallest structure in Paris. Its base is square, measuring 125 metres (410 ft) on each side. During its construction, the Eiffel Tower surpassed the Washington Monument to become the tallest man-made structure in the world, a title it held for 41 years until the Chrysler Building in New York City was finished in 1930. It was the first structure to reach a height of 300 metres. Due to the addition of a broadcasting aerial at the top of the tower in 1957, it is now taller than the Chrysler Building by 5.2 metres (17 ft). Excluding transmitters, the Eiffel Tower is the second tallest free-standing structure in France after the Millau Viaduct.google/pegasus-pubmedsummary:the tower of so paulo ( tower of so paulo , brazil ) is the tallest building in the brazilian amazon .its height is determined by measuring the distance from the top of the building to the base of the building .the building is located in so paulo , brazil , on the so paulo river .its height is determined by measuring the distance from the top of the building to the base of the building .
2020-08-28T00:18:26Z
I cant tell from that. It might be so different from a pubmed article that the model gets confused. Try finding the pubmed dataset and pasting in an example.
2020-08-28T18:46:51Z
Yes ! tested with authors script/finetuned checkpoints and got similar results…Thanks !
2020-09-12T14:11:49Z
@sshleiferI was trying to convert pegasus TF checkpoint to pytorch usingthisscript in transformers.Can you please help me with below queries:1.While replacing the keys i noticed below replacement which is not matching with BART keys so i have changed it accordingly:Pegasus:decoder/LayerNorm/beta -> decoder.layer_norm.biasdecoder/LayerNorm/gamma -> decoder.layer_norm.weightBART:decoder.layernorm_embedding.weightdecoder.layernorm_embedding.bias2.In below code all pegasus dict_keys should be present in BART keys right ? but as Pegasus has more number of layers compared to BART…so it will never get keys like “decoder.layers.12.final_layer_norm.bias” in BART keysdef convert_pegasus_to_bart(tf_weights: dict, cfg_updates: dict) -> PegasusForConditionalGeneration:
    cfg_kwargs = DEFAULTS.copy()
    cfg_kwargs.update(cfg_updates)
    cfg = PegasusConfig(**cfg_updates)
    bart = PegasusForConditionalGeneration(cfg)
    sd = bart.model.state_dict()
    mapping = {}
    for k, v in tf_weights.items():
        new_k = rename_state_dict_key(k)
        if new_k not in sd:
            raise ValueError(f"could not find new key {new_k} in state dict. (converted from {k})")
        if "dense" in k or "proj" in new_k:
            v = v.T
2020-09-12T17:36:30Z
rename_state_dict_keyshould handle thatcfg.encoder_layerswill be 16 there, sodecoder.layers.12will definitely exist.If the code breaks, pls send a command and traceback.
2020-09-12T18:01:45Z
(update: works for me now)cfg_kwargs.update(cfg_updates)cfg = PegasusConfig(**cfg_updates) -> cfg = PegasusConfig(**cfg_kwargs)Also, even got an error with model name in tokenizer which i later changed from “sshleifer/pegasus” to “google/pegasus-aeslc”command:python convert_pegasus_tf_to_pytorch.py gs://pegasus_ckpt/arxiv/model.ckpt-32000 save_dirThanks !
2020-09-12T20:41:45Z
great catch, PR’d fixhttps://github.com/huggingface/transformers/pull/7094
2020-09-12T20:42:03Z
You may know this but we already havegoogle/pegasus-arxivavailable fwiw
2020-09-12T20:55:47Z
Yes, actually i wanted to add pegasus for qa and paraphrasing
2020-09-15T14:44:06Z
Are there any script available to distill model and how long does it take? I need the small model trained on multi-news, but found nothing.
2020-09-15T15:03:19Z
Pegasus distillation is now supported in the seq2seq examples.https://github.com/huggingface/transformers/tree/master/examples/seq2seqYou’ll need to pass pegaus model path instead of bart and multi-news directory formatted according to the format specified in the readme.
2020-11-03T07:18:38Z
Hi Sshleifer,@sshleiferSorry in advance if this is a silly question. When inferring on pegasus-arxiv, specially on long documents like the Arxiv dataset, can we only infer on the maximum input length (ie: 1024) or the whole document at once. Or do we have to send in the input in batches according to max length of 1024?
No date available
No reply text available