How to train GPT-2 for text summarization?
I have scrapped some data wherein I have some text paragraphs followed by one line summary. I am trying to finetune GPT-2 using this dataset for text summarization. I followed the demo available for text summarization atlink- It works perfectly fine, however, uses T5 model. So, I replaced T5 model and corresponding tokenzier with ‘GPT-2 medium’ model and GPT tokenizer. The data preprocessing code I used is exactly the same as the one given for T5 as in the tutorial referenced above. But, it does not work and throws errors.Below is the code I am using :# define tokenizer
tokenizer = AutoTokenizer.from_pretrained("gpt2-medium", model_max_length = 512)

# define model
# model = AutoModelForSeq2SeqLM.from_pretrained("gpt2-medium")
model = GPT2LMHeadModel.from_pretrained('gpt2-medium', pad_token_id=tokenizer.eos_token_id)

# preprocess input
prefix = "summarize: "
def preprocess_function(examples):
  inputs = [ prefix + doc for doc in examples['abstract'] ]
  model_inputs = tokenizer(inputs, max_length=512)

  labels = tokenizer(text_target=examples['title'], max_length=128)
  model_inputs['labels'] = labels['input_ids']
  
  return model_inputs

tokenized_train = train_ds.map(preprocess_function, batched = True )
tokenized_test = test_ds.map(preprocess_function, batched=True)
data_collator = DataCollatorForSeq2Seq(tokenizer=tokenizer, model=model)
rouge = evaluate.load("rouge")This works fine. But, when I proceed to run the training using below code :training_args = Seq2SeqTrainingArguments(
    output_dir = "textSummary",
    evaluation_strategy="epoch",
    learning_rate=2e-5,
    per_device_train_batch_size=16,
    per_device_eval_batch_size=16,
    weight_decay=0.01,
    save_total_limit=3,
    num_train_epochs=3,
    predict_with_generate=True, 
    fp16=True, 
    #report_to="wandb",
    #run_name="text_summary_gpt2-medium" 
)

trainer = Seq2SeqTrainer(
    model=model,
    args=training_args,
    train_dataset=tokenized_train,
    eval_dataset=tokenized_test,
    tokenizer=tokenizer,
    data_collator=data_collator,
    compute_metrics=compute_metrics,
)

trainer.train()I get an error as follows :ValueError: Asking to pad but the tokenizer does not have a padding token. Please select a token to use aspad_token(tokenizer.pad_token = tokenizer.eos_token e.g.)or add a new pad token viatokenizer.add_special_tokens({'pad_token': '[PAD]'}).How to resolve this error ? Also, if there is any resource, where I can study this in detail, please provide the link to the same.
2023-03-10T14:37:21Z
Not completely sure, but it looks like you need to add a pad token to the tokenizer, something liketokenizer = AutoTokenizer.from_pretrained(“gpt2-medium”, model_max_length = 1024)tokenizer.add_special_tokens({‘pad_token’: ‘[PAD]’})There are probably other changes needed as well to adapt from T5 to GPT-2, will see if I can take a look at this.
2023-03-17T15:16:16Z
there’s a mentioning that you could simply add “TL;DR” at the end of the input text to get summaries. I didn’t try it, but check it outhuggingface.coSummarization - Hugging Face CourseWe’re on a journey to advance and democratize artificial intelligence through open source and open science.Although trained as an auto-regressive language model, you can make GPT-2 generate summaries by appending “TL;DR” at the end of the input text.Please notice that GPT-2 is not encoder-decoder so the architecture is not possibly the best one for generating summaries. And for same reason I think you cannot use Seq2Seq trainer or datacollator for fine tuning.Here’s one article on fine tuning the model:philschmid blog – 6 Sep 20Fine-tune a non-English GPT-2 Model with HuggingfaceFine-tune non-English, German GPT-2 model with Huggingface on German recipes. Using their Trainer class and Pipeline objects.…but for summarization you probably are better of by fine tuning for example “Flan-T5” (such as “google/flan-t5-small”) by google. Good luck
2023-08-01T07:28:15Z
brewmaster321:There are probably other changes needed as well to adapt from T5 to Ghi,@brewmaster321Have you taken a look at it? I am also facing the same issue. I have added the pad_token and am now getting a CUDA out-of-memory error after shrinking my dataset as well. and my model size is not much, though.