Training issue of a Transformer based Encoder-Decoder model based on pre-trained BanglaBERT
I just tried to train an EncoderDeoder model forsummarizationtask based on pre-trainedBanglaBERT, which is anELECTRAdiscriminator model pre-trained with the Replaced Token Detection (RTD) objective. Surprisingly, after spending 4500 steps on 10k training data, the model wasn’t trained a bit since theROUGE-2scores were just0.0000. To make sure I used that 4500-checkpoint to generate summaries for testing purposes; it generated a fixed-length (50) output (even if I change the test input of different lengths) containing the[CLS]token49times and a[SEP]tokenlastly. Basically, I followed theWarm-starting encoder-decoder models withTransformersnotebook. Can anybody give any clue what could be the issue here? Thanks in advance.In my case,TokenizationBanglaBERTmodel:tokenizer = AutoTokenizer.from_pretrained("csebuetnlp/banglabert")
tokenizer.bos_token = tokenizer.cls_token
tokenizer.eos_token = tokenizer.sep_tokenInput pre-processing function:def process_data_to_model_inputs(batch):
    inputs = tokenizer(batch['text'], padding="max_length", truncation=True, max_length=encoder_max_length)
    outputs = tokenizer(batch['summary'], padding="max_length", truncation=True, max_length=decoder_max_length)

    batch["input_ids"] = inputs.input_ids
    batch["attention_mask"] = inputs.attention_mask
    batch["decoder_input_ids"] = outputs.input_ids
    batch["decoder_attention_mask"] = outputs.attention_mask
    batch["labels"] = outputs.input_ids.copy()

    batch["labels"] = [[-100 if token == tokenizer.pad_token_id else token for token in labels] for labels in batch["labels"]]
    return batchMapping the pre-processing function to the batches of examples:train_data = train_data.map(
    process_data_to_model_inputs, 
    batched=True,
    batch_size=batch_size,
    remove_columns=["text", "summary"]
)
train_data.set_format(
    type="torch", columns=["input_ids", "attention_mask", "decoder_input_ids", "decoder_attention_mask", "labels"],
)

valid_data = valid_data.map(
    process_data_to_model_inputs, 
    batched=True, 
    batch_size=batch_size,
    remove_columns=["text", "summary"]
)
valid_data.set_format(
    type="torch", columns=["input_ids", "attention_mask", "decoder_input_ids", "decoder_attention_mask", "labels"],
)BanglaBERTmodel and its config settings:bert2bert = EncoderDecoderModel.from_encoder_decoder_pretrained("csebuetnlp/banglabert", "csebuetnlp/banglabert")
bert2bert.config.decoder_start_token_id = tokenizer.bos_token_id
bert2bert.config.eos_token_id = tokenizer.eos_token_id
bert2bert.config.pad_token_id = tokenizer.pad_token_id

bert2bert.config.vocab_size = bert2bert.config.decoder.vocab_size
bert2bert.config.max_length = 128
bert2bert.config.min_length = 42
bert2bert.config.early_stopping = True
bert2bert.config.length_penalty = 2.0
bert2bert.config.num_beams = 8
bert2bert.config.remove_invalid_values = True
bert2bert.config.repetition_penalty = 2.0
bert2bert.config.length_penalty = 2.0I used theSeq2SeqTrainerfor training. TheSeq2SeqTrainingArgumentswere as follows:evaluation_strategy = "steps",
    per_device_train_batch_size = batch_size,
    per_device_eval_batch_size = batch_size,
    predict_with_generate = True,
    logging_steps = 1000, 
    save_steps = 500, 
    eval_steps = 5000, 
    warmup_steps = 500,
    overwrite_output_dir = True,
    save_total_limit = 2,
    num_train_epochs = 20,
    fp16 = True
2022-05-12T17:23:40Z
see hereencoder-decoder (bert2bert) model for summarization task doesn't work in v4.18 · Issue #292 · huggingface/blog · GitHub