Longformer seemingly initializing global attention mask for every step
image803×431 62.8 KBIncluded an image of the problem. I’m using the LongformerForSequenceClassification model and Trainer to train the model, can’t figure out what is going wrong.  Here’s the model initialization and the training argument, in case that’s relevantmodel = LongformerForSequenceClassification.from_pretrained('allenai/longformer-base-4096', num_labels = 25)

training_args = TrainingArguments(
    output_dir='./drive/MyDrive/checkpoints',          # output directory
    num_train_epochs=3,              # total number of training epochs
    per_device_train_batch_size=4,  # batch size per device during training
    per_device_eval_batch_size=8,   # batch size for evaluation
    warmup_steps=200,                # number of warmup steps for learning rate scheduler
    weight_decay=0.01,               # strength of weight decay
    logging_dir='./drive/MyDrive/logs',            # directory for storing logs
    logging_steps=500,
    save_strategy="epoch",
    fp16 = True
)

trainer = Trainer(
    model=model,                         # the instantiated 🤗 Transformers model to be trained
    args=training_args,                  # training arguments, defined above
    train_dataset=train_dataset,         # training dataset
    eval_dataset=test_dataset,             # evaluation dataset
    compute_metrics=compute_metrics,
)

trainer.train()