How calculate loss.backward() in MLM by PyTorch
I am going to train an MLM model by pytorch, but in the training part, I do not know how to calculate theloss.backward.model_name = "distilbert-base-uncased"

model = AutoModelForMaskedLM.from_pretrained(model_name, max_length=256,  return_dict=True)

model.train()
device = torch.device("cpu")

for epoch in range(epochs):
    loop = tqdm(dataloader)
    for batch in loop:
        optimizer.zero_grad()
        
        input_ids = batch['input_ids'].to(device)
        labels = batch['labels'].to(device)
        attention_mask = batch['attention_mask'].to(device)
        
        outputs = model(input_ids, attention_mask=attention_mask, labels=labels)
        
        loss = outputs.loss
        loss.backwards()
        optimizer.step()@lewtun
2022-01-21T14:29:04Z
Hey@MahdiAwhat you have looks pretty close - I think you just needloss.backward()instead ofloss.backwards()Does that solve your issue?