Continue pre-training BERT
Hello, I have a small portion of label data, and a much bigger set of unlabeled observations. I want to use the unlabeled samples in order to continue the pre-training of BERT, and then built a classifier on top of it.Following this postContinue pre-training of Greek BERT with domain specific dataset [clarified]BeginnersHello, 
I want to further pre-train Greek BERT of the library on a domain specific dataset that is not annotated in MLM task. The downstream task of BERT is Name Entity Recognition where I have a second annotated dataset to subsequently fine tune the model. I found that the library providesscriptsfor that also. In the example RoBERTa is further trained on wikitext-2-raw-v1. As I sawhere, the dataset is formatted as: 
{ 
“text”: “” The gold dollar or gold one @-@ dollar … 
} 
although, I down…I tried to use BertModel.from_pretrained(‘bert-base-uncased’), and specificallymodel = BertModel.from_pretrained(HF_BERT_MODEL)
    model.cuda()

    optimizer = AdamW(model.parameters(),
                  lr = 2e-5, 
                  eps = 1e-8 
                )
    
    model.train()

    # For each batch of training data...
    for step, batch in enumerate(train_dataloader):

        b_input_ids = batch[0].to(device)
        b_input_mask = batch[1].to(device)
        b_labels = batch[2].to(device)

        model.zero_grad()

        result = model(b_input_ids,
                       token_type_ids=None,
                       attention_mask=b_input_mask,
                       return_dict=True)

        loss = result.loss
        torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
        optimizer.step()
        scheduler.step()

>>>'BaseModelOutputWithPoolingAndCrossAttentions' object has no attribute 'loss'I get the above error.My question is, how to do the fine-tuning?
2023-11-13T12:16:52Z
Loss is available only when labels are provided to the model.
2023-11-13T13:24:44Z
Thanks for your answer. The question is, though, how can we train the embeddings (using gradient descent) if there is no loss?
2023-11-13T14:13:45Z
I don’t understand the question. Don’t you need to provide labels in order to calculate a loss? You either provide labels to the model so it can calculate the loss or you calculate loss yourself using the output of the model.So in your code above, for training I would expect to pass in the labels to model, the. get the loss as you are expecting and then  callloss.backward
2023-11-13T14:42:24Z
The idea was to continue the pre-training, which, according to BERT is masking words (around 15% of the provided texts) and trying to predict the masked tokens. To the best of my knowledge, this process is considered to be “self-supervised” and therefore you don’t implicitly provide labels, but instead they are inferred from the data. In this case, you still have a loss (otherwise how could you learn the embeddings).From your question though, I understand that I might need to mask tokens myself and add the masked token as the label. Am I correct? Care you to refer me to a notebook that shows an example?
2023-11-13T18:34:14Z
Medium – 2 Sep 21Masked-Language Modelling With BERTTransformer models like BERT are incredibly powerful when fine-tuned for specific tasks using masked-language modeling (MLM), we explore how here.Reading time: 7 min readOr you can use the trainer class from huggingface as described in their guidehuggingface.coMasked language modelingWe’re on a journey to advance and democratize artificial intelligence through open source and open science.