Does AutoTokenizer.from_pretrained add [cls] tokens?
Hello,I am currently working on a classification problem using ProtBERT and I am following theFine-Tuning Tutorial. I have called the tokenised usingtokenizer = AutoTokenizer.from_pretrainedand then tokenised like the tutorial  saystrain_encodings = tokenizer(seq_train, truncation=True, padding=True, 
max_length=1024, return_tensors="pt")Unfortunately, the model doesn’t seem to be learning (I froze the BERT layers).  From reading around, I saw that I need to add the [CLS] token and found such an option usingtokenised.encode(add_special_tokens=True)Yet the tutorial I am following doesn’t seem to require and I was wondering wyy is there a discrepancy and perhaps maybe this is why my model isn’t learning.Thank you
2021-03-01T15:10:43Z
Hi@theudster, I’m pretty sure that ProtBERT has aCLStoken since you can see it in the tokenizer’s special tokens map:from transformers import AutoTokenizer, AutoModelForMaskedLM

tokenizer = AutoTokenizer.from_pretrained("Rostlab/prot_bert")
# returns {'unk_token': '[UNK]', 'sep_token': '[SEP]', 'pad_token': '[PAD]', 'cls_token': '[CLS]', 'mask_token': '[MASK]'}
tokenizer.special_tokens_mapYou can also see it by encoding a text and then decoding it:text = "I love Adelaide!"
# add_special_tokens=True is set by default
text_enc = tokenizer.encode(text)

for tok in text_enc:
    print(tok, tokenizer.decode(tok))You say you froze the BERT layers, so I’m wondering how you’re doing fine-tuning? I’ve sometimes found that the tutorials in the docs aren’t always complete, so for fine-tuning with text classification I would recommend following Sylvain’s tutorial here:https://colab.research.google.com/github/huggingface/notebooks/blob/master/examples/token_classification.ipynb
2021-03-01T15:31:21Z
Thank you@lewtun.So are you saying thattokenizedadds [CLS]?Also, I am freezing the BERT layer but fine-tuning the classification layer. The reason I am doing that is because Colab crashes as it runs out of GPU if I don’t. If I try using just the CPU, I get “Your session crashed after using all available RAM.”I therefore addfor param in model.bert.parameters():
        param.requires_grad = FalseWhich fixes that, but then the model doesn’t seem to learn anything
2021-03-01T20:38:34Z
Yes, the tokenizers intransformersadd the special tokens by default (see the docshere).I’m not familiar with ProtBERT but I’m surprised its crashing Colab because the repo has some Colab examples:ProtTrans/ProtBert-BFD-FineTuning-MS.ipynb at master · agemagician/ProtTrans · GitHubIf you’re still having problems fine-tuning with a GPU, perhaps you can reduce the batch size to avoid the OOM errors
2021-03-01T22:52:45Z
thank you@lewtun. I have gone through their implementation and noticed that their per_device_train_batch_size is 1. Once I changed that, It works and I am actually getting some amazing results (its on its final epoch now, but currently hitting 80% accuracy).I was hoping you could perhaps explain to me what this per_device_train_batch_size does and why was that the issue?
2021-03-02T10:18:50Z
theper_device_train_batch_sizespecifies the batch size for the device you are training on (e.g. GPU/TPU/CPU), so if your training set has 1,000 examples and andper_device_train_batch_size=1then it will take 1,000 steps to complete one epoch.by increasing the value ofper_device_train_batch_sizeyou are able to train faster since it takes less steps to complete each epoch (e.g. ifper_device_train_batch_size=4then we only need 250 steps / epoch in our example), but this can sometimes lead to worse performance since the gradients are averaged / summed in a minibatch.in your case, my guess is that withper_device_train_batch_size=1you need to train for a very long time to see the model learn anything.
2021-03-02T10:32:11Z
well, on Colab, it took a bit over an hour to train 3 epochs, not sure if that is long or not
2021-03-02T12:48:27Z
oh what i meant by “long” is that you may need to run for many epochs before you start seeing any flattening out of you training / validation loss (i.e “convergence”). if you saw the validation loss drop during those 3 epochs then i am not sure what else might have gone wrong in your ProtBERT example