Should cls_token be [CLS] or <cls>?
If I want to create a tokenizer from scratch, should the special tokens use square brackets with uppercase letters, or angle brackets with lowercase letters?That is, should cls_token be [CLS] or<cls>?BertTokenizerFastuses the former one:github.comhuggingface/transformers/blob/599db139f921f3af535052c860cb685cadae6fcd/src/transformers/tokenization_bert.py#L547def __init__(self,vocab_file,do_lower_case=True,do_basic_tokenize=True,never_split=None,unk_token="[UNK]",sep_token="[SEP]",pad_token="[PAD]",cls_token="[CLS]",mask_token="[MASK]",tokenize_chinese_chars=True,max_length=None,pad_to_max_length=False,stride=0,truncation_strategy="longest_first",add_special_tokens=True,**kwargs):super(BertTokenizerFast, self).__init__(while many other examples on the Internet uses the latter one.
2023-10-11T08:34:53Z
It could be anything you want it to be since you’re making the tokenizer from scratch.
2023-10-11T08:44:05Z
thanks. Also, isbos_tokenandeos_tokennecessary? In Bert’s implementation, these two tokens are not defined.
2023-10-11T09:23:19Z
It’s not needed for Bert, which does not generate text auto-regressively. So it depends on your application I guess.