Multi-label token classification: "-100" special label
Hello!I am implementing a HF-based model augmented with native Pytorch-code to classify tokens (not the document!) into one or more classes.Now, based on single labels and using the AutoTokenizer and after aligning subwords with their labels, I get the following output, where the labels -100 correspond to the [CLS] and [SEP] special tokens as well as subwords starting with ## (not seen below). Apparently, the -100 labels tell the model to ignore the token when calculating loss.{‘input_ids’: [2,4759,4683,3],‘token_type_ids’: [0,0,0,0],‘attention_mask’: [1,1,1,1],‘labels’: [-100,3,4,-100]}However, with multi-label classification, my labels are lists of one-hot encodings as required for calculating nn.CEWithLogitsLoss():‘labels’: [XXX,[1, 0, 1],[1, 1, 0],[1, 0, 0],XXX]]What should I put here for XXX instead of the “-100” special label to tell the model to ignore special tokens as well as subword tokens?
2023-09-18T10:30:28Z
Perhaps you can try to create a vector consisting of -100 values?So instead of XXX you can try [-100 -100 -100]Let me know if you get some results.