BertForSequenceClassification only seems to have linear activation at the end - is this a bug?
Hi,theBertForSequenceClassificationclass seems to have a linear activation at the end of the head.See here:https://github.com/huggingface/transformers/blob/3323146e904a1092def4e8527de9d2a7479c1c14/src/transformers/modeling_bert.py#L1351IMO for a binary classification it should have a sigmoid function at the end and for a one of multiple classes classification there should be a softmax at the end. How does this come?ThanksPhilip
2020-09-30T14:54:42Z
Hi@PhilipMayThis is not a bug, At L1351 the pooled output is passed through classification head (linear layer) to get thelogitsCrossEntropyLossdoes not requiresoftmax, it calculates loss usinglogitsBertForSequenceClassificationreturnslogits, then you can the applysoftmaxon the returnedlogitsto get the class scores.