How do I do multi Class (multi head) classification?
Hi,I want to use BERT models to do multi Class (multi head) classification. I have text and want to do a binary classification for churn and one binary classification for sentiment.Is that possible “out of the box”? Or do I have to develop my own “BertForSequenceClassification” Class?ThanksPhilip
2020-09-16T01:05:03Z
This is all in the loss function, so you can definitely useBertForSequenceClassificationwith two labels, then use the proper loss function (probablyBCEWithLogitsLoss).
2020-09-30T13:05:42Z
Well - this is connected to this question:BertForSequenceClassification only seems to have linear activation at the end - is this a bug?Why is it only a thing of the loss function? IMO the different classification methods need different last layer activation functions. Binary Class needs sigmoid, one of multiple classes needs softmax and multiple of multi class needs sigmoid again. But somehow you always seem to have a linear (no) activation at the end.@sguggerIsn’t this a bug?
2020-09-30T13:45:59Z
In PyTorch, the activation is often combined with the loss for numerical stability and speed. That’s why I say it’s all in the loss.
2020-10-01T08:24:23Z
Ahh ok. Thanks. When I want to calculate the metrics on the predictions in most cases a simplepreds = pred.predictions.argmax(-1)is enough. There is no need to apply softmax or sigmoid infront of it. Only in the following cases I have to apply an “activation function” with numpy:when I have multi label lassification I can not doargmaxwhen I want to compute metrics like ROC-AUC where I need “probability estimates”@sgugger@valhallais that right?
2020-10-01T09:01:04Z
If you have in fact two binary classifications, wouldn’t you be better off using two binary classifiers instead of one with four labels?
2022-10-18T19:11:21Z
Hello all, I need to do classification tasks with BERT but with a single BERT as the feature extractor and multiple classification heads. I guess this is similar to multi-task learning but all the tasks are classification. Can anyone help me with how to do that and if there is a notebook that I can start from?