DistilBertModel to sequence classification
Hi,I’m trying to create aDistilBertModelmodel for sequence classification, such thatmax_position_embeddings=1024(otherwise I would have usedDistilBertForSequenceClassificationwhich is defult tomax_position_embeddings=512)I define the model in the following way:configuration = DistilBertConfig(max_position_embeddings=1024)
 model = DistilBertModel(configuration)When forwarding an input to the model in the following way:output = model(ids, attention_mask = mask, return_dict=False)[0]such thatids.shape = (batch_size, 1024)andmask.shape = (batch_size, 1024)the shape of the output is(batch_size, 1024, 768).My question is: What is the best practice to convert this output into a probability vector over the number of labels such the modified output shape would be(batch_size, num_labels)?I thought of a few options including flattening the current output + an additional FC layer, but I’m not sure this is the best practice.Thank you in advance