Info regarding sentence-transformers
sentence-transformerswere recently added in the model hub. I wanted to know the following if :all these models were trained on NLI (which dataset ? MNLI)for how many epochs ?If I want to use it for inference on NLI task, do I need to train an additional linear layer ? So freeze theAutoModel, train ann.Linearfor 1 epoch on themeanoutput and then perform inference. Can you please confirm this usage ?tfmr = AutoModel.from_pretrained("sentence-transformers/bert-base-nli-mean-tokens")

class Model(nn.Module):
    def __init__(self, model):
        super(model, self).__init__()
        self.encoder = model
        for param in self.encoder.parameters():
            param.requires_grad = False
        self.classifier = nn.Linear(768, 3)
        self.criterion = nn.CrossEntropy()

    def forward(self, inputs): # x will come from dataloader using default_data_collator

        labels = inputs.pop('labels')
        model_output = self.encoder(**inputs)
        sentence_embeddings = mean_pooling(model_output,  inputs['attention_mask'])
        logits = self.classifier(sentence_embeddings)
        loss = self.criterion(logits, labels)
        return loss, logitsmodel = Model(tfmr)
2020-08-21T13:20:00Z
I trained it on MNLI with the model frozen, it gets around 46% accuracy after epoch 1. Seems like it has not been trained on MNLI. Can anyone confirm ?
2020-08-21T15:57:35Z
from the paper, section 3.1We train SBERT on the combination of the SNLI(Bowman et al., 2015) and the Multi-Genre NLI dataset.So yes, it’s trained on MNLI .My guess is as it’s trained using Triplet loss to produce embeddings that are semantically meaningfuland can be compared with cosine-similarity, it might nor perform well on just classification .
2020-08-22T02:43:01Z
Thanks for the information. When I read the paper, I initially had the impression that it was trained on STS task. Given that it has been trained on both SNLI and MNLI (not sure about the percentage), I’m getting 46% accuracy when I train a MLP for one epoch. I know the objective is a bit different (not classification) but I’d expect it to do well (in frozen condition) given that it has seen both NLI datasets. Things don’t improve much even after 3 epochs.
2020-08-23T11:18:40Z
On what objective is this model trained on ? Section 3 of the paper mentions three objective functionsclassification,regressionandtriplet. As per Fig 2. they seem to be using regression for computing similarity scores at inference. As per Fig 1. it says that they usedclassificationobjective for fine-tuning. Based on the performance I’m getting, I doubt that these weights are coming from model trained onclassification. Can someone please clarify ?
2020-08-23T16:57:08Z
I have the same confusion, it’s not clear what was the final objective from those 3
2020-08-25T06:16:06Z
@joeddavIf you’ve some idea, can you please confirm this ?
2020-08-25T14:28:16Z
I’m afraid I don’t have any extra insight here. I might head over to thesentence-transformersrepo and see if you can find an answer or open an issue there (and then loop us back in over here once you have an answer)
2020-08-26T04:21:21Z
I’ve opened anissue. You can follow it if interested.