BERT next sentence prediction: bert-base always returns false
I am experimenting with the variousBertFor*model classes with pre-trained modelsUsingbert-base-uncasedwithBertForNextSentencePredictionI have the following code which seems to work nicely:from transformers import BertTokenizer, BertForNextSentencePrediction
import torch

# Load pre-trained tokenizer and model
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = BertForNextSentencePrediction.from_pretrained('bert-base-uncased').eval()

# Tokenize input sentences
sentence_a = "The cat sat on the mat."
sentence_b = "It was a nice day outside."
encoded = tokenizer.encode_plus(sentence_a, sentence_b, return_tensors='pt')

# Get model predictions
with torch.no_grad():
    outputs = model(**encoded)
logits = outputs[0]
probs = torch.softmax(logits, dim=1)
is_next_sentence = torch.argmax(probs, dim=1).bool().item()is_next_sentenceis eitherTrueorFalseThe only problem is that I can’t seem to find any pair of sentences which returnTrue.(Does anybody have any examples? Is the code above wrong?)I am guessing that maybe the base model is just not able to do this task and I need to use a fine-tuned variant?Does anyone have abert-basevariant on HF Hub that is fine-tuned on NSP?
2023-04-26T16:11:12Z
Or… training my own would be a good exercise I guessCan anyone recommend an HF dataset for NSP training?
2023-04-26T16:15:53Z
I am starting to think maybe no one actually has a use for NSP inference… is this just a pre-training technique?
2023-04-26T16:44:41Z
Turns out the code above is wrong somehowWith help fromhappy-transformer/happy_next_sentence.py at master · EricFillion/happy-transformer · GitHubI found working code:encoded = tokenizer(sentence_a, sentence_b, return_tensors='pt')

with torch.no_grad():
    outputs = model(**encoded)

probabilities = torch.softmax(outputs[0], dim=1)
prediction = torch.argmax(probabilities)
# 0 = yes, 1 = no
is_next_sentence = prediction.item() == 0The main difference I see in this code is usingtokenizer(...)rather thantokenizer.encode_plus