Interesting (but puzzling) cosine-similarity comparison with distilbert
Consider the simple example below.I consider three simple sentences, find their embeddings and and compute their cosine-similarity with each other.I am puzzled by the results. I would have expectedthis is a catto be very similar tothis is a dogand boththis is a catandthis is a dogto be dissimilar tothis is a banana.However, taken at face value,this is a bananais more similar tothis is a dogthan the two animals sentences together…Is this expected?! What do you think?import tensorflow as tf
from transformers import pipeline
from numpy import dot
from numpy.linalg import norm

def mycos(x,y):
    return dot(x, y)/(norm(x)*norm(y))

mypipe = pipeline('feature-extraction', 'distilbert-base-uncased-finetuned-sst-2-english')

one = mypipe('this is a cat')[0][0]
two = mypipe('this is a dog')[0][0]
three = mypipe('this is a banana')[0][0]

mycos(one,two)
Out[55]: 0.5795413454711928

mycos(one,three)
Out[56]: 0.19475422728604236

mycos(two,three)
Out[57]: 0.5881860164213862Thanks!