Sentence similarity models not capturing opposite sentences
I have tried different models forsentence similarity, namely:distilbert-base-uncasedbert-base-uncasedsentence-transformers/all-mpnet-base-v2I used them together with the packagessentence-similarityandsentence-transformers, which simplify the programming.I have also tried Universal Sentence Encoders (en_use_mdanden_use_cmlm_lg).However, while these models generally correctly detect similarity for equivalent sentences, they all fail when inputting negated sentences. E.g., theseopposite sentences:“I like rainy days because they make me feel relaxed.”“I don’t like rainy days because they don’t make me feel relaxed.”return asimilarity of 0.993with the modeldistilbert-base-uncased.However, sentences that could be consideredvery similar:“I like rainy days because they make me feel relaxed.”“I enjoy rainy days because they make me feel calm.”return asimilarity of 0.996, which is barely higher.To my understanding, opposite sentences should have a small similarity, especially when semantics are taken into account.My question is: Are there any models/approaches that are able to capture the affirmative/negative nature of sentences when calculating similarity?
2021-09-29T11:31:16Z
This relates back to a discussion that was had on Twitter not too long ago. The problem is that “similarity” is ill-defined. You can read throughthe thread. I did not add much but the discussion between Nils Reimers and Yoav Goldberg is interesting.It is a good mind exercise to think outside of what you’d want it to mean, and what the models are actually paying attention to. In your example, it is likely that content words receive the most attention and are responsible for a lot of the “meaning” (representation vector). On top of that, lexical overlap inevitably contributes to this value as well. That means that “(rainy) days” and “because they make me feel” already overlap. Yes, in terms of semantics and the whole context the meaning is different, but for the model these sentences are very “similar”.These models do not (necessarily) do sentiment analysis and comparisonwhich seems to be what you are after.You may wish  to look for sentiment modelsinstead.
2021-09-29T12:09:48Z
Thank you for your answer. The Twitter thread was very interesting.Actually, I’m not really after sentiment analysis; the models also fail with factual sentences with opposite meanings. What I would like to achieve is inferring when two sentences are opposite and assign them a low similarity score (again, this is within the discussion of what “similarity” means, since someone could argue that two sentences that only differ in a “not” or a “don’t” are still very similar).
2022-09-06T21:08:46Z
Hey dmlls,I’m also interested in similar problem and trying to find a way to distinguish between two opposite sentences. By far, I have created a fine-tuned sentence bert model that have shown improvement but yet far from what we want. I would like to know, have you made any progress in this area or do you have any suggestions that I can try to tackle this issue.Thank you.
2022-09-07T12:40:36Z
Hi@yzm0034,I thought of some obvious but simple approaches such as writing a simple regex-based, sentence negation algorithm in order to bootstrap a labeled dataset.This algorithm would look at sentences containing “easy-to-negate” words/spans, such as modal verbs (can↔can’t/cannot,will↔won’t/will not,should↔shouldn’t/should not, etc.) or auxiliary verbs (is↔isn’t / is not, etc.) and replace them by their opposite counterpart.This idea could be taken further by performing POS tagging, finding the verb (e.g.,like) and negating it (don't/doesn't like).Obviously, things are not always that easy. For example, applying the previous approach to:There are some issueswould result inThere are not some issues, which sounds odd (one would sayThere are not any issuesor simplyThere are no issues). Additional regular expressions could be written for these cases.However, it’s probably not necessary that the generated dataset covers every single case in order to improve over the current baseline
2023-02-13T00:29:47Z
You can try a sentiment classifier on top of a semantic similarity model. Semantically similar but opposite in sentiment. This is could be an Idea.
2023-09-01T19:18:26Z
This is one of the topics which I am interested in. I would classify this topic as “negation” contextual understanding. We might not have anything in natural language domain other than some datasets in medical domain. let me know if anyone finds one
2023-09-29T15:11:53Z
Hi@hashemi786, we recently published a paper digging into the topic:This is not correct! Negation-aware Evaluation of Language Generation Systems.The paper introduces theCANNOTdataset, which focuses on negated textual pairs. It currently contains 77,376 samples, of which roughly of them are negated pairs of sentences, and the other half are not (they are paraphrased versions of each other).We also released the modelNegBLEURT, finetuned on the CANNOT dataset, making it significantly more sensitive to negations than its base model. Additionally, I also finetuned a Sentence Transformer:all-mpnet-base-v2-negation(it can be used with thesentence-transformersmodule). Again, it deals much better with negations.
2023-10-03T04:26:17Z
Thanks for your prompt response, I was able to try NegBLEURT, NegMPNet, clinical-assertion-negation-bert.I was not able to try outall-mpnet-base-v2-negationas the config file/actual model is missing.
2023-10-04T15:28:55Z
Apologies, apparently I never got to upload the model…You can check again atdmlls/all-mpnet-base-v2-negation. I also added a few examples to the widget so that anyone can play with the model directly from the model page. I recommend comparing scores with its base model (sentence-transformers/all-mpnet-base-v2) to verify that the finetuning did work.