
# all-mpnet-base-v2-negation
**This is a fine-tuned [sentence-transformers](https://www.SBERT.net) model to perform better on negated pairs of sentences.**

It maps sentences and paragraphs to a 768 dimensional dense vector space and can be used for tasks like clustering or semantic search.

## Usage (Sentence-Transformers)
Using this model becomes easy when you have [sentence-transformers](https://www.SBERT.net) installed:

```
pip install -U sentence-transformers
```

Then you can use the model like this:
```python
from sentence_transformers import SentenceTransformer

sentences = [
    "I like rainy days because they make me feel relaxed.",
    "I don't like rainy days because they don't make me feel relaxed."
]

model = SentenceTransformer('dmlls/all-mpnet-base-v2-negation')
embeddings = model.encode(sentences)
print(embeddings)
```

## Usage (HuggingFace Transformers)
Without [sentence-transformers](https://www.SBERT.net), you can use the model like this: First, you pass your input through the transformer model, then you have to apply the right pooling-operation on-top of the contextualized word embeddings.

```python
from transformers import AutoTokenizer, AutoModel
import torch
import torch.nn.functional as F

# Mean Pooling - Take attention mask into account for correct averaging
def mean_pooling(model_output, attention_mask):
    token_embeddings = model_output[0] #First element of model_output contains all token embeddings
    input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
    return torch.sum(token_embeddings * input_mask_expanded, 1) / torch.clamp(input_mask_expanded.sum(1), min=1e-9)


# Sentences we want sentence embeddings for
sentences = [
    "I like rainy days because they make me feel relaxed.",
    "I don't like rainy days because they don't make me feel relaxed."
]

# Load model from HuggingFace Hub
tokenizer = AutoTokenizer.from_pretrained('dmlls/all-mpnet-base-v2-negation')
model = AutoModel.from_pretrained('dmlls/all-mpnet-base-v2-negation')

# Tokenize sentences
encoded_input = tokenizer(sentences, padding=True, truncation=True, return_tensors='pt')

# Compute token embeddings
with torch.no_grad():
    model_output = model(**encoded_input)

# Perform pooling
sentence_embeddings = mean_pooling(model_output, encoded_input['attention_mask'])

# Normalize embeddings
sentence_embeddings = F.normalize(sentence_embeddings, p=2, dim=1)

print(sentence_embeddings)
```

------

## Background

This model was finetuned within the context of the [*This is not correct! Negation-aware Evaluation of Language Generation Systems*](https://arxiv.org/abs/2307.13989) paper.


## Intended uses

Our model is intended to be used as a sentence and short paragraph encoder, performing well (i.e., reporting lower similarity scores) on negated pairs of sentences when compared to its base model.

Given an input text, it outputs a vector which captures the semantic information. The sentence vector may be used for information retrieval, clustering or sentence similarity tasks.

By default, input text longer than 384 word pieces is truncated.


## Training procedure

### Pre-training 

We used [`sentence-transformers/all-mpnet-base-v2`](https://huggingface.co/sentence-transformers/all-mpnet-base-v2) as base model.

### Fine-tuning 

We fine-tuned the model on the [CANNOT dataset](https://huggingface.co/datasets/tum-nlp/cannot-dataset) using a contrastive objective. Formally, we compute the cosine similarity from each possible sentence pairs from the batch. We then apply the cross entropy loss by comparing with true pairs.

#### Hyper parameters
We followed an analogous approach to [how other Sentence Transformers were trained](https://github.com/UKPLab/sentence-transformers/blob/3e1929fddef16df94f8bc6e3b10598a98f46e62d/examples/training/nli/training_nli_v2.py). We took the first 90% of samples from the CANNOT dataset as the training split. 
We used a batch size of 64 and trained for 1 epoch.
{"language": "en", "license": "apache-2.0", "tags": ["sentence-transformers", "feature-extraction", "sentence-similarity"], "datasets": ["s2orc", "flax-sentence-embeddings/stackexchange_xml", "ms_marco", "gooaq", "yahoo_answers_topics", "code_search_net", "search_qa", "eli5", "snli", "multi_nli", "wikihow", "natural_questions", "trivia_qa", "embedding-data/sentence-compression", "embedding-data/flickr30k-captions", "embedding-data/altlex", "embedding-data/simple-wiki", "embedding-data/QQP", "embedding-data/SPECTER", "embedding-data/PAQ_pairs", "embedding-data/WikiAnswers", "tum-nlp/cannot-dataset"], "pipeline_tag": "sentence-similarity", "inference": true, "widget": [{"source_sentence": "That is a happy person.", "sentences": ["That is a cheerful person.", "That is not a happy person.", "That is a sad person."], "example_title": "Example 1"}, {"source_sentence": "I like rainy days because they make me feel relaxed.", "sentences": ["I like rainy days because they make me feel chill.", "I don't like rainy days because they don't make me feel relaxed.", "I don't like rainy days because they make me feel stressed out."], "example_title": "Example 2"}, {"source_sentence": "This model should work well with negations.", "sentences": ["This model should work well with negated sentences.", "This model shouldn't work well with negations.", "This model should work terribly with negations."], "example_title": "Example 3"}], "model-index": [{"name": "all-mpnet-base-v2-negation", "results": [{"task": {"type": "Classification"}, "dataset": {"name": "MTEB AmazonCounterfactualClassification (en)", "type": "mteb/amazon_counterfactual", "config": "en", "split": "test", "revision": "e8379541af4e31359cca9fbcf4b00f2671dba205"}, "metrics": [{"type": "accuracy", "value": 72.6268656716418}, {"type": "ap", "value": 36.40585820220466}, {"type": "f1", "value": 67.06383995428979}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB AmazonPolarityClassification", "type": "mteb/amazon_polarity", "config": "default", "split": "test", "revision": "e2d317d38cd51312af73b3d32a06d1a08b442046"}, "metrics": [{"type": "accuracy", "value": 85.11834999999999}, {"type": "ap", "value": 79.72843246428603}, {"type": "f1", "value": 85.08938287851875}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB AmazonReviewsClassification (en)", "type": "mteb/amazon_reviews_multi", "config": "en", "split": "test", "revision": "1399c76144fd37290681b995c656ef9b2e06e26d"}, "metrics": [{"type": "accuracy", "value": 37.788000000000004}, {"type": "f1", "value": 37.40475118737949}]}, {"task": {"type": "Clustering"}, "dataset": {"name": "MTEB ArxivClusteringP2P", "type": "mteb/arxiv-clustering-p2p", "config": "default", "split": "test", "revision": "a122ad7f3f0291bf49cc6f4d32aa80929df69d5d"}, "metrics": [{"type": "v_measure", "value": 45.73138953773995}]}, {"task": {"type": "Clustering"}, "dataset": {"name": "MTEB ArxivClusteringS2S", "type": "mteb/arxiv-clustering-s2s", "config": "default", "split": "test", "revision": "f910caf1a6075f7329cdf8c1a6135696f37dbd53"}, "metrics": [{"type": "v_measure", "value": 39.13609863309245}]}, {"task": {"type": "Reranking"}, "dataset": {"name": "MTEB AskUbuntuDupQuestions", "type": "mteb/askubuntudupquestions-reranking", "config": "default", "split": "test", "revision": "2000358ca161889fa9c082cb41daa8dcfb161a54"}, "metrics": [{"type": "map", "value": 65.56639026991134}, {"type": "mrr", "value": 77.8122938926263}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB BIOSSES", "type": "mteb/biosses-sts", "config": "default", "split": "test", "revision": "d3fb88f8f02e40887cd149695127462bbcf29b4a"}, "metrics": [{"type": "cos_sim_pearson", "value": 72.27098152643569}, {"type": "cos_sim_spearman", "value": 71.13475338373253}, {"type": "euclidean_pearson", "value": 70.48545151074218}, {"type": "euclidean_spearman", "value": 69.49917394727082}, {"type": "manhattan_pearson", "value": 69.2653740752147}, {"type": "manhattan_spearman", "value": 68.59192435931085}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB Banking77Classification", "type": "mteb/banking77", "config": "default", "split": "test", "revision": "0fd18e25b25c072e09e0d92ab615fda904d66300"}, "metrics": [{"type": "accuracy", "value": 84.7012987012987}, {"type": "f1", "value": 84.61766470772943}]}, {"task": {"type": "Clustering"}, "dataset": {"name": "MTEB BiorxivClusteringP2P", "type": "mteb/biorxiv-clustering-p2p", "config": "default", "split": "test", "revision": "65b79d1d13f80053f67aca9498d9402c2d9f1f40"}, "metrics": [{"type": "v_measure", "value": 37.61314886948818}]}, {"task": {"type": "Clustering"}, "dataset": {"name": "MTEB BiorxivClusteringS2S", "type": "mteb/biorxiv-clustering-s2s", "config": "default", "split": "test", "revision": "258694dd0231531bc1fd9de6ceb52a0853c6d908"}, "metrics": [{"type": "v_measure", "value": 34.496442588205205}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB EmotionClassification", "type": "mteb/emotion", "config": "default", "split": "test", "revision": "4f58c6b202a23cf9a4da393831edf4f9183cad37"}, "metrics": [{"type": "accuracy", "value": 45.63}, {"type": "f1", "value": 40.24119129248194}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB ImdbClassification", "type": "mteb/imdb", "config": "default", "split": "test", "revision": "3d86128a09e091d6018b6d26cad27f2739fc2db7"}, "metrics": [{"type": "accuracy", "value": 74.73479999999999}, {"type": "ap", "value": 68.80435332319863}, {"type": "f1", "value": 74.66014345440416}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MTOPDomainClassification (en)", "type": "mteb/mtop_domain", "config": "en", "split": "test", "revision": "d80d48c1eb48d3562165c59d59d0034df9fff0bf"}, "metrics": [{"type": "accuracy", "value": 93.06429548563612}, {"type": "f1", "value": 92.91686969560733}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MTOPIntentClassification (en)", "type": "mteb/mtop_intent", "config": "en", "split": "test", "revision": "ae001d0e6b1228650b7bd1c2c65fb50ad11a8aba"}, "metrics": [{"type": "accuracy", "value": 78.19197446420428}, {"type": "f1", "value": 61.50020940946492}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (en)", "type": "mteb/amazon_massive_intent", "config": "en", "split": "test", "revision": "31efe3c427b0bae9c22cbb560b8f15491cc6bed7"}, "metrics": [{"type": "accuracy", "value": 73.86684599865502}, {"type": "f1", "value": 72.11245795864379}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (en)", "type": "mteb/amazon_massive_scenario", "config": "en", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 77.53866845998655}, {"type": "f1", "value": 77.51746806908895}]}, {"task": {"type": "Clustering"}, "dataset": {"name": "MTEB MedrxivClusteringP2P", "type": "mteb/medrxiv-clustering-p2p", "config": "default", "split": "test", "revision": "e7a26af6f3ae46b30dde8737f02c07b1505bcc73"}, "metrics": [{"type": "v_measure", "value": 33.66744884855605}]}, {"task": {"type": "Clustering"}, "dataset": {"name": "MTEB MedrxivClusteringS2S", "type": "mteb/medrxiv-clustering-s2s", "config": "default", "split": "test", "revision": "35191c8c0dca72d8ff3efcd72aa802307d469663"}, "metrics": [{"type": "v_measure", "value": 31.951900966550262}]}, {"task": {"type": "Reranking"}, "dataset": {"name": "MTEB MindSmallReranking", "type": "mteb/mind_small", "config": "default", "split": "test", "revision": "3bdac13927fdc888b903db93b2ffdbd90b295a69"}, "metrics": [{"type": "map", "value": 29.34485636178124}, {"type": "mrr", "value": 30.118035109577022}]}, {"task": {"type": "Clustering"}, "dataset": {"name": "MTEB RedditClustering", "type": "mteb/reddit-clustering", "config": "default", "split": "test", "revision": "24640382cdbf8abc73003fb0fa6d111a705499eb"}, "metrics": [{"type": "v_measure", "value": 47.14306531904168}]}, {"task": {"type": "Clustering"}, "dataset": {"name": "MTEB RedditClusteringP2P", "type": "mteb/reddit-clustering-p2p", "config": "default", "split": "test", "revision": "282350215ef01743dc01b456c7f5241fa8937f16"}, "metrics": [{"type": "v_measure", "value": 51.59878183893005}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB SICK-R", "type": "mteb/sickr-sts", "config": "default", "split": "test", "revision": "a6ea5a8cab320b040a23452cc28066d9beae2cee"}, "metrics": [{"type": "cos_sim_pearson", "value": 78.5530506834234}, {"type": "cos_sim_spearman", "value": 77.45787185404667}, {"type": "euclidean_pearson", "value": 76.37727601604011}, {"type": "euclidean_spearman", "value": 77.14250754925013}, {"type": "manhattan_pearson", "value": 75.85855462882735}, {"type": "manhattan_spearman", "value": 76.6223895689777}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STS12", "type": "mteb/sts12-sts", "config": "default", "split": "test", "revision": "a0d554a64d88156834ff5ae9920b964011b16384"}, "metrics": [{"type": "cos_sim_pearson", "value": 83.1019526956277}, {"type": "cos_sim_spearman", "value": 72.98362332123834}, {"type": "euclidean_pearson", "value": 78.42992808997602}, {"type": "euclidean_spearman", "value": 70.79569301491145}, {"type": "manhattan_pearson", "value": 77.96413528436207}, {"type": "manhattan_spearman", "value": 70.34707852104586}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STS13", "type": "mteb/sts13-sts", "config": "default", "split": "test", "revision": "7e90230a92c190f1bf69ae9002b8cea547a64cca"}, "metrics": [{"type": "cos_sim_pearson", "value": 85.09200805966644}, {"type": "cos_sim_spearman", "value": 85.52497834636847}, {"type": "euclidean_pearson", "value": 84.20407512505086}, {"type": "euclidean_spearman", "value": 85.35640946044332}, {"type": "manhattan_pearson", "value": 83.79425758102826}, {"type": "manhattan_spearman", "value": 84.9531731481683}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STS14", "type": "mteb/sts14-sts", "config": "default", "split": "test", "revision": "6031580fec1f6af667f0bd2da0a551cf4f0b2375"}, "metrics": [{"type": "cos_sim_pearson", "value": 82.43419245577238}, {"type": "cos_sim_spearman", "value": 79.87215923164575}, {"type": "euclidean_pearson", "value": 80.99628882719712}, {"type": "euclidean_spearman", "value": 79.2671186335978}, {"type": "manhattan_pearson", "value": 80.47076166661054}, {"type": "manhattan_spearman", "value": 78.82329686631051}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STS15", "type": "mteb/sts15-sts", "config": "default", "split": "test", "revision": "ae752c7c21bf194d8b67fd573edf7ae58183cbe3"}, "metrics": [{"type": "cos_sim_pearson", "value": 84.67294508915346}, {"type": "cos_sim_spearman", "value": 85.34528695616378}, {"type": "euclidean_pearson", "value": 83.65270617275111}, {"type": "euclidean_spearman", "value": 84.64456096952591}, {"type": "manhattan_pearson", "value": 83.26416114783083}, {"type": "manhattan_spearman", "value": 84.26944094512996}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STS16", "type": "mteb/sts16-sts", "config": "default", "split": "test", "revision": "4d8694f8f0e0100860b497b999b3dbed754a0513"}, "metrics": [{"type": "cos_sim_pearson", "value": 80.70172607906416}, {"type": "cos_sim_spearman", "value": 81.96031310316046}, {"type": "euclidean_pearson", "value": 82.34820192315314}, {"type": "euclidean_spearman", "value": 82.72576940549405}, {"type": "manhattan_pearson", "value": 81.93093910116202}, {"type": "manhattan_spearman", "value": 82.25431799152639}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STS17 (en-en)", "type": "mteb/sts17-crosslingual-sts", "config": "en-en", "split": "test", "revision": "af5e6fb845001ecf41f4c1e033ce921939a2a68d"}, "metrics": [{"type": "cos_sim_pearson", "value": 90.43640731744911}, {"type": "cos_sim_spearman", "value": 90.16343998541602}, {"type": "euclidean_pearson", "value": 89.49834342254633}, {"type": "euclidean_spearman", "value": 90.17304989919288}, {"type": "manhattan_pearson", "value": 89.32424382015218}, {"type": "manhattan_spearman", "value": 89.91884845996768}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STS22 (en)", "type": "mteb/sts22-crosslingual-sts", "config": "en", "split": "test", "revision": "6d1ba47164174a496b7fa5d3569dae26a6813b80"}, "metrics": [{"type": "cos_sim_pearson", "value": 62.06205206393254}, {"type": "cos_sim_spearman", "value": 60.920792876665885}, {"type": "euclidean_pearson", "value": 60.49188637403393}, {"type": "euclidean_spearman", "value": 60.73500415357452}, {"type": "manhattan_pearson", "value": 59.94692152491976}, {"type": "manhattan_spearman", "value": 60.215426858338994}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STSBenchmark", "type": "mteb/stsbenchmark-sts", "config": "default", "split": "test", "revision": "b0fddb56ed78048fa8b90373c8a3cfc37b684831"}, "metrics": [{"type": "cos_sim_pearson", "value": 84.78948820087687}, {"type": "cos_sim_spearman", "value": 84.64531509697663}, {"type": "euclidean_pearson", "value": 84.77264321816324}, {"type": "euclidean_spearman", "value": 84.67485410196043}, {"type": "manhattan_pearson", "value": 84.43100272264775}, {"type": "manhattan_spearman", "value": 84.29254033404217}]}, {"task": {"type": "Reranking"}, "dataset": {"name": "MTEB SciDocsRR", "type": "mteb/scidocs-reranking", "config": "default", "split": "test", "revision": "d3c5e1fc0b855ab6097bf1cda04dd73947d7caab"}, "metrics": [{"type": "map", "value": 88.39411601972704}, {"type": "mrr", "value": 96.49192583016112}]}, {"task": {"type": "PairClassification"}, "dataset": {"name": "MTEB SprintDuplicateQuestions", "type": "mteb/sprintduplicatequestions-pairclassification", "config": "default", "split": "test", "revision": "d66bd1f72af766a5cc4b0ca5e00c162f89e8cc46"}, "metrics": [{"type": "cos_sim_accuracy", "value": 99.55445544554455}, {"type": "cos_sim_ap", "value": 84.82462858434408}, {"type": "cos_sim_f1", "value": 76.11464968152866}, {"type": "cos_sim_precision", "value": 81.10859728506787}, {"type": "cos_sim_recall", "value": 71.7}, {"type": "dot_accuracy", "value": 99.48613861386139}, {"type": "dot_ap", "value": 80.97278220281665}, {"type": "dot_f1", "value": 72.2914669223394}, {"type": "dot_precision", "value": 69.42909760589319}, {"type": "dot_recall", "value": 75.4}, {"type": "euclidean_accuracy", "value": 99.56138613861386}, {"type": "euclidean_ap", "value": 85.21566333946467}, {"type": "euclidean_f1", "value": 76.60239708181345}, {"type": "euclidean_precision", "value": 79.97823721436343}, {"type": "euclidean_recall", "value": 73.5}, {"type": "manhattan_accuracy", "value": 99.55148514851486}, {"type": "manhattan_ap", "value": 84.49960192851891}, {"type": "manhattan_f1", "value": 75.9681697612732}, {"type": "manhattan_precision", "value": 80.90395480225989}, {"type": "manhattan_recall", "value": 71.6}, {"type": "max_accuracy", "value": 99.56138613861386}, {"type": "max_ap", "value": 85.21566333946467}, {"type": "max_f1", "value": 76.60239708181345}]}, {"task": {"type": "Clustering"}, "dataset": {"name": "MTEB StackExchangeClustering", "type": "mteb/stackexchange-clustering", "config": "default", "split": "test", "revision": "6cbc1f7b2bc0622f2e39d2c77fa502909748c259"}, "metrics": [{"type": "v_measure", "value": 49.33929838947165}]}, {"task": {"type": "Clustering"}, "dataset": {"name": "MTEB StackExchangeClusteringP2P", "type": "mteb/stackexchange-clustering-p2p", "config": "default", "split": "test", "revision": "815ca46b2622cec33ccafc3735d572c266efdb44"}, "metrics": [{"type": "v_measure", "value": 31.523973661953686}]}, {"task": {"type": "Reranking"}, "dataset": {"name": "MTEB StackOverflowDupQuestions", "type": "mteb/stackoverflowdupquestions-reranking", "config": "default", "split": "test", "revision": "e185fbe320c72810689fc5848eb6114e1ef5ec69"}, "metrics": [{"type": "map", "value": 52.22408767861519}, {"type": "mrr", "value": 53.16279921059333}]}, {"task": {"type": "Summarization"}, "dataset": {"name": "MTEB SummEval", "type": "mteb/summeval", "config": "default", "split": "test", "revision": "cda12ad7615edc362dbf25a00fdd61d3b1eaf93c"}, "metrics": [{"type": "cos_sim_pearson", "value": 28.128173244098726}, {"type": "cos_sim_spearman", "value": 30.149225143523662}, {"type": "dot_pearson", "value": 24.322914168643386}, {"type": "dot_spearman", "value": 26.38194545372431}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB ToxicConversationsClassification", "type": "mteb/toxic_conversations_50k", "config": "default", "split": "test", "revision": "d7c0de2777da35d6aae2200a62c6e0e5af397c4c"}, "metrics": [{"type": "accuracy", "value": 67.6684}, {"type": "ap", "value": 12.681984793717413}, {"type": "f1", "value": 51.97637585601529}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB TweetSentimentExtractionClassification", "type": "mteb/tweet_sentiment_extraction", "config": "default", "split": "test", "revision": "d604517c81ca91fe16a244d1248fc021f9ecee7a"}, "metrics": [{"type": "accuracy", "value": 58.44086021505377}, {"type": "f1", "value": 58.68058329615692}]}, {"task": {"type": "Clustering"}, "dataset": {"name": "MTEB TwentyNewsgroupsClustering", "type": "mteb/twentynewsgroups-clustering", "config": "default", "split": "test", "revision": "6125ec4e24fa026cec8a478383ee943acfbd5449"}, "metrics": [{"type": "v_measure", "value": 44.226944341054015}]}, {"task": {"type": "PairClassification"}, "dataset": {"name": "MTEB TwitterSemEval2015", "type": "mteb/twittersemeval2015-pairclassification", "config": "default", "split": "test", "revision": "70970daeab8776df92f5ea462b6173c0b46fd2d1"}, "metrics": [{"type": "cos_sim_accuracy", "value": 86.87488823985218}, {"type": "cos_sim_ap", "value": 76.85283892335002}, {"type": "cos_sim_f1", "value": 70.42042042042041}, {"type": "cos_sim_precision", "value": 66.96811042360781}, {"type": "cos_sim_recall", "value": 74.24802110817942}, {"type": "dot_accuracy", "value": 84.85426476724086}, {"type": "dot_ap", "value": 70.77036812650887}, {"type": "dot_f1", "value": 66.4901577069184}, {"type": "dot_precision", "value": 58.97488258117215}, {"type": "dot_recall", "value": 76.2005277044855}, {"type": "euclidean_accuracy", "value": 86.95833581689217}, {"type": "euclidean_ap", "value": 77.05903224969623}, {"type": "euclidean_f1", "value": 70.75323419175432}, {"type": "euclidean_precision", "value": 65.2979245704084}, {"type": "euclidean_recall", "value": 77.20316622691293}, {"type": "manhattan_accuracy", "value": 86.88084878106932}, {"type": "manhattan_ap", "value": 76.95056209047733}, {"type": "manhattan_f1", "value": 70.61542203843348}, {"type": "manhattan_precision", "value": 65.50090252707581}, {"type": "manhattan_recall", "value": 76.59630606860158}, {"type": "max_accuracy", "value": 86.95833581689217}, {"type": "max_ap", "value": 77.05903224969623}, {"type": "max_f1", "value": 70.75323419175432}]}, {"task": {"type": "PairClassification"}, "dataset": {"name": "MTEB TwitterURLCorpus", "type": "mteb/twitterurlcorpus-pairclassification", "config": "default", "split": "test", "revision": "8b6510b0b1fa4e4c4f879467980e9be563ec1cdf"}, "metrics": [{"type": "cos_sim_accuracy", "value": 88.43870066363954}, {"type": "cos_sim_ap", "value": 84.77197321507954}, {"type": "cos_sim_f1", "value": 76.91440595175472}, {"type": "cos_sim_precision", "value": 75.11375311903713}, {"type": "cos_sim_recall", "value": 78.80351093316908}, {"type": "dot_accuracy", "value": 87.60624054022587}, {"type": "dot_ap", "value": 83.16574114504616}, {"type": "dot_f1", "value": 75.5050226294293}, {"type": "dot_precision", "value": 72.30953555571217}, {"type": "dot_recall", "value": 78.99599630428088}, {"type": "euclidean_accuracy", "value": 88.2951061435169}, {"type": "euclidean_ap", "value": 84.28559058741602}, {"type": "euclidean_f1", "value": 76.7921146953405}, {"type": "euclidean_precision", "value": 74.54334589736156}, {"type": "euclidean_recall", "value": 79.1807822605482}, {"type": "manhattan_accuracy", "value": 88.23883261536074}, {"type": "manhattan_ap", "value": 84.20593815258039}, {"type": "manhattan_f1", "value": 76.74366281685916}, {"type": "manhattan_precision", "value": 74.80263157894737}, {"type": "manhattan_recall", "value": 78.78811210348013}, {"type": "max_accuracy", "value": 88.43870066363954}, {"type": "max_ap", "value": 84.77197321507954}, {"type": "max_f1", "value": 76.91440595175472}]}]}]}
sentence-transformers pytorch mpnet feature-extraction sentence-similarity en dataset:s2orc dataset:flax-sentence-embeddings/stackexchange_xml dataset:ms_marco dataset:gooaq dataset:yahoo_answers_topics dataset:code_search_net dataset:search_qa dataset:eli5 dataset:snli dataset:multi_nli dataset:wikihow dataset:natural_questions dataset:trivia_qa dataset:embedding-data/sentence-compression dataset:embedding-data/flickr30k-captions dataset:embedding-data/altlex dataset:embedding-data/simple-wiki dataset:embedding-data/QQP dataset:embedding-data/SPECTER dataset:embedding-data/PAQ_pairs dataset:embedding-data/WikiAnswers dataset:tum-nlp/cannot-dataset arxiv:2307.13989 license:apache-2.0 model-index endpoints_compatible region:us
TAGS
#sentence-transformers #pytorch #mpnet #feature-extraction #sentence-similarity #en #dataset-s2orc #dataset-flax-sentence-embeddings/stackexchange_xml #dataset-ms_marco #dataset-gooaq #dataset-yahoo_answers_topics #dataset-code_search_net #dataset-search_qa #dataset-eli5 #dataset-snli #dataset-multi_nli #dataset-wikihow #dataset-natural_questions #dataset-trivia_qa #dataset-embedding-data/sentence-compression #dataset-embedding-data/flickr30k-captions #dataset-embedding-data/altlex #dataset-embedding-data/simple-wiki #dataset-embedding-data/QQP #dataset-embedding-data/SPECTER #dataset-embedding-data/PAQ_pairs #dataset-embedding-data/WikiAnswers #dataset-tum-nlp/cannot-dataset #arxiv-2307.13989 #license-apache-2.0 #model-index #endpoints_compatible #region-us 


# all-mpnet-base-v2-negation
This is a fine-tuned sentence-transformers model to perform better on negated pairs of sentences.

It maps sentences and paragraphs to a 768 dimensional dense vector space and can be used for tasks like clustering or semantic search.

## Usage (Sentence-Transformers)
Using this model becomes easy when you have sentence-transformers installed:



Then you can use the model like this:


## Usage (HuggingFace Transformers)
Without sentence-transformers, you can use the model like this: First, you pass your input through the transformer model, then you have to apply the right pooling-operation on-top of the contextualized word embeddings.



------

## Background

This model was finetuned within the context of the *This is not correct! Negation-aware Evaluation of Language Generation Systems* paper.


## Intended uses

Our model is intended to be used as a sentence and short paragraph encoder, performing well (i.e., reporting lower similarity scores) on negated pairs of sentences when compared to its base model.

Given an input text, it outputs a vector which captures the semantic information. The sentence vector may be used for information retrieval, clustering or sentence similarity tasks.

By default, input text longer than 384 word pieces is truncated.


## Training procedure

### Pre-training 

We used 'sentence-transformers/all-mpnet-base-v2' as base model.

### Fine-tuning 

We fine-tuned the model on the CANNOT dataset using a contrastive objective. Formally, we compute the cosine similarity from each possible sentence pairs from the batch. We then apply the cross entropy loss by comparing with true pairs.

#### Hyper parameters
We followed an analogous approach to how other Sentence Transformers were trained. We took the first 90% of samples from the CANNOT dataset as the training split. 
We used a batch size of 64 and trained for 1 epoch.
# all-mpnet-base-v2-negation
This is a fine-tuned sentence-transformers model to perform better on negated pairs of sentences.

It maps sentences and paragraphs to a 768 dimensional dense vector space and can be used for tasks like clustering or semantic search. ## Usage (Sentence-Transformers)
Using this model becomes easy when you have sentence-transformers installed:



Then you can use the model like this: ## Usage (HuggingFace Transformers)
Without sentence-transformers, you can use the model like this: First, you pass your input through the transformer model, then you have to apply the right pooling-operation on-top of the contextualized word embeddings.



------ ## Background

This model was finetuned within the context of the *This is not correct! Negation-aware Evaluation of Language Generation Systems* paper. ## Intended uses

Our model is intended to be used as a sentence and short paragraph encoder, performing well (i.e., reporting lower similarity scores) on negated pairs of sentences when compared to its base model.

Given an input text, it outputs a vector which captures the semantic information. The sentence vector may be used for information retrieval, clustering or sentence similarity tasks.

By default, input text longer than 384 word pieces is truncated. ## Training procedure ### Pre-training 

We used 'sentence-transformers/all-mpnet-base-v2' as base model. ### Fine-tuning 

We fine-tuned the model on the CANNOT dataset using a contrastive objective. Formally, we compute the cosine similarity from each possible sentence pairs from the batch. We then apply the cross entropy loss by comparing with true pairs. #### Hyper parameters
We followed an analogous approach to how other Sentence Transformers were trained. We took the first 90% of samples from the CANNOT dataset as the training split. 
We used a batch size of 64 and trained for 1 epoch.
TAGS
#sentence-transformers #pytorch #mpnet #feature-extraction #sentence-similarity #en #dataset-s2orc #dataset-flax-sentence-embeddings/stackexchange_xml #dataset-ms_marco #dataset-gooaq #dataset-yahoo_answers_topics #dataset-code_search_net #dataset-search_qa #dataset-eli5 #dataset-snli #dataset-multi_nli #dataset-wikihow #dataset-natural_questions #dataset-trivia_qa #dataset-embedding-data/sentence-compression #dataset-embedding-data/flickr30k-captions #dataset-embedding-data/altlex #dataset-embedding-data/simple-wiki #dataset-embedding-data/QQP #dataset-embedding-data/SPECTER #dataset-embedding-data/PAQ_pairs #dataset-embedding-data/WikiAnswers #dataset-tum-nlp/cannot-dataset #arxiv-2307.13989 #license-apache-2.0 #model-index #endpoints_compatible #region-us 
 # all-mpnet-base-v2-negation
This is a fine-tuned sentence-transformers model to perform better on negated pairs of sentences.

It maps sentences and paragraphs to a 768 dimensional dense vector space and can be used for tasks like clustering or semantic search. ## Usage (Sentence-Transformers)
Using this model becomes easy when you have sentence-transformers installed:



Then you can use the model like this: ## Usage (HuggingFace Transformers)
Without sentence-transformers, you can use the model like this: First, you pass your input through the transformer model, then you have to apply the right pooling-operation on-top of the contextualized word embeddings.



------ ## Background

This model was finetuned within the context of the *This is not correct! Negation-aware Evaluation of Language Generation Systems* paper. ## Intended uses

Our model is intended to be used as a sentence and short paragraph encoder, performing well (i.e., reporting lower similarity scores) on negated pairs of sentences when compared to its base model.

Given an input text, it outputs a vector which captures the semantic information. The sentence vector may be used for information retrieval, clustering or sentence similarity tasks.

By default, input text longer than 384 word pieces is truncated. ## Training procedure ### Pre-training 

We used 'sentence-transformers/all-mpnet-base-v2' as base model. ### Fine-tuning 

We fine-tuned the model on the CANNOT dataset using a contrastive objective. Formally, we compute the cosine similarity from each possible sentence pairs from the batch. We then apply the cross entropy loss by comparing with true pairs. #### Hyper parameters
We followed an analogous approach to how other Sentence Transformers were trained. We took the first 90% of samples from the CANNOT dataset as the training split. 
We used a batch size of 64 and trained for 1 epoch.
TAGS
#sentence-transformers #pytorch #mpnet #feature-extraction #sentence-similarity #en #dataset-s2orc #dataset-flax-sentence-embeddings/stackexchange_xml #dataset-ms_marco #dataset-gooaq #dataset-yahoo_answers_topics #dataset-code_search_net #dataset-search_qa #dataset-eli5 #dataset-snli #dataset-multi_nli #dataset-wikihow #dataset-natural_questions #dataset-trivia_qa #dataset-embedding-data/sentence-compression #dataset-embedding-data/flickr30k-captions #dataset-embedding-data/altlex #dataset-embedding-data/simple-wiki #dataset-embedding-data/QQP #dataset-embedding-data/SPECTER #dataset-embedding-data/PAQ_pairs #dataset-embedding-data/WikiAnswers #dataset-tum-nlp/cannot-dataset #arxiv-2307.13989 #license-apache-2.0 #model-index #endpoints_compatible #region-us 
# all-mpnet-base-v2-negation
This is a fine-tuned sentence-transformers model to perform better on negated pairs of sentences.

It maps sentences and paragraphs to a 768 dimensional dense vector space and can be used for tasks like clustering or semantic search.## Usage (Sentence-Transformers)
Using this model becomes easy when you have sentence-transformers installed:



Then you can use the model like this:## Usage (HuggingFace Transformers)
Without sentence-transformers, you can use the model like this: First, you pass your input through the transformer model, then you have to apply the right pooling-operation on-top of the contextualized word embeddings.



------## Background

This model was finetuned within the context of the *This is not correct! Negation-aware Evaluation of Language Generation Systems* paper.## Intended uses

Our model is intended to be used as a sentence and short paragraph encoder, performing well (i.e., reporting lower similarity scores) on negated pairs of sentences when compared to its base model.

Given an input text, it outputs a vector which captures the semantic information. The sentence vector may be used for information retrieval, clustering or sentence similarity tasks.

By default, input text longer than 384 word pieces is truncated.## Training procedure### Pre-training 

We used 'sentence-transformers/all-mpnet-base-v2' as base model.### Fine-tuning 

We fine-tuned the model on the CANNOT dataset using a contrastive objective. Formally, we compute the cosine similarity from each possible sentence pairs from the batch. We then apply the cross entropy loss by comparing with true pairs.#### Hyper parameters
We followed an analogous approach to how other Sentence Transformers were trained. We took the first 90% of samples from the CANNOT dataset as the training split. 
We used a batch size of 64 and trained for 1 epoch.