
<h1 align="center">MMLW-e5-large</h1>

MMLW (muszę mieć lepszą wiadomość) are neural text encoders for Polish.
This is a distilled model that can be used to generate embeddings applicable to many tasks such as semantic similarity, clustering, information retrieval. The model can also serve as a base for further fine-tuning. 
It transforms texts to 1024 dimensional vectors. 
The model was initialized with multilingual E5 checkpoint, and then trained with [multilingual knowledge distillation method](https://aclanthology.org/2020.emnlp-main.365/) on a diverse corpus of 60 million Polish-English text pairs. We utilised [English FlagEmbeddings (BGE)](https://huggingface.co/BAAI/bge-base-en) as teacher models for distillation. 

## Usage (Sentence-Transformers)

⚠️ Our embedding models require the use of specific prefixes and suffixes when encoding texts. For this model, queries should be prefixed with **"query: "** and passages with **"passage: "** ⚠️

You can use the model like this with [sentence-transformers](https://www.SBERT.net):

```python
from sentence_transformers import SentenceTransformer
from sentence_transformers.util import cos_sim

query_prefix = "query: "
answer_prefix = "passage: "
queries = [query_prefix + "Jak dożyć 100 lat?"]
answers = [
    answer_prefix + "Trzeba zdrowo się odżywiać i uprawiać sport.",
    answer_prefix + "Trzeba pić alkohol, imprezować i jeździć szybkimi autami.",
    answer_prefix + "Gdy trwała kampania politycy zapewniali, że rozprawią się z zakazem niedzielnego handlu."
]
model = SentenceTransformer("sdadas/mmlw-e5-large")
queries_emb = model.encode(queries, convert_to_tensor=True, show_progress_bar=False)
answers_emb = model.encode(answers, convert_to_tensor=True, show_progress_bar=False)

best_answer = cos_sim(queries_emb, answers_emb).argmax().item()
print(answers[best_answer])
# Trzeba zdrowo się odżywiać i uprawiać sport.
```

## Evaluation Results

- The model achieves an **Average Score** of **61.17** on the Polish Massive Text Embedding Benchmark (MTEB). See [MTEB Leaderboard](https://huggingface.co/spaces/mteb/leaderboard) for detailed results.
- The model achieves **NDCG@10** of **56.09** on the Polish Information Retrieval Benchmark. See [PIRB Leaderboard](https://huggingface.co/spaces/sdadas/pirb) for detailed results.

## Acknowledgements
This model was trained with the A100 GPU cluster support delivered by the Gdansk University of Technology within the TASK center initiative.
{"language": "pl", "license": "apache-2.0", "tags": ["sentence-transformers", "feature-extraction", "sentence-similarity", "transformers", "mteb"], "pipeline_tag": "sentence-similarity", "widget": [{"source_sentence": "query: Jak do\u017cy\u0107 100 lat?", "sentences": ["passage: Trzeba zdrowo si\u0119 od\u017cywia\u0107 i uprawia\u0107 sport.", "passage: Trzeba pi\u0107 alkohol, imprezowa\u0107 i je\u017adzi\u0107 szybkimi autami.", "passage: Gdy trwa\u0142a kampania politycy zapewniali, \u017ce rozprawi\u0105 si\u0119 z zakazem niedzielnego handlu."]}], "model-index": [{"name": "mmlw-e5-large", "results": [{"task": {"type": "Clustering"}, "dataset": {"name": "MTEB 8TagsClustering", "type": "PL-MTEB/8tags-clustering", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "v_measure", "value": 30.623921415441725}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB AllegroReviews", "type": "PL-MTEB/allegro-reviews", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "accuracy", "value": 37.683896620278325}, {"type": "f1", "value": 34.19193027014284}]}, {"task": {"type": "Retrieval"}, "dataset": {"name": "MTEB ArguAna-PL", "type": "arguana-pl", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "map_at_1", "value": 38.407000000000004}, {"type": "map_at_10", "value": 55.147}, {"type": "map_at_100", "value": 55.757}, {"type": "map_at_1000", "value": 55.761}, {"type": "map_at_3", "value": 51.268}, {"type": "map_at_5", "value": 53.696999999999996}, {"type": "mrr_at_1", "value": 40.043}, {"type": "mrr_at_10", "value": 55.840999999999994}, {"type": "mrr_at_100", "value": 56.459}, {"type": "mrr_at_1000", "value": 56.462999999999994}, {"type": "mrr_at_3", "value": 52.074}, {"type": "mrr_at_5", "value": 54.364999999999995}, {"type": "ndcg_at_1", "value": 38.407000000000004}, {"type": "ndcg_at_10", "value": 63.248000000000005}, {"type": "ndcg_at_100", "value": 65.717}, {"type": "ndcg_at_1000", "value": 65.79}, {"type": "ndcg_at_3", "value": 55.403999999999996}, {"type": "ndcg_at_5", "value": 59.760000000000005}, {"type": "precision_at_1", "value": 38.407000000000004}, {"type": "precision_at_10", "value": 8.862}, {"type": "precision_at_100", "value": 0.991}, {"type": "precision_at_1000", "value": 0.1}, {"type": "precision_at_3", "value": 22.451}, {"type": "precision_at_5", "value": 15.576}, {"type": "recall_at_1", "value": 38.407000000000004}, {"type": "recall_at_10", "value": 88.62}, {"type": "recall_at_100", "value": 99.075}, {"type": "recall_at_1000", "value": 99.57300000000001}, {"type": "recall_at_3", "value": 67.354}, {"type": "recall_at_5", "value": 77.881}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB CBD", "type": "PL-MTEB/cbd", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "accuracy", "value": 66.14999999999999}, {"type": "ap", "value": 21.69513674684204}, {"type": "f1", "value": 56.48142830893528}]}, {"task": {"type": "PairClassification"}, "dataset": {"name": "MTEB CDSC-E", "type": "PL-MTEB/cdsce-pairclassification", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "cos_sim_accuracy", "value": 89.4}, {"type": "cos_sim_ap", "value": 76.83228768203222}, {"type": "cos_sim_f1", "value": 65.3658536585366}, {"type": "cos_sim_precision", "value": 60.909090909090914}, {"type": "cos_sim_recall", "value": 70.52631578947368}, {"type": "dot_accuracy", "value": 84.1}, {"type": "dot_ap", "value": 57.26072201751864}, {"type": "dot_f1", "value": 62.75395033860045}, {"type": "dot_precision", "value": 54.9407114624506}, {"type": "dot_recall", "value": 73.15789473684211}, {"type": "euclidean_accuracy", "value": 89.4}, {"type": "euclidean_ap", "value": 76.59095263388942}, {"type": "euclidean_f1", "value": 65.21739130434783}, {"type": "euclidean_precision", "value": 60.26785714285714}, {"type": "euclidean_recall", "value": 71.05263157894737}, {"type": "manhattan_accuracy", "value": 89.4}, {"type": "manhattan_ap", "value": 76.58825999753456}, {"type": "manhattan_f1", "value": 64.72019464720195}, {"type": "manhattan_precision", "value": 60.18099547511312}, {"type": "manhattan_recall", "value": 70.0}, {"type": "max_accuracy", "value": 89.4}, {"type": "max_ap", "value": 76.83228768203222}, {"type": "max_f1", "value": 65.3658536585366}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB CDSC-R", "type": "PL-MTEB/cdscr-sts", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "cos_sim_pearson", "value": 93.73949495291659}, {"type": "cos_sim_spearman", "value": 93.50397366192922}, {"type": "euclidean_pearson", "value": 92.47498888987636}, {"type": "euclidean_spearman", "value": 93.39315936230747}, {"type": "manhattan_pearson", "value": 92.47250250777654}, {"type": "manhattan_spearman", "value": 93.36739690549109}]}, {"task": {"type": "Retrieval"}, "dataset": {"name": "MTEB DBPedia-PL", "type": "dbpedia-pl", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "map_at_1", "value": 8.434}, {"type": "map_at_10", "value": 18.424}, {"type": "map_at_100", "value": 26.428}, {"type": "map_at_1000", "value": 28.002}, {"type": "map_at_3", "value": 13.502}, {"type": "map_at_5", "value": 15.577}, {"type": "mrr_at_1", "value": 63.0}, {"type": "mrr_at_10", "value": 72.714}, {"type": "mrr_at_100", "value": 73.021}, {"type": "mrr_at_1000", "value": 73.028}, {"type": "mrr_at_3", "value": 70.75}, {"type": "mrr_at_5", "value": 72.3}, {"type": "ndcg_at_1", "value": 52.75}, {"type": "ndcg_at_10", "value": 39.839999999999996}, {"type": "ndcg_at_100", "value": 44.989000000000004}, {"type": "ndcg_at_1000", "value": 52.532999999999994}, {"type": "ndcg_at_3", "value": 45.198}, {"type": "ndcg_at_5", "value": 42.015}, {"type": "precision_at_1", "value": 63.0}, {"type": "precision_at_10", "value": 31.05}, {"type": "precision_at_100", "value": 10.26}, {"type": "precision_at_1000", "value": 1.9879999999999998}, {"type": "precision_at_3", "value": 48.25}, {"type": "precision_at_5", "value": 40.45}, {"type": "recall_at_1", "value": 8.434}, {"type": "recall_at_10", "value": 24.004}, {"type": "recall_at_100", "value": 51.428}, {"type": "recall_at_1000", "value": 75.712}, {"type": "recall_at_3", "value": 15.015}, {"type": "recall_at_5", "value": 18.282999999999998}]}, {"task": {"type": "Retrieval"}, "dataset": {"name": "MTEB FiQA-PL", "type": "fiqa-pl", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "map_at_1", "value": 19.088}, {"type": "map_at_10", "value": 31.818}, {"type": "map_at_100", "value": 33.689}, {"type": "map_at_1000", "value": 33.86}, {"type": "map_at_3", "value": 27.399}, {"type": "map_at_5", "value": 29.945}, {"type": "mrr_at_1", "value": 38.117000000000004}, {"type": "mrr_at_10", "value": 47.668}, {"type": "mrr_at_100", "value": 48.428}, {"type": "mrr_at_1000", "value": 48.475}, {"type": "mrr_at_3", "value": 45.242}, {"type": "mrr_at_5", "value": 46.716}, {"type": "ndcg_at_1", "value": 38.272}, {"type": "ndcg_at_10", "value": 39.903}, {"type": "ndcg_at_100", "value": 46.661}, {"type": "ndcg_at_1000", "value": 49.625}, {"type": "ndcg_at_3", "value": 35.921}, {"type": "ndcg_at_5", "value": 37.558}, {"type": "precision_at_1", "value": 38.272}, {"type": "precision_at_10", "value": 11.358}, {"type": "precision_at_100", "value": 1.8190000000000002}, {"type": "precision_at_1000", "value": 0.23500000000000001}, {"type": "precision_at_3", "value": 24.434}, {"type": "precision_at_5", "value": 18.395}, {"type": "recall_at_1", "value": 19.088}, {"type": "recall_at_10", "value": 47.355999999999995}, {"type": "recall_at_100", "value": 72.451}, {"type": "recall_at_1000", "value": 90.257}, {"type": "recall_at_3", "value": 32.931}, {"type": "recall_at_5", "value": 39.878}]}, {"task": {"type": "Retrieval"}, "dataset": {"name": "MTEB HotpotQA-PL", "type": "hotpotqa-pl", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "map_at_1", "value": 39.095}, {"type": "map_at_10", "value": 62.529}, {"type": "map_at_100", "value": 63.425}, {"type": "map_at_1000", "value": 63.483000000000004}, {"type": "map_at_3", "value": 58.887}, {"type": "map_at_5", "value": 61.18599999999999}, {"type": "mrr_at_1", "value": 78.123}, {"type": "mrr_at_10", "value": 84.231}, {"type": "mrr_at_100", "value": 84.408}, {"type": "mrr_at_1000", "value": 84.414}, {"type": "mrr_at_3", "value": 83.286}, {"type": "mrr_at_5", "value": 83.94}, {"type": "ndcg_at_1", "value": 78.19}, {"type": "ndcg_at_10", "value": 70.938}, {"type": "ndcg_at_100", "value": 73.992}, {"type": "ndcg_at_1000", "value": 75.1}, {"type": "ndcg_at_3", "value": 65.863}, {"type": "ndcg_at_5", "value": 68.755}, {"type": "precision_at_1", "value": 78.19}, {"type": "precision_at_10", "value": 14.949000000000002}, {"type": "precision_at_100", "value": 1.733}, {"type": "precision_at_1000", "value": 0.188}, {"type": "precision_at_3", "value": 42.381}, {"type": "precision_at_5", "value": 27.711000000000002}, {"type": "recall_at_1", "value": 39.095}, {"type": "recall_at_10", "value": 74.747}, {"type": "recall_at_100", "value": 86.631}, {"type": "recall_at_1000", "value": 93.923}, {"type": "recall_at_3", "value": 63.571999999999996}, {"type": "recall_at_5", "value": 69.27799999999999}]}, {"task": {"type": "Retrieval"}, "dataset": {"name": "MTEB MSMARCO-PL", "type": "msmarco-pl", "config": "default", "split": "validation", "revision": "None"}, "metrics": [{"type": "map_at_1", "value": 19.439999999999998}, {"type": "map_at_10", "value": 30.264000000000003}, {"type": "map_at_100", "value": 31.438}, {"type": "map_at_1000", "value": 31.495}, {"type": "map_at_3", "value": 26.735}, {"type": "map_at_5", "value": 28.716}, {"type": "mrr_at_1", "value": 19.914}, {"type": "mrr_at_10", "value": 30.753999999999998}, {"type": "mrr_at_100", "value": 31.877}, {"type": "mrr_at_1000", "value": 31.929000000000002}, {"type": "mrr_at_3", "value": 27.299}, {"type": "mrr_at_5", "value": 29.254}, {"type": "ndcg_at_1", "value": 20.014000000000003}, {"type": "ndcg_at_10", "value": 36.472}, {"type": "ndcg_at_100", "value": 42.231}, {"type": "ndcg_at_1000", "value": 43.744}, {"type": "ndcg_at_3", "value": 29.268}, {"type": "ndcg_at_5", "value": 32.79}, {"type": "precision_at_1", "value": 20.014000000000003}, {"type": "precision_at_10", "value": 5.814}, {"type": "precision_at_100", "value": 0.8710000000000001}, {"type": "precision_at_1000", "value": 0.1}, {"type": "precision_at_3", "value": 12.426}, {"type": "precision_at_5", "value": 9.238}, {"type": "recall_at_1", "value": 19.439999999999998}, {"type": "recall_at_10", "value": 55.535000000000004}, {"type": "recall_at_100", "value": 82.44399999999999}, {"type": "recall_at_1000", "value": 94.217}, {"type": "recall_at_3", "value": 35.963}, {"type": "recall_at_5", "value": 44.367000000000004}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (pl)", "type": "mteb/amazon_massive_intent", "config": "pl", "split": "test", "revision": "31efe3c427b0bae9c22cbb560b8f15491cc6bed7"}, "metrics": [{"type": "accuracy", "value": 72.01412239408205}, {"type": "f1", "value": 70.04544187503352}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (pl)", "type": "mteb/amazon_massive_scenario", "config": "pl", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 75.26899798251513}, {"type": "f1", "value": 75.55876166863844}]}, {"task": {"type": "Retrieval"}, "dataset": {"name": "MTEB NFCorpus-PL", "type": "nfcorpus-pl", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "map_at_1", "value": 5.772}, {"type": "map_at_10", "value": 12.708}, {"type": "map_at_100", "value": 16.194}, {"type": "map_at_1000", "value": 17.630000000000003}, {"type": "map_at_3", "value": 9.34}, {"type": "map_at_5", "value": 10.741}, {"type": "mrr_at_1", "value": 43.344}, {"type": "mrr_at_10", "value": 53.429}, {"type": "mrr_at_100", "value": 53.88699999999999}, {"type": "mrr_at_1000", "value": 53.925}, {"type": "mrr_at_3", "value": 51.342}, {"type": "mrr_at_5", "value": 52.456}, {"type": "ndcg_at_1", "value": 41.641}, {"type": "ndcg_at_10", "value": 34.028000000000006}, {"type": "ndcg_at_100", "value": 31.613000000000003}, {"type": "ndcg_at_1000", "value": 40.428}, {"type": "ndcg_at_3", "value": 38.991}, {"type": "ndcg_at_5", "value": 36.704}, {"type": "precision_at_1", "value": 43.034}, {"type": "precision_at_10", "value": 25.324999999999996}, {"type": "precision_at_100", "value": 7.889}, {"type": "precision_at_1000", "value": 2.069}, {"type": "precision_at_3", "value": 36.739}, {"type": "precision_at_5", "value": 32.074000000000005}, {"type": "recall_at_1", "value": 5.772}, {"type": "recall_at_10", "value": 16.827}, {"type": "recall_at_100", "value": 32.346000000000004}, {"type": "recall_at_1000", "value": 62.739}, {"type": "recall_at_3", "value": 10.56}, {"type": "recall_at_5", "value": 12.655}]}, {"task": {"type": "Retrieval"}, "dataset": {"name": "MTEB NQ-PL", "type": "nq-pl", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "map_at_1", "value": 26.101000000000003}, {"type": "map_at_10", "value": 39.912}, {"type": "map_at_100", "value": 41.037}, {"type": "map_at_1000", "value": 41.077000000000005}, {"type": "map_at_3", "value": 35.691}, {"type": "map_at_5", "value": 38.155}, {"type": "mrr_at_1", "value": 29.403000000000002}, {"type": "mrr_at_10", "value": 42.376999999999995}, {"type": "mrr_at_100", "value": 43.248999999999995}, {"type": "mrr_at_1000", "value": 43.277}, {"type": "mrr_at_3", "value": 38.794000000000004}, {"type": "mrr_at_5", "value": 40.933}, {"type": "ndcg_at_1", "value": 29.519000000000002}, {"type": "ndcg_at_10", "value": 47.33}, {"type": "ndcg_at_100", "value": 52.171}, {"type": "ndcg_at_1000", "value": 53.125}, {"type": "ndcg_at_3", "value": 39.316}, {"type": "ndcg_at_5", "value": 43.457}, {"type": "precision_at_1", "value": 29.519000000000002}, {"type": "precision_at_10", "value": 8.03}, {"type": "precision_at_100", "value": 1.075}, {"type": "precision_at_1000", "value": 0.117}, {"type": "precision_at_3", "value": 18.009}, {"type": "precision_at_5", "value": 13.221}, {"type": "recall_at_1", "value": 26.101000000000003}, {"type": "recall_at_10", "value": 67.50399999999999}, {"type": "recall_at_100", "value": 88.64699999999999}, {"type": "recall_at_1000", "value": 95.771}, {"type": "recall_at_3", "value": 46.669}, {"type": "recall_at_5", "value": 56.24}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB PAC", "type": "laugustyniak/abusive-clauses-pl", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "accuracy", "value": 63.76773819866782}, {"type": "ap", "value": 74.87896817642536}, {"type": "f1", "value": 61.420506092721425}]}, {"task": {"type": "PairClassification"}, "dataset": {"name": "MTEB PPC", "type": "PL-MTEB/ppc-pairclassification", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "cos_sim_accuracy", "value": 82.1}, {"type": "cos_sim_ap", "value": 91.09417013497443}, {"type": "cos_sim_f1", "value": 84.78437754271766}, {"type": "cos_sim_precision", "value": 83.36}, {"type": "cos_sim_recall", "value": 86.25827814569537}, {"type": "dot_accuracy", "value": 75.9}, {"type": "dot_ap", "value": 86.82680649789796}, {"type": "dot_f1", "value": 80.5379746835443}, {"type": "dot_precision", "value": 77.12121212121212}, {"type": "dot_recall", "value": 84.27152317880795}, {"type": "euclidean_accuracy", "value": 81.6}, {"type": "euclidean_ap", "value": 90.81248760600693}, {"type": "euclidean_f1", "value": 84.35374149659863}, {"type": "euclidean_precision", "value": 86.7132867132867}, {"type": "euclidean_recall", "value": 82.11920529801324}, {"type": "manhattan_accuracy", "value": 81.6}, {"type": "manhattan_ap", "value": 90.81272803548767}, {"type": "manhattan_f1", "value": 84.33530906011855}, {"type": "manhattan_precision", "value": 86.30849220103987}, {"type": "manhattan_recall", "value": 82.45033112582782}, {"type": "max_accuracy", "value": 82.1}, {"type": "max_ap", "value": 91.09417013497443}, {"type": "max_f1", "value": 84.78437754271766}]}, {"task": {"type": "PairClassification"}, "dataset": {"name": "MTEB PSC", "type": "PL-MTEB/psc-pairclassification", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "cos_sim_accuracy", "value": 98.05194805194806}, {"type": "cos_sim_ap", "value": 99.52709687103496}, {"type": "cos_sim_f1", "value": 96.83257918552036}, {"type": "cos_sim_precision", "value": 95.82089552238806}, {"type": "cos_sim_recall", "value": 97.86585365853658}, {"type": "dot_accuracy", "value": 92.30055658627087}, {"type": "dot_ap", "value": 94.12759311032353}, {"type": "dot_f1", "value": 87.00906344410878}, {"type": "dot_precision", "value": 86.22754491017965}, {"type": "dot_recall", "value": 87.8048780487805}, {"type": "euclidean_accuracy", "value": 98.05194805194806}, {"type": "euclidean_ap", "value": 99.49402675624125}, {"type": "euclidean_f1", "value": 96.8133535660091}, {"type": "euclidean_precision", "value": 96.37462235649546}, {"type": "euclidean_recall", "value": 97.2560975609756}, {"type": "manhattan_accuracy", "value": 98.05194805194806}, {"type": "manhattan_ap", "value": 99.50120505935962}, {"type": "manhattan_f1", "value": 96.8133535660091}, {"type": "manhattan_precision", "value": 96.37462235649546}, {"type": "manhattan_recall", "value": 97.2560975609756}, {"type": "max_accuracy", "value": 98.05194805194806}, {"type": "max_ap", "value": 99.52709687103496}, {"type": "max_f1", "value": 96.83257918552036}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB PolEmo2.0-IN", "type": "PL-MTEB/polemo2_in", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "accuracy", "value": 69.45983379501385}, {"type": "f1", "value": 68.60917948426784}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB PolEmo2.0-OUT", "type": "PL-MTEB/polemo2_out", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "accuracy", "value": 43.13765182186235}, {"type": "f1", "value": 36.15557441785656}]}, {"task": {"type": "Retrieval"}, "dataset": {"name": "MTEB Quora-PL", "type": "quora-pl", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "map_at_1", "value": 67.448}, {"type": "map_at_10", "value": 81.566}, {"type": "map_at_100", "value": 82.284}, {"type": "map_at_1000", "value": 82.301}, {"type": "map_at_3", "value": 78.425}, {"type": "map_at_5", "value": 80.43400000000001}, {"type": "mrr_at_1", "value": 77.61}, {"type": "mrr_at_10", "value": 84.467}, {"type": "mrr_at_100", "value": 84.63199999999999}, {"type": "mrr_at_1000", "value": 84.634}, {"type": "mrr_at_3", "value": 83.288}, {"type": "mrr_at_5", "value": 84.095}, {"type": "ndcg_at_1", "value": 77.66}, {"type": "ndcg_at_10", "value": 85.63199999999999}, {"type": "ndcg_at_100", "value": 87.166}, {"type": "ndcg_at_1000", "value": 87.306}, {"type": "ndcg_at_3", "value": 82.32300000000001}, {"type": "ndcg_at_5", "value": 84.22}, {"type": "precision_at_1", "value": 77.66}, {"type": "precision_at_10", "value": 13.136000000000001}, {"type": "precision_at_100", "value": 1.522}, {"type": "precision_at_1000", "value": 0.156}, {"type": "precision_at_3", "value": 36.153}, {"type": "precision_at_5", "value": 23.982}, {"type": "recall_at_1", "value": 67.448}, {"type": "recall_at_10", "value": 93.83200000000001}, {"type": "recall_at_100", "value": 99.212}, {"type": "recall_at_1000", "value": 99.94}, {"type": "recall_at_3", "value": 84.539}, {"type": "recall_at_5", "value": 89.71000000000001}]}, {"task": {"type": "Retrieval"}, "dataset": {"name": "MTEB SCIDOCS-PL", "type": "scidocs-pl", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "map_at_1", "value": 4.393}, {"type": "map_at_10", "value": 11.472}, {"type": "map_at_100", "value": 13.584999999999999}, {"type": "map_at_1000", "value": 13.918}, {"type": "map_at_3", "value": 8.212}, {"type": "map_at_5", "value": 9.864}, {"type": "mrr_at_1", "value": 21.7}, {"type": "mrr_at_10", "value": 32.268}, {"type": "mrr_at_100", "value": 33.495000000000005}, {"type": "mrr_at_1000", "value": 33.548}, {"type": "mrr_at_3", "value": 29.15}, {"type": "mrr_at_5", "value": 30.91}, {"type": "ndcg_at_1", "value": 21.6}, {"type": "ndcg_at_10", "value": 19.126}, {"type": "ndcg_at_100", "value": 27.496}, {"type": "ndcg_at_1000", "value": 33.274}, {"type": "ndcg_at_3", "value": 18.196}, {"type": "ndcg_at_5", "value": 15.945}, {"type": "precision_at_1", "value": 21.6}, {"type": "precision_at_10", "value": 9.94}, {"type": "precision_at_100", "value": 2.1999999999999997}, {"type": "precision_at_1000", "value": 0.359}, {"type": "precision_at_3", "value": 17.2}, {"type": "precision_at_5", "value": 14.12}, {"type": "recall_at_1", "value": 4.393}, {"type": "recall_at_10", "value": 20.166999999999998}, {"type": "recall_at_100", "value": 44.678000000000004}, {"type": "recall_at_1000", "value": 72.868}, {"type": "recall_at_3", "value": 10.473}, {"type": "recall_at_5", "value": 14.313}]}, {"task": {"type": "PairClassification"}, "dataset": {"name": "MTEB SICK-E-PL", "type": "PL-MTEB/sicke-pl-pairclassification", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "cos_sim_accuracy", "value": 82.65389319200979}, {"type": "cos_sim_ap", "value": 76.13749398520014}, {"type": "cos_sim_f1", "value": 66.64355062413314}, {"type": "cos_sim_precision", "value": 64.93243243243244}, {"type": "cos_sim_recall", "value": 68.44729344729345}, {"type": "dot_accuracy", "value": 76.0905014268243}, {"type": "dot_ap", "value": 58.058968583382494}, {"type": "dot_f1", "value": 61.181080324657145}, {"type": "dot_precision", "value": 50.391885661595204}, {"type": "dot_recall", "value": 77.84900284900284}, {"type": "euclidean_accuracy", "value": 82.61312678353036}, {"type": "euclidean_ap", "value": 76.10290283033221}, {"type": "euclidean_f1", "value": 66.50782845473111}, {"type": "euclidean_precision", "value": 63.6897001303781}, {"type": "euclidean_recall", "value": 69.58689458689459}, {"type": "manhattan_accuracy", "value": 82.6742763962495}, {"type": "manhattan_ap", "value": 76.12712309700966}, {"type": "manhattan_f1", "value": 66.59700452803902}, {"type": "manhattan_precision", "value": 65.16700749829583}, {"type": "manhattan_recall", "value": 68.09116809116809}, {"type": "max_accuracy", "value": 82.6742763962495}, {"type": "max_ap", "value": 76.13749398520014}, {"type": "max_f1", "value": 66.64355062413314}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB SICK-R-PL", "type": "PL-MTEB/sickr-pl-sts", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "cos_sim_pearson", "value": 81.23898481255246}, {"type": "cos_sim_spearman", "value": 76.0416957474899}, {"type": "euclidean_pearson", "value": 78.96475496102107}, {"type": "euclidean_spearman", "value": 76.07208683063504}, {"type": "manhattan_pearson", "value": 78.92666424673251}, {"type": "manhattan_spearman", "value": 76.04968227583831}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STS22 (pl)", "type": "mteb/sts22-crosslingual-sts", "config": "pl", "split": "test", "revision": "6d1ba47164174a496b7fa5d3569dae26a6813b80"}, "metrics": [{"type": "cos_sim_pearson", "value": 39.13987124398541}, {"type": "cos_sim_spearman", "value": 40.40194528288759}, {"type": "euclidean_pearson", "value": 29.14566247168167}, {"type": "euclidean_spearman", "value": 39.97389932591777}, {"type": "manhattan_pearson", "value": 29.172993134388935}, {"type": "manhattan_spearman", "value": 39.85681935287037}]}, {"task": {"type": "Retrieval"}, "dataset": {"name": "MTEB SciFact-PL", "type": "scifact-pl", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "map_at_1", "value": 57.260999999999996}, {"type": "map_at_10", "value": 66.92399999999999}, {"type": "map_at_100", "value": 67.443}, {"type": "map_at_1000", "value": 67.47800000000001}, {"type": "map_at_3", "value": 64.859}, {"type": "map_at_5", "value": 65.71900000000001}, {"type": "mrr_at_1", "value": 60.333000000000006}, {"type": "mrr_at_10", "value": 67.95400000000001}, {"type": "mrr_at_100", "value": 68.42}, {"type": "mrr_at_1000", "value": 68.45}, {"type": "mrr_at_3", "value": 66.444}, {"type": "mrr_at_5", "value": 67.128}, {"type": "ndcg_at_1", "value": 60.333000000000006}, {"type": "ndcg_at_10", "value": 71.209}, {"type": "ndcg_at_100", "value": 73.37}, {"type": "ndcg_at_1000", "value": 74.287}, {"type": "ndcg_at_3", "value": 67.66799999999999}, {"type": "ndcg_at_5", "value": 68.644}, {"type": "precision_at_1", "value": 60.333000000000006}, {"type": "precision_at_10", "value": 9.467}, {"type": "precision_at_100", "value": 1.053}, {"type": "precision_at_1000", "value": 0.11299999999999999}, {"type": "precision_at_3", "value": 26.778000000000002}, {"type": "precision_at_5", "value": 16.933}, {"type": "recall_at_1", "value": 57.260999999999996}, {"type": "recall_at_10", "value": 83.256}, {"type": "recall_at_100", "value": 92.767}, {"type": "recall_at_1000", "value": 100.0}, {"type": "recall_at_3", "value": 72.933}, {"type": "recall_at_5", "value": 75.744}]}, {"task": {"type": "Retrieval"}, "dataset": {"name": "MTEB TRECCOVID-PL", "type": "trec-covid-pl", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "map_at_1", "value": 0.22}, {"type": "map_at_10", "value": 1.693}, {"type": "map_at_100", "value": 9.281}, {"type": "map_at_1000", "value": 21.462999999999997}, {"type": "map_at_3", "value": 0.609}, {"type": "map_at_5", "value": 0.9570000000000001}, {"type": "mrr_at_1", "value": 80.0}, {"type": "mrr_at_10", "value": 88.73299999999999}, {"type": "mrr_at_100", "value": 88.73299999999999}, {"type": "mrr_at_1000", "value": 88.73299999999999}, {"type": "mrr_at_3", "value": 88.333}, {"type": "mrr_at_5", "value": 88.73299999999999}, {"type": "ndcg_at_1", "value": 79.0}, {"type": "ndcg_at_10", "value": 71.177}, {"type": "ndcg_at_100", "value": 52.479}, {"type": "ndcg_at_1000", "value": 45.333}, {"type": "ndcg_at_3", "value": 77.48}, {"type": "ndcg_at_5", "value": 76.137}, {"type": "precision_at_1", "value": 82.0}, {"type": "precision_at_10", "value": 74.0}, {"type": "precision_at_100", "value": 53.68000000000001}, {"type": "precision_at_1000", "value": 19.954}, {"type": "precision_at_3", "value": 80.667}, {"type": "precision_at_5", "value": 80.80000000000001}, {"type": "recall_at_1", "value": 0.22}, {"type": "recall_at_10", "value": 1.934}, {"type": "recall_at_100", "value": 12.728}, {"type": "recall_at_1000", "value": 41.869}, {"type": "recall_at_3", "value": 0.637}, {"type": "recall_at_5", "value": 1.042}]}]}]}
sentence-transformers pytorch xlm-roberta feature-extraction sentence-similarity transformers mteb pl license:apache-2.0 model-index endpoints_compatible region:us