
<h1 align="center">MMLW-e5-small</h1>

MMLW (muszę mieć lepszą wiadomość) are neural text encoders for Polish.
This is a distilled model that can be used to generate embeddings applicable to many tasks such as semantic similarity, clustering, information retrieval. The model can also serve as a base for further fine-tuning. 
It transforms texts to 384 dimensional vectors. 
The model was initialized with multilingual E5 checkpoint, and then trained with [multilingual knowledge distillation method](https://aclanthology.org/2020.emnlp-main.365/) on a diverse corpus of 60 million Polish-English text pairs. We utilised [English FlagEmbeddings (BGE)](https://huggingface.co/BAAI/bge-base-en) as teacher models for distillation. 

## Usage (Sentence-Transformers)

⚠️ Our embedding models require the use of specific prefixes and suffixes when encoding texts. For this model, queries should be prefixed with **"query: "** and passages with **"passage: "** ⚠️

You can use the model like this with [sentence-transformers](https://www.SBERT.net):

```python
from sentence_transformers import SentenceTransformer
from sentence_transformers.util import cos_sim

query_prefix = "query: "
answer_prefix = "passage: "
queries = [query_prefix + "Jak dożyć 100 lat?"]
answers = [
    answer_prefix + "Trzeba zdrowo się odżywiać i uprawiać sport.",
    answer_prefix + "Trzeba pić alkohol, imprezować i jeździć szybkimi autami.",
    answer_prefix + "Gdy trwała kampania politycy zapewniali, że rozprawią się z zakazem niedzielnego handlu."
]
model = SentenceTransformer("sdadas/mmlw-e5-small")
queries_emb = model.encode(queries, convert_to_tensor=True, show_progress_bar=False)
answers_emb = model.encode(answers, convert_to_tensor=True, show_progress_bar=False)

best_answer = cos_sim(queries_emb, answers_emb).argmax().item()
print(answers[best_answer])
# Trzeba zdrowo się odżywiać i uprawiać sport.
```

## Evaluation Results

- The model achieves an **Average Score** of **55.84** on the Polish Massive Text Embedding Benchmark (MTEB). See [MTEB Leaderboard](https://huggingface.co/spaces/mteb/leaderboard) for detailed results.
- The model achieves **NDCG@10** of **47.64** on the Polish Information Retrieval Benchmark. See [PIRB Leaderboard](https://huggingface.co/spaces/sdadas/pirb) for detailed results.

## Acknowledgements
This model was trained with the A100 GPU cluster support delivered by the Gdansk University of Technology within the TASK center initiative.
{"language": "pl", "license": "apache-2.0", "tags": ["sentence-transformers", "feature-extraction", "sentence-similarity", "transformers", "mteb"], "pipeline_tag": "sentence-similarity", "widget": [{"source_sentence": "query: Jak do\u017cy\u0107 100 lat?", "sentences": ["passage: Trzeba zdrowo si\u0119 od\u017cywia\u0107 i uprawia\u0107 sport.", "passage: Trzeba pi\u0107 alkohol, imprezowa\u0107 i je\u017adzi\u0107 szybkimi autami.", "passage: Gdy trwa\u0142a kampania politycy zapewniali, \u017ce rozprawi\u0105 si\u0119 z zakazem niedzielnego handlu."]}], "model-index": [{"name": "mmlw-e5-small", "results": [{"task": {"type": "Clustering"}, "dataset": {"name": "MTEB 8TagsClustering", "type": "PL-MTEB/8tags-clustering", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "v_measure", "value": 31.772224277808153}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB AllegroReviews", "type": "PL-MTEB/allegro-reviews", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "accuracy", "value": 33.03180914512922}, {"type": "f1", "value": 29.800304217426167}]}, {"task": {"type": "Retrieval"}, "dataset": {"name": "MTEB ArguAna-PL", "type": "arguana-pl", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "map_at_1", "value": 28.804999999999996}, {"type": "map_at_10", "value": 45.327}, {"type": "map_at_100", "value": 46.17}, {"type": "map_at_1000", "value": 46.177}, {"type": "map_at_3", "value": 40.528999999999996}, {"type": "map_at_5", "value": 43.335}, {"type": "mrr_at_1", "value": 30.299}, {"type": "mrr_at_10", "value": 45.763}, {"type": "mrr_at_100", "value": 46.641}, {"type": "mrr_at_1000", "value": 46.648}, {"type": "mrr_at_3", "value": 41.074}, {"type": "mrr_at_5", "value": 43.836999999999996}, {"type": "ndcg_at_1", "value": 28.804999999999996}, {"type": "ndcg_at_10", "value": 54.308}, {"type": "ndcg_at_100", "value": 57.879000000000005}, {"type": "ndcg_at_1000", "value": 58.048}, {"type": "ndcg_at_3", "value": 44.502}, {"type": "ndcg_at_5", "value": 49.519000000000005}, {"type": "precision_at_1", "value": 28.804999999999996}, {"type": "precision_at_10", "value": 8.286}, {"type": "precision_at_100", "value": 0.984}, {"type": "precision_at_1000", "value": 0.1}, {"type": "precision_at_3", "value": 18.682000000000002}, {"type": "precision_at_5", "value": 13.627}, {"type": "recall_at_1", "value": 28.804999999999996}, {"type": "recall_at_10", "value": 82.85900000000001}, {"type": "recall_at_100", "value": 98.36399999999999}, {"type": "recall_at_1000", "value": 99.644}, {"type": "recall_at_3", "value": 56.04599999999999}, {"type": "recall_at_5", "value": 68.137}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB CBD", "type": "PL-MTEB/cbd", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "accuracy", "value": 64.24}, {"type": "ap", "value": 17.967103105024705}, {"type": "f1", "value": 52.97375416129459}]}, {"task": {"type": "PairClassification"}, "dataset": {"name": "MTEB CDSC-E", "type": "PL-MTEB/cdsce-pairclassification", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "cos_sim_accuracy", "value": 88.8}, {"type": "cos_sim_ap", "value": 76.68028778789487}, {"type": "cos_sim_f1", "value": 66.82352941176471}, {"type": "cos_sim_precision", "value": 60.42553191489362}, {"type": "cos_sim_recall", "value": 74.73684210526315}, {"type": "dot_accuracy", "value": 88.1}, {"type": "dot_ap", "value": 72.04910086070551}, {"type": "dot_f1", "value": 66.66666666666667}, {"type": "dot_precision", "value": 69.31818181818183}, {"type": "dot_recall", "value": 64.21052631578948}, {"type": "euclidean_accuracy", "value": 88.8}, {"type": "euclidean_ap", "value": 76.63591858340688}, {"type": "euclidean_f1", "value": 67.13286713286713}, {"type": "euclidean_precision", "value": 60.25104602510461}, {"type": "euclidean_recall", "value": 75.78947368421053}, {"type": "manhattan_accuracy", "value": 88.9}, {"type": "manhattan_ap", "value": 76.54552849815124}, {"type": "manhattan_f1", "value": 66.66666666666667}, {"type": "manhattan_precision", "value": 60.51502145922747}, {"type": "manhattan_recall", "value": 74.21052631578947}, {"type": "max_accuracy", "value": 88.9}, {"type": "max_ap", "value": 76.68028778789487}, {"type": "max_f1", "value": 67.13286713286713}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB CDSC-R", "type": "PL-MTEB/cdscr-sts", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "cos_sim_pearson", "value": 91.64169404461497}, {"type": "cos_sim_spearman", "value": 91.9755161377078}, {"type": "euclidean_pearson", "value": 90.87481478491249}, {"type": "euclidean_spearman", "value": 91.92362666383987}, {"type": "manhattan_pearson", "value": 90.8415510499638}, {"type": "manhattan_spearman", "value": 91.85927127194698}]}, {"task": {"type": "Retrieval"}, "dataset": {"name": "MTEB DBPedia-PL", "type": "dbpedia-pl", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "map_at_1", "value": 6.148}, {"type": "map_at_10", "value": 12.870999999999999}, {"type": "map_at_100", "value": 18.04}, {"type": "map_at_1000", "value": 19.286}, {"type": "map_at_3", "value": 9.156}, {"type": "map_at_5", "value": 10.857999999999999}, {"type": "mrr_at_1", "value": 53.25}, {"type": "mrr_at_10", "value": 61.016999999999996}, {"type": "mrr_at_100", "value": 61.48400000000001}, {"type": "mrr_at_1000", "value": 61.507999999999996}, {"type": "mrr_at_3", "value": 58.75}, {"type": "mrr_at_5", "value": 60.375}, {"type": "ndcg_at_1", "value": 41.0}, {"type": "ndcg_at_10", "value": 30.281000000000002}, {"type": "ndcg_at_100", "value": 33.955999999999996}, {"type": "ndcg_at_1000", "value": 40.77}, {"type": "ndcg_at_3", "value": 34.127}, {"type": "ndcg_at_5", "value": 32.274}, {"type": "precision_at_1", "value": 52.5}, {"type": "precision_at_10", "value": 24.525}, {"type": "precision_at_100", "value": 8.125}, {"type": "precision_at_1000", "value": 1.728}, {"type": "precision_at_3", "value": 37.083}, {"type": "precision_at_5", "value": 32.15}, {"type": "recall_at_1", "value": 6.148}, {"type": "recall_at_10", "value": 17.866}, {"type": "recall_at_100", "value": 39.213}, {"type": "recall_at_1000", "value": 61.604000000000006}, {"type": "recall_at_3", "value": 10.084}, {"type": "recall_at_5", "value": 13.333999999999998}]}, {"task": {"type": "Retrieval"}, "dataset": {"name": "MTEB FiQA-PL", "type": "fiqa-pl", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "map_at_1", "value": 14.643}, {"type": "map_at_10", "value": 23.166}, {"type": "map_at_100", "value": 24.725}, {"type": "map_at_1000", "value": 24.92}, {"type": "map_at_3", "value": 20.166}, {"type": "map_at_5", "value": 22.003}, {"type": "mrr_at_1", "value": 29.630000000000003}, {"type": "mrr_at_10", "value": 37.632}, {"type": "mrr_at_100", "value": 38.512}, {"type": "mrr_at_1000", "value": 38.578}, {"type": "mrr_at_3", "value": 35.391}, {"type": "mrr_at_5", "value": 36.857}, {"type": "ndcg_at_1", "value": 29.166999999999998}, {"type": "ndcg_at_10", "value": 29.749}, {"type": "ndcg_at_100", "value": 35.983}, {"type": "ndcg_at_1000", "value": 39.817}, {"type": "ndcg_at_3", "value": 26.739}, {"type": "ndcg_at_5", "value": 27.993000000000002}, {"type": "precision_at_1", "value": 29.166999999999998}, {"type": "precision_at_10", "value": 8.333}, {"type": "precision_at_100", "value": 1.448}, {"type": "precision_at_1000", "value": 0.213}, {"type": "precision_at_3", "value": 17.747}, {"type": "precision_at_5", "value": 13.58}, {"type": "recall_at_1", "value": 14.643}, {"type": "recall_at_10", "value": 35.247}, {"type": "recall_at_100", "value": 59.150999999999996}, {"type": "recall_at_1000", "value": 82.565}, {"type": "recall_at_3", "value": 24.006}, {"type": "recall_at_5", "value": 29.383}]}, {"task": {"type": "Retrieval"}, "dataset": {"name": "MTEB HotpotQA-PL", "type": "hotpotqa-pl", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "map_at_1", "value": 32.627}, {"type": "map_at_10", "value": 48.041}, {"type": "map_at_100", "value": 49.008}, {"type": "map_at_1000", "value": 49.092999999999996}, {"type": "map_at_3", "value": 44.774}, {"type": "map_at_5", "value": 46.791}, {"type": "mrr_at_1", "value": 65.28}, {"type": "mrr_at_10", "value": 72.53500000000001}, {"type": "mrr_at_100", "value": 72.892}, {"type": "mrr_at_1000", "value": 72.909}, {"type": "mrr_at_3", "value": 71.083}, {"type": "mrr_at_5", "value": 71.985}, {"type": "ndcg_at_1", "value": 65.253}, {"type": "ndcg_at_10", "value": 57.13700000000001}, {"type": "ndcg_at_100", "value": 60.783}, {"type": "ndcg_at_1000", "value": 62.507000000000005}, {"type": "ndcg_at_3", "value": 52.17}, {"type": "ndcg_at_5", "value": 54.896}, {"type": "precision_at_1", "value": 65.253}, {"type": "precision_at_10", "value": 12.088000000000001}, {"type": "precision_at_100", "value": 1.496}, {"type": "precision_at_1000", "value": 0.172}, {"type": "precision_at_3", "value": 32.96}, {"type": "precision_at_5", "value": 21.931}, {"type": "recall_at_1", "value": 32.627}, {"type": "recall_at_10", "value": 60.439}, {"type": "recall_at_100", "value": 74.80799999999999}, {"type": "recall_at_1000", "value": 86.219}, {"type": "recall_at_3", "value": 49.44}, {"type": "recall_at_5", "value": 54.827999999999996}]}, {"task": {"type": "Retrieval"}, "dataset": {"name": "MTEB MSMARCO-PL", "type": "msmarco-pl", "config": "default", "split": "validation", "revision": "None"}, "metrics": [{"type": "map_at_1", "value": 13.150999999999998}, {"type": "map_at_10", "value": 21.179000000000002}, {"type": "map_at_100", "value": 22.227}, {"type": "map_at_1000", "value": 22.308}, {"type": "map_at_3", "value": 18.473}, {"type": "map_at_5", "value": 19.942999999999998}, {"type": "mrr_at_1", "value": 13.467}, {"type": "mrr_at_10", "value": 21.471}, {"type": "mrr_at_100", "value": 22.509}, {"type": "mrr_at_1000", "value": 22.585}, {"type": "mrr_at_3", "value": 18.789}, {"type": "mrr_at_5", "value": 20.262}, {"type": "ndcg_at_1", "value": 13.539000000000001}, {"type": "ndcg_at_10", "value": 25.942999999999998}, {"type": "ndcg_at_100", "value": 31.386999999999997}, {"type": "ndcg_at_1000", "value": 33.641}, {"type": "ndcg_at_3", "value": 20.368}, {"type": "ndcg_at_5", "value": 23.003999999999998}, {"type": "precision_at_1", "value": 13.539000000000001}, {"type": "precision_at_10", "value": 4.249}, {"type": "precision_at_100", "value": 0.7040000000000001}, {"type": "precision_at_1000", "value": 0.09}, {"type": "precision_at_3", "value": 8.782}, {"type": "precision_at_5", "value": 6.6049999999999995}, {"type": "recall_at_1", "value": 13.150999999999998}, {"type": "recall_at_10", "value": 40.698}, {"type": "recall_at_100", "value": 66.71000000000001}, {"type": "recall_at_1000", "value": 84.491}, {"type": "recall_at_3", "value": 25.452}, {"type": "recall_at_5", "value": 31.791000000000004}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (pl)", "type": "mteb/amazon_massive_intent", "config": "pl", "split": "test", "revision": "31efe3c427b0bae9c22cbb560b8f15491cc6bed7"}, "metrics": [{"type": "accuracy", "value": 67.3537323470074}, {"type": "f1", "value": 64.67852047603644}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (pl)", "type": "mteb/amazon_massive_scenario", "config": "pl", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 72.12508406186953}, {"type": "f1", "value": 71.55887309568853}]}, {"task": {"type": "Retrieval"}, "dataset": {"name": "MTEB NFCorpus-PL", "type": "nfcorpus-pl", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "map_at_1", "value": 4.18}, {"type": "map_at_10", "value": 9.524000000000001}, {"type": "map_at_100", "value": 12.272}, {"type": "map_at_1000", "value": 13.616}, {"type": "map_at_3", "value": 6.717}, {"type": "map_at_5", "value": 8.172}, {"type": "mrr_at_1", "value": 37.152}, {"type": "mrr_at_10", "value": 45.068000000000005}, {"type": "mrr_at_100", "value": 46.026}, {"type": "mrr_at_1000", "value": 46.085}, {"type": "mrr_at_3", "value": 43.344}, {"type": "mrr_at_5", "value": 44.412}, {"type": "ndcg_at_1", "value": 34.52}, {"type": "ndcg_at_10", "value": 27.604}, {"type": "ndcg_at_100", "value": 26.012999999999998}, {"type": "ndcg_at_1000", "value": 35.272}, {"type": "ndcg_at_3", "value": 31.538}, {"type": "ndcg_at_5", "value": 30.165999999999997}, {"type": "precision_at_1", "value": 36.223}, {"type": "precision_at_10", "value": 21.053}, {"type": "precision_at_100", "value": 7.08}, {"type": "precision_at_1000", "value": 1.9929999999999999}, {"type": "precision_at_3", "value": 30.031000000000002}, {"type": "precision_at_5", "value": 26.997}, {"type": "recall_at_1", "value": 4.18}, {"type": "recall_at_10", "value": 12.901000000000002}, {"type": "recall_at_100", "value": 27.438000000000002}, {"type": "recall_at_1000", "value": 60.768}, {"type": "recall_at_3", "value": 7.492}, {"type": "recall_at_5", "value": 10.05}]}, {"task": {"type": "Retrieval"}, "dataset": {"name": "MTEB NQ-PL", "type": "nq-pl", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "map_at_1", "value": 17.965}, {"type": "map_at_10", "value": 28.04}, {"type": "map_at_100", "value": 29.217}, {"type": "map_at_1000", "value": 29.285}, {"type": "map_at_3", "value": 24.818}, {"type": "map_at_5", "value": 26.617}, {"type": "mrr_at_1", "value": 20.22}, {"type": "mrr_at_10", "value": 30.148000000000003}, {"type": "mrr_at_100", "value": 31.137999999999998}, {"type": "mrr_at_1000", "value": 31.19}, {"type": "mrr_at_3", "value": 27.201999999999998}, {"type": "mrr_at_5", "value": 28.884999999999998}, {"type": "ndcg_at_1", "value": 20.365}, {"type": "ndcg_at_10", "value": 33.832}, {"type": "ndcg_at_100", "value": 39.33}, {"type": "ndcg_at_1000", "value": 41.099999999999994}, {"type": "ndcg_at_3", "value": 27.46}, {"type": "ndcg_at_5", "value": 30.584}, {"type": "precision_at_1", "value": 20.365}, {"type": "precision_at_10", "value": 5.849}, {"type": "precision_at_100", "value": 0.8959999999999999}, {"type": "precision_at_1000", "value": 0.107}, {"type": "precision_at_3", "value": 12.64}, {"type": "precision_at_5", "value": 9.334000000000001}, {"type": "recall_at_1", "value": 17.965}, {"type": "recall_at_10", "value": 49.503}, {"type": "recall_at_100", "value": 74.351}, {"type": "recall_at_1000", "value": 87.766}, {"type": "recall_at_3", "value": 32.665}, {"type": "recall_at_5", "value": 39.974}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB PAC", "type": "laugustyniak/abusive-clauses-pl", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "accuracy", "value": 63.11323486823051}, {"type": "ap", "value": 74.53486257377787}, {"type": "f1", "value": 60.631005373417736}]}, {"task": {"type": "PairClassification"}, "dataset": {"name": "MTEB PPC", "type": "PL-MTEB/ppc-pairclassification", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "cos_sim_accuracy", "value": 80.10000000000001}, {"type": "cos_sim_ap", "value": 89.69526236458292}, {"type": "cos_sim_f1", "value": 83.37468982630274}, {"type": "cos_sim_precision", "value": 83.30578512396694}, {"type": "cos_sim_recall", "value": 83.44370860927152}, {"type": "dot_accuracy", "value": 77.8}, {"type": "dot_ap", "value": 87.72366051496104}, {"type": "dot_f1", "value": 82.83752860411899}, {"type": "dot_precision", "value": 76.80339462517681}, {"type": "dot_recall", "value": 89.90066225165563}, {"type": "euclidean_accuracy", "value": 80.10000000000001}, {"type": "euclidean_ap", "value": 89.61317191870039}, {"type": "euclidean_f1", "value": 83.40214698596202}, {"type": "euclidean_precision", "value": 83.19604612850083}, {"type": "euclidean_recall", "value": 83.6092715231788}, {"type": "manhattan_accuracy", "value": 79.60000000000001}, {"type": "manhattan_ap", "value": 89.48363786968471}, {"type": "manhattan_f1", "value": 82.96296296296296}, {"type": "manhattan_precision", "value": 82.48772504091653}, {"type": "manhattan_recall", "value": 83.44370860927152}, {"type": "max_accuracy", "value": 80.10000000000001}, {"type": "max_ap", "value": 89.69526236458292}, {"type": "max_f1", "value": 83.40214698596202}]}, {"task": {"type": "PairClassification"}, "dataset": {"name": "MTEB PSC", "type": "PL-MTEB/psc-pairclassification", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "cos_sim_accuracy", "value": 96.93877551020408}, {"type": "cos_sim_ap", "value": 98.86489482248999}, {"type": "cos_sim_f1", "value": 95.11111111111113}, {"type": "cos_sim_precision", "value": 92.507204610951}, {"type": "cos_sim_recall", "value": 97.86585365853658}, {"type": "dot_accuracy", "value": 95.73283858998145}, {"type": "dot_ap", "value": 97.8261652492545}, {"type": "dot_f1", "value": 93.21533923303835}, {"type": "dot_precision", "value": 90.28571428571428}, {"type": "dot_recall", "value": 96.34146341463415}, {"type": "euclidean_accuracy", "value": 96.93877551020408}, {"type": "euclidean_ap", "value": 98.84837797066623}, {"type": "euclidean_f1", "value": 95.11111111111113}, {"type": "euclidean_precision", "value": 92.507204610951}, {"type": "euclidean_recall", "value": 97.86585365853658}, {"type": "manhattan_accuracy", "value": 96.84601113172542}, {"type": "manhattan_ap", "value": 98.78659090944161}, {"type": "manhattan_f1", "value": 94.9404761904762}, {"type": "manhattan_precision", "value": 92.73255813953489}, {"type": "manhattan_recall", "value": 97.2560975609756}, {"type": "max_accuracy", "value": 96.93877551020408}, {"type": "max_ap", "value": 98.86489482248999}, {"type": "max_f1", "value": 95.11111111111113}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB PolEmo2.0-IN", "type": "PL-MTEB/polemo2_in", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "accuracy", "value": 63.961218836565095}, {"type": "f1", "value": 64.3979989243291}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB PolEmo2.0-OUT", "type": "PL-MTEB/polemo2_out", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "accuracy", "value": 40.32388663967612}, {"type": "f1", "value": 32.339117999015755}]}, {"task": {"type": "Retrieval"}, "dataset": {"name": "MTEB Quora-PL", "type": "quora-pl", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "map_at_1", "value": 62.757}, {"type": "map_at_10", "value": 76.55999999999999}, {"type": "map_at_100", "value": 77.328}, {"type": "map_at_1000", "value": 77.35499999999999}, {"type": "map_at_3", "value": 73.288}, {"type": "map_at_5", "value": 75.25500000000001}, {"type": "mrr_at_1", "value": 72.28}, {"type": "mrr_at_10", "value": 79.879}, {"type": "mrr_at_100", "value": 80.121}, {"type": "mrr_at_1000", "value": 80.12700000000001}, {"type": "mrr_at_3", "value": 78.40700000000001}, {"type": "mrr_at_5", "value": 79.357}, {"type": "ndcg_at_1", "value": 72.33000000000001}, {"type": "ndcg_at_10", "value": 81.151}, {"type": "ndcg_at_100", "value": 83.107}, {"type": "ndcg_at_1000", "value": 83.397}, {"type": "ndcg_at_3", "value": 77.3}, {"type": "ndcg_at_5", "value": 79.307}, {"type": "precision_at_1", "value": 72.33000000000001}, {"type": "precision_at_10", "value": 12.587000000000002}, {"type": "precision_at_100", "value": 1.488}, {"type": "precision_at_1000", "value": 0.155}, {"type": "precision_at_3", "value": 33.943}, {"type": "precision_at_5", "value": 22.61}, {"type": "recall_at_1", "value": 62.757}, {"type": "recall_at_10", "value": 90.616}, {"type": "recall_at_100", "value": 97.905}, {"type": "recall_at_1000", "value": 99.618}, {"type": "recall_at_3", "value": 79.928}, {"type": "recall_at_5", "value": 85.30499999999999}]}, {"task": {"type": "Retrieval"}, "dataset": {"name": "MTEB SCIDOCS-PL", "type": "scidocs-pl", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "map_at_1", "value": 3.313}, {"type": "map_at_10", "value": 8.559999999999999}, {"type": "map_at_100", "value": 10.177999999999999}, {"type": "map_at_1000", "value": 10.459999999999999}, {"type": "map_at_3", "value": 6.094}, {"type": "map_at_5", "value": 7.323}, {"type": "mrr_at_1", "value": 16.3}, {"type": "mrr_at_10", "value": 25.579}, {"type": "mrr_at_100", "value": 26.717000000000002}, {"type": "mrr_at_1000", "value": 26.799}, {"type": "mrr_at_3", "value": 22.583000000000002}, {"type": "mrr_at_5", "value": 24.298000000000002}, {"type": "ndcg_at_1", "value": 16.3}, {"type": "ndcg_at_10", "value": 14.789}, {"type": "ndcg_at_100", "value": 21.731}, {"type": "ndcg_at_1000", "value": 27.261999999999997}, {"type": "ndcg_at_3", "value": 13.74}, {"type": "ndcg_at_5", "value": 12.199}, {"type": "precision_at_1", "value": 16.3}, {"type": "precision_at_10", "value": 7.779999999999999}, {"type": "precision_at_100", "value": 1.79}, {"type": "precision_at_1000", "value": 0.313}, {"type": "precision_at_3", "value": 12.933}, {"type": "precision_at_5", "value": 10.86}, {"type": "recall_at_1", "value": 3.313}, {"type": "recall_at_10", "value": 15.772}, {"type": "recall_at_100", "value": 36.392}, {"type": "recall_at_1000", "value": 63.525}, {"type": "recall_at_3", "value": 7.863}, {"type": "recall_at_5", "value": 11.003}]}, {"task": {"type": "PairClassification"}, "dataset": {"name": "MTEB SICK-E-PL", "type": "PL-MTEB/sicke-pl-pairclassification", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "cos_sim_accuracy", "value": 81.7977986139421}, {"type": "cos_sim_ap", "value": 73.21294750778902}, {"type": "cos_sim_f1", "value": 66.57391304347826}, {"type": "cos_sim_precision", "value": 65.05778382053025}, {"type": "cos_sim_recall", "value": 68.16239316239316}, {"type": "dot_accuracy", "value": 78.67916836526702}, {"type": "dot_ap", "value": 63.61943815978181}, {"type": "dot_f1", "value": 62.45014245014245}, {"type": "dot_precision", "value": 52.04178537511871}, {"type": "dot_recall", "value": 78.06267806267806}, {"type": "euclidean_accuracy", "value": 81.7774154097024}, {"type": "euclidean_ap", "value": 73.25053778387148}, {"type": "euclidean_f1", "value": 66.55064392620953}, {"type": "euclidean_precision", "value": 65.0782845473111}, {"type": "euclidean_recall", "value": 68.09116809116809}, {"type": "manhattan_accuracy", "value": 81.63473298002447}, {"type": "manhattan_ap", "value": 72.99781945530033}, {"type": "manhattan_f1", "value": 66.3623595505618}, {"type": "manhattan_precision", "value": 65.4432132963989}, {"type": "manhattan_recall", "value": 67.3076923076923}, {"type": "max_accuracy", "value": 81.7977986139421}, {"type": "max_ap", "value": 73.25053778387148}, {"type": "max_f1", "value": 66.57391304347826}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB SICK-R-PL", "type": "PL-MTEB/sickr-pl-sts", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "cos_sim_pearson", "value": 79.62332929388755}, {"type": "cos_sim_spearman", "value": 73.70598290849304}, {"type": "euclidean_pearson", "value": 77.3603286710006}, {"type": "euclidean_spearman", "value": 73.74420279933932}, {"type": "manhattan_pearson", "value": 77.12735032552482}, {"type": "manhattan_spearman", "value": 73.53014836690127}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STS22 (pl)", "type": "mteb/sts22-crosslingual-sts", "config": "pl", "split": "test", "revision": "6d1ba47164174a496b7fa5d3569dae26a6813b80"}, "metrics": [{"type": "cos_sim_pearson", "value": 37.696942928686724}, {"type": "cos_sim_spearman", "value": 40.6271445245692}, {"type": "euclidean_pearson", "value": 30.212734461370832}, {"type": "euclidean_spearman", "value": 40.66643376699638}, {"type": "manhattan_pearson", "value": 29.90223716230108}, {"type": "manhattan_spearman", "value": 40.35576319091178}]}, {"task": {"type": "Retrieval"}, "dataset": {"name": "MTEB SciFact-PL", "type": "scifact-pl", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "map_at_1", "value": 43.528}, {"type": "map_at_10", "value": 53.290000000000006}, {"type": "map_at_100", "value": 54.342}, {"type": "map_at_1000", "value": 54.376999999999995}, {"type": "map_at_3", "value": 50.651999999999994}, {"type": "map_at_5", "value": 52.248000000000005}, {"type": "mrr_at_1", "value": 46.666999999999994}, {"type": "mrr_at_10", "value": 55.286}, {"type": "mrr_at_100", "value": 56.094}, {"type": "mrr_at_1000", "value": 56.125}, {"type": "mrr_at_3", "value": 53.222}, {"type": "mrr_at_5", "value": 54.339000000000006}, {"type": "ndcg_at_1", "value": 46.0}, {"type": "ndcg_at_10", "value": 58.142}, {"type": "ndcg_at_100", "value": 62.426}, {"type": "ndcg_at_1000", "value": 63.395999999999994}, {"type": "ndcg_at_3", "value": 53.53}, {"type": "ndcg_at_5", "value": 55.842000000000006}, {"type": "precision_at_1", "value": 46.0}, {"type": "precision_at_10", "value": 7.9670000000000005}, {"type": "precision_at_100", "value": 1.023}, {"type": "precision_at_1000", "value": 0.11100000000000002}, {"type": "precision_at_3", "value": 21.444}, {"type": "precision_at_5", "value": 14.333000000000002}, {"type": "recall_at_1", "value": 43.528}, {"type": "recall_at_10", "value": 71.511}, {"type": "recall_at_100", "value": 89.93299999999999}, {"type": "recall_at_1000", "value": 97.667}, {"type": "recall_at_3", "value": 59.067}, {"type": "recall_at_5", "value": 64.789}]}, {"task": {"type": "Retrieval"}, "dataset": {"name": "MTEB TRECCOVID-PL", "type": "trec-covid-pl", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "map_at_1", "value": 0.22699999999999998}, {"type": "map_at_10", "value": 1.3379999999999999}, {"type": "map_at_100", "value": 6.965000000000001}, {"type": "map_at_1000", "value": 17.135}, {"type": "map_at_3", "value": 0.53}, {"type": "map_at_5", "value": 0.799}, {"type": "mrr_at_1", "value": 84.0}, {"type": "mrr_at_10", "value": 88.083}, {"type": "mrr_at_100", "value": 88.432}, {"type": "mrr_at_1000", "value": 88.432}, {"type": "mrr_at_3", "value": 87.333}, {"type": "mrr_at_5", "value": 87.833}, {"type": "ndcg_at_1", "value": 76.0}, {"type": "ndcg_at_10", "value": 58.199}, {"type": "ndcg_at_100", "value": 43.230000000000004}, {"type": "ndcg_at_1000", "value": 39.751}, {"type": "ndcg_at_3", "value": 63.743}, {"type": "ndcg_at_5", "value": 60.42999999999999}, {"type": "precision_at_1", "value": 84.0}, {"type": "precision_at_10", "value": 62.0}, {"type": "precision_at_100", "value": 44.519999999999996}, {"type": "precision_at_1000", "value": 17.746000000000002}, {"type": "precision_at_3", "value": 67.333}, {"type": "precision_at_5", "value": 63.2}, {"type": "recall_at_1", "value": 0.22699999999999998}, {"type": "recall_at_10", "value": 1.627}, {"type": "recall_at_100", "value": 10.600999999999999}, {"type": "recall_at_1000", "value": 37.532}, {"type": "recall_at_3", "value": 0.547}, {"type": "recall_at_5", "value": 0.864}]}]}]}
sentence-transformers pytorch bert feature-extraction sentence-similarity transformers mteb pl license:apache-2.0 model-index endpoints_compatible region:us