
<h1 align="center">MMLW-roberta-large</h1>

MMLW (muszę mieć lepszą wiadomość) are neural text encoders for Polish.
This is a distilled model that can be used to generate embeddings applicable to many tasks such as semantic similarity, clustering, information retrieval. The model can also serve as a base for further fine-tuning. 
It transforms texts to 1024 dimensional vectors. 
The model was initialized with Polish RoBERTa checkpoint, and then trained with [multilingual knowledge distillation method](https://aclanthology.org/2020.emnlp-main.365/) on a diverse corpus of 60 million Polish-English text pairs. We utilised [English FlagEmbeddings (BGE)](https://huggingface.co/BAAI/bge-base-en) as teacher models for distillation. 

## Usage (Sentence-Transformers)

⚠️ Our embedding models require the use of specific prefixes and suffixes when encoding texts. For this model, each query should be preceded by the prefix **"zapytanie: "** ⚠️

You can use the model like this with [sentence-transformers](https://www.SBERT.net):

```python
from sentence_transformers import SentenceTransformer
from sentence_transformers.util import cos_sim

query_prefix = "zapytanie: "
answer_prefix = ""
queries = [query_prefix + "Jak dożyć 100 lat?"]
answers = [
    answer_prefix + "Trzeba zdrowo się odżywiać i uprawiać sport.",
    answer_prefix + "Trzeba pić alkohol, imprezować i jeździć szybkimi autami.",
    answer_prefix + "Gdy trwała kampania politycy zapewniali, że rozprawią się z zakazem niedzielnego handlu."
]
model = SentenceTransformer("sdadas/mmlw-roberta-large")
queries_emb = model.encode(queries, convert_to_tensor=True, show_progress_bar=False)
answers_emb = model.encode(answers, convert_to_tensor=True, show_progress_bar=False)

best_answer = cos_sim(queries_emb, answers_emb).argmax().item()
print(answers[best_answer])
# Trzeba zdrowo się odżywiać i uprawiać sport.
```

## Evaluation Results

- The model achieves an **Average Score** of **63.23** on the Polish Massive Text Embedding Benchmark (MTEB). See [MTEB Leaderboard](https://huggingface.co/spaces/mteb/leaderboard) for detailed results.
- The model achieves **NDCG@10** of **55.95** on the Polish Information Retrieval Benchmark. See [PIRB Leaderboard](https://huggingface.co/spaces/sdadas/pirb) for detailed results.

## Acknowledgements
This model was trained with the A100 GPU cluster support delivered by the Gdansk University of Technology within the TASK center initiative.
{"language": "pl", "license": "apache-2.0", "tags": ["sentence-transformers", "feature-extraction", "sentence-similarity", "transformers", "mteb"], "pipeline_tag": "sentence-similarity", "widget": [{"source_sentence": "zapytanie: Jak do\u017cy\u0107 100 lat?", "sentences": ["Trzeba zdrowo si\u0119 od\u017cywia\u0107 i uprawia\u0107 sport.", "Trzeba pi\u0107 alkohol, imprezowa\u0107 i je\u017adzi\u0107 szybkimi autami.", "Gdy trwa\u0142a kampania politycy zapewniali, \u017ce rozprawi\u0105 si\u0119 z zakazem niedzielnego handlu."]}], "model-index": [{"name": "mmlw-roberta-large", "results": [{"task": {"type": "Clustering"}, "dataset": {"name": "MTEB 8TagsClustering", "type": "PL-MTEB/8tags-clustering", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "v_measure", "value": 31.16472823814849}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB AllegroReviews", "type": "PL-MTEB/allegro-reviews", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "accuracy", "value": 47.48508946322067}, {"type": "f1", "value": 42.33327527584009}]}, {"task": {"type": "Retrieval"}, "dataset": {"name": "MTEB ArguAna-PL", "type": "arguana-pl", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "map_at_1", "value": 38.834}, {"type": "map_at_10", "value": 55.22899999999999}, {"type": "map_at_100", "value": 55.791999999999994}, {"type": "map_at_1000", "value": 55.794}, {"type": "map_at_3", "value": 51.233}, {"type": "map_at_5", "value": 53.772}, {"type": "mrr_at_1", "value": 39.687}, {"type": "mrr_at_10", "value": 55.596000000000004}, {"type": "mrr_at_100", "value": 56.157000000000004}, {"type": "mrr_at_1000", "value": 56.157999999999994}, {"type": "mrr_at_3", "value": 51.66}, {"type": "mrr_at_5", "value": 54.135}, {"type": "ndcg_at_1", "value": 38.834}, {"type": "ndcg_at_10", "value": 63.402}, {"type": "ndcg_at_100", "value": 65.78}, {"type": "ndcg_at_1000", "value": 65.816}, {"type": "ndcg_at_3", "value": 55.349000000000004}, {"type": "ndcg_at_5", "value": 59.892}, {"type": "precision_at_1", "value": 38.834}, {"type": "precision_at_10", "value": 8.905000000000001}, {"type": "precision_at_100", "value": 0.9939999999999999}, {"type": "precision_at_1000", "value": 0.1}, {"type": "precision_at_3", "value": 22.428}, {"type": "precision_at_5", "value": 15.647}, {"type": "recall_at_1", "value": 38.834}, {"type": "recall_at_10", "value": 89.047}, {"type": "recall_at_100", "value": 99.36}, {"type": "recall_at_1000", "value": 99.644}, {"type": "recall_at_3", "value": 67.283}, {"type": "recall_at_5", "value": 78.236}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB CBD", "type": "PL-MTEB/cbd", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "accuracy", "value": 69.33}, {"type": "ap", "value": 22.972409521444508}, {"type": "f1", "value": 58.91072163784952}]}, {"task": {"type": "PairClassification"}, "dataset": {"name": "MTEB CDSC-E", "type": "PL-MTEB/cdsce-pairclassification", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "cos_sim_accuracy", "value": 89.8}, {"type": "cos_sim_ap", "value": 79.87039801032493}, {"type": "cos_sim_f1", "value": 68.53932584269663}, {"type": "cos_sim_precision", "value": 73.49397590361446}, {"type": "cos_sim_recall", "value": 64.21052631578948}, {"type": "dot_accuracy", "value": 86.1}, {"type": "dot_ap", "value": 63.684975861694035}, {"type": "dot_f1", "value": 63.61746361746362}, {"type": "dot_precision", "value": 52.57731958762887}, {"type": "dot_recall", "value": 80.52631578947368}, {"type": "euclidean_accuracy", "value": 89.8}, {"type": "euclidean_ap", "value": 79.7527126811392}, {"type": "euclidean_f1", "value": 68.46361185983827}, {"type": "euclidean_precision", "value": 70.1657458563536}, {"type": "euclidean_recall", "value": 66.84210526315789}, {"type": "manhattan_accuracy", "value": 89.7}, {"type": "manhattan_ap", "value": 79.64632771093657}, {"type": "manhattan_f1", "value": 68.4931506849315}, {"type": "manhattan_precision", "value": 71.42857142857143}, {"type": "manhattan_recall", "value": 65.78947368421053}, {"type": "max_accuracy", "value": 89.8}, {"type": "max_ap", "value": 79.87039801032493}, {"type": "max_f1", "value": 68.53932584269663}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB CDSC-R", "type": "PL-MTEB/cdscr-sts", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "cos_sim_pearson", "value": 92.1088892402831}, {"type": "cos_sim_spearman", "value": 92.54126377343101}, {"type": "euclidean_pearson", "value": 91.99022371986013}, {"type": "euclidean_spearman", "value": 92.55235973775511}, {"type": "manhattan_pearson", "value": 91.92170171331357}, {"type": "manhattan_spearman", "value": 92.47797623672449}]}, {"task": {"type": "Retrieval"}, "dataset": {"name": "MTEB DBPedia-PL", "type": "dbpedia-pl", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "map_at_1", "value": 8.683}, {"type": "map_at_10", "value": 18.9}, {"type": "map_at_100", "value": 26.933}, {"type": "map_at_1000", "value": 28.558}, {"type": "map_at_3", "value": 13.638}, {"type": "map_at_5", "value": 15.9}, {"type": "mrr_at_1", "value": 63.74999999999999}, {"type": "mrr_at_10", "value": 73.566}, {"type": "mrr_at_100", "value": 73.817}, {"type": "mrr_at_1000", "value": 73.824}, {"type": "mrr_at_3", "value": 71.875}, {"type": "mrr_at_5", "value": 73.2}, {"type": "ndcg_at_1", "value": 53.125}, {"type": "ndcg_at_10", "value": 40.271}, {"type": "ndcg_at_100", "value": 45.51}, {"type": "ndcg_at_1000", "value": 52.968}, {"type": "ndcg_at_3", "value": 45.122}, {"type": "ndcg_at_5", "value": 42.306}, {"type": "precision_at_1", "value": 63.74999999999999}, {"type": "precision_at_10", "value": 31.55}, {"type": "precision_at_100", "value": 10.440000000000001}, {"type": "precision_at_1000", "value": 2.01}, {"type": "precision_at_3", "value": 48.333}, {"type": "precision_at_5", "value": 40.5}, {"type": "recall_at_1", "value": 8.683}, {"type": "recall_at_10", "value": 24.63}, {"type": "recall_at_100", "value": 51.762}, {"type": "recall_at_1000", "value": 75.64999999999999}, {"type": "recall_at_3", "value": 15.136}, {"type": "recall_at_5", "value": 18.678}]}, {"task": {"type": "Retrieval"}, "dataset": {"name": "MTEB FiQA-PL", "type": "fiqa-pl", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "map_at_1", "value": 19.872999999999998}, {"type": "map_at_10", "value": 32.923}, {"type": "map_at_100", "value": 34.819}, {"type": "map_at_1000", "value": 34.99}, {"type": "map_at_3", "value": 28.500999999999998}, {"type": "map_at_5", "value": 31.087999999999997}, {"type": "mrr_at_1", "value": 40.432}, {"type": "mrr_at_10", "value": 49.242999999999995}, {"type": "mrr_at_100", "value": 50.014}, {"type": "mrr_at_1000", "value": 50.05500000000001}, {"type": "mrr_at_3", "value": 47.144999999999996}, {"type": "mrr_at_5", "value": 48.171}, {"type": "ndcg_at_1", "value": 40.586}, {"type": "ndcg_at_10", "value": 40.887}, {"type": "ndcg_at_100", "value": 47.701}, {"type": "ndcg_at_1000", "value": 50.624}, {"type": "ndcg_at_3", "value": 37.143}, {"type": "ndcg_at_5", "value": 38.329}, {"type": "precision_at_1", "value": 40.586}, {"type": "precision_at_10", "value": 11.497}, {"type": "precision_at_100", "value": 1.838}, {"type": "precision_at_1000", "value": 0.23700000000000002}, {"type": "precision_at_3", "value": 25.0}, {"type": "precision_at_5", "value": 18.549}, {"type": "recall_at_1", "value": 19.872999999999998}, {"type": "recall_at_10", "value": 48.073}, {"type": "recall_at_100", "value": 73.473}, {"type": "recall_at_1000", "value": 90.94}, {"type": "recall_at_3", "value": 33.645}, {"type": "recall_at_5", "value": 39.711}]}, {"task": {"type": "Retrieval"}, "dataset": {"name": "MTEB HotpotQA-PL", "type": "hotpotqa-pl", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "map_at_1", "value": 39.399}, {"type": "map_at_10", "value": 62.604000000000006}, {"type": "map_at_100", "value": 63.475}, {"type": "map_at_1000", "value": 63.534}, {"type": "map_at_3", "value": 58.870999999999995}, {"type": "map_at_5", "value": 61.217}, {"type": "mrr_at_1", "value": 78.758}, {"type": "mrr_at_10", "value": 84.584}, {"type": "mrr_at_100", "value": 84.753}, {"type": "mrr_at_1000", "value": 84.759}, {"type": "mrr_at_3", "value": 83.65700000000001}, {"type": "mrr_at_5", "value": 84.283}, {"type": "ndcg_at_1", "value": 78.798}, {"type": "ndcg_at_10", "value": 71.04}, {"type": "ndcg_at_100", "value": 74.048}, {"type": "ndcg_at_1000", "value": 75.163}, {"type": "ndcg_at_3", "value": 65.862}, {"type": "ndcg_at_5", "value": 68.77600000000001}, {"type": "precision_at_1", "value": 78.798}, {"type": "precision_at_10", "value": 14.949000000000002}, {"type": "precision_at_100", "value": 1.7309999999999999}, {"type": "precision_at_1000", "value": 0.188}, {"type": "precision_at_3", "value": 42.237}, {"type": "precision_at_5", "value": 27.634999999999998}, {"type": "recall_at_1", "value": 39.399}, {"type": "recall_at_10", "value": 74.747}, {"type": "recall_at_100", "value": 86.529}, {"type": "recall_at_1000", "value": 93.849}, {"type": "recall_at_3", "value": 63.356}, {"type": "recall_at_5", "value": 69.08800000000001}]}, {"task": {"type": "Retrieval"}, "dataset": {"name": "MTEB MSMARCO-PL", "type": "msmarco-pl", "config": "default", "split": "validation", "revision": "None"}, "metrics": [{"type": "map_at_1", "value": 19.598}, {"type": "map_at_10", "value": 30.453999999999997}, {"type": "map_at_100", "value": 31.601000000000003}, {"type": "map_at_1000", "value": 31.66}, {"type": "map_at_3", "value": 27.118}, {"type": "map_at_5", "value": 28.943}, {"type": "mrr_at_1", "value": 20.1}, {"type": "mrr_at_10", "value": 30.978}, {"type": "mrr_at_100", "value": 32.057}, {"type": "mrr_at_1000", "value": 32.112}, {"type": "mrr_at_3", "value": 27.679}, {"type": "mrr_at_5", "value": 29.493000000000002}, {"type": "ndcg_at_1", "value": 20.158}, {"type": "ndcg_at_10", "value": 36.63}, {"type": "ndcg_at_100", "value": 42.291000000000004}, {"type": "ndcg_at_1000", "value": 43.828}, {"type": "ndcg_at_3", "value": 29.744999999999997}, {"type": "ndcg_at_5", "value": 33.024}, {"type": "precision_at_1", "value": 20.158}, {"type": "precision_at_10", "value": 5.811999999999999}, {"type": "precision_at_100", "value": 0.868}, {"type": "precision_at_1000", "value": 0.1}, {"type": "precision_at_3", "value": 12.689}, {"type": "precision_at_5", "value": 9.295}, {"type": "recall_at_1", "value": 19.598}, {"type": "recall_at_10", "value": 55.596999999999994}, {"type": "recall_at_100", "value": 82.143}, {"type": "recall_at_1000", "value": 94.015}, {"type": "recall_at_3", "value": 36.720000000000006}, {"type": "recall_at_5", "value": 44.606}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (pl)", "type": "mteb/amazon_massive_intent", "config": "pl", "split": "test", "revision": "31efe3c427b0bae9c22cbb560b8f15491cc6bed7"}, "metrics": [{"type": "accuracy", "value": 74.8117014122394}, {"type": "f1", "value": 72.0259730121889}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (pl)", "type": "mteb/amazon_massive_scenario", "config": "pl", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 77.84465366509752}, {"type": "f1", "value": 77.73439218970051}]}, {"task": {"type": "Retrieval"}, "dataset": {"name": "MTEB NFCorpus-PL", "type": "nfcorpus-pl", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "map_at_1", "value": 5.604}, {"type": "map_at_10", "value": 12.684000000000001}, {"type": "map_at_100", "value": 16.274}, {"type": "map_at_1000", "value": 17.669}, {"type": "map_at_3", "value": 9.347}, {"type": "map_at_5", "value": 10.752}, {"type": "mrr_at_1", "value": 43.963}, {"type": "mrr_at_10", "value": 52.94}, {"type": "mrr_at_100", "value": 53.571000000000005}, {"type": "mrr_at_1000", "value": 53.613}, {"type": "mrr_at_3", "value": 51.032}, {"type": "mrr_at_5", "value": 52.193}, {"type": "ndcg_at_1", "value": 41.486000000000004}, {"type": "ndcg_at_10", "value": 33.937}, {"type": "ndcg_at_100", "value": 31.726}, {"type": "ndcg_at_1000", "value": 40.331}, {"type": "ndcg_at_3", "value": 39.217}, {"type": "ndcg_at_5", "value": 36.521}, {"type": "precision_at_1", "value": 43.034}, {"type": "precision_at_10", "value": 25.324999999999996}, {"type": "precision_at_100", "value": 8.022}, {"type": "precision_at_1000", "value": 2.0629999999999997}, {"type": "precision_at_3", "value": 36.945}, {"type": "precision_at_5", "value": 31.517}, {"type": "recall_at_1", "value": 5.604}, {"type": "recall_at_10", "value": 16.554}, {"type": "recall_at_100", "value": 33.113}, {"type": "recall_at_1000", "value": 62.832}, {"type": "recall_at_3", "value": 10.397}, {"type": "recall_at_5", "value": 12.629999999999999}]}, {"task": {"type": "Retrieval"}, "dataset": {"name": "MTEB NQ-PL", "type": "nq-pl", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "map_at_1", "value": 26.642}, {"type": "map_at_10", "value": 40.367999999999995}, {"type": "map_at_100", "value": 41.487}, {"type": "map_at_1000", "value": 41.528}, {"type": "map_at_3", "value": 36.292}, {"type": "map_at_5", "value": 38.548}, {"type": "mrr_at_1", "value": 30.156}, {"type": "mrr_at_10", "value": 42.853}, {"type": "mrr_at_100", "value": 43.742}, {"type": "mrr_at_1000", "value": 43.772}, {"type": "mrr_at_3", "value": 39.47}, {"type": "mrr_at_5", "value": 41.366}, {"type": "ndcg_at_1", "value": 30.214000000000002}, {"type": "ndcg_at_10", "value": 47.620000000000005}, {"type": "ndcg_at_100", "value": 52.486}, {"type": "ndcg_at_1000", "value": 53.482}, {"type": "ndcg_at_3", "value": 39.864}, {"type": "ndcg_at_5", "value": 43.645}, {"type": "precision_at_1", "value": 30.214000000000002}, {"type": "precision_at_10", "value": 8.03}, {"type": "precision_at_100", "value": 1.0739999999999998}, {"type": "precision_at_1000", "value": 0.117}, {"type": "precision_at_3", "value": 18.183}, {"type": "precision_at_5", "value": 13.105}, {"type": "recall_at_1", "value": 26.642}, {"type": "recall_at_10", "value": 67.282}, {"type": "recall_at_100", "value": 88.632}, {"type": "recall_at_1000", "value": 96.109}, {"type": "recall_at_3", "value": 47.048}, {"type": "recall_at_5", "value": 55.791000000000004}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB PAC", "type": "laugustyniak/abusive-clauses-pl", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "accuracy", "value": 64.69446857804807}, {"type": "ap", "value": 75.58028779280512}, {"type": "f1", "value": 62.3610392963539}]}, {"task": {"type": "PairClassification"}, "dataset": {"name": "MTEB PPC", "type": "PL-MTEB/ppc-pairclassification", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "cos_sim_accuracy", "value": 88.4}, {"type": "cos_sim_ap", "value": 93.56462741831817}, {"type": "cos_sim_f1", "value": 90.73634204275535}, {"type": "cos_sim_precision", "value": 86.94992412746586}, {"type": "cos_sim_recall", "value": 94.86754966887418}, {"type": "dot_accuracy", "value": 75.3}, {"type": "dot_ap", "value": 83.06945936688015}, {"type": "dot_f1", "value": 81.50887573964496}, {"type": "dot_precision", "value": 73.66310160427807}, {"type": "dot_recall", "value": 91.22516556291392}, {"type": "euclidean_accuracy", "value": 88.8}, {"type": "euclidean_ap", "value": 93.53974198044985}, {"type": "euclidean_f1", "value": 90.87947882736157}, {"type": "euclidean_precision", "value": 89.42307692307693}, {"type": "euclidean_recall", "value": 92.3841059602649}, {"type": "manhattan_accuracy", "value": 88.8}, {"type": "manhattan_ap", "value": 93.54209967780366}, {"type": "manhattan_f1", "value": 90.85072231139645}, {"type": "manhattan_precision", "value": 88.1619937694704}, {"type": "manhattan_recall", "value": 93.70860927152319}, {"type": "max_accuracy", "value": 88.8}, {"type": "max_ap", "value": 93.56462741831817}, {"type": "max_f1", "value": 90.87947882736157}]}, {"task": {"type": "PairClassification"}, "dataset": {"name": "MTEB PSC", "type": "PL-MTEB/psc-pairclassification", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "cos_sim_accuracy", "value": 97.03153988868274}, {"type": "cos_sim_ap", "value": 98.63208302459417}, {"type": "cos_sim_f1", "value": 95.06172839506173}, {"type": "cos_sim_precision", "value": 96.25}, {"type": "cos_sim_recall", "value": 93.90243902439023}, {"type": "dot_accuracy", "value": 86.82745825602969}, {"type": "dot_ap", "value": 83.77450133931302}, {"type": "dot_f1", "value": 79.3053545586107}, {"type": "dot_precision", "value": 75.48209366391184}, {"type": "dot_recall", "value": 83.53658536585365}, {"type": "euclidean_accuracy", "value": 97.03153988868274}, {"type": "euclidean_ap", "value": 98.80678168225653}, {"type": "euclidean_f1", "value": 95.20958083832335}, {"type": "euclidean_precision", "value": 93.52941176470588}, {"type": "euclidean_recall", "value": 96.95121951219512}, {"type": "manhattan_accuracy", "value": 97.21706864564007}, {"type": "manhattan_ap", "value": 98.82279484224186}, {"type": "manhattan_f1", "value": 95.44072948328268}, {"type": "manhattan_precision", "value": 95.15151515151516}, {"type": "manhattan_recall", "value": 95.73170731707317}, {"type": "max_accuracy", "value": 97.21706864564007}, {"type": "max_ap", "value": 98.82279484224186}, {"type": "max_f1", "value": 95.44072948328268}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB PolEmo2.0-IN", "type": "PL-MTEB/polemo2_in", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "accuracy", "value": 76.84210526315789}, {"type": "f1", "value": 75.49713789106988}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB PolEmo2.0-OUT", "type": "PL-MTEB/polemo2_out", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "accuracy", "value": 53.7246963562753}, {"type": "f1", "value": 43.060592194322986}]}, {"task": {"type": "Retrieval"}, "dataset": {"name": "MTEB Quora-PL", "type": "quora-pl", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "map_at_1", "value": 67.021}, {"type": "map_at_10", "value": 81.362}, {"type": "map_at_100", "value": 82.06700000000001}, {"type": "map_at_1000", "value": 82.084}, {"type": "map_at_3", "value": 78.223}, {"type": "map_at_5", "value": 80.219}, {"type": "mrr_at_1", "value": 77.17}, {"type": "mrr_at_10", "value": 84.222}, {"type": "mrr_at_100", "value": 84.37599999999999}, {"type": "mrr_at_1000", "value": 84.379}, {"type": "mrr_at_3", "value": 83.003}, {"type": "mrr_at_5", "value": 83.834}, {"type": "ndcg_at_1", "value": 77.29}, {"type": "ndcg_at_10", "value": 85.506}, {"type": "ndcg_at_100", "value": 87.0}, {"type": "ndcg_at_1000", "value": 87.143}, {"type": "ndcg_at_3", "value": 82.17}, {"type": "ndcg_at_5", "value": 84.057}, {"type": "precision_at_1", "value": 77.29}, {"type": "precision_at_10", "value": 13.15}, {"type": "precision_at_100", "value": 1.522}, {"type": "precision_at_1000", "value": 0.156}, {"type": "precision_at_3", "value": 36.173}, {"type": "precision_at_5", "value": 23.988}, {"type": "recall_at_1", "value": 67.021}, {"type": "recall_at_10", "value": 93.943}, {"type": "recall_at_100", "value": 99.167}, {"type": "recall_at_1000", "value": 99.929}, {"type": "recall_at_3", "value": 84.55799999999999}, {"type": "recall_at_5", "value": 89.697}]}, {"task": {"type": "Retrieval"}, "dataset": {"name": "MTEB SCIDOCS-PL", "type": "scidocs-pl", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "map_at_1", "value": 4.523}, {"type": "map_at_10", "value": 11.584}, {"type": "map_at_100", "value": 13.705}, {"type": "map_at_1000", "value": 14.038999999999998}, {"type": "map_at_3", "value": 8.187999999999999}, {"type": "map_at_5", "value": 9.922}, {"type": "mrr_at_1", "value": 22.1}, {"type": "mrr_at_10", "value": 32.946999999999996}, {"type": "mrr_at_100", "value": 34.11}, {"type": "mrr_at_1000", "value": 34.163}, {"type": "mrr_at_3", "value": 29.633}, {"type": "mrr_at_5", "value": 31.657999999999998}, {"type": "ndcg_at_1", "value": 22.2}, {"type": "ndcg_at_10", "value": 19.466}, {"type": "ndcg_at_100", "value": 27.725}, {"type": "ndcg_at_1000", "value": 33.539}, {"type": "ndcg_at_3", "value": 18.26}, {"type": "ndcg_at_5", "value": 16.265}, {"type": "precision_at_1", "value": 22.2}, {"type": "precision_at_10", "value": 10.11}, {"type": "precision_at_100", "value": 2.204}, {"type": "precision_at_1000", "value": 0.36}, {"type": "precision_at_3", "value": 17.1}, {"type": "precision_at_5", "value": 14.44}, {"type": "recall_at_1", "value": 4.523}, {"type": "recall_at_10", "value": 20.497}, {"type": "recall_at_100", "value": 44.757000000000005}, {"type": "recall_at_1000", "value": 73.14699999999999}, {"type": "recall_at_3", "value": 10.413}, {"type": "recall_at_5", "value": 14.638000000000002}]}, {"task": {"type": "PairClassification"}, "dataset": {"name": "MTEB SICK-E-PL", "type": "PL-MTEB/sicke-pl-pairclassification", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "cos_sim_accuracy", "value": 87.4235629841011}, {"type": "cos_sim_ap", "value": 84.46531935663157}, {"type": "cos_sim_f1", "value": 77.18910963944077}, {"type": "cos_sim_precision", "value": 79.83257229832572}, {"type": "cos_sim_recall", "value": 74.71509971509973}, {"type": "dot_accuracy", "value": 81.10476966979209}, {"type": "dot_ap", "value": 71.12231750543143}, {"type": "dot_f1", "value": 68.13455657492355}, {"type": "dot_precision", "value": 59.69989281886387}, {"type": "dot_recall", "value": 79.34472934472934}, {"type": "euclidean_accuracy", "value": 87.21973094170403}, {"type": "euclidean_ap", "value": 84.33077991405355}, {"type": "euclidean_f1", "value": 76.81931132410365}, {"type": "euclidean_precision", "value": 76.57466383581033}, {"type": "euclidean_recall", "value": 77.06552706552706}, {"type": "manhattan_accuracy", "value": 87.21973094170403}, {"type": "manhattan_ap", "value": 84.35651252115137}, {"type": "manhattan_f1", "value": 76.87004481213376}, {"type": "manhattan_precision", "value": 74.48229792919172}, {"type": "manhattan_recall", "value": 79.41595441595442}, {"type": "max_accuracy", "value": 87.4235629841011}, {"type": "max_ap", "value": 84.46531935663157}, {"type": "max_f1", "value": 77.18910963944077}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB SICK-R-PL", "type": "PL-MTEB/sickr-pl-sts", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "cos_sim_pearson", "value": 83.05629619004273}, {"type": "cos_sim_spearman", "value": 79.90632583043678}, {"type": "euclidean_pearson", "value": 81.56426663515931}, {"type": "euclidean_spearman", "value": 80.05439220131294}, {"type": "manhattan_pearson", "value": 81.52958181013108}, {"type": "manhattan_spearman", "value": 80.0387467163383}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STS22 (pl)", "type": "mteb/sts22-crosslingual-sts", "config": "pl", "split": "test", "revision": "6d1ba47164174a496b7fa5d3569dae26a6813b80"}, "metrics": [{"type": "cos_sim_pearson", "value": 35.93847200513348}, {"type": "cos_sim_spearman", "value": 39.31543525546526}, {"type": "euclidean_pearson", "value": 30.19743936591465}, {"type": "euclidean_spearman", "value": 39.966612599252095}, {"type": "manhattan_pearson", "value": 30.195614462473387}, {"type": "manhattan_spearman", "value": 39.822552043685754}]}, {"task": {"type": "Retrieval"}, "dataset": {"name": "MTEB SciFact-PL", "type": "scifact-pl", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "map_at_1", "value": 56.05}, {"type": "map_at_10", "value": 65.93299999999999}, {"type": "map_at_100", "value": 66.571}, {"type": "map_at_1000", "value": 66.60000000000001}, {"type": "map_at_3", "value": 63.489}, {"type": "map_at_5", "value": 64.91799999999999}, {"type": "mrr_at_1", "value": 59.0}, {"type": "mrr_at_10", "value": 67.026}, {"type": "mrr_at_100", "value": 67.559}, {"type": "mrr_at_1000", "value": 67.586}, {"type": "mrr_at_3", "value": 65.444}, {"type": "mrr_at_5", "value": 66.278}, {"type": "ndcg_at_1", "value": 59.0}, {"type": "ndcg_at_10", "value": 70.233}, {"type": "ndcg_at_100", "value": 72.789}, {"type": "ndcg_at_1000", "value": 73.637}, {"type": "ndcg_at_3", "value": 66.40700000000001}, {"type": "ndcg_at_5", "value": 68.206}, {"type": "precision_at_1", "value": 59.0}, {"type": "precision_at_10", "value": 9.367}, {"type": "precision_at_100", "value": 1.06}, {"type": "precision_at_1000", "value": 0.11299999999999999}, {"type": "precision_at_3", "value": 26.222}, {"type": "precision_at_5", "value": 17.067}, {"type": "recall_at_1", "value": 56.05}, {"type": "recall_at_10", "value": 82.089}, {"type": "recall_at_100", "value": 93.167}, {"type": "recall_at_1000", "value": 100.0}, {"type": "recall_at_3", "value": 71.822}, {"type": "recall_at_5", "value": 76.483}]}, {"task": {"type": "Retrieval"}, "dataset": {"name": "MTEB TRECCOVID-PL", "type": "trec-covid-pl", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "map_at_1", "value": 0.21}, {"type": "map_at_10", "value": 1.7680000000000002}, {"type": "map_at_100", "value": 9.447999999999999}, {"type": "map_at_1000", "value": 21.728}, {"type": "map_at_3", "value": 0.603}, {"type": "map_at_5", "value": 0.9610000000000001}, {"type": "mrr_at_1", "value": 80.0}, {"type": "mrr_at_10", "value": 88.667}, {"type": "mrr_at_100", "value": 88.667}, {"type": "mrr_at_1000", "value": 88.667}, {"type": "mrr_at_3", "value": 87.667}, {"type": "mrr_at_5", "value": 88.667}, {"type": "ndcg_at_1", "value": 77.0}, {"type": "ndcg_at_10", "value": 70.814}, {"type": "ndcg_at_100", "value": 52.532000000000004}, {"type": "ndcg_at_1000", "value": 45.635999999999996}, {"type": "ndcg_at_3", "value": 76.542}, {"type": "ndcg_at_5", "value": 73.24000000000001}, {"type": "precision_at_1", "value": 80.0}, {"type": "precision_at_10", "value": 75.0}, {"type": "precision_at_100", "value": 53.879999999999995}, {"type": "precision_at_1000", "value": 20.002}, {"type": "precision_at_3", "value": 80.0}, {"type": "precision_at_5", "value": 76.4}, {"type": "recall_at_1", "value": 0.21}, {"type": "recall_at_10", "value": 2.012}, {"type": "recall_at_100", "value": 12.781999999999998}, {"type": "recall_at_1000", "value": 42.05}, {"type": "recall_at_3", "value": 0.644}, {"type": "recall_at_5", "value": 1.04}]}]}]}
sentence-transformers pytorch roberta feature-extraction sentence-similarity transformers mteb pl license:apache-2.0 model-index endpoints_compatible has_space region:us