|
--- |
|
pipeline_tag: sentence-similarity |
|
tags: |
|
- sentence-transformers |
|
- feature-extraction |
|
- sentence-similarity |
|
- transformers |
|
- mteb |
|
model-index: |
|
- name: mmlw-e5-small |
|
results: |
|
- task: |
|
type: Clustering |
|
dataset: |
|
type: PL-MTEB/8tags-clustering |
|
name: MTEB 8TagsClustering |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: v_measure |
|
value: 31.772224277808153 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: PL-MTEB/allegro-reviews |
|
name: MTEB AllegroReviews |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: accuracy |
|
value: 33.03180914512922 |
|
- type: f1 |
|
value: 29.800304217426167 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: arguana-pl |
|
name: MTEB ArguAna-PL |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 28.804999999999996 |
|
- type: map_at_10 |
|
value: 45.327 |
|
- type: map_at_100 |
|
value: 46.17 |
|
- type: map_at_1000 |
|
value: 46.177 |
|
- type: map_at_3 |
|
value: 40.528999999999996 |
|
- type: map_at_5 |
|
value: 43.335 |
|
- type: mrr_at_1 |
|
value: 30.299 |
|
- type: mrr_at_10 |
|
value: 45.763 |
|
- type: mrr_at_100 |
|
value: 46.641 |
|
- type: mrr_at_1000 |
|
value: 46.648 |
|
- type: mrr_at_3 |
|
value: 41.074 |
|
- type: mrr_at_5 |
|
value: 43.836999999999996 |
|
- type: ndcg_at_1 |
|
value: 28.804999999999996 |
|
- type: ndcg_at_10 |
|
value: 54.308 |
|
- type: ndcg_at_100 |
|
value: 57.879000000000005 |
|
- type: ndcg_at_1000 |
|
value: 58.048 |
|
- type: ndcg_at_3 |
|
value: 44.502 |
|
- type: ndcg_at_5 |
|
value: 49.519000000000005 |
|
- type: precision_at_1 |
|
value: 28.804999999999996 |
|
- type: precision_at_10 |
|
value: 8.286 |
|
- type: precision_at_100 |
|
value: 0.984 |
|
- type: precision_at_1000 |
|
value: 0.1 |
|
- type: precision_at_3 |
|
value: 18.682000000000002 |
|
- type: precision_at_5 |
|
value: 13.627 |
|
- type: recall_at_1 |
|
value: 28.804999999999996 |
|
- type: recall_at_10 |
|
value: 82.85900000000001 |
|
- type: recall_at_100 |
|
value: 98.36399999999999 |
|
- type: recall_at_1000 |
|
value: 99.644 |
|
- type: recall_at_3 |
|
value: 56.04599999999999 |
|
- type: recall_at_5 |
|
value: 68.137 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: PL-MTEB/cbd |
|
name: MTEB CBD |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: accuracy |
|
value: 64.24 |
|
- type: ap |
|
value: 17.967103105024705 |
|
- type: f1 |
|
value: 52.97375416129459 |
|
- task: |
|
type: PairClassification |
|
dataset: |
|
type: PL-MTEB/cdsce-pairclassification |
|
name: MTEB CDSC-E |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: cos_sim_accuracy |
|
value: 88.8 |
|
- type: cos_sim_ap |
|
value: 76.68028778789487 |
|
- type: cos_sim_f1 |
|
value: 66.82352941176471 |
|
- type: cos_sim_precision |
|
value: 60.42553191489362 |
|
- type: cos_sim_recall |
|
value: 74.73684210526315 |
|
- type: dot_accuracy |
|
value: 88.1 |
|
- type: dot_ap |
|
value: 72.04910086070551 |
|
- type: dot_f1 |
|
value: 66.66666666666667 |
|
- type: dot_precision |
|
value: 69.31818181818183 |
|
- type: dot_recall |
|
value: 64.21052631578948 |
|
- type: euclidean_accuracy |
|
value: 88.8 |
|
- type: euclidean_ap |
|
value: 76.63591858340688 |
|
- type: euclidean_f1 |
|
value: 67.13286713286713 |
|
- type: euclidean_precision |
|
value: 60.25104602510461 |
|
- type: euclidean_recall |
|
value: 75.78947368421053 |
|
- type: manhattan_accuracy |
|
value: 88.9 |
|
- type: manhattan_ap |
|
value: 76.54552849815124 |
|
- type: manhattan_f1 |
|
value: 66.66666666666667 |
|
- type: manhattan_precision |
|
value: 60.51502145922747 |
|
- type: manhattan_recall |
|
value: 74.21052631578947 |
|
- type: max_accuracy |
|
value: 88.9 |
|
- type: max_ap |
|
value: 76.68028778789487 |
|
- type: max_f1 |
|
value: 67.13286713286713 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: PL-MTEB/cdscr-sts |
|
name: MTEB CDSC-R |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 91.64169404461497 |
|
- type: cos_sim_spearman |
|
value: 91.9755161377078 |
|
- type: euclidean_pearson |
|
value: 90.87481478491249 |
|
- type: euclidean_spearman |
|
value: 91.92362666383987 |
|
- type: manhattan_pearson |
|
value: 90.8415510499638 |
|
- type: manhattan_spearman |
|
value: 91.85927127194698 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: dbpedia-pl |
|
name: MTEB DBPedia-PL |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 6.148 |
|
- type: map_at_10 |
|
value: 12.870999999999999 |
|
- type: map_at_100 |
|
value: 18.04 |
|
- type: map_at_1000 |
|
value: 19.286 |
|
- type: map_at_3 |
|
value: 9.156 |
|
- type: map_at_5 |
|
value: 10.857999999999999 |
|
- type: mrr_at_1 |
|
value: 53.25 |
|
- type: mrr_at_10 |
|
value: 61.016999999999996 |
|
- type: mrr_at_100 |
|
value: 61.48400000000001 |
|
- type: mrr_at_1000 |
|
value: 61.507999999999996 |
|
- type: mrr_at_3 |
|
value: 58.75 |
|
- type: mrr_at_5 |
|
value: 60.375 |
|
- type: ndcg_at_1 |
|
value: 41.0 |
|
- type: ndcg_at_10 |
|
value: 30.281000000000002 |
|
- type: ndcg_at_100 |
|
value: 33.955999999999996 |
|
- type: ndcg_at_1000 |
|
value: 40.77 |
|
- type: ndcg_at_3 |
|
value: 34.127 |
|
- type: ndcg_at_5 |
|
value: 32.274 |
|
- type: precision_at_1 |
|
value: 52.5 |
|
- type: precision_at_10 |
|
value: 24.525 |
|
- type: precision_at_100 |
|
value: 8.125 |
|
- type: precision_at_1000 |
|
value: 1.728 |
|
- type: precision_at_3 |
|
value: 37.083 |
|
- type: precision_at_5 |
|
value: 32.15 |
|
- type: recall_at_1 |
|
value: 6.148 |
|
- type: recall_at_10 |
|
value: 17.866 |
|
- type: recall_at_100 |
|
value: 39.213 |
|
- type: recall_at_1000 |
|
value: 61.604000000000006 |
|
- type: recall_at_3 |
|
value: 10.084 |
|
- type: recall_at_5 |
|
value: 13.333999999999998 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: fiqa-pl |
|
name: MTEB FiQA-PL |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 14.643 |
|
- type: map_at_10 |
|
value: 23.166 |
|
- type: map_at_100 |
|
value: 24.725 |
|
- type: map_at_1000 |
|
value: 24.92 |
|
- type: map_at_3 |
|
value: 20.166 |
|
- type: map_at_5 |
|
value: 22.003 |
|
- type: mrr_at_1 |
|
value: 29.630000000000003 |
|
- type: mrr_at_10 |
|
value: 37.632 |
|
- type: mrr_at_100 |
|
value: 38.512 |
|
- type: mrr_at_1000 |
|
value: 38.578 |
|
- type: mrr_at_3 |
|
value: 35.391 |
|
- type: mrr_at_5 |
|
value: 36.857 |
|
- type: ndcg_at_1 |
|
value: 29.166999999999998 |
|
- type: ndcg_at_10 |
|
value: 29.749 |
|
- type: ndcg_at_100 |
|
value: 35.983 |
|
- type: ndcg_at_1000 |
|
value: 39.817 |
|
- type: ndcg_at_3 |
|
value: 26.739 |
|
- type: ndcg_at_5 |
|
value: 27.993000000000002 |
|
- type: precision_at_1 |
|
value: 29.166999999999998 |
|
- type: precision_at_10 |
|
value: 8.333 |
|
- type: precision_at_100 |
|
value: 1.448 |
|
- type: precision_at_1000 |
|
value: 0.213 |
|
- type: precision_at_3 |
|
value: 17.747 |
|
- type: precision_at_5 |
|
value: 13.58 |
|
- type: recall_at_1 |
|
value: 14.643 |
|
- type: recall_at_10 |
|
value: 35.247 |
|
- type: recall_at_100 |
|
value: 59.150999999999996 |
|
- type: recall_at_1000 |
|
value: 82.565 |
|
- type: recall_at_3 |
|
value: 24.006 |
|
- type: recall_at_5 |
|
value: 29.383 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: hotpotqa-pl |
|
name: MTEB HotpotQA-PL |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 32.627 |
|
- type: map_at_10 |
|
value: 48.041 |
|
- type: map_at_100 |
|
value: 49.008 |
|
- type: map_at_1000 |
|
value: 49.092999999999996 |
|
- type: map_at_3 |
|
value: 44.774 |
|
- type: map_at_5 |
|
value: 46.791 |
|
- type: mrr_at_1 |
|
value: 65.28 |
|
- type: mrr_at_10 |
|
value: 72.53500000000001 |
|
- type: mrr_at_100 |
|
value: 72.892 |
|
- type: mrr_at_1000 |
|
value: 72.909 |
|
- type: mrr_at_3 |
|
value: 71.083 |
|
- type: mrr_at_5 |
|
value: 71.985 |
|
- type: ndcg_at_1 |
|
value: 65.253 |
|
- type: ndcg_at_10 |
|
value: 57.13700000000001 |
|
- type: ndcg_at_100 |
|
value: 60.783 |
|
- type: ndcg_at_1000 |
|
value: 62.507000000000005 |
|
- type: ndcg_at_3 |
|
value: 52.17 |
|
- type: ndcg_at_5 |
|
value: 54.896 |
|
- type: precision_at_1 |
|
value: 65.253 |
|
- type: precision_at_10 |
|
value: 12.088000000000001 |
|
- type: precision_at_100 |
|
value: 1.496 |
|
- type: precision_at_1000 |
|
value: 0.172 |
|
- type: precision_at_3 |
|
value: 32.96 |
|
- type: precision_at_5 |
|
value: 21.931 |
|
- type: recall_at_1 |
|
value: 32.627 |
|
- type: recall_at_10 |
|
value: 60.439 |
|
- type: recall_at_100 |
|
value: 74.80799999999999 |
|
- type: recall_at_1000 |
|
value: 86.219 |
|
- type: recall_at_3 |
|
value: 49.44 |
|
- type: recall_at_5 |
|
value: 54.827999999999996 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: msmarco-pl |
|
name: MTEB MSMARCO-PL |
|
config: default |
|
split: validation |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 13.150999999999998 |
|
- type: map_at_10 |
|
value: 21.179000000000002 |
|
- type: map_at_100 |
|
value: 22.227 |
|
- type: map_at_1000 |
|
value: 22.308 |
|
- type: map_at_3 |
|
value: 18.473 |
|
- type: map_at_5 |
|
value: 19.942999999999998 |
|
- type: mrr_at_1 |
|
value: 13.467 |
|
- type: mrr_at_10 |
|
value: 21.471 |
|
- type: mrr_at_100 |
|
value: 22.509 |
|
- type: mrr_at_1000 |
|
value: 22.585 |
|
- type: mrr_at_3 |
|
value: 18.789 |
|
- type: mrr_at_5 |
|
value: 20.262 |
|
- type: ndcg_at_1 |
|
value: 13.539000000000001 |
|
- type: ndcg_at_10 |
|
value: 25.942999999999998 |
|
- type: ndcg_at_100 |
|
value: 31.386999999999997 |
|
- type: ndcg_at_1000 |
|
value: 33.641 |
|
- type: ndcg_at_3 |
|
value: 20.368 |
|
- type: ndcg_at_5 |
|
value: 23.003999999999998 |
|
- type: precision_at_1 |
|
value: 13.539000000000001 |
|
- type: precision_at_10 |
|
value: 4.249 |
|
- type: precision_at_100 |
|
value: 0.7040000000000001 |
|
- type: precision_at_1000 |
|
value: 0.09 |
|
- type: precision_at_3 |
|
value: 8.782 |
|
- type: precision_at_5 |
|
value: 6.6049999999999995 |
|
- type: recall_at_1 |
|
value: 13.150999999999998 |
|
- type: recall_at_10 |
|
value: 40.698 |
|
- type: recall_at_100 |
|
value: 66.71000000000001 |
|
- type: recall_at_1000 |
|
value: 84.491 |
|
- type: recall_at_3 |
|
value: 25.452 |
|
- type: recall_at_5 |
|
value: 31.791000000000004 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_intent |
|
name: MTEB MassiveIntentClassification (pl) |
|
config: pl |
|
split: test |
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
|
metrics: |
|
- type: accuracy |
|
value: 67.3537323470074 |
|
- type: f1 |
|
value: 64.67852047603644 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_scenario |
|
name: MTEB MassiveScenarioClassification (pl) |
|
config: pl |
|
split: test |
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
|
metrics: |
|
- type: accuracy |
|
value: 72.12508406186953 |
|
- type: f1 |
|
value: 71.55887309568853 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: nfcorpus-pl |
|
name: MTEB NFCorpus-PL |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 4.18 |
|
- type: map_at_10 |
|
value: 9.524000000000001 |
|
- type: map_at_100 |
|
value: 12.272 |
|
- type: map_at_1000 |
|
value: 13.616 |
|
- type: map_at_3 |
|
value: 6.717 |
|
- type: map_at_5 |
|
value: 8.172 |
|
- type: mrr_at_1 |
|
value: 37.152 |
|
- type: mrr_at_10 |
|
value: 45.068000000000005 |
|
- type: mrr_at_100 |
|
value: 46.026 |
|
- type: mrr_at_1000 |
|
value: 46.085 |
|
- type: mrr_at_3 |
|
value: 43.344 |
|
- type: mrr_at_5 |
|
value: 44.412 |
|
- type: ndcg_at_1 |
|
value: 34.52 |
|
- type: ndcg_at_10 |
|
value: 27.604 |
|
- type: ndcg_at_100 |
|
value: 26.012999999999998 |
|
- type: ndcg_at_1000 |
|
value: 35.272 |
|
- type: ndcg_at_3 |
|
value: 31.538 |
|
- type: ndcg_at_5 |
|
value: 30.165999999999997 |
|
- type: precision_at_1 |
|
value: 36.223 |
|
- type: precision_at_10 |
|
value: 21.053 |
|
- type: precision_at_100 |
|
value: 7.08 |
|
- type: precision_at_1000 |
|
value: 1.9929999999999999 |
|
- type: precision_at_3 |
|
value: 30.031000000000002 |
|
- type: precision_at_5 |
|
value: 26.997 |
|
- type: recall_at_1 |
|
value: 4.18 |
|
- type: recall_at_10 |
|
value: 12.901000000000002 |
|
- type: recall_at_100 |
|
value: 27.438000000000002 |
|
- type: recall_at_1000 |
|
value: 60.768 |
|
- type: recall_at_3 |
|
value: 7.492 |
|
- type: recall_at_5 |
|
value: 10.05 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: nq-pl |
|
name: MTEB NQ-PL |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 17.965 |
|
- type: map_at_10 |
|
value: 28.04 |
|
- type: map_at_100 |
|
value: 29.217 |
|
- type: map_at_1000 |
|
value: 29.285 |
|
- type: map_at_3 |
|
value: 24.818 |
|
- type: map_at_5 |
|
value: 26.617 |
|
- type: mrr_at_1 |
|
value: 20.22 |
|
- type: mrr_at_10 |
|
value: 30.148000000000003 |
|
- type: mrr_at_100 |
|
value: 31.137999999999998 |
|
- type: mrr_at_1000 |
|
value: 31.19 |
|
- type: mrr_at_3 |
|
value: 27.201999999999998 |
|
- type: mrr_at_5 |
|
value: 28.884999999999998 |
|
- type: ndcg_at_1 |
|
value: 20.365 |
|
- type: ndcg_at_10 |
|
value: 33.832 |
|
- type: ndcg_at_100 |
|
value: 39.33 |
|
- type: ndcg_at_1000 |
|
value: 41.099999999999994 |
|
- type: ndcg_at_3 |
|
value: 27.46 |
|
- type: ndcg_at_5 |
|
value: 30.584 |
|
- type: precision_at_1 |
|
value: 20.365 |
|
- type: precision_at_10 |
|
value: 5.849 |
|
- type: precision_at_100 |
|
value: 0.8959999999999999 |
|
- type: precision_at_1000 |
|
value: 0.107 |
|
- type: precision_at_3 |
|
value: 12.64 |
|
- type: precision_at_5 |
|
value: 9.334000000000001 |
|
- type: recall_at_1 |
|
value: 17.965 |
|
- type: recall_at_10 |
|
value: 49.503 |
|
- type: recall_at_100 |
|
value: 74.351 |
|
- type: recall_at_1000 |
|
value: 87.766 |
|
- type: recall_at_3 |
|
value: 32.665 |
|
- type: recall_at_5 |
|
value: 39.974 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: laugustyniak/abusive-clauses-pl |
|
name: MTEB PAC |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: accuracy |
|
value: 63.11323486823051 |
|
- type: ap |
|
value: 74.53486257377787 |
|
- type: f1 |
|
value: 60.631005373417736 |
|
- task: |
|
type: PairClassification |
|
dataset: |
|
type: PL-MTEB/ppc-pairclassification |
|
name: MTEB PPC |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: cos_sim_accuracy |
|
value: 80.10000000000001 |
|
- type: cos_sim_ap |
|
value: 89.69526236458292 |
|
- type: cos_sim_f1 |
|
value: 83.37468982630274 |
|
- type: cos_sim_precision |
|
value: 83.30578512396694 |
|
- type: cos_sim_recall |
|
value: 83.44370860927152 |
|
- type: dot_accuracy |
|
value: 77.8 |
|
- type: dot_ap |
|
value: 87.72366051496104 |
|
- type: dot_f1 |
|
value: 82.83752860411899 |
|
- type: dot_precision |
|
value: 76.80339462517681 |
|
- type: dot_recall |
|
value: 89.90066225165563 |
|
- type: euclidean_accuracy |
|
value: 80.10000000000001 |
|
- type: euclidean_ap |
|
value: 89.61317191870039 |
|
- type: euclidean_f1 |
|
value: 83.40214698596202 |
|
- type: euclidean_precision |
|
value: 83.19604612850083 |
|
- type: euclidean_recall |
|
value: 83.6092715231788 |
|
- type: manhattan_accuracy |
|
value: 79.60000000000001 |
|
- type: manhattan_ap |
|
value: 89.48363786968471 |
|
- type: manhattan_f1 |
|
value: 82.96296296296296 |
|
- type: manhattan_precision |
|
value: 82.48772504091653 |
|
- type: manhattan_recall |
|
value: 83.44370860927152 |
|
- type: max_accuracy |
|
value: 80.10000000000001 |
|
- type: max_ap |
|
value: 89.69526236458292 |
|
- type: max_f1 |
|
value: 83.40214698596202 |
|
- task: |
|
type: PairClassification |
|
dataset: |
|
type: PL-MTEB/psc-pairclassification |
|
name: MTEB PSC |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: cos_sim_accuracy |
|
value: 96.93877551020408 |
|
- type: cos_sim_ap |
|
value: 98.86489482248999 |
|
- type: cos_sim_f1 |
|
value: 95.11111111111113 |
|
- type: cos_sim_precision |
|
value: 92.507204610951 |
|
- type: cos_sim_recall |
|
value: 97.86585365853658 |
|
- type: dot_accuracy |
|
value: 95.73283858998145 |
|
- type: dot_ap |
|
value: 97.8261652492545 |
|
- type: dot_f1 |
|
value: 93.21533923303835 |
|
- type: dot_precision |
|
value: 90.28571428571428 |
|
- type: dot_recall |
|
value: 96.34146341463415 |
|
- type: euclidean_accuracy |
|
value: 96.93877551020408 |
|
- type: euclidean_ap |
|
value: 98.84837797066623 |
|
- type: euclidean_f1 |
|
value: 95.11111111111113 |
|
- type: euclidean_precision |
|
value: 92.507204610951 |
|
- type: euclidean_recall |
|
value: 97.86585365853658 |
|
- type: manhattan_accuracy |
|
value: 96.84601113172542 |
|
- type: manhattan_ap |
|
value: 98.78659090944161 |
|
- type: manhattan_f1 |
|
value: 94.9404761904762 |
|
- type: manhattan_precision |
|
value: 92.73255813953489 |
|
- type: manhattan_recall |
|
value: 97.2560975609756 |
|
- type: max_accuracy |
|
value: 96.93877551020408 |
|
- type: max_ap |
|
value: 98.86489482248999 |
|
- type: max_f1 |
|
value: 95.11111111111113 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: PL-MTEB/polemo2_in |
|
name: MTEB PolEmo2.0-IN |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: accuracy |
|
value: 63.961218836565095 |
|
- type: f1 |
|
value: 64.3979989243291 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: PL-MTEB/polemo2_out |
|
name: MTEB PolEmo2.0-OUT |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: accuracy |
|
value: 40.32388663967612 |
|
- type: f1 |
|
value: 32.339117999015755 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: quora-pl |
|
name: MTEB Quora-PL |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 62.757 |
|
- type: map_at_10 |
|
value: 76.55999999999999 |
|
- type: map_at_100 |
|
value: 77.328 |
|
- type: map_at_1000 |
|
value: 77.35499999999999 |
|
- type: map_at_3 |
|
value: 73.288 |
|
- type: map_at_5 |
|
value: 75.25500000000001 |
|
- type: mrr_at_1 |
|
value: 72.28 |
|
- type: mrr_at_10 |
|
value: 79.879 |
|
- type: mrr_at_100 |
|
value: 80.121 |
|
- type: mrr_at_1000 |
|
value: 80.12700000000001 |
|
- type: mrr_at_3 |
|
value: 78.40700000000001 |
|
- type: mrr_at_5 |
|
value: 79.357 |
|
- type: ndcg_at_1 |
|
value: 72.33000000000001 |
|
- type: ndcg_at_10 |
|
value: 81.151 |
|
- type: ndcg_at_100 |
|
value: 83.107 |
|
- type: ndcg_at_1000 |
|
value: 83.397 |
|
- type: ndcg_at_3 |
|
value: 77.3 |
|
- type: ndcg_at_5 |
|
value: 79.307 |
|
- type: precision_at_1 |
|
value: 72.33000000000001 |
|
- type: precision_at_10 |
|
value: 12.587000000000002 |
|
- type: precision_at_100 |
|
value: 1.488 |
|
- type: precision_at_1000 |
|
value: 0.155 |
|
- type: precision_at_3 |
|
value: 33.943 |
|
- type: precision_at_5 |
|
value: 22.61 |
|
- type: recall_at_1 |
|
value: 62.757 |
|
- type: recall_at_10 |
|
value: 90.616 |
|
- type: recall_at_100 |
|
value: 97.905 |
|
- type: recall_at_1000 |
|
value: 99.618 |
|
- type: recall_at_3 |
|
value: 79.928 |
|
- type: recall_at_5 |
|
value: 85.30499999999999 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: scidocs-pl |
|
name: MTEB SCIDOCS-PL |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 3.313 |
|
- type: map_at_10 |
|
value: 8.559999999999999 |
|
- type: map_at_100 |
|
value: 10.177999999999999 |
|
- type: map_at_1000 |
|
value: 10.459999999999999 |
|
- type: map_at_3 |
|
value: 6.094 |
|
- type: map_at_5 |
|
value: 7.323 |
|
- type: mrr_at_1 |
|
value: 16.3 |
|
- type: mrr_at_10 |
|
value: 25.579 |
|
- type: mrr_at_100 |
|
value: 26.717000000000002 |
|
- type: mrr_at_1000 |
|
value: 26.799 |
|
- type: mrr_at_3 |
|
value: 22.583000000000002 |
|
- type: mrr_at_5 |
|
value: 24.298000000000002 |
|
- type: ndcg_at_1 |
|
value: 16.3 |
|
- type: ndcg_at_10 |
|
value: 14.789 |
|
- type: ndcg_at_100 |
|
value: 21.731 |
|
- type: ndcg_at_1000 |
|
value: 27.261999999999997 |
|
- type: ndcg_at_3 |
|
value: 13.74 |
|
- type: ndcg_at_5 |
|
value: 12.199 |
|
- type: precision_at_1 |
|
value: 16.3 |
|
- type: precision_at_10 |
|
value: 7.779999999999999 |
|
- type: precision_at_100 |
|
value: 1.79 |
|
- type: precision_at_1000 |
|
value: 0.313 |
|
- type: precision_at_3 |
|
value: 12.933 |
|
- type: precision_at_5 |
|
value: 10.86 |
|
- type: recall_at_1 |
|
value: 3.313 |
|
- type: recall_at_10 |
|
value: 15.772 |
|
- type: recall_at_100 |
|
value: 36.392 |
|
- type: recall_at_1000 |
|
value: 63.525 |
|
- type: recall_at_3 |
|
value: 7.863 |
|
- type: recall_at_5 |
|
value: 11.003 |
|
- task: |
|
type: PairClassification |
|
dataset: |
|
type: PL-MTEB/sicke-pl-pairclassification |
|
name: MTEB SICK-E-PL |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: cos_sim_accuracy |
|
value: 81.7977986139421 |
|
- type: cos_sim_ap |
|
value: 73.21294750778902 |
|
- type: cos_sim_f1 |
|
value: 66.57391304347826 |
|
- type: cos_sim_precision |
|
value: 65.05778382053025 |
|
- type: cos_sim_recall |
|
value: 68.16239316239316 |
|
- type: dot_accuracy |
|
value: 78.67916836526702 |
|
- type: dot_ap |
|
value: 63.61943815978181 |
|
- type: dot_f1 |
|
value: 62.45014245014245 |
|
- type: dot_precision |
|
value: 52.04178537511871 |
|
- type: dot_recall |
|
value: 78.06267806267806 |
|
- type: euclidean_accuracy |
|
value: 81.7774154097024 |
|
- type: euclidean_ap |
|
value: 73.25053778387148 |
|
- type: euclidean_f1 |
|
value: 66.55064392620953 |
|
- type: euclidean_precision |
|
value: 65.0782845473111 |
|
- type: euclidean_recall |
|
value: 68.09116809116809 |
|
- type: manhattan_accuracy |
|
value: 81.63473298002447 |
|
- type: manhattan_ap |
|
value: 72.99781945530033 |
|
- type: manhattan_f1 |
|
value: 66.3623595505618 |
|
- type: manhattan_precision |
|
value: 65.4432132963989 |
|
- type: manhattan_recall |
|
value: 67.3076923076923 |
|
- type: max_accuracy |
|
value: 81.7977986139421 |
|
- type: max_ap |
|
value: 73.25053778387148 |
|
- type: max_f1 |
|
value: 66.57391304347826 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: PL-MTEB/sickr-pl-sts |
|
name: MTEB SICK-R-PL |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 79.62332929388755 |
|
- type: cos_sim_spearman |
|
value: 73.70598290849304 |
|
- type: euclidean_pearson |
|
value: 77.3603286710006 |
|
- type: euclidean_spearman |
|
value: 73.74420279933932 |
|
- type: manhattan_pearson |
|
value: 77.12735032552482 |
|
- type: manhattan_spearman |
|
value: 73.53014836690127 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/sts22-crosslingual-sts |
|
name: MTEB STS22 (pl) |
|
config: pl |
|
split: test |
|
revision: 6d1ba47164174a496b7fa5d3569dae26a6813b80 |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 37.696942928686724 |
|
- type: cos_sim_spearman |
|
value: 40.6271445245692 |
|
- type: euclidean_pearson |
|
value: 30.212734461370832 |
|
- type: euclidean_spearman |
|
value: 40.66643376699638 |
|
- type: manhattan_pearson |
|
value: 29.90223716230108 |
|
- type: manhattan_spearman |
|
value: 40.35576319091178 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: scifact-pl |
|
name: MTEB SciFact-PL |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 43.528 |
|
- type: map_at_10 |
|
value: 53.290000000000006 |
|
- type: map_at_100 |
|
value: 54.342 |
|
- type: map_at_1000 |
|
value: 54.376999999999995 |
|
- type: map_at_3 |
|
value: 50.651999999999994 |
|
- type: map_at_5 |
|
value: 52.248000000000005 |
|
- type: mrr_at_1 |
|
value: 46.666999999999994 |
|
- type: mrr_at_10 |
|
value: 55.286 |
|
- type: mrr_at_100 |
|
value: 56.094 |
|
- type: mrr_at_1000 |
|
value: 56.125 |
|
- type: mrr_at_3 |
|
value: 53.222 |
|
- type: mrr_at_5 |
|
value: 54.339000000000006 |
|
- type: ndcg_at_1 |
|
value: 46.0 |
|
- type: ndcg_at_10 |
|
value: 58.142 |
|
- type: ndcg_at_100 |
|
value: 62.426 |
|
- type: ndcg_at_1000 |
|
value: 63.395999999999994 |
|
- type: ndcg_at_3 |
|
value: 53.53 |
|
- type: ndcg_at_5 |
|
value: 55.842000000000006 |
|
- type: precision_at_1 |
|
value: 46.0 |
|
- type: precision_at_10 |
|
value: 7.9670000000000005 |
|
- type: precision_at_100 |
|
value: 1.023 |
|
- type: precision_at_1000 |
|
value: 0.11100000000000002 |
|
- type: precision_at_3 |
|
value: 21.444 |
|
- type: precision_at_5 |
|
value: 14.333000000000002 |
|
- type: recall_at_1 |
|
value: 43.528 |
|
- type: recall_at_10 |
|
value: 71.511 |
|
- type: recall_at_100 |
|
value: 89.93299999999999 |
|
- type: recall_at_1000 |
|
value: 97.667 |
|
- type: recall_at_3 |
|
value: 59.067 |
|
- type: recall_at_5 |
|
value: 64.789 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: trec-covid-pl |
|
name: MTEB TRECCOVID-PL |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 0.22699999999999998 |
|
- type: map_at_10 |
|
value: 1.3379999999999999 |
|
- type: map_at_100 |
|
value: 6.965000000000001 |
|
- type: map_at_1000 |
|
value: 17.135 |
|
- type: map_at_3 |
|
value: 0.53 |
|
- type: map_at_5 |
|
value: 0.799 |
|
- type: mrr_at_1 |
|
value: 84.0 |
|
- type: mrr_at_10 |
|
value: 88.083 |
|
- type: mrr_at_100 |
|
value: 88.432 |
|
- type: mrr_at_1000 |
|
value: 88.432 |
|
- type: mrr_at_3 |
|
value: 87.333 |
|
- type: mrr_at_5 |
|
value: 87.833 |
|
- type: ndcg_at_1 |
|
value: 76.0 |
|
- type: ndcg_at_10 |
|
value: 58.199 |
|
- type: ndcg_at_100 |
|
value: 43.230000000000004 |
|
- type: ndcg_at_1000 |
|
value: 39.751 |
|
- type: ndcg_at_3 |
|
value: 63.743 |
|
- type: ndcg_at_5 |
|
value: 60.42999999999999 |
|
- type: precision_at_1 |
|
value: 84.0 |
|
- type: precision_at_10 |
|
value: 62.0 |
|
- type: precision_at_100 |
|
value: 44.519999999999996 |
|
- type: precision_at_1000 |
|
value: 17.746000000000002 |
|
- type: precision_at_3 |
|
value: 67.333 |
|
- type: precision_at_5 |
|
value: 63.2 |
|
- type: recall_at_1 |
|
value: 0.22699999999999998 |
|
- type: recall_at_10 |
|
value: 1.627 |
|
- type: recall_at_100 |
|
value: 10.600999999999999 |
|
- type: recall_at_1000 |
|
value: 37.532 |
|
- type: recall_at_3 |
|
value: 0.547 |
|
- type: recall_at_5 |
|
value: 0.864 |
|
language: pl |
|
license: apache-2.0 |
|
widget: |
|
- source_sentence: "query: Jak dożyć 100 lat?" |
|
sentences: |
|
- "passage: Trzeba zdrowo się odżywiać i uprawiać sport." |
|
- "passage: Trzeba pić alkohol, imprezować i jeździć szybkimi autami." |
|
- "passage: Gdy trwała kampania politycy zapewniali, że rozprawią się z zakazem niedzielnego handlu." |
|
|
|
--- |
|
|
|
<h1 align="center">MMLW-e5-small</h1> |
|
|
|
MMLW (muszę mieć lepszą wiadomość) are neural text encoders for Polish. |
|
This is a distilled model that can be used to generate embeddings applicable to many tasks such as semantic similarity, clustering, information retrieval. The model can also serve as a base for further fine-tuning. |
|
It transforms texts to 384 dimensional vectors. |
|
The model was initialized with multilingual E5 checkpoint, and then trained with [multilingual knowledge distillation method](https://aclanthology.org/2020.emnlp-main.365/) on a diverse corpus of 60 million Polish-English text pairs. We utilised [English FlagEmbeddings (BGE)](https://huggingface.co/BAAI/bge-base-en) as teacher models for distillation. |
|
|
|
## Usage (Sentence-Transformers) |
|
|
|
⚠️ Our embedding models require the use of specific prefixes and suffixes when encoding texts. For this model, queries should be prefixed with **"query: "** and passages with **"passage: "** ⚠️ |
|
|
|
You can use the model like this with [sentence-transformers](https://www.SBERT.net): |
|
|
|
```python |
|
from sentence_transformers import SentenceTransformer |
|
from sentence_transformers.util import cos_sim |
|
|
|
query_prefix = "query: " |
|
answer_prefix = "passage: " |
|
queries = [query_prefix + "Jak dożyć 100 lat?"] |
|
answers = [ |
|
answer_prefix + "Trzeba zdrowo się odżywiać i uprawiać sport.", |
|
answer_prefix + "Trzeba pić alkohol, imprezować i jeździć szybkimi autami.", |
|
answer_prefix + "Gdy trwała kampania politycy zapewniali, że rozprawią się z zakazem niedzielnego handlu." |
|
] |
|
model = SentenceTransformer("sdadas/mmlw-e5-small") |
|
queries_emb = model.encode(queries, convert_to_tensor=True, show_progress_bar=False) |
|
answers_emb = model.encode(answers, convert_to_tensor=True, show_progress_bar=False) |
|
|
|
best_answer = cos_sim(queries_emb, answers_emb).argmax().item() |
|
print(answers[best_answer]) |
|
# Trzeba zdrowo się odżywiać i uprawiać sport. |
|
``` |
|
|
|
## Evaluation Results |
|
|
|
- The model achieves an **Average Score** of **55.84** on the Polish Massive Text Embedding Benchmark (MTEB). See [MTEB Leaderboard](https://huggingface.co/spaces/mteb/leaderboard) for detailed results. |
|
- The model achieves **NDCG@10** of **47.64** on the Polish Information Retrieval Benchmark. See [PIRB Leaderboard](https://huggingface.co/spaces/sdadas/pirb) for detailed results. |
|
|
|
## Acknowledgements |
|
This model was trained with the A100 GPU cluster support delivered by the Gdansk University of Technology within the TASK center initiative. |
|
|
|
## Citation |
|
|
|
```bibtex |
|
@article{dadas2024pirb, |
|
title={{PIRB}: A Comprehensive Benchmark of Polish Dense and Hybrid Text Retrieval Methods}, |
|
author={Sławomir Dadas and Michał Perełkiewicz and Rafał Poświata}, |
|
year={2024}, |
|
eprint={2402.13350}, |
|
archivePrefix={arXiv}, |
|
primaryClass={cs.CL} |
|
} |
|
``` |
