
## E5-mistral-7b-instruct

[Improving Text Embeddings with Large Language Models](https://arxiv.org/pdf/2401.00368.pdf). Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, Furu Wei, arXiv 2024

This model has 32 layers and the embedding size is 4096.

## Usage

Below is an example to encode queries and passages from the MS-MARCO passage ranking dataset.

```python
import torch
import torch.nn.functional as F

from torch import Tensor
from transformers import AutoTokenizer, AutoModel


def last_token_pool(last_hidden_states: Tensor,
                 attention_mask: Tensor) -> Tensor:
    left_padding = (attention_mask[:, -1].sum() == attention_mask.shape[0])
    if left_padding:
        return last_hidden_states[:, -1]
    else:
        sequence_lengths = attention_mask.sum(dim=1) - 1
        batch_size = last_hidden_states.shape[0]
        return last_hidden_states[torch.arange(batch_size, device=last_hidden_states.device), sequence_lengths]


def get_detailed_instruct(task_description: str, query: str) -> str:
    return f'Instruct: {task_description}\nQuery: {query}'


# Each query must come with a one-sentence instruction that describes the task
task = 'Given a web search query, retrieve relevant passages that answer the query'
queries = [
    get_detailed_instruct(task, 'how much protein should a female eat'),
    get_detailed_instruct(task, 'summit define')
]
# No need to add instruction for retrieval documents
documents = [
    "As a general guideline, the CDC's average requirement of protein for women ages 19 to 70 is 46 grams per day. But, as you can see from this chart, you'll need to increase that if you're expecting or training for a marathon. Check out the chart below to see how much protein you should be eating each day.",
    "Definition of summit for English Language Learners. : 1  the highest point of a mountain : the top of a mountain. : 2  the highest level. : 3  a meeting or series of meetings between the leaders of two or more governments."
]
input_texts = queries + documents

tokenizer = AutoTokenizer.from_pretrained('intfloat/e5-mistral-7b-instruct')
model = AutoModel.from_pretrained('intfloat/e5-mistral-7b-instruct')

max_length = 4096
# Tokenize the input texts
batch_dict = tokenizer(input_texts, max_length=max_length - 1, return_attention_mask=False, padding=False, truncation=True)
# append eos_token_id to every input_ids
batch_dict['input_ids'] = [input_ids + [tokenizer.eos_token_id] for input_ids in batch_dict['input_ids']]
batch_dict = tokenizer.pad(batch_dict, padding=True, return_attention_mask=True, return_tensors='pt')

outputs = model(**batch_dict)
embeddings = last_token_pool(outputs.last_hidden_state, batch_dict['attention_mask'])

# normalize embeddings
embeddings = F.normalize(embeddings, p=2, dim=1)
scores = (embeddings[:2] @ embeddings[2:].T) * 100
print(scores.tolist())
```

## Supported Languages

This model is initialized from [Mistral-7B-v0.1](https://huggingface.co/mistralai/Mistral-7B-v0.1)
and fine-tuned on a mixture of multilingual datasets.
As a result, it has some multilingual capability.
However, since Mistral-7B-v0.1 is mainly trained on English data, we recommend using this model for English only.
For multilingual use cases, please refer to [multilingual-e5-large](https://huggingface.co/intfloat/multilingual-e5-large).

## MTEB Benchmark Evaluation

Check out [unilm/e5](https://github.com/microsoft/unilm/tree/master/e5) to reproduce evaluation results 
on the [BEIR](https://arxiv.org/abs/2104.08663) and [MTEB benchmark](https://arxiv.org/abs/2210.07316).

## FAQ

**1. Do I need to add instructions to the query?**

Yes, this is how the model is trained, otherwise you will see a performance degradation.
The task definition should be a one-sentence instruction that describes the task.
This is a way to customize text embeddings for different scenarios through natural language instructions.

Please check out [unilm/e5/utils.py](https://github.com/microsoft/unilm/blob/9c0f1ff7ca53431fe47d2637dfe253643d94185b/e5/utils.py#L106) for instructions we used for evaluation.

On the other hand, there is no need to add instructions to the document side.

**2. Why are my reproduced results slightly different from reported in the model card?**

Different versions of `transformers` and `pytorch` could cause negligible but non-zero performance differences.

**3. Where are the LoRA-only weights?**

You can find the LoRA-only weights at [https://huggingface.co/intfloat/e5-mistral-7b-instruct/tree/main/lora](https://huggingface.co/intfloat/e5-mistral-7b-instruct/tree/main/lora).

## Citation

If you find our paper or models helpful, please consider cite as follows:

```bibtex
@article{wang2023improving,
  title={Improving Text Embeddings with Large Language Models},
  author={Wang, Liang and Yang, Nan and Huang, Xiaolong and Yang, Linjun and Majumder, Rangan and Wei, Furu},
  journal={arXiv preprint arXiv:2401.00368},
  year={2023}
}

@article{wang2022text,
  title={Text Embeddings by Weakly-Supervised Contrastive Pre-training},
  author={Wang, Liang and Yang, Nan and Huang, Xiaolong and Jiao, Binxing and Yang, Linjun and Jiang, Daxin and Majumder, Rangan and Wei, Furu},
  journal={arXiv preprint arXiv:2212.03533},
  year={2022}
}
```

## Limitations

Using this model for inputs longer than 4096 tokens is not recommended.

This model's multilingual capability is still inferior to [multilingual-e5-large](https://huggingface.co/intfloat/multilingual-e5-large) for some cases.


{"language": ["en"], "license": "mit", "tags": ["mteb"], "model-index": [{"name": "e5-mistral-7b-instruct", "results": [{"task": {"type": "STS"}, "dataset": {"name": "MTEB AFQMC", "type": "C-MTEB/AFQMC", "config": "default", "split": "validation", "revision": "None"}, "metrics": [{"type": "cos_sim_pearson", "value": 37.863226091673866}, {"type": "cos_sim_spearman", "value": 38.98733013335281}, {"type": "euclidean_pearson", "value": 37.51783380497874}, {"type": "euclidean_spearman", "value": 38.98733012753365}, {"type": "manhattan_pearson", "value": 37.26706888081721}, {"type": "manhattan_spearman", "value": 38.709750161903834}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB ATEC", "type": "C-MTEB/ATEC", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "cos_sim_pearson", "value": 43.33924583134623}, {"type": "cos_sim_spearman", "value": 42.84316155158754}, {"type": "euclidean_pearson", "value": 45.62709879515238}, {"type": "euclidean_spearman", "value": 42.843155921732404}, {"type": "manhattan_pearson", "value": 45.4786950991229}, {"type": "manhattan_spearman", "value": 42.657334751855984}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB AmazonCounterfactualClassification (en)", "type": "mteb/amazon_counterfactual", "config": "en", "split": "test", "revision": "e8379541af4e31359cca9fbcf4b00f2671dba205"}, "metrics": [{"type": "accuracy", "value": 78.68656716417911}, {"type": "ap", "value": 41.71522322900398}, {"type": "f1", "value": 72.37207703532552}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB AmazonCounterfactualClassification (de)", "type": "mteb/amazon_counterfactual", "config": "de", "split": "test", "revision": "e8379541af4e31359cca9fbcf4b00f2671dba205"}, "metrics": [{"type": "accuracy", "value": 74.04710920770879}, {"type": "ap", "value": 83.42622221864045}, {"type": "f1", "value": 72.14388257905772}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB AmazonCounterfactualClassification (en-ext)", "type": "mteb/amazon_counterfactual", "config": "en-ext", "split": "test", "revision": "e8379541af4e31359cca9fbcf4b00f2671dba205"}, "metrics": [{"type": "accuracy", "value": 77.93103448275862}, {"type": "ap", "value": 26.039284760509513}, {"type": "f1", "value": 64.81092954450712}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB AmazonCounterfactualClassification (ja)", "type": "mteb/amazon_counterfactual", "config": "ja", "split": "test", "revision": "e8379541af4e31359cca9fbcf4b00f2671dba205"}, "metrics": [{"type": "accuracy", "value": 77.21627408993577}, {"type": "ap", "value": 24.876490553983036}, {"type": "f1", "value": 63.8773359684989}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB AmazonPolarityClassification", "type": "mteb/amazon_polarity", "config": "default", "split": "test", "revision": "e2d317d38cd51312af73b3d32a06d1a08b442046"}, "metrics": [{"type": "accuracy", "value": 95.90679999999999}, {"type": "ap", "value": 94.32357863164454}, {"type": "f1", "value": 95.90485634708557}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB AmazonReviewsClassification (en)", "type": "mteb/amazon_reviews_multi", "config": "en", "split": "test", "revision": "1399c76144fd37290681b995c656ef9b2e06e26d"}, "metrics": [{"type": "accuracy", "value": 55.786}, {"type": "f1", "value": 55.31211995815146}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB AmazonReviewsClassification (de)", "type": "mteb/amazon_reviews_multi", "config": "de", "split": "test", "revision": "1399c76144fd37290681b995c656ef9b2e06e26d"}, "metrics": [{"type": "accuracy", "value": 53.26}, {"type": "f1", "value": 52.156230111544986}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB AmazonReviewsClassification (es)", "type": "mteb/amazon_reviews_multi", "config": "es", "split": "test", "revision": "1399c76144fd37290681b995c656ef9b2e06e26d"}, "metrics": [{"type": "accuracy", "value": 50.33}, {"type": "f1", "value": 49.195023008878145}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB AmazonReviewsClassification (fr)", "type": "mteb/amazon_reviews_multi", "config": "fr", "split": "test", "revision": "1399c76144fd37290681b995c656ef9b2e06e26d"}, "metrics": [{"type": "accuracy", "value": 49.3}, {"type": "f1", "value": 48.434470184108}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB AmazonReviewsClassification (ja)", "type": "mteb/amazon_reviews_multi", "config": "ja", "split": "test", "revision": "1399c76144fd37290681b995c656ef9b2e06e26d"}, "metrics": [{"type": "accuracy", "value": 48.68599999999999}, {"type": "f1", "value": 47.62681775202072}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB AmazonReviewsClassification (zh)", "type": "mteb/amazon_reviews_multi", "config": "zh", "split": "test", "revision": "1399c76144fd37290681b995c656ef9b2e06e26d"}, "metrics": [{"type": "accuracy", "value": 46.238}, {"type": "f1", "value": 45.014030559653705}]}, {"task": {"type": "Retrieval"}, "dataset": {"name": "MTEB ArguAna", "type": "arguana", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "map_at_1", "value": 36.486000000000004}, {"type": "map_at_10", "value": 53.076}, {"type": "map_at_100", "value": 53.657999999999994}, {"type": "map_at_1000", "value": 53.659}, {"type": "map_at_3", "value": 48.234}, {"type": "map_at_5", "value": 51.121}, {"type": "mrr_at_1", "value": 37.269000000000005}, {"type": "mrr_at_10", "value": 53.335}, {"type": "mrr_at_100", "value": 53.916}, {"type": "mrr_at_1000", "value": 53.918}, {"type": "mrr_at_3", "value": 48.518}, {"type": "mrr_at_5", "value": 51.406}, {"type": "ndcg_at_1", "value": 36.486000000000004}, {"type": "ndcg_at_10", "value": 61.882000000000005}, {"type": "ndcg_at_100", "value": 64.165}, {"type": "ndcg_at_1000", "value": 64.203}, {"type": "ndcg_at_3", "value": 52.049}, {"type": "ndcg_at_5", "value": 57.199}, {"type": "precision_at_1", "value": 36.486000000000004}, {"type": "precision_at_10", "value": 8.982999999999999}, {"type": "precision_at_100", "value": 0.9939999999999999}, {"type": "precision_at_1000", "value": 0.1}, {"type": "precision_at_3", "value": 21.029}, {"type": "precision_at_5", "value": 15.092}, {"type": "recall_at_1", "value": 36.486000000000004}, {"type": "recall_at_10", "value": 89.82900000000001}, {"type": "recall_at_100", "value": 99.36}, {"type": "recall_at_1000", "value": 99.644}, {"type": "recall_at_3", "value": 63.087}, {"type": "recall_at_5", "value": 75.46199999999999}]}, {"task": {"type": "Clustering"}, "dataset": {"name": "MTEB ArxivClusteringP2P", "type": "mteb/arxiv-clustering-p2p", "config": "default", "split": "test", "revision": "a122ad7f3f0291bf49cc6f4d32aa80929df69d5d"}, "metrics": [{"type": "v_measure", "value": 50.45119266859667}]}, {"task": {"type": "Clustering"}, "dataset": {"name": "MTEB ArxivClusteringS2S", "type": "mteb/arxiv-clustering-s2s", "config": "default", "split": "test", "revision": "f910caf1a6075f7329cdf8c1a6135696f37dbd53"}, "metrics": [{"type": "v_measure", "value": 45.4958298992051}]}, {"task": {"type": "Reranking"}, "dataset": {"name": "MTEB AskUbuntuDupQuestions", "type": "mteb/askubuntudupquestions-reranking", "config": "default", "split": "test", "revision": "2000358ca161889fa9c082cb41daa8dcfb161a54"}, "metrics": [{"type": "map", "value": 66.98177472838887}, {"type": "mrr", "value": 79.91854636591478}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB BIOSSES", "type": "mteb/biosses-sts", "config": "default", "split": "test", "revision": "d3fb88f8f02e40887cd149695127462bbcf29b4a"}, "metrics": [{"type": "cos_sim_pearson", "value": 87.67086498650698}, {"type": "cos_sim_spearman", "value": 85.54773239564638}, {"type": "euclidean_pearson", "value": 86.48229161588425}, {"type": "euclidean_spearman", "value": 85.54773239564638}, {"type": "manhattan_pearson", "value": 86.67533327742343}, {"type": "manhattan_spearman", "value": 85.76099026691983}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB BQ", "type": "C-MTEB/BQ", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "cos_sim_pearson", "value": 50.31998888922809}, {"type": "cos_sim_spearman", "value": 50.6369940530675}, {"type": "euclidean_pearson", "value": 50.055544636296055}, {"type": "euclidean_spearman", "value": 50.63699405154838}, {"type": "manhattan_pearson", "value": 50.00739378036807}, {"type": "manhattan_spearman", "value": 50.607237418676945}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB BUCC (de-en)", "type": "mteb/bucc-bitext-mining", "config": "de-en", "split": "test", "revision": "d51519689f32196a32af33b075a01d0e7c51e252"}, "metrics": [{"type": "accuracy", "value": 99.5615866388309}, {"type": "f1", "value": 99.49895615866389}, {"type": "precision", "value": 99.46764091858039}, {"type": "recall", "value": 99.5615866388309}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB BUCC (fr-en)", "type": "mteb/bucc-bitext-mining", "config": "fr-en", "split": "test", "revision": "d51519689f32196a32af33b075a01d0e7c51e252"}, "metrics": [{"type": "accuracy", "value": 99.19656614571869}, {"type": "f1", "value": 99.08650671362535}, {"type": "precision", "value": 99.0314769975787}, {"type": "recall", "value": 99.19656614571869}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB BUCC (ru-en)", "type": "mteb/bucc-bitext-mining", "config": "ru-en", "split": "test", "revision": "d51519689f32196a32af33b075a01d0e7c51e252"}, "metrics": [{"type": "accuracy", "value": 98.0256321440942}, {"type": "f1", "value": 97.83743216718624}, {"type": "precision", "value": 97.74390947927492}, {"type": "recall", "value": 98.0256321440942}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB BUCC (zh-en)", "type": "mteb/bucc-bitext-mining", "config": "zh-en", "split": "test", "revision": "d51519689f32196a32af33b075a01d0e7c51e252"}, "metrics": [{"type": "accuracy", "value": 99.26276987888363}, {"type": "f1", "value": 99.22766368264}, {"type": "precision", "value": 99.21011058451816}, {"type": "recall", "value": 99.26276987888363}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB Banking77Classification", "type": "mteb/banking77", "config": "default", "split": "test", "revision": "0fd18e25b25c072e09e0d92ab615fda904d66300"}, "metrics": [{"type": "accuracy", "value": 88.22727272727272}, {"type": "f1", "value": 88.17411732496673}]}, {"task": {"type": "Clustering"}, "dataset": {"name": "MTEB BiorxivClusteringP2P", "type": "mteb/biorxiv-clustering-p2p", "config": "default", "split": "test", "revision": "65b79d1d13f80053f67aca9498d9402c2d9f1f40"}, "metrics": [{"type": "v_measure", "value": 43.530637846246975}]}, {"task": {"type": "Clustering"}, "dataset": {"name": "MTEB BiorxivClusteringS2S", "type": "mteb/biorxiv-clustering-s2s", "config": "default", "split": "test", "revision": "258694dd0231531bc1fd9de6ceb52a0853c6d908"}, "metrics": [{"type": "v_measure", "value": 40.23505728593893}]}, {"task": {"type": "Clustering"}, "dataset": {"name": "MTEB CLSClusteringP2P", "type": "C-MTEB/CLSClusteringP2P", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "v_measure", "value": 44.419028279451275}]}, {"task": {"type": "Clustering"}, "dataset": {"name": "MTEB CLSClusteringS2S", "type": "C-MTEB/CLSClusteringS2S", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "v_measure", "value": 42.5820277929776}]}, {"task": {"type": "Reranking"}, "dataset": {"name": "MTEB CMedQAv1", "type": "C-MTEB/CMedQAv1-reranking", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "map", "value": 77.67811726152972}, {"type": "mrr", "value": 80.99003968253969}]}, {"task": {"type": "Reranking"}, "dataset": {"name": "MTEB CMedQAv2", "type": "C-MTEB/CMedQAv2-reranking", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "map", "value": 78.66055354534922}, {"type": "mrr", "value": 81.66119047619047}]}, {"task": {"type": "Retrieval"}, "dataset": {"name": "MTEB CQADupstackRetrieval", "type": "BeIR/cqadupstack", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "map_at_1", "value": 27.162333333333333}, {"type": "map_at_10", "value": 37.22291666666667}, {"type": "map_at_100", "value": 38.56733333333333}, {"type": "map_at_1000", "value": 38.684250000000006}, {"type": "map_at_3", "value": 34.22858333333333}, {"type": "map_at_5", "value": 35.852500000000006}, {"type": "mrr_at_1", "value": 32.459833333333336}, {"type": "mrr_at_10", "value": 41.65358333333333}, {"type": "mrr_at_100", "value": 42.566916666666664}, {"type": "mrr_at_1000", "value": 42.61766666666667}, {"type": "mrr_at_3", "value": 39.210499999999996}, {"type": "mrr_at_5", "value": 40.582166666666666}, {"type": "ndcg_at_1", "value": 32.459833333333336}, {"type": "ndcg_at_10", "value": 42.96758333333333}, {"type": "ndcg_at_100", "value": 48.5065}, {"type": "ndcg_at_1000", "value": 50.556583333333336}, {"type": "ndcg_at_3", "value": 38.004416666666664}, {"type": "ndcg_at_5", "value": 40.25916666666667}, {"type": "precision_at_1", "value": 32.459833333333336}, {"type": "precision_at_10", "value": 7.664583333333333}, {"type": "precision_at_100", "value": 1.2349999999999999}, {"type": "precision_at_1000", "value": 0.15966666666666668}, {"type": "precision_at_3", "value": 17.731166666666663}, {"type": "precision_at_5", "value": 12.575333333333335}, {"type": "recall_at_1", "value": 27.162333333333333}, {"type": "recall_at_10", "value": 55.44158333333334}, {"type": "recall_at_100", "value": 79.56966666666666}, {"type": "recall_at_1000", "value": 93.45224999999999}, {"type": "recall_at_3", "value": 41.433083333333336}, {"type": "recall_at_5", "value": 47.31108333333333}]}, {"task": {"type": "Retrieval"}, "dataset": {"name": "MTEB ClimateFEVER", "type": "climate-fever", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "map_at_1", "value": 16.539}, {"type": "map_at_10", "value": 28.494999999999997}, {"type": "map_at_100", "value": 30.568}, {"type": "map_at_1000", "value": 30.741000000000003}, {"type": "map_at_3", "value": 23.846999999999998}, {"type": "map_at_5", "value": 26.275}, {"type": "mrr_at_1", "value": 37.394}, {"type": "mrr_at_10", "value": 50.068}, {"type": "mrr_at_100", "value": 50.727}, {"type": "mrr_at_1000", "value": 50.751000000000005}, {"type": "mrr_at_3", "value": 46.938}, {"type": "mrr_at_5", "value": 48.818}, {"type": "ndcg_at_1", "value": 37.394}, {"type": "ndcg_at_10", "value": 38.349}, {"type": "ndcg_at_100", "value": 45.512}, {"type": "ndcg_at_1000", "value": 48.321}, {"type": "ndcg_at_3", "value": 32.172}, {"type": "ndcg_at_5", "value": 34.265}, {"type": "precision_at_1", "value": 37.394}, {"type": "precision_at_10", "value": 11.927999999999999}, {"type": "precision_at_100", "value": 1.966}, {"type": "precision_at_1000", "value": 0.25}, {"type": "precision_at_3", "value": 24.126}, {"type": "precision_at_5", "value": 18.306}, {"type": "recall_at_1", "value": 16.539}, {"type": "recall_at_10", "value": 44.504}, {"type": "recall_at_100", "value": 68.605}, {"type": "recall_at_1000", "value": 84.1}, {"type": "recall_at_3", "value": 29.008}, {"type": "recall_at_5", "value": 35.58}]}, {"task": {"type": "Retrieval"}, "dataset": {"name": "MTEB CmedqaRetrieval", "type": "C-MTEB/CmedqaRetrieval", "config": "default", "split": "dev", "revision": "None"}, "metrics": [{"type": "map_at_1", "value": 19.482}, {"type": "map_at_10", "value": 28.622999999999998}, {"type": "map_at_100", "value": 30.262}, {"type": "map_at_1000", "value": 30.432}, {"type": "map_at_3", "value": 25.647}, {"type": "map_at_5", "value": 27.128000000000004}, {"type": "mrr_at_1", "value": 30.408}, {"type": "mrr_at_10", "value": 37.188}, {"type": "mrr_at_100", "value": 38.196000000000005}, {"type": "mrr_at_1000", "value": 38.273}, {"type": "mrr_at_3", "value": 35.067}, {"type": "mrr_at_5", "value": 36.124}, {"type": "ndcg_at_1", "value": 30.408}, {"type": "ndcg_at_10", "value": 34.215}, {"type": "ndcg_at_100", "value": 41.349999999999994}, {"type": "ndcg_at_1000", "value": 44.689}, {"type": "ndcg_at_3", "value": 30.264999999999997}, {"type": "ndcg_at_5", "value": 31.572}, {"type": "precision_at_1", "value": 30.408}, {"type": "precision_at_10", "value": 7.6770000000000005}, {"type": "precision_at_100", "value": 1.352}, {"type": "precision_at_1000", "value": 0.178}, {"type": "precision_at_3", "value": 17.213}, {"type": "precision_at_5", "value": 12.198}, {"type": "recall_at_1", "value": 19.482}, {"type": "recall_at_10", "value": 42.368}, {"type": "recall_at_100", "value": 72.694}, {"type": "recall_at_1000", "value": 95.602}, {"type": "recall_at_3", "value": 30.101}, {"type": "recall_at_5", "value": 34.708}]}, {"task": {"type": "PairClassification"}, "dataset": {"name": "MTEB Cmnli", "type": "C-MTEB/CMNLI", "config": "default", "split": "validation", "revision": "None"}, "metrics": [{"type": "cos_sim_accuracy", "value": 71.16055321707758}, {"type": "cos_sim_ap", "value": 80.21073839711723}, {"type": "cos_sim_f1", "value": 72.9740932642487}, {"type": "cos_sim_precision", "value": 65.53136050623488}, {"type": "cos_sim_recall", "value": 82.3240589198036}, {"type": "dot_accuracy", "value": 71.16055321707758}, {"type": "dot_ap", "value": 80.212299264122}, {"type": "dot_f1", "value": 72.9740932642487}, {"type": "dot_precision", "value": 65.53136050623488}, {"type": "dot_recall", "value": 82.3240589198036}, {"type": "euclidean_accuracy", "value": 71.16055321707758}, {"type": "euclidean_ap", "value": 80.21076298680417}, {"type": "euclidean_f1", "value": 72.9740932642487}, {"type": "euclidean_precision", "value": 65.53136050623488}, {"type": "euclidean_recall", "value": 82.3240589198036}, {"type": "manhattan_accuracy", "value": 70.71557426337944}, {"type": "manhattan_ap", "value": 79.93448977199749}, {"type": "manhattan_f1", "value": 72.83962726826877}, {"type": "manhattan_precision", "value": 62.7407908077053}, {"type": "manhattan_recall", "value": 86.81318681318682}, {"type": "max_accuracy", "value": 71.16055321707758}, {"type": "max_ap", "value": 80.212299264122}, {"type": "max_f1", "value": 72.9740932642487}]}, {"task": {"type": "Retrieval"}, "dataset": {"name": "MTEB CovidRetrieval", "type": "C-MTEB/CovidRetrieval", "config": "default", "split": "dev", "revision": "None"}, "metrics": [{"type": "map_at_1", "value": 60.643}, {"type": "map_at_10", "value": 69.011}, {"type": "map_at_100", "value": 69.533}, {"type": "map_at_1000", "value": 69.545}, {"type": "map_at_3", "value": 67.167}, {"type": "map_at_5", "value": 68.12700000000001}, {"type": "mrr_at_1", "value": 60.801}, {"type": "mrr_at_10", "value": 69.111}, {"type": "mrr_at_100", "value": 69.6}, {"type": "mrr_at_1000", "value": 69.611}, {"type": "mrr_at_3", "value": 67.229}, {"type": "mrr_at_5", "value": 68.214}, {"type": "ndcg_at_1", "value": 60.801}, {"type": "ndcg_at_10", "value": 73.128}, {"type": "ndcg_at_100", "value": 75.614}, {"type": "ndcg_at_1000", "value": 75.92}, {"type": "ndcg_at_3", "value": 69.261}, {"type": "ndcg_at_5", "value": 70.973}, {"type": "precision_at_1", "value": 60.801}, {"type": "precision_at_10", "value": 8.662}, {"type": "precision_at_100", "value": 0.9860000000000001}, {"type": "precision_at_1000", "value": 0.101}, {"type": "precision_at_3", "value": 25.149}, {"type": "precision_at_5", "value": 15.953999999999999}, {"type": "recall_at_1", "value": 60.643}, {"type": "recall_at_10", "value": 85.959}, {"type": "recall_at_100", "value": 97.576}, {"type": "recall_at_1000", "value": 100.0}, {"type": "recall_at_3", "value": 75.184}, {"type": "recall_at_5", "value": 79.32000000000001}]}, {"task": {"type": "Retrieval"}, "dataset": {"name": "MTEB DBPedia", "type": "dbpedia-entity", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "map_at_1", "value": 10.183}, {"type": "map_at_10", "value": 23.958}, {"type": "map_at_100", "value": 34.354}, {"type": "map_at_1000", "value": 36.442}, {"type": "map_at_3", "value": 16.345000000000002}, {"type": "map_at_5", "value": 19.647000000000002}, {"type": "mrr_at_1", "value": 74.25}, {"type": "mrr_at_10", "value": 80.976}, {"type": "mrr_at_100", "value": 81.256}, {"type": "mrr_at_1000", "value": 81.262}, {"type": "mrr_at_3", "value": 79.958}, {"type": "mrr_at_5", "value": 80.37100000000001}, {"type": "ndcg_at_1", "value": 62.0}, {"type": "ndcg_at_10", "value": 48.894999999999996}, {"type": "ndcg_at_100", "value": 53.867}, {"type": "ndcg_at_1000", "value": 61.304}, {"type": "ndcg_at_3", "value": 53.688}, {"type": "ndcg_at_5", "value": 50.900999999999996}, {"type": "precision_at_1", "value": 74.25}, {"type": "precision_at_10", "value": 39.525}, {"type": "precision_at_100", "value": 12.323}, {"type": "precision_at_1000", "value": 2.539}, {"type": "precision_at_3", "value": 57.49999999999999}, {"type": "precision_at_5", "value": 49.1}, {"type": "recall_at_1", "value": 10.183}, {"type": "recall_at_10", "value": 29.296}, {"type": "recall_at_100", "value": 60.394999999999996}, {"type": "recall_at_1000", "value": 83.12}, {"type": "recall_at_3", "value": 17.495}, {"type": "recall_at_5", "value": 22.235}]}, {"task": {"type": "Retrieval"}, "dataset": {"name": "MTEB DuRetrieval", "type": "C-MTEB/DuRetrieval", "config": "default", "split": "dev", "revision": "None"}, "metrics": [{"type": "map_at_1", "value": 26.613999999999997}, {"type": "map_at_10", "value": 79.77300000000001}, {"type": "map_at_100", "value": 82.71}, {"type": "map_at_1000", "value": 82.75}, {"type": "map_at_3", "value": 55.92700000000001}, {"type": "map_at_5", "value": 70.085}, {"type": "mrr_at_1", "value": 90.7}, {"type": "mrr_at_10", "value": 93.438}, {"type": "mrr_at_100", "value": 93.504}, {"type": "mrr_at_1000", "value": 93.50699999999999}, {"type": "mrr_at_3", "value": 93.125}, {"type": "mrr_at_5", "value": 93.34}, {"type": "ndcg_at_1", "value": 90.7}, {"type": "ndcg_at_10", "value": 87.023}, {"type": "ndcg_at_100", "value": 90.068}, {"type": "ndcg_at_1000", "value": 90.43299999999999}, {"type": "ndcg_at_3", "value": 86.339}, {"type": "ndcg_at_5", "value": 85.013}, {"type": "precision_at_1", "value": 90.7}, {"type": "precision_at_10", "value": 41.339999999999996}, {"type": "precision_at_100", "value": 4.806}, {"type": "precision_at_1000", "value": 0.48900000000000005}, {"type": "precision_at_3", "value": 76.983}, {"type": "precision_at_5", "value": 64.69}, {"type": "recall_at_1", "value": 26.613999999999997}, {"type": "recall_at_10", "value": 87.681}, {"type": "recall_at_100", "value": 97.44699999999999}, {"type": "recall_at_1000", "value": 99.348}, {"type": "recall_at_3", "value": 57.809999999999995}, {"type": "recall_at_5", "value": 74.258}]}, {"task": {"type": "Retrieval"}, "dataset": {"name": "MTEB EcomRetrieval", "type": "C-MTEB/EcomRetrieval", "config": "default", "split": "dev", "revision": "None"}, "metrics": [{"type": "map_at_1", "value": 30.9}, {"type": "map_at_10", "value": 40.467}, {"type": "map_at_100", "value": 41.423}, {"type": "map_at_1000", "value": 41.463}, {"type": "map_at_3", "value": 37.25}, {"type": "map_at_5", "value": 39.31}, {"type": "mrr_at_1", "value": 30.9}, {"type": "mrr_at_10", "value": 40.467}, {"type": "mrr_at_100", "value": 41.423}, {"type": "mrr_at_1000", "value": 41.463}, {"type": "mrr_at_3", "value": 37.25}, {"type": "mrr_at_5", "value": 39.31}, {"type": "ndcg_at_1", "value": 30.9}, {"type": "ndcg_at_10", "value": 45.957}, {"type": "ndcg_at_100", "value": 50.735}, {"type": "ndcg_at_1000", "value": 51.861999999999995}, {"type": "ndcg_at_3", "value": 39.437}, {"type": "ndcg_at_5", "value": 43.146}, {"type": "precision_at_1", "value": 30.9}, {"type": "precision_at_10", "value": 6.35}, {"type": "precision_at_100", "value": 0.861}, {"type": "precision_at_1000", "value": 0.095}, {"type": "precision_at_3", "value": 15.267}, {"type": "precision_at_5", "value": 10.96}, {"type": "recall_at_1", "value": 30.9}, {"type": "recall_at_10", "value": 63.5}, {"type": "recall_at_100", "value": 86.1}, {"type": "recall_at_1000", "value": 95.1}, {"type": "recall_at_3", "value": 45.800000000000004}, {"type": "recall_at_5", "value": 54.800000000000004}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB EmotionClassification", "type": "mteb/emotion", "config": "default", "split": "test", "revision": "4f58c6b202a23cf9a4da393831edf4f9183cad37"}, "metrics": [{"type": "accuracy", "value": 49.765}, {"type": "f1", "value": 45.93242203574485}]}, {"task": {"type": "Retrieval"}, "dataset": {"name": "MTEB FEVER", "type": "fever", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "map_at_1", "value": 75.138}, {"type": "map_at_10", "value": 84.21300000000001}, {"type": "map_at_100", "value": 84.43}, {"type": "map_at_1000", "value": 84.441}, {"type": "map_at_3", "value": 83.071}, {"type": "map_at_5", "value": 83.853}, {"type": "mrr_at_1", "value": 80.948}, {"type": "mrr_at_10", "value": 88.175}, {"type": "mrr_at_100", "value": 88.24}, {"type": "mrr_at_1000", "value": 88.241}, {"type": "mrr_at_3", "value": 87.516}, {"type": "mrr_at_5", "value": 87.997}, {"type": "ndcg_at_1", "value": 80.948}, {"type": "ndcg_at_10", "value": 87.84100000000001}, {"type": "ndcg_at_100", "value": 88.576}, {"type": "ndcg_at_1000", "value": 88.75699999999999}, {"type": "ndcg_at_3", "value": 86.176}, {"type": "ndcg_at_5", "value": 87.214}, {"type": "precision_at_1", "value": 80.948}, {"type": "precision_at_10", "value": 10.632}, {"type": "precision_at_100", "value": 1.123}, {"type": "precision_at_1000", "value": 0.11499999999999999}, {"type": "precision_at_3", "value": 33.193}, {"type": "precision_at_5", "value": 20.663}, {"type": "recall_at_1", "value": 75.138}, {"type": "recall_at_10", "value": 94.89699999999999}, {"type": "recall_at_100", "value": 97.751}, {"type": "recall_at_1000", "value": 98.833}, {"type": "recall_at_3", "value": 90.455}, {"type": "recall_at_5", "value": 93.085}]}, {"task": {"type": "Retrieval"}, "dataset": {"name": "MTEB FiQA2018", "type": "fiqa", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "map_at_1", "value": 29.45}, {"type": "map_at_10", "value": 48.596000000000004}, {"type": "map_at_100", "value": 50.70400000000001}, {"type": "map_at_1000", "value": 50.83800000000001}, {"type": "map_at_3", "value": 42.795}, {"type": "map_at_5", "value": 46.085}, {"type": "mrr_at_1", "value": 56.172999999999995}, {"type": "mrr_at_10", "value": 64.35300000000001}, {"type": "mrr_at_100", "value": 64.947}, {"type": "mrr_at_1000", "value": 64.967}, {"type": "mrr_at_3", "value": 62.653999999999996}, {"type": "mrr_at_5", "value": 63.534}, {"type": "ndcg_at_1", "value": 56.172999999999995}, {"type": "ndcg_at_10", "value": 56.593}, {"type": "ndcg_at_100", "value": 62.942}, {"type": "ndcg_at_1000", "value": 64.801}, {"type": "ndcg_at_3", "value": 53.024}, {"type": "ndcg_at_5", "value": 53.986999999999995}, {"type": "precision_at_1", "value": 56.172999999999995}, {"type": "precision_at_10", "value": 15.494}, {"type": "precision_at_100", "value": 2.222}, {"type": "precision_at_1000", "value": 0.254}, {"type": "precision_at_3", "value": 35.185}, {"type": "precision_at_5", "value": 25.556}, {"type": "recall_at_1", "value": 29.45}, {"type": "recall_at_10", "value": 62.882000000000005}, {"type": "recall_at_100", "value": 85.56099999999999}, {"type": "recall_at_1000", "value": 96.539}, {"type": "recall_at_3", "value": 47.911}, {"type": "recall_at_5", "value": 54.52}]}, {"task": {"type": "Retrieval"}, "dataset": {"name": "MTEB HotpotQA", "type": "hotpotqa", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "map_at_1", "value": 39.581}, {"type": "map_at_10", "value": 68.401}, {"type": "map_at_100", "value": 69.207}, {"type": "map_at_1000", "value": 69.25200000000001}, {"type": "map_at_3", "value": 64.689}, {"type": "map_at_5", "value": 67.158}, {"type": "mrr_at_1", "value": 79.163}, {"type": "mrr_at_10", "value": 85.22999999999999}, {"type": "mrr_at_100", "value": 85.386}, {"type": "mrr_at_1000", "value": 85.39099999999999}, {"type": "mrr_at_3", "value": 84.432}, {"type": "mrr_at_5", "value": 84.952}, {"type": "ndcg_at_1", "value": 79.163}, {"type": "ndcg_at_10", "value": 75.721}, {"type": "ndcg_at_100", "value": 78.411}, {"type": "ndcg_at_1000", "value": 79.23599999999999}, {"type": "ndcg_at_3", "value": 70.68799999999999}, {"type": "ndcg_at_5", "value": 73.694}, {"type": "precision_at_1", "value": 79.163}, {"type": "precision_at_10", "value": 16.134}, {"type": "precision_at_100", "value": 1.821}, {"type": "precision_at_1000", "value": 0.193}, {"type": "precision_at_3", "value": 46.446}, {"type": "precision_at_5", "value": 30.242}, {"type": "recall_at_1", "value": 39.581}, {"type": "recall_at_10", "value": 80.66799999999999}, {"type": "recall_at_100", "value": 91.033}, {"type": "recall_at_1000", "value": 96.408}, {"type": "recall_at_3", "value": 69.669}, {"type": "recall_at_5", "value": 75.604}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB IFlyTek", "type": "C-MTEB/IFlyTek-classification", "config": "default", "split": "validation", "revision": "None"}, "metrics": [{"type": "accuracy", "value": 45.04809542131589}, {"type": "f1", "value": 37.01181779071118}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB ImdbClassification", "type": "mteb/imdb", "config": "default", "split": "test", "revision": "3d86128a09e091d6018b6d26cad27f2739fc2db7"}, "metrics": [{"type": "accuracy", "value": 94.78120000000001}, {"type": "ap", "value": 92.52931921594387}, {"type": "f1", "value": 94.77902110732532}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB JDReview", "type": "C-MTEB/JDReview-classification", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "accuracy", "value": 85.81613508442777}, {"type": "ap", "value": 52.430320593468394}, {"type": "f1", "value": 79.95467268178068}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB LCQMC", "type": "C-MTEB/LCQMC", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "cos_sim_pearson", "value": 71.05801751913393}, {"type": "cos_sim_spearman", "value": 75.47954644971965}, {"type": "euclidean_pearson", "value": 74.27472296759713}, {"type": "euclidean_spearman", "value": 75.47954201369866}, {"type": "manhattan_pearson", "value": 74.30508190186474}, {"type": "manhattan_spearman", "value": 75.51326518159436}]}, {"task": {"type": "Reranking"}, "dataset": {"name": "MTEB MMarcoReranking", "type": "C-MTEB/Mmarco-reranking", "config": "default", "split": "dev", "revision": "None"}, "metrics": [{"type": "map", "value": 24.21110921666315}, {"type": "mrr", "value": 22.863492063492064}]}, {"task": {"type": "Retrieval"}, "dataset": {"name": "MTEB MMarcoRetrieval", "type": "C-MTEB/MMarcoRetrieval", "config": "default", "split": "dev", "revision": "None"}, "metrics": [{"type": "map_at_1", "value": 61.38400000000001}, {"type": "map_at_10", "value": 70.895}, {"type": "map_at_100", "value": 71.314}, {"type": "map_at_1000", "value": 71.331}, {"type": "map_at_3", "value": 69.016}, {"type": "map_at_5", "value": 70.179}, {"type": "mrr_at_1", "value": 63.481}, {"type": "mrr_at_10", "value": 71.543}, {"type": "mrr_at_100", "value": 71.91300000000001}, {"type": "mrr_at_1000", "value": 71.928}, {"type": "mrr_at_3", "value": 69.90899999999999}, {"type": "mrr_at_5", "value": 70.907}, {"type": "ndcg_at_1", "value": 63.481}, {"type": "ndcg_at_10", "value": 74.833}, {"type": "ndcg_at_100", "value": 76.705}, {"type": "ndcg_at_1000", "value": 77.13600000000001}, {"type": "ndcg_at_3", "value": 71.236}, {"type": "ndcg_at_5", "value": 73.199}, {"type": "precision_at_1", "value": 63.481}, {"type": "precision_at_10", "value": 9.179}, {"type": "precision_at_100", "value": 1.011}, {"type": "precision_at_1000", "value": 0.105}, {"type": "precision_at_3", "value": 27.044}, {"type": "precision_at_5", "value": 17.272000000000002}, {"type": "recall_at_1", "value": 61.38400000000001}, {"type": "recall_at_10", "value": 86.318}, {"type": "recall_at_100", "value": 94.786}, {"type": "recall_at_1000", "value": 98.14500000000001}, {"type": "recall_at_3", "value": 76.717}, {"type": "recall_at_5", "value": 81.416}]}, {"task": {"type": "Retrieval"}, "dataset": {"name": "MTEB MSMARCO", "type": "msmarco", "config": "default", "split": "dev", "revision": "None"}, "metrics": [{"type": "map_at_1", "value": 23.363999999999997}, {"type": "map_at_10", "value": 36.022}, {"type": "map_at_100", "value": 37.229}, {"type": "map_at_1000", "value": 37.274}, {"type": "map_at_3", "value": 32.131}, {"type": "map_at_5", "value": 34.391}, {"type": "mrr_at_1", "value": 24.069}, {"type": "mrr_at_10", "value": 36.620000000000005}, {"type": "mrr_at_100", "value": 37.769999999999996}, {"type": "mrr_at_1000", "value": 37.809}, {"type": "mrr_at_3", "value": 32.846}, {"type": "mrr_at_5", "value": 35.02}, {"type": "ndcg_at_1", "value": 24.069}, {"type": "ndcg_at_10", "value": 43.056}, {"type": "ndcg_at_100", "value": 48.754}, {"type": "ndcg_at_1000", "value": 49.829}, {"type": "ndcg_at_3", "value": 35.167}, {"type": "ndcg_at_5", "value": 39.168}, {"type": "precision_at_1", "value": 24.069}, {"type": "precision_at_10", "value": 6.762}, {"type": "precision_at_100", "value": 0.96}, {"type": "precision_at_1000", "value": 0.105}, {"type": "precision_at_3", "value": 14.957}, {"type": "precision_at_5", "value": 11.023}, {"type": "recall_at_1", "value": 23.363999999999997}, {"type": "recall_at_10", "value": 64.696}, {"type": "recall_at_100", "value": 90.795}, {"type": "recall_at_1000", "value": 98.892}, {"type": "recall_at_3", "value": 43.247}, {"type": "recall_at_5", "value": 52.86300000000001}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MTOPDomainClassification (en)", "type": "mteb/mtop_domain", "config": "en", "split": "test", "revision": "d80d48c1eb48d3562165c59d59d0034df9fff0bf"}, "metrics": [{"type": "accuracy", "value": 96.11947104423166}, {"type": "f1", "value": 95.89561841159332}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MTOPDomainClassification (de)", "type": "mteb/mtop_domain", "config": "de", "split": "test", "revision": "d80d48c1eb48d3562165c59d59d0034df9fff0bf"}, "metrics": [{"type": "accuracy", "value": 92.97548605240912}, {"type": "f1", "value": 92.17133696717212}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MTOPDomainClassification (es)", "type": "mteb/mtop_domain", "config": "es", "split": "test", "revision": "d80d48c1eb48d3562165c59d59d0034df9fff0bf"}, "metrics": [{"type": "accuracy", "value": 93.37224816544364}, {"type": "f1", "value": 93.19978829237863}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MTOPDomainClassification (fr)", "type": "mteb/mtop_domain", "config": "fr", "split": "test", "revision": "d80d48c1eb48d3562165c59d59d0034df9fff0bf"}, "metrics": [{"type": "accuracy", "value": 91.28719072972127}, {"type": "f1", "value": 91.28448045979604}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MTOPDomainClassification (hi)", "type": "mteb/mtop_domain", "config": "hi", "split": "test", "revision": "d80d48c1eb48d3562165c59d59d0034df9fff0bf"}, "metrics": [{"type": "accuracy", "value": 88.8131946934385}, {"type": "f1", "value": 88.27883019362747}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MTOPDomainClassification (th)", "type": "mteb/mtop_domain", "config": "th", "split": "test", "revision": "d80d48c1eb48d3562165c59d59d0034df9fff0bf"}, "metrics": [{"type": "accuracy", "value": 85.52260397830018}, {"type": "f1", "value": 85.15528226728568}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MTOPIntentClassification (en)", "type": "mteb/mtop_intent", "config": "en", "split": "test", "revision": "ae001d0e6b1228650b7bd1c2c65fb50ad11a8aba"}, "metrics": [{"type": "accuracy", "value": 86.10807113543093}, {"type": "f1", "value": 70.88498219072167}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MTOPIntentClassification (de)", "type": "mteb/mtop_intent", "config": "de", "split": "test", "revision": "ae001d0e6b1228650b7bd1c2c65fb50ad11a8aba"}, "metrics": [{"type": "accuracy", "value": 77.77120315581854}, {"type": "f1", "value": 57.97153920153224}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MTOPIntentClassification (es)", "type": "mteb/mtop_intent", "config": "es", "split": "test", "revision": "ae001d0e6b1228650b7bd1c2c65fb50ad11a8aba"}, "metrics": [{"type": "accuracy", "value": 79.93995997331554}, {"type": "f1", "value": 58.839203810064866}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MTOPIntentClassification (fr)", "type": "mteb/mtop_intent", "config": "fr", "split": "test", "revision": "ae001d0e6b1228650b7bd1c2c65fb50ad11a8aba"}, "metrics": [{"type": "accuracy", "value": 77.801440651425}, {"type": "f1", "value": 58.68009647839332}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MTOPIntentClassification (hi)", "type": "mteb/mtop_intent", "config": "hi", "split": "test", "revision": "ae001d0e6b1228650b7bd1c2c65fb50ad11a8aba"}, "metrics": [{"type": "accuracy", "value": 72.90785227680172}, {"type": "f1", "value": 49.83760954655788}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MTOPIntentClassification (th)", "type": "mteb/mtop_intent", "config": "th", "split": "test", "revision": "ae001d0e6b1228650b7bd1c2c65fb50ad11a8aba"}, "metrics": [{"type": "accuracy", "value": 73.24050632911391}, {"type": "f1", "value": 52.0562553541082}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (af)", "type": "mteb/amazon_massive_intent", "config": "af", "split": "test", "revision": "31efe3c427b0bae9c22cbb560b8f15491cc6bed7"}, "metrics": [{"type": "accuracy", "value": 66.47948890383321}, {"type": "f1", "value": 63.334877563135485}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (am)", "type": "mteb/amazon_massive_intent", "config": "am", "split": "test", "revision": "31efe3c427b0bae9c22cbb560b8f15491cc6bed7"}, "metrics": [{"type": "accuracy", "value": 44.2871553463349}, {"type": "f1", "value": 43.17658050605427}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (ar)", "type": "mteb/amazon_massive_intent", "config": "ar", "split": "test", "revision": "31efe3c427b0bae9c22cbb560b8f15491cc6bed7"}, "metrics": [{"type": "accuracy", "value": 63.174176193678555}, {"type": "f1", "value": 59.236659587042425}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (az)", "type": "mteb/amazon_massive_intent", "config": "az", "split": "test", "revision": "31efe3c427b0bae9c22cbb560b8f15491cc6bed7"}, "metrics": [{"type": "accuracy", "value": 64.226630800269}, {"type": "f1", "value": 60.951842696956184}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (bn)", "type": "mteb/amazon_massive_intent", "config": "bn", "split": "test", "revision": "31efe3c427b0bae9c22cbb560b8f15491cc6bed7"}, "metrics": [{"type": "accuracy", "value": 64.94283792871555}, {"type": "f1", "value": 61.40057652844215}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (cy)", "type": "mteb/amazon_massive_intent", "config": "cy", "split": "test", "revision": "31efe3c427b0bae9c22cbb560b8f15491cc6bed7"}, "metrics": [{"type": "accuracy", "value": 55.480833893745796}, {"type": "f1", "value": 52.5298332072816}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (da)", "type": "mteb/amazon_massive_intent", "config": "da", "split": "test", "revision": "31efe3c427b0bae9c22cbb560b8f15491cc6bed7"}, "metrics": [{"type": "accuracy", "value": 72.52858103564223}, {"type": "f1", "value": 69.3770851919204}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (de)", "type": "mteb/amazon_massive_intent", "config": "de", "split": "test", "revision": "31efe3c427b0bae9c22cbb560b8f15491cc6bed7"}, "metrics": [{"type": "accuracy", "value": 74.09213180901143}, {"type": "f1", "value": 71.13518469365879}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (el)", "type": "mteb/amazon_massive_intent", "config": "el", "split": "test", "revision": "31efe3c427b0bae9c22cbb560b8f15491cc6bed7"}, "metrics": [{"type": "accuracy", "value": 68.31203765971756}, {"type": "f1", "value": 66.05906970865144}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (en)", "type": "mteb/amazon_massive_intent", "config": "en", "split": "test", "revision": "31efe3c427b0bae9c22cbb560b8f15491cc6bed7"}, "metrics": [{"type": "accuracy", "value": 80.57162071284465}, {"type": "f1", "value": 77.7866172598823}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (es)", "type": "mteb/amazon_massive_intent", "config": "es", "split": "test", "revision": "31efe3c427b0bae9c22cbb560b8f15491cc6bed7"}, "metrics": [{"type": "accuracy", "value": 75.09414929388029}, {"type": "f1", "value": 72.5712594833695}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (fa)", "type": "mteb/amazon_massive_intent", "config": "fa", "split": "test", "revision": "31efe3c427b0bae9c22cbb560b8f15491cc6bed7"}, "metrics": [{"type": "accuracy", "value": 72.20914593140553}, {"type": "f1", "value": 68.90619124909186}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (fi)", "type": "mteb/amazon_massive_intent", "config": "fi", "split": "test", "revision": "31efe3c427b0bae9c22cbb560b8f15491cc6bed7"}, "metrics": [{"type": "accuracy", "value": 68.74243443174176}, {"type": "f1", "value": 64.72743141749955}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (fr)", "type": "mteb/amazon_massive_intent", "config": "fr", "split": "test", "revision": "31efe3c427b0bae9c22cbb560b8f15491cc6bed7"}, "metrics": [{"type": "accuracy", "value": 75.11096166778749}, {"type": "f1", "value": 72.61849933064694}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (he)", "type": "mteb/amazon_massive_intent", "config": "he", "split": "test", "revision": "31efe3c427b0bae9c22cbb560b8f15491cc6bed7"}, "metrics": [{"type": "accuracy", "value": 66.22394082044384}, {"type": "f1", "value": 62.43648797607235}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (hi)", "type": "mteb/amazon_massive_intent", "config": "hi", "split": "test", "revision": "31efe3c427b0bae9c22cbb560b8f15491cc6bed7"}, "metrics": [{"type": "accuracy", "value": 69.44855413584399}, {"type": "f1", "value": 66.56851670913659}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (hu)", "type": "mteb/amazon_massive_intent", "config": "hu", "split": "test", "revision": "31efe3c427b0bae9c22cbb560b8f15491cc6bed7"}, "metrics": [{"type": "accuracy", "value": 69.4149293880296}, {"type": "f1", "value": 66.12960877904776}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (hy)", "type": "mteb/amazon_massive_intent", "config": "hy", "split": "test", "revision": "31efe3c427b0bae9c22cbb560b8f15491cc6bed7"}, "metrics": [{"type": "accuracy", "value": 56.916610625420304}, {"type": "f1", "value": 54.02534600927991}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (id)", "type": "mteb/amazon_massive_intent", "config": "id", "split": "test", "revision": "31efe3c427b0bae9c22cbb560b8f15491cc6bed7"}, "metrics": [{"type": "accuracy", "value": 72.71351714862138}, {"type": "f1", "value": 69.70227985126316}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (is)", "type": "mteb/amazon_massive_intent", "config": "is", "split": "test", "revision": "31efe3c427b0bae9c22cbb560b8f15491cc6bed7"}, "metrics": [{"type": "accuracy", "value": 59.91257565568257}, {"type": "f1", "value": 57.06811572144974}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (it)", "type": "mteb/amazon_massive_intent", "config": "it", "split": "test", "revision": "31efe3c427b0bae9c22cbb560b8f15491cc6bed7"}, "metrics": [{"type": "accuracy", "value": 75.25218560860793}, {"type": "f1", "value": 72.48057563104247}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (ja)", "type": "mteb/amazon_massive_intent", "config": "ja", "split": "test", "revision": "31efe3c427b0bae9c22cbb560b8f15491cc6bed7"}, "metrics": [{"type": "accuracy", "value": 76.35507733691998}, {"type": "f1", "value": 73.03024649541128}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (jv)", "type": "mteb/amazon_massive_intent", "config": "jv", "split": "test", "revision": "31efe3c427b0bae9c22cbb560b8f15491cc6bed7"}, "metrics": [{"type": "accuracy", "value": 57.918628110289184}, {"type": "f1", "value": 54.75590124456177}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (ka)", "type": "mteb/amazon_massive_intent", "config": "ka", "split": "test", "revision": "31efe3c427b0bae9c22cbb560b8f15491cc6bed7"}, "metrics": [{"type": "accuracy", "value": 52.548755884330866}, {"type": "f1", "value": 51.5356975360209}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (km)", "type": "mteb/amazon_massive_intent", "config": "km", "split": "test", "revision": "31efe3c427b0bae9c22cbb560b8f15491cc6bed7"}, "metrics": [{"type": "accuracy", "value": 46.44922663080027}, {"type": "f1", "value": 44.561114416830975}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (kn)", "type": "mteb/amazon_massive_intent", "config": "kn", "split": "test", "revision": "31efe3c427b0bae9c22cbb560b8f15491cc6bed7"}, "metrics": [{"type": "accuracy", "value": 53.95763281775386}, {"type": "f1", "value": 50.68367245122476}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (ko)", "type": "mteb/amazon_massive_intent", "config": "ko", "split": "test", "revision": "31efe3c427b0bae9c22cbb560b8f15491cc6bed7"}, "metrics": [{"type": "accuracy", "value": 74.20645595158035}, {"type": "f1", "value": 71.78450093258185}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (lv)", "type": "mteb/amazon_massive_intent", "config": "lv", "split": "test", "revision": "31efe3c427b0bae9c22cbb560b8f15491cc6bed7"}, "metrics": [{"type": "accuracy", "value": 59.226630800269}, {"type": "f1", "value": 57.53988988993337}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (ml)", "type": "mteb/amazon_massive_intent", "config": "ml", "split": "test", "revision": "31efe3c427b0bae9c22cbb560b8f15491cc6bed7"}, "metrics": [{"type": "accuracy", "value": 51.44922663080027}, {"type": "f1", "value": 48.58809018065056}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (mn)", "type": "mteb/amazon_massive_intent", "config": "mn", "split": "test", "revision": "31efe3c427b0bae9c22cbb560b8f15491cc6bed7"}, "metrics": [{"type": "accuracy", "value": 51.3752521856086}, {"type": "f1", "value": 49.91373941436425}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (ms)", "type": "mteb/amazon_massive_intent", "config": "ms", "split": "test", "revision": "31efe3c427b0bae9c22cbb560b8f15491cc6bed7"}, "metrics": [{"type": "accuracy", "value": 69.85205110961668}, {"type": "f1", "value": 67.05660019588582}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (my)", "type": "mteb/amazon_massive_intent", "config": "my", "split": "test", "revision": "31efe3c427b0bae9c22cbb560b8f15491cc6bed7"}, "metrics": [{"type": "accuracy", "value": 49.1492938802959}, {"type": "f1", "value": 46.717578025393195}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (nb)", "type": "mteb/amazon_massive_intent", "config": "nb", "split": "test", "revision": "31efe3c427b0bae9c22cbb560b8f15491cc6bed7"}, "metrics": [{"type": "accuracy", "value": 70.93140551445865}, {"type": "f1", "value": 67.45406609372205}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (nl)", "type": "mteb/amazon_massive_intent", "config": "nl", "split": "test", "revision": "31efe3c427b0bae9c22cbb560b8f15491cc6bed7"}, "metrics": [{"type": "accuracy", "value": 74.82851378614662}, {"type": "f1", "value": 71.15951964393868}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (pl)", "type": "mteb/amazon_massive_intent", "config": "pl", "split": "test", "revision": "31efe3c427b0bae9c22cbb560b8f15491cc6bed7"}, "metrics": [{"type": "accuracy", "value": 74.84868863483524}, {"type": "f1", "value": 71.76056802364877}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (pt)", "type": "mteb/amazon_massive_intent", "config": "pt", "split": "test", "revision": "31efe3c427b0bae9c22cbb560b8f15491cc6bed7"}, "metrics": [{"type": "accuracy", "value": 75.27236045729657}, {"type": "f1", "value": 72.48733090101163}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (ro)", "type": "mteb/amazon_massive_intent", "config": "ro", "split": "test", "revision": "31efe3c427b0bae9c22cbb560b8f15491cc6bed7"}, "metrics": [{"type": "accuracy", "value": 69.63012777404168}, {"type": "f1", "value": 66.56444015346203}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (ru)", "type": "mteb/amazon_massive_intent", "config": "ru", "split": "test", "revision": "31efe3c427b0bae9c22cbb560b8f15491cc6bed7"}, "metrics": [{"type": "accuracy", "value": 76.62743779421655}, {"type": "f1", "value": 73.82720656992142}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (sl)", "type": "mteb/amazon_massive_intent", "config": "sl", "split": "test", "revision": "31efe3c427b0bae9c22cbb560b8f15491cc6bed7"}, "metrics": [{"type": "accuracy", "value": 67.15198386012105}, {"type": "f1", "value": 64.41418309797744}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (sq)", "type": "mteb/amazon_massive_intent", "config": "sq", "split": "test", "revision": "31efe3c427b0bae9c22cbb560b8f15491cc6bed7"}, "metrics": [{"type": "accuracy", "value": 58.8399462004035}, {"type": "f1", "value": 56.050989519693886}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (sv)", "type": "mteb/amazon_massive_intent", "config": "sv", "split": "test", "revision": "31efe3c427b0bae9c22cbb560b8f15491cc6bed7"}, "metrics": [{"type": "accuracy", "value": 73.86684599865501}, {"type": "f1", "value": 70.80682480844303}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (sw)", "type": "mteb/amazon_massive_intent", "config": "sw", "split": "test", "revision": "31efe3c427b0bae9c22cbb560b8f15491cc6bed7"}, "metrics": [{"type": "accuracy", "value": 57.36718224613316}, {"type": "f1", "value": 54.998746471013774}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (ta)", "type": "mteb/amazon_massive_intent", "config": "ta", "split": "test", "revision": "31efe3c427b0bae9c22cbb560b8f15491cc6bed7"}, "metrics": [{"type": "accuracy", "value": 53.150638870208475}, {"type": "f1", "value": 49.79179342620099}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (te)", "type": "mteb/amazon_massive_intent", "config": "te", "split": "test", "revision": "31efe3c427b0bae9c22cbb560b8f15491cc6bed7"}, "metrics": [{"type": "accuracy", "value": 51.50638870208473}, {"type": "f1", "value": 49.778960742003555}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (th)", "type": "mteb/amazon_massive_intent", "config": "th", "split": "test", "revision": "31efe3c427b0bae9c22cbb560b8f15491cc6bed7"}, "metrics": [{"type": "accuracy", "value": 66.906523201076}, {"type": "f1", "value": 66.75784022138245}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (tl)", "type": "mteb/amazon_massive_intent", "config": "tl", "split": "test", "revision": "31efe3c427b0bae9c22cbb560b8f15491cc6bed7"}, "metrics": [{"type": "accuracy", "value": 68.73234700739744}, {"type": "f1", "value": 65.75016141148413}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (tr)", "type": "mteb/amazon_massive_intent", "config": "tr", "split": "test", "revision": "31efe3c427b0bae9c22cbb560b8f15491cc6bed7"}, "metrics": [{"type": "accuracy", "value": 72.06792199058508}, {"type": "f1", "value": 67.90334782594083}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (ur)", "type": "mteb/amazon_massive_intent", "config": "ur", "split": "test", "revision": "31efe3c427b0bae9c22cbb560b8f15491cc6bed7"}, "metrics": [{"type": "accuracy", "value": 62.09145931405515}, {"type": "f1", "value": 58.88703095210731}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (vi)", "type": "mteb/amazon_massive_intent", "config": "vi", "split": "test", "revision": "31efe3c427b0bae9c22cbb560b8f15491cc6bed7"}, "metrics": [{"type": "accuracy", "value": 71.17014122394083}, {"type": "f1", "value": 68.43676277921544}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (zh-CN)", "type": "mteb/amazon_massive_intent", "config": "zh-CN", "split": "test", "revision": "31efe3c427b0bae9c22cbb560b8f15491cc6bed7"}, "metrics": [{"type": "accuracy", "value": 74.99327505043712}, {"type": "f1", "value": 72.26813373392943}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (zh-TW)", "type": "mteb/amazon_massive_intent", "config": "zh-TW", "split": "test", "revision": "31efe3c427b0bae9c22cbb560b8f15491cc6bed7"}, "metrics": [{"type": "accuracy", "value": 71.13987895090787}, {"type": "f1", "value": 70.29309514467575}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (af)", "type": "mteb/amazon_massive_scenario", "config": "af", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 73.37256220578345}, {"type": "f1", "value": 72.56456170538992}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (am)", "type": "mteb/amazon_massive_scenario", "config": "am", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 47.205783456624076}, {"type": "f1", "value": 45.905999859074434}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (ar)", "type": "mteb/amazon_massive_scenario", "config": "ar", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 69.8352387357095}, {"type": "f1", "value": 69.43553987525273}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (az)", "type": "mteb/amazon_massive_scenario", "config": "az", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 67.00403496973773}, {"type": "f1", "value": 65.97477215779143}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (bn)", "type": "mteb/amazon_massive_scenario", "config": "bn", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 68.04976462676531}, {"type": "f1", "value": 67.24581993778398}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (cy)", "type": "mteb/amazon_massive_scenario", "config": "cy", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 61.882985877605925}, {"type": "f1", "value": 59.995293199988794}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (da)", "type": "mteb/amazon_massive_scenario", "config": "da", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 76.75857431069267}, {"type": "f1", "value": 76.52031675299841}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (de)", "type": "mteb/amazon_massive_scenario", "config": "de", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 79.03496973772697}, {"type": "f1", "value": 79.25548063175344}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (el)", "type": "mteb/amazon_massive_scenario", "config": "el", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 72.96570275722931}, {"type": "f1", "value": 72.19110435289122}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (en)", "type": "mteb/amazon_massive_scenario", "config": "en", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 82.38735709482178}, {"type": "f1", "value": 82.34495627619785}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (es)", "type": "mteb/amazon_massive_scenario", "config": "es", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 78.83994620040352}, {"type": "f1", "value": 78.91526355393667}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (fa)", "type": "mteb/amazon_massive_scenario", "config": "fa", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 76.7350369872226}, {"type": "f1", "value": 75.919437344927}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (fi)", "type": "mteb/amazon_massive_scenario", "config": "fi", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 71.21721587088096}, {"type": "f1", "value": 70.82973286243262}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (fr)", "type": "mteb/amazon_massive_scenario", "config": "fr", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 78.59784801613988}, {"type": "f1", "value": 78.47383161087423}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (he)", "type": "mteb/amazon_massive_scenario", "config": "he", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 69.64021519838602}, {"type": "f1", "value": 68.45118053027653}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (hi)", "type": "mteb/amazon_massive_scenario", "config": "hi", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 73.51042367182245}, {"type": "f1", "value": 72.90013022879003}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (hu)", "type": "mteb/amazon_massive_scenario", "config": "hu", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 74.0551445864156}, {"type": "f1", "value": 73.45871761713292}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (hy)", "type": "mteb/amazon_massive_scenario", "config": "hy", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 59.54606590450571}, {"type": "f1", "value": 57.72711794953869}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (id)", "type": "mteb/amazon_massive_scenario", "config": "id", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 77.40753194351042}, {"type": "f1", "value": 76.8157455506521}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (is)", "type": "mteb/amazon_massive_scenario", "config": "is", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 66.58372562205783}, {"type": "f1", "value": 65.2654868709758}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (it)", "type": "mteb/amazon_massive_scenario", "config": "it", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 78.39273705447208}, {"type": "f1", "value": 78.3592956594837}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (ja)", "type": "mteb/amazon_massive_scenario", "config": "ja", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 79.62004034969739}, {"type": "f1", "value": 79.78673754501855}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (jv)", "type": "mteb/amazon_massive_scenario", "config": "jv", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 64.29051782111634}, {"type": "f1", "value": 63.12502587609454}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (ka)", "type": "mteb/amazon_massive_scenario", "config": "ka", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 57.51849361129791}, {"type": "f1", "value": 56.32320906403241}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (km)", "type": "mteb/amazon_massive_scenario", "config": "km", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 52.41761936785474}, {"type": "f1", "value": 49.113762010098306}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (kn)", "type": "mteb/amazon_massive_scenario", "config": "kn", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 58.547410894418284}, {"type": "f1", "value": 56.87580674198118}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (ko)", "type": "mteb/amazon_massive_scenario", "config": "ko", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 78.89038332212507}, {"type": "f1", "value": 79.09210140529848}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (lv)", "type": "mteb/amazon_massive_scenario", "config": "lv", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 63.503698722259585}, {"type": "f1", "value": 61.45718858568352}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (ml)", "type": "mteb/amazon_massive_scenario", "config": "ml", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 54.02824478816408}, {"type": "f1", "value": 52.732738981386504}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (mn)", "type": "mteb/amazon_massive_scenario", "config": "mn", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 54.23671822461331}, {"type": "f1", "value": 52.688080372545286}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (ms)", "type": "mteb/amazon_massive_scenario", "config": "ms", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 75.5312710154674}, {"type": "f1", "value": 74.59368478550698}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (my)", "type": "mteb/amazon_massive_scenario", "config": "my", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 52.192333557498316}, {"type": "f1", "value": 50.18302290152229}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (nb)", "type": "mteb/amazon_massive_scenario", "config": "nb", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 75.6960322797579}, {"type": "f1", "value": 75.25331182714856}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (nl)", "type": "mteb/amazon_massive_scenario", "config": "nl", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 78.47679892400808}, {"type": "f1", "value": 78.24044732352424}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (pl)", "type": "mteb/amazon_massive_scenario", "config": "pl", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 77.36718224613315}, {"type": "f1", "value": 77.2714452985389}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (pt)", "type": "mteb/amazon_massive_scenario", "config": "pt", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 77.96234028244788}, {"type": "f1", "value": 78.21282127011372}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (ro)", "type": "mteb/amazon_massive_scenario", "config": "ro", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 73.19435104236717}, {"type": "f1", "value": 73.1963711292812}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (ru)", "type": "mteb/amazon_massive_scenario", "config": "ru", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 80.52118359112306}, {"type": "f1", "value": 80.4179964390288}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (sl)", "type": "mteb/amazon_massive_scenario", "config": "sl", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 73.65837256220577}, {"type": "f1", "value": 73.07156989634905}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (sq)", "type": "mteb/amazon_massive_scenario", "config": "sq", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 64.02824478816409}, {"type": "f1", "value": 62.972399027713664}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (sv)", "type": "mteb/amazon_massive_scenario", "config": "sv", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 78.87020847343645}, {"type": "f1", "value": 78.224240866849}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (sw)", "type": "mteb/amazon_massive_scenario", "config": "sw", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 64.6570275722932}, {"type": "f1", "value": 63.274871811412545}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (ta)", "type": "mteb/amazon_massive_scenario", "config": "ta", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 57.760591795561524}, {"type": "f1", "value": 56.73711528075771}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (te)", "type": "mteb/amazon_massive_scenario", "config": "te", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 57.26967047747142}, {"type": "f1", "value": 55.74735330863165}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (th)", "type": "mteb/amazon_massive_scenario", "config": "th", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 72.46133154001345}, {"type": "f1", "value": 71.9644168952811}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (tl)", "type": "mteb/amazon_massive_scenario", "config": "tl", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 73.70880968392737}, {"type": "f1", "value": 73.61543141070884}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (tr)", "type": "mteb/amazon_massive_scenario", "config": "tr", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 75.0437121721587}, {"type": "f1", "value": 74.83359868879921}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (ur)", "type": "mteb/amazon_massive_scenario", "config": "ur", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 67.05110961667788}, {"type": "f1", "value": 66.25869819274315}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (vi)", "type": "mteb/amazon_massive_scenario", "config": "vi", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 75.52118359112306}, {"type": "f1", "value": 75.92098546052303}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (zh-CN)", "type": "mteb/amazon_massive_scenario", "config": "zh-CN", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 79.92938802958977}, {"type": "f1", "value": 79.79833572573796}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (zh-TW)", "type": "mteb/amazon_massive_scenario", "config": "zh-TW", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 76.86617350369872}, {"type": "f1", "value": 77.42645654909516}]}, {"task": {"type": "Retrieval"}, "dataset": {"name": "MTEB MedicalRetrieval", "type": "C-MTEB/MedicalRetrieval", "config": "default", "split": "dev", "revision": "None"}, "metrics": [{"type": "map_at_1", "value": 44.6}, {"type": "map_at_10", "value": 50.019000000000005}, {"type": "map_at_100", "value": 50.611}, {"type": "map_at_1000", "value": 50.67}, {"type": "map_at_3", "value": 48.699999999999996}, {"type": "map_at_5", "value": 49.455}, {"type": "mrr_at_1", "value": 44.800000000000004}, {"type": "mrr_at_10", "value": 50.119}, {"type": "mrr_at_100", "value": 50.711}, {"type": "mrr_at_1000", "value": 50.77}, {"type": "mrr_at_3", "value": 48.8}, {"type": "mrr_at_5", "value": 49.555}, {"type": "ndcg_at_1", "value": 44.6}, {"type": "ndcg_at_10", "value": 52.754}, {"type": "ndcg_at_100", "value": 55.935}, {"type": "ndcg_at_1000", "value": 57.607}, {"type": "ndcg_at_3", "value": 50.012}, {"type": "ndcg_at_5", "value": 51.393}, {"type": "precision_at_1", "value": 44.6}, {"type": "precision_at_10", "value": 6.140000000000001}, {"type": "precision_at_100", "value": 0.77}, {"type": "precision_at_1000", "value": 0.09}, {"type": "precision_at_3", "value": 17.933}, {"type": "precision_at_5", "value": 11.44}, {"type": "recall_at_1", "value": 44.6}, {"type": "recall_at_10", "value": 61.4}, {"type": "recall_at_100", "value": 77.0}, {"type": "recall_at_1000", "value": 90.4}, {"type": "recall_at_3", "value": 53.800000000000004}, {"type": "recall_at_5", "value": 57.199999999999996}]}, {"task": {"type": "Clustering"}, "dataset": {"name": "MTEB MedrxivClusteringP2P", "type": "mteb/medrxiv-clustering-p2p", "config": "default", "split": "test", "revision": "e7a26af6f3ae46b30dde8737f02c07b1505bcc73"}, "metrics": [{"type": "v_measure", "value": 38.192667527616315}]}, {"task": {"type": "Clustering"}, "dataset": {"name": "MTEB MedrxivClusteringS2S", "type": "mteb/medrxiv-clustering-s2s", "config": "default", "split": "test", "revision": "35191c8c0dca72d8ff3efcd72aa802307d469663"}, "metrics": [{"type": "v_measure", "value": 37.44738902946689}]}, {"task": {"type": "Reranking"}, "dataset": {"name": "MTEB MindSmallReranking", "type": "mteb/mind_small", "config": "default", "split": "test", "revision": "3bdac13927fdc888b903db93b2ffdbd90b295a69"}, "metrics": [{"type": "map", "value": 32.59661273103955}, {"type": "mrr", "value": 33.82024242497473}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MultilingualSentiment", "type": "C-MTEB/MultilingualSentiment-classification", "config": "default", "split": "validation", "revision": "None"}, "metrics": [{"type": "accuracy", "value": 73.31333333333335}, {"type": "f1", "value": 73.0873466527602}]}, {"task": {"type": "Retrieval"}, "dataset": {"name": "MTEB NFCorpus", "type": "nfcorpus", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "map_at_1", "value": 5.471}, {"type": "map_at_10", "value": 14.142}, {"type": "map_at_100", "value": 18.179000000000002}, {"type": "map_at_1000", "value": 19.772000000000002}, {"type": "map_at_3", "value": 9.716}, {"type": "map_at_5", "value": 11.763}, {"type": "mrr_at_1", "value": 51.393}, {"type": "mrr_at_10", "value": 58.814}, {"type": "mrr_at_100", "value": 59.330000000000005}, {"type": "mrr_at_1000", "value": 59.35}, {"type": "mrr_at_3", "value": 56.398}, {"type": "mrr_at_5", "value": 58.038999999999994}, {"type": "ndcg_at_1", "value": 49.69}, {"type": "ndcg_at_10", "value": 38.615}, {"type": "ndcg_at_100", "value": 35.268}, {"type": "ndcg_at_1000", "value": 43.745}, {"type": "ndcg_at_3", "value": 43.187}, {"type": "ndcg_at_5", "value": 41.528999999999996}, {"type": "precision_at_1", "value": 51.083999999999996}, {"type": "precision_at_10", "value": 29.474}, {"type": "precision_at_100", "value": 9.167}, {"type": "precision_at_1000", "value": 2.2089999999999996}, {"type": "precision_at_3", "value": 40.351}, {"type": "precision_at_5", "value": 36.285000000000004}, {"type": "recall_at_1", "value": 5.471}, {"type": "recall_at_10", "value": 19.242}, {"type": "recall_at_100", "value": 37.14}, {"type": "recall_at_1000", "value": 68.35900000000001}, {"type": "recall_at_3", "value": 10.896}, {"type": "recall_at_5", "value": 14.75}]}, {"task": {"type": "Retrieval"}, "dataset": {"name": "MTEB NQ", "type": "nq", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "map_at_1", "value": 39.499}, {"type": "map_at_10", "value": 55.862}, {"type": "map_at_100", "value": 56.667}, {"type": "map_at_1000", "value": 56.684999999999995}, {"type": "map_at_3", "value": 51.534}, {"type": "map_at_5", "value": 54.2}, {"type": "mrr_at_1", "value": 44.351}, {"type": "mrr_at_10", "value": 58.567}, {"type": "mrr_at_100", "value": 59.099000000000004}, {"type": "mrr_at_1000", "value": 59.109}, {"type": "mrr_at_3", "value": 55.218999999999994}, {"type": "mrr_at_5", "value": 57.391999999999996}, {"type": "ndcg_at_1", "value": 44.322}, {"type": "ndcg_at_10", "value": 63.535}, {"type": "ndcg_at_100", "value": 66.654}, {"type": "ndcg_at_1000", "value": 66.991}, {"type": "ndcg_at_3", "value": 55.701}, {"type": "ndcg_at_5", "value": 60.06700000000001}, {"type": "precision_at_1", "value": 44.322}, {"type": "precision_at_10", "value": 10.026}, {"type": "precision_at_100", "value": 1.18}, {"type": "precision_at_1000", "value": 0.121}, {"type": "precision_at_3", "value": 24.865000000000002}, {"type": "precision_at_5", "value": 17.48}, {"type": "recall_at_1", "value": 39.499}, {"type": "recall_at_10", "value": 84.053}, {"type": "recall_at_100", "value": 97.11}, {"type": "recall_at_1000", "value": 99.493}, {"type": "recall_at_3", "value": 64.091}, {"type": "recall_at_5", "value": 74.063}]}, {"task": {"type": "PairClassification"}, "dataset": {"name": "MTEB Ocnli", "type": "C-MTEB/OCNLI", "config": "default", "split": "validation", "revision": "None"}, "metrics": [{"type": "cos_sim_accuracy", "value": 61.18029236599891}, {"type": "cos_sim_ap", "value": 64.18398769398412}, {"type": "cos_sim_f1", "value": 67.96347757046446}, {"type": "cos_sim_precision", "value": 54.4529262086514}, {"type": "cos_sim_recall", "value": 90.3907074973601}, {"type": "dot_accuracy", "value": 61.18029236599891}, {"type": "dot_ap", "value": 64.18393484706077}, {"type": "dot_f1", "value": 67.96347757046446}, {"type": "dot_precision", "value": 54.4529262086514}, {"type": "dot_recall", "value": 90.3907074973601}, {"type": "euclidean_accuracy", "value": 61.18029236599891}, {"type": "euclidean_ap", "value": 64.18395024821486}, {"type": "euclidean_f1", "value": 67.96347757046446}, {"type": "euclidean_precision", "value": 54.4529262086514}, {"type": "euclidean_recall", "value": 90.3907074973601}, {"type": "manhattan_accuracy", "value": 61.451001624255554}, {"type": "manhattan_ap", "value": 64.38232708763513}, {"type": "manhattan_f1", "value": 68.05860805860804}, {"type": "manhattan_precision", "value": 52.10319685922602}, {"type": "manhattan_recall", "value": 98.09926082365365}, {"type": "max_accuracy", "value": 61.451001624255554}, {"type": "max_ap", "value": 64.38232708763513}, {"type": "max_f1", "value": 68.05860805860804}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB OnlineShopping", "type": "C-MTEB/OnlineShopping-classification", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "accuracy", "value": 92.19000000000001}, {"type": "ap", "value": 89.73918431886767}, {"type": "f1", "value": 92.17175032574507}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB PAWSX", "type": "C-MTEB/PAWSX", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "cos_sim_pearson", "value": 15.079320253752224}, {"type": "cos_sim_spearman", "value": 16.813772504404263}, {"type": "euclidean_pearson", "value": 19.476541162041762}, {"type": "euclidean_spearman", "value": 16.813772498098782}, {"type": "manhattan_pearson", "value": 19.497429832915277}, {"type": "manhattan_spearman", "value": 16.869600674180607}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB QBQTC", "type": "C-MTEB/QBQTC", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "cos_sim_pearson", "value": 30.36139599797913}, {"type": "cos_sim_spearman", "value": 31.80296402851347}, {"type": "euclidean_pearson", "value": 30.10387888252793}, {"type": "euclidean_spearman", "value": 31.80297780103808}, {"type": "manhattan_pearson", "value": 30.86720382849436}, {"type": "manhattan_spearman", "value": 32.70491131366606}]}, {"task": {"type": "Retrieval"}, "dataset": {"name": "MTEB QuoraRetrieval", "type": "quora", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "map_at_1", "value": 71.911}, {"type": "map_at_10", "value": 86.087}, {"type": "map_at_100", "value": 86.701}, {"type": "map_at_1000", "value": 86.715}, {"type": "map_at_3", "value": 83.231}, {"type": "map_at_5", "value": 85.051}, {"type": "mrr_at_1", "value": 82.75}, {"type": "mrr_at_10", "value": 88.759}, {"type": "mrr_at_100", "value": 88.844}, {"type": "mrr_at_1000", "value": 88.844}, {"type": "mrr_at_3", "value": 87.935}, {"type": "mrr_at_5", "value": 88.504}, {"type": "ndcg_at_1", "value": 82.75}, {"type": "ndcg_at_10", "value": 89.605}, {"type": "ndcg_at_100", "value": 90.664}, {"type": "ndcg_at_1000", "value": 90.733}, {"type": "ndcg_at_3", "value": 87.03}, {"type": "ndcg_at_5", "value": 88.473}, {"type": "precision_at_1", "value": 82.75}, {"type": "precision_at_10", "value": 13.575000000000001}, {"type": "precision_at_100", "value": 1.539}, {"type": "precision_at_1000", "value": 0.157}, {"type": "precision_at_3", "value": 38.153}, {"type": "precision_at_5", "value": 25.008000000000003}, {"type": "recall_at_1", "value": 71.911}, {"type": "recall_at_10", "value": 96.261}, {"type": "recall_at_100", "value": 99.72800000000001}, {"type": "recall_at_1000", "value": 99.993}, {"type": "recall_at_3", "value": 88.762}, {"type": "recall_at_5", "value": 92.949}]}, {"task": {"type": "Clustering"}, "dataset": {"name": "MTEB RedditClustering", "type": "mteb/reddit-clustering", "config": "default", "split": "test", "revision": "24640382cdbf8abc73003fb0fa6d111a705499eb"}, "metrics": [{"type": "v_measure", "value": 57.711581165572376}]}, {"task": {"type": "Clustering"}, "dataset": {"name": "MTEB RedditClusteringP2P", "type": "mteb/reddit-clustering-p2p", "config": "default", "split": "test", "revision": "282350215ef01743dc01b456c7f5241fa8937f16"}, "metrics": [{"type": "v_measure", "value": 66.48938885750297}]}, {"task": {"type": "Retrieval"}, "dataset": {"name": "MTEB SCIDOCS", "type": "scidocs", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "map_at_1", "value": 3.7379999999999995}, {"type": "map_at_10", "value": 9.261}, {"type": "map_at_100", "value": 11.001}, {"type": "map_at_1000", "value": 11.262}, {"type": "map_at_3", "value": 6.816}, {"type": "map_at_5", "value": 8.0}, {"type": "mrr_at_1", "value": 18.4}, {"type": "mrr_at_10", "value": 28.755999999999997}, {"type": "mrr_at_100", "value": 29.892000000000003}, {"type": "mrr_at_1000", "value": 29.961}, {"type": "mrr_at_3", "value": 25.467000000000002}, {"type": "mrr_at_5", "value": 27.332}, {"type": "ndcg_at_1", "value": 18.4}, {"type": "ndcg_at_10", "value": 16.296}, {"type": "ndcg_at_100", "value": 23.52}, {"type": "ndcg_at_1000", "value": 28.504}, {"type": "ndcg_at_3", "value": 15.485}, {"type": "ndcg_at_5", "value": 13.471}, {"type": "precision_at_1", "value": 18.4}, {"type": "precision_at_10", "value": 8.469999999999999}, {"type": "precision_at_100", "value": 1.8950000000000002}, {"type": "precision_at_1000", "value": 0.309}, {"type": "precision_at_3", "value": 14.6}, {"type": "precision_at_5", "value": 11.84}, {"type": "recall_at_1", "value": 3.7379999999999995}, {"type": "recall_at_10", "value": 17.185}, {"type": "recall_at_100", "value": 38.397}, {"type": "recall_at_1000", "value": 62.798}, {"type": "recall_at_3", "value": 8.896999999999998}, {"type": "recall_at_5", "value": 12.021999999999998}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB SICK-R", "type": "mteb/sickr-sts", "config": "default", "split": "test", "revision": "a6ea5a8cab320b040a23452cc28066d9beae2cee"}, "metrics": [{"type": "cos_sim_pearson", "value": 86.43977757480083}, {"type": "cos_sim_spearman", "value": 82.64182475199533}, {"type": "euclidean_pearson", "value": 83.71756009999591}, {"type": "euclidean_spearman", "value": 82.64182331395057}, {"type": "manhattan_pearson", "value": 83.8028936913025}, {"type": "manhattan_spearman", "value": 82.71024597804252}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STS12", "type": "mteb/sts12-sts", "config": "default", "split": "test", "revision": "a0d554a64d88156834ff5ae9920b964011b16384"}, "metrics": [{"type": "cos_sim_pearson", "value": 86.85653060698912}, {"type": "cos_sim_spearman", "value": 79.65598885228324}, {"type": "euclidean_pearson", "value": 83.1205137628455}, {"type": "euclidean_spearman", "value": 79.65629387709038}, {"type": "manhattan_pearson", "value": 83.71108853545837}, {"type": "manhattan_spearman", "value": 80.25617619716708}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STS13", "type": "mteb/sts13-sts", "config": "default", "split": "test", "revision": "7e90230a92c190f1bf69ae9002b8cea547a64cca"}, "metrics": [{"type": "cos_sim_pearson", "value": 88.22921688565664}, {"type": "cos_sim_spearman", "value": 88.42662103041957}, {"type": "euclidean_pearson", "value": 87.91679798473325}, {"type": "euclidean_spearman", "value": 88.42662103041957}, {"type": "manhattan_pearson", "value": 88.16927537961303}, {"type": "manhattan_spearman", "value": 88.81581680062541}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STS14", "type": "mteb/sts14-sts", "config": "default", "split": "test", "revision": "6031580fec1f6af667f0bd2da0a551cf4f0b2375"}, "metrics": [{"type": "cos_sim_pearson", "value": 86.77261424554293}, {"type": "cos_sim_spearman", "value": 84.53930146434155}, {"type": "euclidean_pearson", "value": 85.67420491389697}, {"type": "euclidean_spearman", "value": 84.53929771783851}, {"type": "manhattan_pearson", "value": 85.74306784515618}, {"type": "manhattan_spearman", "value": 84.7399304675314}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STS15", "type": "mteb/sts15-sts", "config": "default", "split": "test", "revision": "ae752c7c21bf194d8b67fd573edf7ae58183cbe3"}, "metrics": [{"type": "cos_sim_pearson", "value": 89.86138395166455}, {"type": "cos_sim_spearman", "value": 90.42577823022054}, {"type": "euclidean_pearson", "value": 89.8787763797515}, {"type": "euclidean_spearman", "value": 90.42577823022054}, {"type": "manhattan_pearson", "value": 89.9592937492158}, {"type": "manhattan_spearman", "value": 90.63535505335524}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STS16", "type": "mteb/sts16-sts", "config": "default", "split": "test", "revision": "4d8694f8f0e0100860b497b999b3dbed754a0513"}, "metrics": [{"type": "cos_sim_pearson", "value": 86.5176674585941}, {"type": "cos_sim_spearman", "value": 87.6842917085397}, {"type": "euclidean_pearson", "value": 86.70213081520711}, {"type": "euclidean_spearman", "value": 87.6842917085397}, {"type": "manhattan_pearson", "value": 86.83702628983627}, {"type": "manhattan_spearman", "value": 87.87791000374443}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STS17 (ko-ko)", "type": "mteb/sts17-crosslingual-sts", "config": "ko-ko", "split": "test", "revision": "af5e6fb845001ecf41f4c1e033ce921939a2a68d"}, "metrics": [{"type": "cos_sim_pearson", "value": 83.86395454805867}, {"type": "cos_sim_spearman", "value": 83.69454595252267}, {"type": "euclidean_pearson", "value": 83.04743892608313}, {"type": "euclidean_spearman", "value": 83.69454026433006}, {"type": "manhattan_pearson", "value": 83.4032095553322}, {"type": "manhattan_spearman", "value": 84.11527379013802}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STS17 (ar-ar)", "type": "mteb/sts17-crosslingual-sts", "config": "ar-ar", "split": "test", "revision": "af5e6fb845001ecf41f4c1e033ce921939a2a68d"}, "metrics": [{"type": "cos_sim_pearson", "value": 81.80249894729546}, {"type": "cos_sim_spearman", "value": 81.87004960533409}, {"type": "euclidean_pearson", "value": 80.0392760044179}, {"type": "euclidean_spearman", "value": 81.87004960533409}, {"type": "manhattan_pearson", "value": 80.38096542355912}, {"type": "manhattan_spearman", "value": 82.40774679630341}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STS17 (en-ar)", "type": "mteb/sts17-crosslingual-sts", "config": "en-ar", "split": "test", "revision": "af5e6fb845001ecf41f4c1e033ce921939a2a68d"}, "metrics": [{"type": "cos_sim_pearson", "value": 77.6158201787172}, {"type": "cos_sim_spearman", "value": 77.934651044009}, {"type": "euclidean_pearson", "value": 77.7874683895269}, {"type": "euclidean_spearman", "value": 77.934651044009}, {"type": "manhattan_pearson", "value": 78.36151849193052}, {"type": "manhattan_spearman", "value": 78.52439586349938}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STS17 (en-de)", "type": "mteb/sts17-crosslingual-sts", "config": "en-de", "split": "test", "revision": "af5e6fb845001ecf41f4c1e033ce921939a2a68d"}, "metrics": [{"type": "cos_sim_pearson", "value": 87.04363311392207}, {"type": "cos_sim_spearman", "value": 87.30483659369973}, {"type": "euclidean_pearson", "value": 87.62634489502616}, {"type": "euclidean_spearman", "value": 87.30483659369973}, {"type": "manhattan_pearson", "value": 88.02340837141445}, {"type": "manhattan_spearman", "value": 87.55012003294}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STS17 (en-en)", "type": "mteb/sts17-crosslingual-sts", "config": "en-en", "split": "test", "revision": "af5e6fb845001ecf41f4c1e033ce921939a2a68d"}, "metrics": [{"type": "cos_sim_pearson", "value": 91.69172851958248}, {"type": "cos_sim_spearman", "value": 91.7546879482416}, {"type": "euclidean_pearson", "value": 91.84843039183963}, {"type": "euclidean_spearman", "value": 91.7546879482416}, {"type": "manhattan_pearson", "value": 91.72325753804357}, {"type": "manhattan_spearman", "value": 91.55330259513397}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STS17 (en-tr)", "type": "mteb/sts17-crosslingual-sts", "config": "en-tr", "split": "test", "revision": "af5e6fb845001ecf41f4c1e033ce921939a2a68d"}, "metrics": [{"type": "cos_sim_pearson", "value": 73.95572901084864}, {"type": "cos_sim_spearman", "value": 72.56217821552626}, {"type": "euclidean_pearson", "value": 74.24242980323574}, {"type": "euclidean_spearman", "value": 72.56217821552626}, {"type": "manhattan_pearson", "value": 74.57473362519922}, {"type": "manhattan_spearman", "value": 72.76048826648497}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STS17 (es-en)", "type": "mteb/sts17-crosslingual-sts", "config": "es-en", "split": "test", "revision": "af5e6fb845001ecf41f4c1e033ce921939a2a68d"}, "metrics": [{"type": "cos_sim_pearson", "value": 86.93329396008296}, {"type": "cos_sim_spearman", "value": 88.2406635486219}, {"type": "euclidean_pearson", "value": 87.49687343908533}, {"type": "euclidean_spearman", "value": 88.2406635486219}, {"type": "manhattan_pearson", "value": 88.14088309231084}, {"type": "manhattan_spearman", "value": 88.93314020908534}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STS17 (es-es)", "type": "mteb/sts17-crosslingual-sts", "config": "es-es", "split": "test", "revision": "af5e6fb845001ecf41f4c1e033ce921939a2a68d"}, "metrics": [{"type": "cos_sim_pearson", "value": 88.70124451546057}, {"type": "cos_sim_spearman", "value": 87.45988160052252}, {"type": "euclidean_pearson", "value": 88.44395505247728}, {"type": "euclidean_spearman", "value": 87.45988160052252}, {"type": "manhattan_pearson", "value": 88.69269783495425}, {"type": "manhattan_spearman", "value": 87.65383425621}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STS17 (fr-en)", "type": "mteb/sts17-crosslingual-sts", "config": "fr-en", "split": "test", "revision": "af5e6fb845001ecf41f4c1e033ce921939a2a68d"}, "metrics": [{"type": "cos_sim_pearson", "value": 87.64109149761346}, {"type": "cos_sim_spearman", "value": 88.06459637689733}, {"type": "euclidean_pearson", "value": 88.02313315797703}, {"type": "euclidean_spearman", "value": 88.06459637689733}, {"type": "manhattan_pearson", "value": 88.28328539133253}, {"type": "manhattan_spearman", "value": 88.06605708379142}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STS17 (it-en)", "type": "mteb/sts17-crosslingual-sts", "config": "it-en", "split": "test", "revision": "af5e6fb845001ecf41f4c1e033ce921939a2a68d"}, "metrics": [{"type": "cos_sim_pearson", "value": 88.9040028177525}, {"type": "cos_sim_spearman", "value": 89.68152202933464}, {"type": "euclidean_pearson", "value": 89.23684469601253}, {"type": "euclidean_spearman", "value": 89.68152202933464}, {"type": "manhattan_pearson", "value": 89.59504307277454}, {"type": "manhattan_spearman", "value": 89.88060100313582}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STS17 (nl-en)", "type": "mteb/sts17-crosslingual-sts", "config": "nl-en", "split": "test", "revision": "af5e6fb845001ecf41f4c1e033ce921939a2a68d"}, "metrics": [{"type": "cos_sim_pearson", "value": 87.69891585325125}, {"type": "cos_sim_spearman", "value": 88.25252785071736}, {"type": "euclidean_pearson", "value": 87.99932873748662}, {"type": "euclidean_spearman", "value": 88.25252785071736}, {"type": "manhattan_pearson", "value": 88.26959683009446}, {"type": "manhattan_spearman", "value": 88.32583227300715}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STS22 (en)", "type": "mteb/sts22-crosslingual-sts", "config": "en", "split": "test", "revision": "6d1ba47164174a496b7fa5d3569dae26a6813b80"}, "metrics": [{"type": "cos_sim_pearson", "value": 67.53235909794135}, {"type": "cos_sim_spearman", "value": 66.97521740529574}, {"type": "euclidean_pearson", "value": 68.19502223613912}, {"type": "euclidean_spearman", "value": 66.97521740529574}, {"type": "manhattan_pearson", "value": 68.39070714774539}, {"type": "manhattan_spearman", "value": 67.1072812364868}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STS22 (de)", "type": "mteb/sts22-crosslingual-sts", "config": "de", "split": "test", "revision": "6d1ba47164174a496b7fa5d3569dae26a6813b80"}, "metrics": [{"type": "cos_sim_pearson", "value": 43.715742021204775}, {"type": "cos_sim_spearman", "value": 49.12255971271453}, {"type": "euclidean_pearson", "value": 40.76848562610837}, {"type": "euclidean_spearman", "value": 49.12255971271453}, {"type": "manhattan_pearson", "value": 40.92204625614112}, {"type": "manhattan_spearman", "value": 49.23333793661129}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STS22 (es)", "type": "mteb/sts22-crosslingual-sts", "config": "es", "split": "test", "revision": "6d1ba47164174a496b7fa5d3569dae26a6813b80"}, "metrics": [{"type": "cos_sim_pearson", "value": 63.35268345563588}, {"type": "cos_sim_spearman", "value": 66.99661626042061}, {"type": "euclidean_pearson", "value": 65.85589122857066}, {"type": "euclidean_spearman", "value": 66.99661626042061}, {"type": "manhattan_pearson", "value": 66.78454301512294}, {"type": "manhattan_spearman", "value": 67.17570330149233}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STS22 (pl)", "type": "mteb/sts22-crosslingual-sts", "config": "pl", "split": "test", "revision": "6d1ba47164174a496b7fa5d3569dae26a6813b80"}, "metrics": [{"type": "cos_sim_pearson", "value": 33.36599908204445}, {"type": "cos_sim_spearman", "value": 39.20768331939503}, {"type": "euclidean_pearson", "value": 22.16066769530468}, {"type": "euclidean_spearman", "value": 39.20768331939503}, {"type": "manhattan_pearson", "value": 22.386053195546022}, {"type": "manhattan_spearman", "value": 39.70172817465986}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STS22 (tr)", "type": "mteb/sts22-crosslingual-sts", "config": "tr", "split": "test", "revision": "6d1ba47164174a496b7fa5d3569dae26a6813b80"}, "metrics": [{"type": "cos_sim_pearson", "value": 63.06813956986753}, {"type": "cos_sim_spearman", "value": 68.72065117995668}, {"type": "euclidean_pearson", "value": 66.97373456344194}, {"type": "euclidean_spearman", "value": 68.72065117995668}, {"type": "manhattan_pearson", "value": 67.34907265771595}, {"type": "manhattan_spearman", "value": 68.73705769957843}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STS22 (ar)", "type": "mteb/sts22-crosslingual-sts", "config": "ar", "split": "test", "revision": "6d1ba47164174a496b7fa5d3569dae26a6813b80"}, "metrics": [{"type": "cos_sim_pearson", "value": 47.17664865207108}, {"type": "cos_sim_spearman", "value": 54.115568323148864}, {"type": "euclidean_pearson", "value": 48.56418162879182}, {"type": "euclidean_spearman", "value": 54.115568323148864}, {"type": "manhattan_pearson", "value": 48.85951643453165}, {"type": "manhattan_spearman", "value": 54.13599784169052}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STS22 (ru)", "type": "mteb/sts22-crosslingual-sts", "config": "ru", "split": "test", "revision": "6d1ba47164174a496b7fa5d3569dae26a6813b80"}, "metrics": [{"type": "cos_sim_pearson", "value": 55.87514136275987}, {"type": "cos_sim_spearman", "value": 60.82923573674973}, {"type": "euclidean_pearson", "value": 53.724183308215615}, {"type": "euclidean_spearman", "value": 60.82923573674973}, {"type": "manhattan_pearson", "value": 53.954305573102445}, {"type": "manhattan_spearman", "value": 60.957483900644526}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STS22 (zh)", "type": "mteb/sts22-crosslingual-sts", "config": "zh", "split": "test", "revision": "6d1ba47164174a496b7fa5d3569dae26a6813b80"}, "metrics": [{"type": "cos_sim_pearson", "value": 59.55001413648593}, {"type": "cos_sim_spearman", "value": 63.395777040381276}, {"type": "euclidean_pearson", "value": 59.869972550293305}, {"type": "euclidean_spearman", "value": 63.395777040381276}, {"type": "manhattan_pearson", "value": 61.16195496847885}, {"type": "manhattan_spearman", "value": 63.41968682525581}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STS22 (fr)", "type": "mteb/sts22-crosslingual-sts", "config": "fr", "split": "test", "revision": "6d1ba47164174a496b7fa5d3569dae26a6813b80"}, "metrics": [{"type": "cos_sim_pearson", "value": 79.13334972675852}, {"type": "cos_sim_spearman", "value": 79.86263136371802}, {"type": "euclidean_pearson", "value": 78.2433603592541}, {"type": "euclidean_spearman", "value": 79.86263136371802}, {"type": "manhattan_pearson", "value": 78.87337106318412}, {"type": "manhattan_spearman", "value": 80.31230584758441}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STS22 (de-en)", "type": "mteb/sts22-crosslingual-sts", "config": "de-en", "split": "test", "revision": "6d1ba47164174a496b7fa5d3569dae26a6813b80"}, "metrics": [{"type": "cos_sim_pearson", "value": 63.559700748242356}, {"type": "cos_sim_spearman", "value": 60.92342109509558}, {"type": "euclidean_pearson", "value": 66.07256437521119}, {"type": "euclidean_spearman", "value": 60.92342109509558}, {"type": "manhattan_pearson", "value": 67.72769744612663}, {"type": "manhattan_spearman", "value": 59.64714507774168}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STS22 (es-en)", "type": "mteb/sts22-crosslingual-sts", "config": "es-en", "split": "test", "revision": "6d1ba47164174a496b7fa5d3569dae26a6813b80"}, "metrics": [{"type": "cos_sim_pearson", "value": 73.93491616145891}, {"type": "cos_sim_spearman", "value": 75.84242594400156}, {"type": "euclidean_pearson", "value": 74.87279745626121}, {"type": "euclidean_spearman", "value": 75.84242594400156}, {"type": "manhattan_pearson", "value": 76.47764144677505}, {"type": "manhattan_spearman", "value": 77.08411157845183}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STS22 (it)", "type": "mteb/sts22-crosslingual-sts", "config": "it", "split": "test", "revision": "6d1ba47164174a496b7fa5d3569dae26a6813b80"}, "metrics": [{"type": "cos_sim_pearson", "value": 72.75624124540954}, {"type": "cos_sim_spearman", "value": 75.8667941654703}, {"type": "euclidean_pearson", "value": 73.74314588451925}, {"type": "euclidean_spearman", "value": 75.8667941654703}, {"type": "manhattan_pearson", "value": 73.99641425871518}, {"type": "manhattan_spearman", "value": 76.1982840205817}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STS22 (pl-en)", "type": "mteb/sts22-crosslingual-sts", "config": "pl-en", "split": "test", "revision": "6d1ba47164174a496b7fa5d3569dae26a6813b80"}, "metrics": [{"type": "cos_sim_pearson", "value": 75.20898141298767}, {"type": "cos_sim_spearman", "value": 73.18060375331436}, {"type": "euclidean_pearson", "value": 75.44489280944619}, {"type": "euclidean_spearman", "value": 73.18060375331436}, {"type": "manhattan_pearson", "value": 75.65451039552286}, {"type": "manhattan_spearman", "value": 72.97744006123156}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STS22 (zh-en)", "type": "mteb/sts22-crosslingual-sts", "config": "zh-en", "split": "test", "revision": "6d1ba47164174a496b7fa5d3569dae26a6813b80"}, "metrics": [{"type": "cos_sim_pearson", "value": 72.04278252247816}, {"type": "cos_sim_spearman", "value": 71.8846446821539}, {"type": "euclidean_pearson", "value": 73.16043307050612}, {"type": "euclidean_spearman", "value": 71.8846446821539}, {"type": "manhattan_pearson", "value": 74.76905116839777}, {"type": "manhattan_spearman", "value": 72.66237093518471}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STS22 (es-it)", "type": "mteb/sts22-crosslingual-sts", "config": "es-it", "split": "test", "revision": "6d1ba47164174a496b7fa5d3569dae26a6813b80"}, "metrics": [{"type": "cos_sim_pearson", "value": 71.71033173838558}, {"type": "cos_sim_spearman", "value": 75.043122881885}, {"type": "euclidean_pearson", "value": 72.77579680345087}, {"type": "euclidean_spearman", "value": 75.043122881885}, {"type": "manhattan_pearson", "value": 72.99901534854922}, {"type": "manhattan_spearman", "value": 75.15418335015957}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STS22 (de-fr)", "type": "mteb/sts22-crosslingual-sts", "config": "de-fr", "split": "test", "revision": "6d1ba47164174a496b7fa5d3569dae26a6813b80"}, "metrics": [{"type": "cos_sim_pearson", "value": 55.75733447190482}, {"type": "cos_sim_spearman", "value": 61.38968334176681}, {"type": "euclidean_pearson", "value": 55.479231520643744}, {"type": "euclidean_spearman", "value": 61.38968334176681}, {"type": "manhattan_pearson", "value": 56.05230571465244}, {"type": "manhattan_spearman", "value": 62.69383054007398}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STS22 (de-pl)", "type": "mteb/sts22-crosslingual-sts", "config": "de-pl", "split": "test", "revision": "6d1ba47164174a496b7fa5d3569dae26a6813b80"}, "metrics": [{"type": "cos_sim_pearson", "value": 41.72244325050302}, {"type": "cos_sim_spearman", "value": 54.47476909084119}, {"type": "euclidean_pearson", "value": 43.94629756436873}, {"type": "euclidean_spearman", "value": 54.47476909084119}, {"type": "manhattan_pearson", "value": 46.36533046394657}, {"type": "manhattan_spearman", "value": 54.87509243633636}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STS22 (fr-pl)", "type": "mteb/sts22-crosslingual-sts", "config": "fr-pl", "split": "test", "revision": "6d1ba47164174a496b7fa5d3569dae26a6813b80"}, "metrics": [{"type": "cos_sim_pearson", "value": 70.75183711835146}, {"type": "cos_sim_spearman", "value": 84.51542547285167}, {"type": "euclidean_pearson", "value": 71.84188960126669}, {"type": "euclidean_spearman", "value": 84.51542547285167}, {"type": "manhattan_pearson", "value": 73.94847166379994}, {"type": "manhattan_spearman", "value": 84.51542547285167}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STSB", "type": "C-MTEB/STSB", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "cos_sim_pearson", "value": 81.78690149086131}, {"type": "cos_sim_spearman", "value": 81.81202616916873}, {"type": "euclidean_pearson", "value": 80.98792254251062}, {"type": "euclidean_spearman", "value": 81.81202616916873}, {"type": "manhattan_pearson", "value": 81.46953021346732}, {"type": "manhattan_spearman", "value": 82.34259562492315}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STSBenchmark", "type": "mteb/stsbenchmark-sts", "config": "default", "split": "test", "revision": "b0fddb56ed78048fa8b90373c8a3cfc37b684831"}, "metrics": [{"type": "cos_sim_pearson", "value": 87.68273341294419}, {"type": "cos_sim_spearman", "value": 88.59927164210958}, {"type": "euclidean_pearson", "value": 88.10745681818025}, {"type": "euclidean_spearman", "value": 88.59927164210958}, {"type": "manhattan_pearson", "value": 88.25166703784649}, {"type": "manhattan_spearman", "value": 88.85343247873482}]}, {"task": {"type": "Reranking"}, "dataset": {"name": "MTEB SciDocsRR", "type": "mteb/scidocs-reranking", "config": "default", "split": "test", "revision": "d3c5e1fc0b855ab6097bf1cda04dd73947d7caab"}, "metrics": [{"type": "map", "value": 86.3340463345719}, {"type": "mrr", "value": 96.5182611506141}]}, {"task": {"type": "Retrieval"}, "dataset": {"name": "MTEB SciFact", "type": "scifact", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "map_at_1", "value": 60.967000000000006}, {"type": "map_at_10", "value": 71.873}, {"type": "map_at_100", "value": 72.271}, {"type": "map_at_1000", "value": 72.292}, {"type": "map_at_3", "value": 69.006}, {"type": "map_at_5", "value": 70.856}, {"type": "mrr_at_1", "value": 63.666999999999994}, {"type": "mrr_at_10", "value": 72.929}, {"type": "mrr_at_100", "value": 73.26}, {"type": "mrr_at_1000", "value": 73.282}, {"type": "mrr_at_3", "value": 71.111}, {"type": "mrr_at_5", "value": 72.328}, {"type": "ndcg_at_1", "value": 63.666999999999994}, {"type": "ndcg_at_10", "value": 76.414}, {"type": "ndcg_at_100", "value": 78.152}, {"type": "ndcg_at_1000", "value": 78.604}, {"type": "ndcg_at_3", "value": 71.841}, {"type": "ndcg_at_5", "value": 74.435}, {"type": "precision_at_1", "value": 63.666999999999994}, {"type": "precision_at_10", "value": 10.067}, {"type": "precision_at_100", "value": 1.097}, {"type": "precision_at_1000", "value": 0.11299999999999999}, {"type": "precision_at_3", "value": 27.667}, {"type": "precision_at_5", "value": 18.467}, {"type": "recall_at_1", "value": 60.967000000000006}, {"type": "recall_at_10", "value": 88.922}, {"type": "recall_at_100", "value": 96.667}, {"type": "recall_at_1000", "value": 100.0}, {"type": "recall_at_3", "value": 77.228}, {"type": "recall_at_5", "value": 83.428}]}, {"task": {"type": "PairClassification"}, "dataset": {"name": "MTEB SprintDuplicateQuestions", "type": "mteb/sprintduplicatequestions-pairclassification", "config": "default", "split": "test", "revision": "d66bd1f72af766a5cc4b0ca5e00c162f89e8cc46"}, "metrics": [{"type": "cos_sim_accuracy", "value": 99.82277227722773}, {"type": "cos_sim_ap", "value": 95.66279851444406}, {"type": "cos_sim_f1", "value": 90.9367088607595}, {"type": "cos_sim_precision", "value": 92.1025641025641}, {"type": "cos_sim_recall", "value": 89.8}, {"type": "dot_accuracy", "value": 99.82277227722773}, {"type": "dot_ap", "value": 95.66279851444406}, {"type": "dot_f1", "value": 90.9367088607595}, {"type": "dot_precision", "value": 92.1025641025641}, {"type": "dot_recall", "value": 89.8}, {"type": "euclidean_accuracy", "value": 99.82277227722773}, {"type": "euclidean_ap", "value": 95.66279851444406}, {"type": "euclidean_f1", "value": 90.9367088607595}, {"type": "euclidean_precision", "value": 92.1025641025641}, {"type": "euclidean_recall", "value": 89.8}, {"type": "manhattan_accuracy", "value": 99.82673267326733}, {"type": "manhattan_ap", "value": 95.86094873177069}, {"type": "manhattan_f1", "value": 91.26788357178096}, {"type": "manhattan_precision", "value": 90.06815968841285}, {"type": "manhattan_recall", "value": 92.5}, {"type": "max_accuracy", "value": 99.82673267326733}, {"type": "max_ap", "value": 95.86094873177069}, {"type": "max_f1", "value": 91.26788357178096}]}, {"task": {"type": "Clustering"}, "dataset": {"name": "MTEB StackExchangeClustering", "type": "mteb/stackexchange-clustering", "config": "default", "split": "test", "revision": "6cbc1f7b2bc0622f2e39d2c77fa502909748c259"}, "metrics": [{"type": "v_measure", "value": 73.09533925852372}]}, {"task": {"type": "Clustering"}, "dataset": {"name": "MTEB StackExchangeClusteringP2P", "type": "mteb/stackexchange-clustering-p2p", "config": "default", "split": "test", "revision": "815ca46b2622cec33ccafc3735d572c266efdb44"}, "metrics": [{"type": "v_measure", "value": 45.90745648090035}]}, {"task": {"type": "Reranking"}, "dataset": {"name": "MTEB StackOverflowDupQuestions", "type": "mteb/stackoverflowdupquestions-reranking", "config": "default", "split": "test", "revision": "e185fbe320c72810689fc5848eb6114e1ef5ec69"}, "metrics": [{"type": "map", "value": 54.91147686504404}, {"type": "mrr", "value": 56.03900082760377}]}, {"task": {"type": "Summarization"}, "dataset": {"name": "MTEB SummEval", "type": "mteb/summeval", "config": "default", "split": "test", "revision": "cda12ad7615edc362dbf25a00fdd61d3b1eaf93c"}, "metrics": [{"type": "cos_sim_pearson", "value": 31.46908662038217}, {"type": "cos_sim_spearman", "value": 31.40325730367437}, {"type": "dot_pearson", "value": 31.469083969291894}, {"type": "dot_spearman", "value": 31.40325730367437}]}, {"task": {"type": "Reranking"}, "dataset": {"name": "MTEB T2Reranking", "type": "C-MTEB/T2Reranking", "config": "default", "split": "dev", "revision": "None"}, "metrics": [{"type": "map", "value": 66.90300783402137}, {"type": "mrr", "value": 77.06451972574179}]}, {"task": {"type": "Retrieval"}, "dataset": {"name": "MTEB T2Retrieval", "type": "C-MTEB/T2Retrieval", "config": "default", "split": "dev", "revision": "None"}, "metrics": [{"type": "map_at_1", "value": 25.82}, {"type": "map_at_10", "value": 72.32300000000001}, {"type": "map_at_100", "value": 76.198}, {"type": "map_at_1000", "value": 76.281}, {"type": "map_at_3", "value": 50.719}, {"type": "map_at_5", "value": 62.326}, {"type": "mrr_at_1", "value": 86.599}, {"type": "mrr_at_10", "value": 89.751}, {"type": "mrr_at_100", "value": 89.876}, {"type": "mrr_at_1000", "value": 89.88000000000001}, {"type": "mrr_at_3", "value": 89.151}, {"type": "mrr_at_5", "value": 89.519}, {"type": "ndcg_at_1", "value": 86.599}, {"type": "ndcg_at_10", "value": 80.676}, {"type": "ndcg_at_100", "value": 85.03}, {"type": "ndcg_at_1000", "value": 85.854}, {"type": "ndcg_at_3", "value": 82.057}, {"type": "ndcg_at_5", "value": 80.537}, {"type": "precision_at_1", "value": 86.599}, {"type": "precision_at_10", "value": 40.373}, {"type": "precision_at_100", "value": 4.95}, {"type": "precision_at_1000", "value": 0.514}, {"type": "precision_at_3", "value": 71.918}, {"type": "precision_at_5", "value": 60.246}, {"type": "recall_at_1", "value": 25.82}, {"type": "recall_at_10", "value": 79.905}, {"type": "recall_at_100", "value": 93.88499999999999}, {"type": "recall_at_1000", "value": 98.073}, {"type": "recall_at_3", "value": 52.623}, {"type": "recall_at_5", "value": 66.233}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB TNews", "type": "C-MTEB/TNews-classification", "config": "default", "split": "validation", "revision": "None"}, "metrics": [{"type": "accuracy", "value": 47.050000000000004}, {"type": "f1", "value": 45.704071498353294}]}, {"task": {"type": "Retrieval"}, "dataset": {"name": "MTEB TRECCOVID", "type": "trec-covid", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "map_at_1", "value": 0.243}, {"type": "map_at_10", "value": 2.278}, {"type": "map_at_100", "value": 14.221}, {"type": "map_at_1000", "value": 33.474}, {"type": "map_at_3", "value": 0.7270000000000001}, {"type": "map_at_5", "value": 1.183}, {"type": "mrr_at_1", "value": 94.0}, {"type": "mrr_at_10", "value": 97.0}, {"type": "mrr_at_100", "value": 97.0}, {"type": "mrr_at_1000", "value": 97.0}, {"type": "mrr_at_3", "value": 97.0}, {"type": "mrr_at_5", "value": 97.0}, {"type": "ndcg_at_1", "value": 90.0}, {"type": "ndcg_at_10", "value": 87.249}, {"type": "ndcg_at_100", "value": 67.876}, {"type": "ndcg_at_1000", "value": 59.205}, {"type": "ndcg_at_3", "value": 90.12299999999999}, {"type": "ndcg_at_5", "value": 89.126}, {"type": "precision_at_1", "value": 94.0}, {"type": "precision_at_10", "value": 90.8}, {"type": "precision_at_100", "value": 69.28}, {"type": "precision_at_1000", "value": 25.85}, {"type": "precision_at_3", "value": 94.667}, {"type": "precision_at_5", "value": 92.80000000000001}, {"type": "recall_at_1", "value": 0.243}, {"type": "recall_at_10", "value": 2.392}, {"type": "recall_at_100", "value": 16.982}, {"type": "recall_at_1000", "value": 55.214}, {"type": "recall_at_3", "value": 0.745}, {"type": "recall_at_5", "value": 1.2229999999999999}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (sqi-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "sqi-eng", "split": "test", "revision": "9080400076fbadbb4c4dcb136ff4eddc40b42553"}, "metrics": [{"type": "accuracy", "value": 70.5}, {"type": "f1", "value": 67.05501804646966}, {"type": "precision", "value": 65.73261904761904}, {"type": "recall", "value": 70.5}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (fry-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "fry-eng", "split": "test", "revision": "9080400076fbadbb4c4dcb136ff4eddc40b42553"}, "metrics": [{"type": "accuracy", "value": 75.14450867052022}, {"type": "f1", "value": 70.98265895953759}, {"type": "precision", "value": 69.26782273603082}, {"type": "recall", "value": 75.14450867052022}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (kur-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "kur-eng", "split": "test", "revision": "9080400076fbadbb4c4dcb136ff4eddc40b42553"}, "metrics": [{"type": "accuracy", "value": 33.170731707317074}, {"type": "f1", "value": 29.92876500193573}, {"type": "precision", "value": 28.669145894755648}, {"type": "recall", "value": 33.170731707317074}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (tur-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "tur-eng", "split": "test", "revision": "9080400076fbadbb4c4dcb136ff4eddc40b42553"}, "metrics": [{"type": "accuracy", "value": 95.5}, {"type": "f1", "value": 94.13333333333333}, {"type": "precision", "value": 93.46666666666667}, {"type": "recall", "value": 95.5}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (deu-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "deu-eng", "split": "test", "revision": "9080400076fbadbb4c4dcb136ff4eddc40b42553"}, "metrics": [{"type": "accuracy", "value": 99.6}, {"type": "f1", "value": 99.46666666666665}, {"type": "precision", "value": 99.4}, {"type": "recall", "value": 99.6}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (nld-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "nld-eng", "split": "test", "revision": "9080400076fbadbb4c4dcb136ff4eddc40b42553"}, "metrics": [{"type": "accuracy", "value": 97.2}, {"type": "f1", "value": 96.39999999999999}, {"type": "precision", "value": 96.0}, {"type": "recall", "value": 97.2}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (ron-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "ron-eng", "split": "test", "revision": "9080400076fbadbb4c4dcb136ff4eddc40b42553"}, "metrics": [{"type": "accuracy", "value": 94.5}, {"type": "f1", "value": 92.99666666666667}, {"type": "precision", "value": 92.31666666666666}, {"type": "recall", "value": 94.5}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (ang-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "ang-eng", "split": "test", "revision": "9080400076fbadbb4c4dcb136ff4eddc40b42553"}, "metrics": [{"type": "accuracy", "value": 85.82089552238806}, {"type": "f1", "value": 81.59203980099502}, {"type": "precision", "value": 79.60199004975124}, {"type": "recall", "value": 85.82089552238806}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (ido-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "ido-eng", "split": "test", "revision": "9080400076fbadbb4c4dcb136ff4eddc40b42553"}, "metrics": [{"type": "accuracy", "value": 79.5}, {"type": "f1", "value": 75.11246031746032}, {"type": "precision", "value": 73.38734126984127}, {"type": "recall", "value": 79.5}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (jav-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "jav-eng", "split": "test", "revision": "9080400076fbadbb4c4dcb136ff4eddc40b42553"}, "metrics": [{"type": "accuracy", "value": 44.390243902439025}, {"type": "f1", "value": 38.48896631823461}, {"type": "precision", "value": 36.57220286488579}, {"type": "recall", "value": 44.390243902439025}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (isl-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "isl-eng", "split": "test", "revision": "9080400076fbadbb4c4dcb136ff4eddc40b42553"}, "metrics": [{"type": "accuracy", "value": 90.2}, {"type": "f1", "value": 87.57333333333334}, {"type": "precision", "value": 86.34166666666665}, {"type": "recall", "value": 90.2}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (slv-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "slv-eng", "split": "test", "revision": "9080400076fbadbb4c4dcb136ff4eddc40b42553"}, "metrics": [{"type": "accuracy", "value": 88.82138517618469}, {"type": "f1", "value": 85.98651854423423}, {"type": "precision", "value": 84.79257073424753}, {"type": "recall", "value": 88.82138517618469}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (cym-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "cym-eng", "split": "test", "revision": "9080400076fbadbb4c4dcb136ff4eddc40b42553"}, "metrics": [{"type": "accuracy", "value": 77.04347826086956}, {"type": "f1", "value": 72.32108147606868}, {"type": "precision", "value": 70.37207357859532}, {"type": "recall", "value": 77.04347826086956}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (kaz-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "kaz-eng", "split": "test", "revision": "9080400076fbadbb4c4dcb136ff4eddc40b42553"}, "metrics": [{"type": "accuracy", "value": 53.04347826086957}, {"type": "f1", "value": 46.88868184955141}, {"type": "precision", "value": 44.71730105643149}, {"type": "recall", "value": 53.04347826086957}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (est-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "est-eng", "split": "test", "revision": "9080400076fbadbb4c4dcb136ff4eddc40b42553"}, "metrics": [{"type": "accuracy", "value": 68.0}, {"type": "f1", "value": 62.891813186813195}, {"type": "precision", "value": 61.037906162464985}, {"type": "recall", "value": 68.0}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (heb-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "heb-eng", "split": "test", "revision": "9080400076fbadbb4c4dcb136ff4eddc40b42553"}, "metrics": [{"type": "accuracy", "value": 86.3}, {"type": "f1", "value": 82.82000000000001}, {"type": "precision", "value": 81.25690476190475}, {"type": "recall", "value": 86.3}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (gla-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "gla-eng", "split": "test", "revision": "9080400076fbadbb4c4dcb136ff4eddc40b42553"}, "metrics": [{"type": "accuracy", "value": 68.87816646562122}, {"type": "f1", "value": 63.53054933272062}, {"type": "precision", "value": 61.47807816331196}, {"type": "recall", "value": 68.87816646562122}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (mar-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "mar-eng", "split": "test", "revision": "9080400076fbadbb4c4dcb136ff4eddc40b42553"}, "metrics": [{"type": "accuracy", "value": 74.4}, {"type": "f1", "value": 68.99388888888889}, {"type": "precision", "value": 66.81035714285713}, {"type": "recall", "value": 74.4}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (lat-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "lat-eng", "split": "test", "revision": "9080400076fbadbb4c4dcb136ff4eddc40b42553"}, "metrics": [{"type": "accuracy", "value": 90.5}, {"type": "f1", "value": 87.93666666666667}, {"type": "precision", "value": 86.825}, {"type": "recall", "value": 90.5}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (bel-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "bel-eng", "split": "test", "revision": "9080400076fbadbb4c4dcb136ff4eddc40b42553"}, "metrics": [{"type": "accuracy", "value": 90.7}, {"type": "f1", "value": 88.09}, {"type": "precision", "value": 86.85833333333333}, {"type": "recall", "value": 90.7}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (pms-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "pms-eng", "split": "test", "revision": "9080400076fbadbb4c4dcb136ff4eddc40b42553"}, "metrics": [{"type": "accuracy", "value": 67.61904761904762}, {"type": "f1", "value": 62.30239247214037}, {"type": "precision", "value": 60.340702947845806}, {"type": "recall", "value": 67.61904761904762}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (gle-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "gle-eng", "split": "test", "revision": "9080400076fbadbb4c4dcb136ff4eddc40b42553"}, "metrics": [{"type": "accuracy", "value": 77.9}, {"type": "f1", "value": 73.81285714285714}, {"type": "precision", "value": 72.21570818070818}, {"type": "recall", "value": 77.9}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (pes-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "pes-eng", "split": "test", "revision": "9080400076fbadbb4c4dcb136ff4eddc40b42553"}, "metrics": [{"type": "accuracy", "value": 91.8}, {"type": "f1", "value": 89.66666666666667}, {"type": "precision", "value": 88.66666666666666}, {"type": "recall", "value": 91.8}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (nob-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "nob-eng", "split": "test", "revision": "9080400076fbadbb4c4dcb136ff4eddc40b42553"}, "metrics": [{"type": "accuracy", "value": 97.6}, {"type": "f1", "value": 96.85666666666665}, {"type": "precision", "value": 96.50833333333333}, {"type": "recall", "value": 97.6}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (bul-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "bul-eng", "split": "test", "revision": "9080400076fbadbb4c4dcb136ff4eddc40b42553"}, "metrics": [{"type": "accuracy", "value": 95.39999999999999}, {"type": "f1", "value": 93.98333333333333}, {"type": "precision", "value": 93.30000000000001}, {"type": "recall", "value": 95.39999999999999}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (cbk-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "cbk-eng", "split": "test", "revision": "9080400076fbadbb4c4dcb136ff4eddc40b42553"}, "metrics": [{"type": "accuracy", "value": 85.0}, {"type": "f1", "value": 81.31538461538462}, {"type": "precision", "value": 79.70666666666666}, {"type": "recall", "value": 85.0}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (hun-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "hun-eng", "split": "test", "revision": "9080400076fbadbb4c4dcb136ff4eddc40b42553"}, "metrics": [{"type": "accuracy", "value": 91.60000000000001}, {"type": "f1", "value": 89.81888888888888}, {"type": "precision", "value": 89.08583333333333}, {"type": "recall", "value": 91.60000000000001}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (uig-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "uig-eng", "split": "test", "revision": "9080400076fbadbb4c4dcb136ff4eddc40b42553"}, "metrics": [{"type": "accuracy", "value": 44.3}, {"type": "f1", "value": 38.8623088023088}, {"type": "precision", "value": 37.03755623461505}, {"type": "recall", "value": 44.3}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (rus-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "rus-eng", "split": "test", "revision": "9080400076fbadbb4c4dcb136ff4eddc40b42553"}, "metrics": [{"type": "accuracy", "value": 95.19999999999999}, {"type": "f1", "value": 93.75}, {"type": "precision", "value": 93.05}, {"type": "recall", "value": 95.19999999999999}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (spa-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "spa-eng", "split": "test", "revision": "9080400076fbadbb4c4dcb136ff4eddc40b42553"}, "metrics": [{"type": "accuracy", "value": 99.1}, {"type": "f1", "value": 98.8}, {"type": "precision", "value": 98.65}, {"type": "recall", "value": 99.1}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (hye-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "hye-eng", "split": "test", "revision": "9080400076fbadbb4c4dcb136ff4eddc40b42553"}, "metrics": [{"type": "accuracy", "value": 69.6765498652291}, {"type": "f1", "value": 63.991785393402644}, {"type": "precision", "value": 61.7343729944808}, {"type": "recall", "value": 69.6765498652291}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (tel-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "tel-eng", "split": "test", "revision": "9080400076fbadbb4c4dcb136ff4eddc40b42553"}, "metrics": [{"type": "accuracy", "value": 50.0}, {"type": "f1", "value": 42.79341029341029}, {"type": "precision", "value": 40.25098358431692}, {"type": "recall", "value": 50.0}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (afr-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "afr-eng", "split": "test", "revision": "9080400076fbadbb4c4dcb136ff4eddc40b42553"}, "metrics": [{"type": "accuracy", "value": 89.7}, {"type": "f1", "value": 87.19023809523809}, {"type": "precision", "value": 86.12595238095237}, {"type": "recall", "value": 89.7}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (mon-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "mon-eng", "split": "test", "revision": "9080400076fbadbb4c4dcb136ff4eddc40b42553"}, "metrics": [{"type": "accuracy", "value": 42.72727272727273}, {"type": "f1", "value": 37.78789518562245}, {"type": "precision", "value": 36.24208471267295}, {"type": "recall", "value": 42.72727272727273}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (arz-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "arz-eng", "split": "test", "revision": "9080400076fbadbb4c4dcb136ff4eddc40b42553"}, "metrics": [{"type": "accuracy", "value": 75.26205450733752}, {"type": "f1", "value": 70.72842833849123}, {"type": "precision", "value": 68.93256464011182}, {"type": "recall", "value": 75.26205450733752}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (hrv-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "hrv-eng", "split": "test", "revision": "9080400076fbadbb4c4dcb136ff4eddc40b42553"}, "metrics": [{"type": "accuracy", "value": 95.19999999999999}, {"type": "f1", "value": 93.96666666666668}, {"type": "precision", "value": 93.42}, {"type": "recall", "value": 95.19999999999999}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (nov-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "nov-eng", "split": "test", "revision": "9080400076fbadbb4c4dcb136ff4eddc40b42553"}, "metrics": [{"type": "accuracy", "value": 76.26459143968872}, {"type": "f1", "value": 72.40190419178747}, {"type": "precision", "value": 70.84954604409856}, {"type": "recall", "value": 76.26459143968872}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (gsw-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "gsw-eng", "split": "test", "revision": "9080400076fbadbb4c4dcb136ff4eddc40b42553"}, "metrics": [{"type": "accuracy", "value": 59.82905982905983}, {"type": "f1", "value": 52.2100122100122}, {"type": "precision", "value": 49.52516619183286}, {"type": "recall", "value": 59.82905982905983}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (nds-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "nds-eng", "split": "test", "revision": "9080400076fbadbb4c4dcb136ff4eddc40b42553"}, "metrics": [{"type": "accuracy", "value": 81.69999999999999}, {"type": "f1", "value": 77.41714285714286}, {"type": "precision", "value": 75.64833333333334}, {"type": "recall", "value": 81.69999999999999}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (ukr-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "ukr-eng", "split": "test", "revision": "9080400076fbadbb4c4dcb136ff4eddc40b42553"}, "metrics": [{"type": "accuracy", "value": 95.5}, {"type": "f1", "value": 94.45}, {"type": "precision", "value": 93.93333333333334}, {"type": "recall", "value": 95.5}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (uzb-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "uzb-eng", "split": "test", "revision": "9080400076fbadbb4c4dcb136ff4eddc40b42553"}, "metrics": [{"type": "accuracy", "value": 58.41121495327103}, {"type": "f1", "value": 52.73495974430554}, {"type": "precision", "value": 50.717067200712066}, {"type": "recall", "value": 58.41121495327103}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (lit-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "lit-eng", "split": "test", "revision": "9080400076fbadbb4c4dcb136ff4eddc40b42553"}, "metrics": [{"type": "accuracy", "value": 73.3}, {"type": "f1", "value": 69.20371794871795}, {"type": "precision", "value": 67.6597557997558}, {"type": "recall", "value": 73.3}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (ina-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "ina-eng", "split": "test", "revision": "9080400076fbadbb4c4dcb136ff4eddc40b42553"}, "metrics": [{"type": "accuracy", "value": 96.5}, {"type": "f1", "value": 95.51666666666667}, {"type": "precision", "value": 95.05}, {"type": "recall", "value": 96.5}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (lfn-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "lfn-eng", "split": "test", "revision": "9080400076fbadbb4c4dcb136ff4eddc40b42553"}, "metrics": [{"type": "accuracy", "value": 78.4}, {"type": "f1", "value": 73.88856643356644}, {"type": "precision", "value": 72.01373015873016}, {"type": "recall", "value": 78.4}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (zsm-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "zsm-eng", "split": "test", "revision": "9080400076fbadbb4c4dcb136ff4eddc40b42553"}, "metrics": [{"type": "accuracy", "value": 95.3}, {"type": "f1", "value": 94.09666666666668}, {"type": "precision", "value": 93.53333333333332}, {"type": "recall", "value": 95.3}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (ita-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "ita-eng", "split": "test", "revision": "9080400076fbadbb4c4dcb136ff4eddc40b42553"}, "metrics": [{"type": "accuracy", "value": 93.7}, {"type": "f1", "value": 91.94}, {"type": "precision", "value": 91.10833333333333}, {"type": "recall", "value": 93.7}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (cmn-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "cmn-eng", "split": "test", "revision": "9080400076fbadbb4c4dcb136ff4eddc40b42553"}, "metrics": [{"type": "accuracy", "value": 96.8}, {"type": "f1", "value": 95.89999999999999}, {"type": "precision", "value": 95.46666666666668}, {"type": "recall", "value": 96.8}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (lvs-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "lvs-eng", "split": "test", "revision": "9080400076fbadbb4c4dcb136ff4eddc40b42553"}, "metrics": [{"type": "accuracy", "value": 70.5}, {"type": "f1", "value": 66.00635642135641}, {"type": "precision", "value": 64.36345238095238}, {"type": "recall", "value": 70.5}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (glg-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "glg-eng", "split": "test", "revision": "9080400076fbadbb4c4dcb136ff4eddc40b42553"}, "metrics": [{"type": "accuracy", "value": 92.4}, {"type": "f1", "value": 90.44388888888889}, {"type": "precision", "value": 89.5767857142857}, {"type": "recall", "value": 92.4}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (ceb-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "ceb-eng", "split": "test", "revision": "9080400076fbadbb4c4dcb136ff4eddc40b42553"}, "metrics": [{"type": "accuracy", "value": 48.0}, {"type": "f1", "value": 43.15372775372776}, {"type": "precision", "value": 41.53152510162313}, {"type": "recall", "value": 48.0}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (bre-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "bre-eng", "split": "test", "revision": "9080400076fbadbb4c4dcb136ff4eddc40b42553"}, "metrics": [{"type": "accuracy", "value": 16.7}, {"type": "f1", "value": 14.198431372549017}, {"type": "precision", "value": 13.411765873015872}, {"type": "recall", "value": 16.7}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (ben-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "ben-eng", "split": "test", "revision": "9080400076fbadbb4c4dcb136ff4eddc40b42553"}, "metrics": [{"type": "accuracy", "value": 85.7}, {"type": "f1", "value": 81.81666666666666}, {"type": "precision", "value": 80.10833333333332}, {"type": "recall", "value": 85.7}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (swg-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "swg-eng", "split": "test", "revision": "9080400076fbadbb4c4dcb136ff4eddc40b42553"}, "metrics": [{"type": "accuracy", "value": 69.64285714285714}, {"type": "f1", "value": 64.745670995671}, {"type": "precision", "value": 62.916666666666664}, {"type": "recall", "value": 69.64285714285714}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (arq-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "arq-eng", "split": "test", "revision": "9080400076fbadbb4c4dcb136ff4eddc40b42553"}, "metrics": [{"type": "accuracy", "value": 54.665203073545555}, {"type": "f1", "value": 48.55366630916923}, {"type": "precision", "value": 46.35683318998357}, {"type": "recall", "value": 54.665203073545555}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (kab-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "kab-eng", "split": "test", "revision": "9080400076fbadbb4c4dcb136ff4eddc40b42553"}, "metrics": [{"type": "accuracy", "value": 4.8}, {"type": "f1", "value": 3.808587223587223}, {"type": "precision", "value": 3.5653174603174604}, {"type": "recall", "value": 4.8}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (fra-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "fra-eng", "split": "test", "revision": "9080400076fbadbb4c4dcb136ff4eddc40b42553"}, "metrics": [{"type": "accuracy", "value": 96.6}, {"type": "f1", "value": 95.77333333333333}, {"type": "precision", "value": 95.39166666666667}, {"type": "recall", "value": 96.6}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (por-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "por-eng", "split": "test", "revision": "9080400076fbadbb4c4dcb136ff4eddc40b42553"}, "metrics": [{"type": "accuracy", "value": 95.39999999999999}, {"type": "f1", "value": 94.44}, {"type": "precision", "value": 93.975}, {"type": "recall", "value": 95.39999999999999}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (tat-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "tat-eng", "split": "test", "revision": "9080400076fbadbb4c4dcb136ff4eddc40b42553"}, "metrics": [{"type": "accuracy", "value": 42.0}, {"type": "f1", "value": 37.024908424908425}, {"type": "precision", "value": 35.365992063492065}, {"type": "recall", "value": 42.0}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (oci-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "oci-eng", "split": "test", "revision": "9080400076fbadbb4c4dcb136ff4eddc40b42553"}, "metrics": [{"type": "accuracy", "value": 66.7}, {"type": "f1", "value": 62.20460835058661}, {"type": "precision", "value": 60.590134587634594}, {"type": "recall", "value": 66.7}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (pol-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "pol-eng", "split": "test", "revision": "9080400076fbadbb4c4dcb136ff4eddc40b42553"}, "metrics": [{"type": "accuracy", "value": 97.3}, {"type": "f1", "value": 96.46666666666667}, {"type": "precision", "value": 96.06666666666668}, {"type": "recall", "value": 97.3}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (war-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "war-eng", "split": "test", "revision": "9080400076fbadbb4c4dcb136ff4eddc40b42553"}, "metrics": [{"type": "accuracy", "value": 47.3}, {"type": "f1", "value": 41.96905408317173}, {"type": "precision", "value": 40.18741402116402}, {"type": "recall", "value": 47.3}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (aze-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "aze-eng", "split": "test", "revision": "9080400076fbadbb4c4dcb136ff4eddc40b42553"}, "metrics": [{"type": "accuracy", "value": 80.2}, {"type": "f1", "value": 76.22690476190476}, {"type": "precision", "value": 74.63539682539682}, {"type": "recall", "value": 80.2}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (vie-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "vie-eng", "split": "test", "revision": "9080400076fbadbb4c4dcb136ff4eddc40b42553"}, "metrics": [{"type": "accuracy", "value": 96.0}, {"type": "f1", "value": 94.83333333333333}, {"type": "precision", "value": 94.26666666666668}, {"type": "recall", "value": 96.0}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (nno-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "nno-eng", "split": "test", "revision": "9080400076fbadbb4c4dcb136ff4eddc40b42553"}, "metrics": [{"type": "accuracy", "value": 89.7}, {"type": "f1", "value": 87.24333333333334}, {"type": "precision", "value": 86.17}, {"type": "recall", "value": 89.7}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (cha-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "cha-eng", "split": "test", "revision": "9080400076fbadbb4c4dcb136ff4eddc40b42553"}, "metrics": [{"type": "accuracy", "value": 50.36496350364964}, {"type": "f1", "value": 44.795520780922246}, {"type": "precision", "value": 43.09002433090024}, {"type": "recall", "value": 50.36496350364964}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (mhr-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "mhr-eng", "split": "test", "revision": "9080400076fbadbb4c4dcb136ff4eddc40b42553"}, "metrics": [{"type": "accuracy", "value": 18.8}, {"type": "f1", "value": 16.242864357864356}, {"type": "precision", "value": 15.466596638655464}, {"type": "recall", "value": 18.8}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (dan-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "dan-eng", "split": "test", "revision": "9080400076fbadbb4c4dcb136ff4eddc40b42553"}, "metrics": [{"type": "accuracy", "value": 95.19999999999999}, {"type": "f1", "value": 93.92333333333333}, {"type": "precision", "value": 93.30833333333332}, {"type": "recall", "value": 95.19999999999999}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (ell-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "ell-eng", "split": "test", "revision": "9080400076fbadbb4c4dcb136ff4eddc40b42553"}, "metrics": [{"type": "accuracy", "value": 93.4}, {"type": "f1", "value": 91.42333333333333}, {"type": "precision", "value": 90.50833333333334}, {"type": "recall", "value": 93.4}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (amh-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "amh-eng", "split": "test", "revision": "9080400076fbadbb4c4dcb136ff4eddc40b42553"}, "metrics": [{"type": "accuracy", "value": 26.190476190476193}, {"type": "f1", "value": 22.05208151636723}, {"type": "precision", "value": 21.09292328042328}, {"type": "recall", "value": 26.190476190476193}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (pam-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "pam-eng", "split": "test", "revision": "9080400076fbadbb4c4dcb136ff4eddc40b42553"}, "metrics": [{"type": "accuracy", "value": 17.2}, {"type": "f1", "value": 14.021009731460952}, {"type": "precision", "value": 13.1389886698243}, {"type": "recall", "value": 17.2}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (hsb-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "hsb-eng", "split": "test", "revision": "9080400076fbadbb4c4dcb136ff4eddc40b42553"}, "metrics": [{"type": "accuracy", "value": 78.67494824016563}, {"type": "f1", "value": 74.24430641821947}, {"type": "precision", "value": 72.50747642051991}, {"type": "recall", "value": 78.67494824016563}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (srp-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "srp-eng", "split": "test", "revision": "9080400076fbadbb4c4dcb136ff4eddc40b42553"}, "metrics": [{"type": "accuracy", "value": 94.19999999999999}, {"type": "f1", "value": 92.54}, {"type": "precision", "value": 91.75833333333334}, {"type": "recall", "value": 94.19999999999999}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (epo-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "epo-eng", "split": "test", "revision": "9080400076fbadbb4c4dcb136ff4eddc40b42553"}, "metrics": [{"type": "accuracy", "value": 90.2}, {"type": "f1", "value": 87.78666666666666}, {"type": "precision", "value": 86.69833333333334}, {"type": "recall", "value": 90.2}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (kzj-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "kzj-eng", "split": "test", "revision": "9080400076fbadbb4c4dcb136ff4eddc40b42553"}, "metrics": [{"type": "accuracy", "value": 14.7}, {"type": "f1", "value": 12.19206214842218}, {"type": "precision", "value": 11.526261904761904}, {"type": "recall", "value": 14.7}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (awa-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "awa-eng", "split": "test", "revision": "9080400076fbadbb4c4dcb136ff4eddc40b42553"}, "metrics": [{"type": "accuracy", "value": 73.16017316017316}, {"type": "f1", "value": 67.44858316286889}, {"type": "precision", "value": 65.23809523809523}, {"type": "recall", "value": 73.16017316017316}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (fao-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "fao-eng", "split": "test", "revision": "9080400076fbadbb4c4dcb136ff4eddc40b42553"}, "metrics": [{"type": "accuracy", "value": 75.19083969465649}, {"type": "f1", "value": 70.33078880407125}, {"type": "precision", "value": 68.3969465648855}, {"type": "recall", "value": 75.19083969465649}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (mal-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "mal-eng", "split": "test", "revision": "9080400076fbadbb4c4dcb136ff4eddc40b42553"}, "metrics": [{"type": "accuracy", "value": 62.154294032023294}, {"type": "f1", "value": 55.86030821838681}, {"type": "precision", "value": 53.53509623160277}, {"type": "recall", "value": 62.154294032023294}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (ile-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "ile-eng", "split": "test", "revision": "9080400076fbadbb4c4dcb136ff4eddc40b42553"}, "metrics": [{"type": "accuracy", "value": 86.8}, {"type": "f1", "value": 83.9652380952381}, {"type": "precision", "value": 82.84242424242424}, {"type": "recall", "value": 86.8}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (bos-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "bos-eng", "split": "test", "revision": "9080400076fbadbb4c4dcb136ff4eddc40b42553"}, "metrics": [{"type": "accuracy", "value": 93.50282485875707}, {"type": "f1", "value": 91.54425612052731}, {"type": "precision", "value": 90.65442561205272}, {"type": "recall", "value": 93.50282485875707}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (cor-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "cor-eng", "split": "test", "revision": "9080400076fbadbb4c4dcb136ff4eddc40b42553"}, "metrics": [{"type": "accuracy", "value": 11.4}, {"type": "f1", "value": 9.189775870222714}, {"type": "precision", "value": 8.66189886502811}, {"type": "recall", "value": 11.4}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (cat-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "cat-eng", "split": "test", "revision": "9080400076fbadbb4c4dcb136ff4eddc40b42553"}, "metrics": [{"type": "accuracy", "value": 93.4}, {"type": "f1", "value": 91.88666666666666}, {"type": "precision", "value": 91.21444444444444}, {"type": "recall", "value": 93.4}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (eus-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "eus-eng", "split": "test", "revision": "9080400076fbadbb4c4dcb136ff4eddc40b42553"}, "metrics": [{"type": "accuracy", "value": 46.0}, {"type": "f1", "value": 40.51069226095542}, {"type": "precision", "value": 38.57804926010808}, {"type": "recall", "value": 46.0}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (yue-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "yue-eng", "split": "test", "revision": "9080400076fbadbb4c4dcb136ff4eddc40b42553"}, "metrics": [{"type": "accuracy", "value": 91.0}, {"type": "f1", "value": 89.11333333333333}, {"type": "precision", "value": 88.27000000000001}, {"type": "recall", "value": 91.0}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (swe-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "swe-eng", "split": "test", "revision": "9080400076fbadbb4c4dcb136ff4eddc40b42553"}, "metrics": [{"type": "accuracy", "value": 94.39999999999999}, {"type": "f1", "value": 92.95}, {"type": "precision", "value": 92.27000000000001}, {"type": "recall", "value": 94.39999999999999}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (dtp-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "dtp-eng", "split": "test", "revision": "9080400076fbadbb4c4dcb136ff4eddc40b42553"}, "metrics": [{"type": "accuracy", "value": 14.2}, {"type": "f1", "value": 11.73701698770113}, {"type": "precision", "value": 11.079207014736676}, {"type": "recall", "value": 14.2}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (kat-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "kat-eng", "split": "test", "revision": "9080400076fbadbb4c4dcb136ff4eddc40b42553"}, "metrics": [{"type": "accuracy", "value": 65.14745308310992}, {"type": "f1", "value": 59.665707393589415}, {"type": "precision", "value": 57.560853653346946}, {"type": "recall", "value": 65.14745308310992}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (jpn-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "jpn-eng", "split": "test", "revision": "9080400076fbadbb4c4dcb136ff4eddc40b42553"}, "metrics": [{"type": "accuracy", "value": 95.39999999999999}, {"type": "f1", "value": 94.0}, {"type": "precision", "value": 93.33333333333333}, {"type": "recall", "value": 95.39999999999999}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (csb-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "csb-eng", "split": "test", "revision": "9080400076fbadbb4c4dcb136ff4eddc40b42553"}, "metrics": [{"type": "accuracy", "value": 69.56521739130434}, {"type": "f1", "value": 62.92490118577074}, {"type": "precision", "value": 60.27009222661397}, {"type": "recall", "value": 69.56521739130434}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (xho-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "xho-eng", "split": "test", "revision": "9080400076fbadbb4c4dcb136ff4eddc40b42553"}, "metrics": [{"type": "accuracy", "value": 40.140845070422536}, {"type": "f1", "value": 35.96411804158283}, {"type": "precision", "value": 34.89075869357559}, {"type": "recall", "value": 40.140845070422536}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (orv-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "orv-eng", "split": "test", "revision": "9080400076fbadbb4c4dcb136ff4eddc40b42553"}, "metrics": [{"type": "accuracy", "value": 65.86826347305389}, {"type": "f1", "value": 59.646248628284546}, {"type": "precision", "value": 57.22982606216139}, {"type": "recall", "value": 65.86826347305389}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (ind-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "ind-eng", "split": "test", "revision": "9080400076fbadbb4c4dcb136ff4eddc40b42553"}, "metrics": [{"type": "accuracy", "value": 94.89999999999999}, {"type": "f1", "value": 93.48333333333333}, {"type": "precision", "value": 92.83666666666667}, {"type": "recall", "value": 94.89999999999999}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (tuk-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "tuk-eng", "split": "test", "revision": "9080400076fbadbb4c4dcb136ff4eddc40b42553"}, "metrics": [{"type": "accuracy", "value": 47.783251231527096}, {"type": "f1", "value": 42.006447302013804}, {"type": "precision", "value": 40.12747105111637}, {"type": "recall", "value": 47.783251231527096}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (max-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "max-eng", "split": "test", "revision": "9080400076fbadbb4c4dcb136ff4eddc40b42553"}, "metrics": [{"type": "accuracy", "value": 69.71830985915493}, {"type": "f1", "value": 64.80266212660578}, {"type": "precision", "value": 63.08098591549296}, {"type": "recall", "value": 69.71830985915493}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (swh-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "swh-eng", "split": "test", "revision": "9080400076fbadbb4c4dcb136ff4eddc40b42553"}, "metrics": [{"type": "accuracy", "value": 67.94871794871796}, {"type": "f1", "value": 61.59912309912309}, {"type": "precision", "value": 59.17338217338218}, {"type": "recall", "value": 67.94871794871796}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (hin-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "hin-eng", "split": "test", "revision": "9080400076fbadbb4c4dcb136ff4eddc40b42553"}, "metrics": [{"type": "accuracy", "value": 96.39999999999999}, {"type": "f1", "value": 95.28333333333335}, {"type": "precision", "value": 94.75}, {"type": "recall", "value": 96.39999999999999}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (dsb-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "dsb-eng", "split": "test", "revision": "9080400076fbadbb4c4dcb136ff4eddc40b42553"}, "metrics": [{"type": "accuracy", "value": 70.14613778705638}, {"type": "f1", "value": 65.4349338900487}, {"type": "precision", "value": 63.57599255302805}, {"type": "recall", "value": 70.14613778705638}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (ber-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "ber-eng", "split": "test", "revision": "9080400076fbadbb4c4dcb136ff4eddc40b42553"}, "metrics": [{"type": "accuracy", "value": 9.2}, {"type": "f1", "value": 7.622184434339607}, {"type": "precision", "value": 7.287048159682417}, {"type": "recall", "value": 9.2}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (tam-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "tam-eng", "split": "test", "revision": "9080400076fbadbb4c4dcb136ff4eddc40b42553"}, "metrics": [{"type": "accuracy", "value": 77.85016286644951}, {"type": "f1", "value": 72.83387622149837}, {"type": "precision", "value": 70.58450959102424}, {"type": "recall", "value": 77.85016286644951}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (slk-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "slk-eng", "split": "test", "revision": "9080400076fbadbb4c4dcb136ff4eddc40b42553"}, "metrics": [{"type": "accuracy", "value": 90.8}, {"type": "f1", "value": 88.84333333333333}, {"type": "precision", "value": 87.96666666666665}, {"type": "recall", "value": 90.8}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (tgl-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "tgl-eng", "split": "test", "revision": "9080400076fbadbb4c4dcb136ff4eddc40b42553"}, "metrics": [{"type": "accuracy", "value": 94.6}, {"type": "f1", "value": 93.14}, {"type": "precision", "value": 92.49833333333333}, {"type": "recall", "value": 94.6}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (ast-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "ast-eng", "split": "test", "revision": "9080400076fbadbb4c4dcb136ff4eddc40b42553"}, "metrics": [{"type": "accuracy", "value": 84.25196850393701}, {"type": "f1", "value": 80.94488188976378}, {"type": "precision", "value": 79.65879265091863}, {"type": "recall", "value": 84.25196850393701}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (mkd-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "mkd-eng", "split": "test", "revision": "9080400076fbadbb4c4dcb136ff4eddc40b42553"}, "metrics": [{"type": "accuracy", "value": 89.5}, {"type": "f1", "value": 86.89666666666666}, {"type": "precision", "value": 85.7}, {"type": "recall", "value": 89.5}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (khm-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "khm-eng", "split": "test", "revision": "9080400076fbadbb4c4dcb136ff4eddc40b42553"}, "metrics": [{"type": "accuracy", "value": 42.797783933518005}, {"type": "f1", "value": 37.30617360155193}, {"type": "precision", "value": 35.34933825792552}, {"type": "recall", "value": 42.797783933518005}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (ces-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "ces-eng", "split": "test", "revision": "9080400076fbadbb4c4dcb136ff4eddc40b42553"}, "metrics": [{"type": "accuracy", "value": 96.1}, {"type": "f1", "value": 94.93333333333332}, {"type": "precision", "value": 94.38333333333333}, {"type": "recall", "value": 96.1}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (tzl-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "tzl-eng", "split": "test", "revision": "9080400076fbadbb4c4dcb136ff4eddc40b42553"}, "metrics": [{"type": "accuracy", "value": 54.807692307692314}, {"type": "f1", "value": 49.506903353057204}, {"type": "precision", "value": 47.54807692307693}, {"type": "recall", "value": 54.807692307692314}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (urd-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "urd-eng", "split": "test", "revision": "9080400076fbadbb4c4dcb136ff4eddc40b42553"}, "metrics": [{"type": "accuracy", "value": 87.1}, {"type": "f1", "value": 83.61857142857143}, {"type": "precision", "value": 81.975}, {"type": "recall", "value": 87.1}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (ara-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "ara-eng", "split": "test", "revision": "9080400076fbadbb4c4dcb136ff4eddc40b42553"}, "metrics": [{"type": "accuracy", "value": 91.10000000000001}, {"type": "f1", "value": 88.76333333333332}, {"type": "precision", "value": 87.67}, {"type": "recall", "value": 91.10000000000001}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (kor-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "kor-eng", "split": "test", "revision": "9080400076fbadbb4c4dcb136ff4eddc40b42553"}, "metrics": [{"type": "accuracy", "value": 93.10000000000001}, {"type": "f1", "value": 91.28999999999999}, {"type": "precision", "value": 90.44500000000001}, {"type": "recall", "value": 93.10000000000001}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (yid-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "yid-eng", "split": "test", "revision": "9080400076fbadbb4c4dcb136ff4eddc40b42553"}, "metrics": [{"type": "accuracy", "value": 39.97641509433962}, {"type": "f1", "value": 33.12271889998028}, {"type": "precision", "value": 30.95185381542554}, {"type": "recall", "value": 39.97641509433962}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (fin-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "fin-eng", "split": "test", "revision": "9080400076fbadbb4c4dcb136ff4eddc40b42553"}, "metrics": [{"type": "accuracy", "value": 92.60000000000001}, {"type": "f1", "value": 90.69}, {"type": "precision", "value": 89.84500000000001}, {"type": "recall", "value": 92.60000000000001}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (tha-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "tha-eng", "split": "test", "revision": "9080400076fbadbb4c4dcb136ff4eddc40b42553"}, "metrics": [{"type": "accuracy", "value": 95.07299270072993}, {"type": "f1", "value": 93.64355231143554}, {"type": "precision", "value": 92.94403892944038}, {"type": "recall", "value": 95.07299270072993}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (wuu-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "wuu-eng", "split": "test", "revision": "9080400076fbadbb4c4dcb136ff4eddc40b42553"}, "metrics": [{"type": "accuracy", "value": 91.9}, {"type": "f1", "value": 89.61333333333333}, {"type": "precision", "value": 88.53333333333333}, {"type": "recall", "value": 91.9}]}, {"task": {"type": "Clustering"}, "dataset": {"name": "MTEB ThuNewsClusteringP2P", "type": "C-MTEB/ThuNewsClusteringP2P", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "v_measure", "value": 64.68478289806511}]}, {"task": {"type": "Clustering"}, "dataset": {"name": "MTEB ThuNewsClusteringS2S", "type": "C-MTEB/ThuNewsClusteringS2S", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "v_measure", "value": 57.53010296184097}]}, {"task": {"type": "Retrieval"}, "dataset": {"name": "MTEB Touche2020", "type": "webis-touche2020", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "map_at_1", "value": 2.519}, {"type": "map_at_10", "value": 10.31}, {"type": "map_at_100", "value": 16.027}, {"type": "map_at_1000", "value": 17.827}, {"type": "map_at_3", "value": 5.721}, {"type": "map_at_5", "value": 7.7829999999999995}, {"type": "mrr_at_1", "value": 34.694}, {"type": "mrr_at_10", "value": 52.642999999999994}, {"type": "mrr_at_100", "value": 53.366}, {"type": "mrr_at_1000", "value": 53.366}, {"type": "mrr_at_3", "value": 48.638999999999996}, {"type": "mrr_at_5", "value": 50.578}, {"type": "ndcg_at_1", "value": 31.633}, {"type": "ndcg_at_10", "value": 26.394000000000002}, {"type": "ndcg_at_100", "value": 36.41}, {"type": "ndcg_at_1000", "value": 49.206}, {"type": "ndcg_at_3", "value": 31.694}, {"type": "ndcg_at_5", "value": 29.529}, {"type": "precision_at_1", "value": 34.694}, {"type": "precision_at_10", "value": 23.469}, {"type": "precision_at_100", "value": 7.286}, {"type": "precision_at_1000", "value": 1.5610000000000002}, {"type": "precision_at_3", "value": 34.014}, {"type": "precision_at_5", "value": 29.796}, {"type": "recall_at_1", "value": 2.519}, {"type": "recall_at_10", "value": 17.091}, {"type": "recall_at_100", "value": 45.429}, {"type": "recall_at_1000", "value": 84.621}, {"type": "recall_at_3", "value": 7.208}, {"type": "recall_at_5", "value": 10.523}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB ToxicConversationsClassification", "type": "mteb/toxic_conversations_50k", "config": "default", "split": "test", "revision": "d7c0de2777da35d6aae2200a62c6e0e5af397c4c"}, "metrics": [{"type": "accuracy", "value": 69.58659999999999}, {"type": "ap", "value": 14.735696532619}, {"type": "f1", "value": 54.23517220069903}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB TweetSentimentExtractionClassification", "type": "mteb/tweet_sentiment_extraction", "config": "default", "split": "test", "revision": "d604517c81ca91fe16a244d1248fc021f9ecee7a"}, "metrics": [{"type": "accuracy", "value": 63.723825693265425}, {"type": "f1", "value": 64.02405729449103}]}, {"task": {"type": "Clustering"}, "dataset": {"name": "MTEB TwentyNewsgroupsClustering", "type": "mteb/twentynewsgroups-clustering", "config": "default", "split": "test", "revision": "6125ec4e24fa026cec8a478383ee943acfbd5449"}, "metrics": [{"type": "v_measure", "value": 54.310161547491006}]}, {"task": {"type": "PairClassification"}, "dataset": {"name": "MTEB TwitterSemEval2015", "type": "mteb/twittersemeval2015-pairclassification", "config": "default", "split": "test", "revision": "70970daeab8776df92f5ea462b6173c0b46fd2d1"}, "metrics": [{"type": "cos_sim_accuracy", "value": 88.77630088812064}, {"type": "cos_sim_ap", "value": 81.61725457333809}, {"type": "cos_sim_f1", "value": 74.91373801916932}, {"type": "cos_sim_precision", "value": 72.63940520446097}, {"type": "cos_sim_recall", "value": 77.33509234828496}, {"type": "dot_accuracy", "value": 88.77630088812064}, {"type": "dot_ap", "value": 81.61725317476251}, {"type": "dot_f1", "value": 74.91373801916932}, {"type": "dot_precision", "value": 72.63940520446097}, {"type": "dot_recall", "value": 77.33509234828496}, {"type": "euclidean_accuracy", "value": 88.77630088812064}, {"type": "euclidean_ap", "value": 81.61724596869566}, {"type": "euclidean_f1", "value": 74.91373801916932}, {"type": "euclidean_precision", "value": 72.63940520446097}, {"type": "euclidean_recall", "value": 77.33509234828496}, {"type": "manhattan_accuracy", "value": 88.67497168742922}, {"type": "manhattan_ap", "value": 81.430251048948}, {"type": "manhattan_f1", "value": 74.79593118171543}, {"type": "manhattan_precision", "value": 71.3635274382938}, {"type": "manhattan_recall", "value": 78.57519788918206}, {"type": "max_accuracy", "value": 88.77630088812064}, {"type": "max_ap", "value": 81.61725457333809}, {"type": "max_f1", "value": 74.91373801916932}]}, {"task": {"type": "PairClassification"}, "dataset": {"name": "MTEB TwitterURLCorpus", "type": "mteb/twitterurlcorpus-pairclassification", "config": "default", "split": "test", "revision": "8b6510b0b1fa4e4c4f879467980e9be563ec1cdf"}, "metrics": [{"type": "cos_sim_accuracy", "value": 89.85136026700819}, {"type": "cos_sim_ap", "value": 87.74656687446567}, {"type": "cos_sim_f1", "value": 80.3221673073403}, {"type": "cos_sim_precision", "value": 76.56871640957633}, {"type": "cos_sim_recall", "value": 84.46258084385587}, {"type": "dot_accuracy", "value": 89.85136026700819}, {"type": "dot_ap", "value": 87.74656471395072}, {"type": "dot_f1", "value": 80.3221673073403}, {"type": "dot_precision", "value": 76.56871640957633}, {"type": "dot_recall", "value": 84.46258084385587}, {"type": "euclidean_accuracy", "value": 89.85136026700819}, {"type": "euclidean_ap", "value": 87.74656885754466}, {"type": "euclidean_f1", "value": 80.3221673073403}, {"type": "euclidean_precision", "value": 76.56871640957633}, {"type": "euclidean_recall", "value": 84.46258084385587}, {"type": "manhattan_accuracy", "value": 89.86300306593705}, {"type": "manhattan_ap", "value": 87.78807479093082}, {"type": "manhattan_f1", "value": 80.31663429471911}, {"type": "manhattan_precision", "value": 76.63472970137772}, {"type": "manhattan_recall", "value": 84.3701878657222}, {"type": "max_accuracy", "value": 89.86300306593705}, {"type": "max_ap", "value": 87.78807479093082}, {"type": "max_f1", "value": 80.3221673073403}]}, {"task": {"type": "Retrieval"}, "dataset": {"name": "MTEB VideoRetrieval", "type": "C-MTEB/VideoRetrieval", "config": "default", "split": "dev", "revision": "None"}, "metrics": [{"type": "map_at_1", "value": 32.4}, {"type": "map_at_10", "value": 40.961999999999996}, {"type": "map_at_100", "value": 41.660000000000004}, {"type": "map_at_1000", "value": 41.721000000000004}, {"type": "map_at_3", "value": 38.550000000000004}, {"type": "map_at_5", "value": 40.06}, {"type": "mrr_at_1", "value": 32.4}, {"type": "mrr_at_10", "value": 40.961999999999996}, {"type": "mrr_at_100", "value": 41.660000000000004}, {"type": "mrr_at_1000", "value": 41.721000000000004}, {"type": "mrr_at_3", "value": 38.550000000000004}, {"type": "mrr_at_5", "value": 40.06}, {"type": "ndcg_at_1", "value": 32.4}, {"type": "ndcg_at_10", "value": 45.388}, {"type": "ndcg_at_100", "value": 49.012}, {"type": "ndcg_at_1000", "value": 50.659}, {"type": "ndcg_at_3", "value": 40.47}, {"type": "ndcg_at_5", "value": 43.232}, {"type": "precision_at_1", "value": 32.4}, {"type": "precision_at_10", "value": 5.94}, {"type": "precision_at_100", "value": 0.769}, {"type": "precision_at_1000", "value": 0.09}, {"type": "precision_at_3", "value": 15.333}, {"type": "precision_at_5", "value": 10.56}, {"type": "recall_at_1", "value": 32.4}, {"type": "recall_at_10", "value": 59.4}, {"type": "recall_at_100", "value": 76.9}, {"type": "recall_at_1000", "value": 90.0}, {"type": "recall_at_3", "value": 46.0}, {"type": "recall_at_5", "value": 52.800000000000004}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB Waimai", "type": "C-MTEB/waimai-classification", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "accuracy", "value": 86.94000000000001}, {"type": "ap", "value": 70.57373468481975}, {"type": "f1", "value": 85.26264784928323}]}]}]}
transformers pytorch safetensors mistral feature-extraction mteb en arxiv:2401.00368 arxiv:2104.08663 arxiv:2210.07316 license:mit model-index endpoints_compatible has_space text-generation-inference region:us