
# 1 开源清单

本次开源2个通用向量编码模型和一个针对dialogue进行编码的向量模型，同时开源全量160万对话重写数据集和20万的难负例的检索数据集。

**开源模型：**

| ModelName                                                                                                     | ModelSize | MaxTokens | EmbeddingDimensions | Language | Scenario | C-MTEB Score |   
|---------------------------------------------------------------------------------------------------------------|-----------|-----------|---------------------|----------|----------|--------------|
| [infgrad/stella-base-zh-v3-1792d](https://huggingface.co/infgrad/stella-base-zh-v3-1792d)                     | 0.4GB     | 512       | 1792                | zh-CN    | 通用文本     | 67.96        |   
| [infgrad/stella-large-zh-v3-1792d](https://huggingface.co/infgrad/stella-large-zh-v3-1792d)                   | 1.3GB     | 512       | 1792                | zh-CN    | 通用文本     | 68.48        |   
| [infgrad/stella-dialogue-large-zh-v3-1792d](https://huggingface.co/infgrad/stella-dialogue-large-zh-v3-1792d) | 1.3GB     | 512       | 1792                | zh-CN    | **对话文本** | 不适用          |

**开源数据：**

1. [全量对话重写数据集](https://huggingface.co/datasets/infgrad/dialogue_rewrite_llm) 约160万
2. [部分带有难负例的检索数据集](https://huggingface.co/datasets/infgrad/retrieval_data_llm) 约20万

上述数据集均使用LLM构造，欢迎各位贡献数据集。

# 2 使用方法

## 2.1 通用编码模型使用方法

直接SentenceTransformer加载即可：

```python
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("infgrad/stella-base-zh-v3-1792d")
# model = SentenceTransformer("infgrad/stella-large-zh-v3-1792d")
vectors = model.encode(["text1", "text2"])
```

## 2.2 dialogue编码模型使用方法

**使用场景：**
**在一段对话中，需要根据用户语句去检索相关文本，但是对话中的用户语句存在大量的指代和省略，导致直接使用通用编码模型效果不好，
可以使用本项目的专门的dialogue编码模型进行编码**

**使用要点：**

1. 对dialogue进行编码时，dialogue中的每个utterance需要是如下格式：`"{ROLE}: {TEXT}"`，然后使用`[SEP]` join一下
2. 整个对话都要送入模型进行编码，如果长度不够就删掉早期的对话，**编码后的向量本质是对话中最后一句话的重写版本的向量！！**
3. 对话用stella-dialogue-large-zh-v3-1792d编码，被检索文本使用stella-large-zh-v3-1792d进行编码，所以本场景是需要2个编码模型的

如果对使用方法还有疑惑，请到下面章节阅读该模型是如何训练的。

使用示例：

```python
from sentence_transformers import SentenceTransformer

dial_model = SentenceTransformer("infgrad/stella-dialogue-large-zh-v3-1792d")
general_model = SentenceTransformer("infgrad/stella-large-zh-v3-1792d")
# dialogue = ["张三: 吃饭吗", "李四: 等会去"]
dialogue = ["A: 最近去打篮球了吗", "B: 没有"]
corpus = ["B没打篮球是因为受伤了。", "B没有打乒乓球"]
last_utterance_vector = dial_model.encode(["[SEP]".join(dialogue)], normalize_embeddings=True)
corpus_vectors = general_model.encode(corpus, normalize_embeddings=True)
# 计算相似度
sims = (last_utterance_vector * corpus_vectors).sum(axis=1)
print(sims)
```

# 3 通用编码模型训练技巧分享

## hard negative

难负例挖掘也是个经典的trick了，几乎总能提升效果

## dropout-1d

dropout已经是深度学习的标配，我们可以稍微改造下使其更适合句向量的训练。
我们在训练时会尝试让每一个token-embedding都可以表征整个句子，而在推理时使用mean_pooling从而达到类似模型融合的效果。
具体操作是在mean_pooling时加入dropout_1d，torch代码如下：

```python
vector_dropout = nn.Dropout1d(0.3)  # 算力有限，试了0.3和0.5 两个参数，其中0.3更优
last_hidden_state = bert_model(...)[0]
last_hidden = last_hidden_state.masked_fill(~attention_mask[..., None].bool(), 0.0)
last_hidden = vector_dropout(last_hidden)
vectors = last_hidden.sum(dim=1) / attention_mask.sum(dim=1)[..., None]
```

# 4 dialogue编码模型细节

## 4.1 为什么需要一个dialogue编码模型？

参见本人历史文章：https://www.zhihu.com/pin/1674913544847077376

## 4.2 训练数据

单条数据示例：

```json
{
  "dialogue": [
    "A: 最近去打篮球了吗",
    "B: 没有"
  ],
  "last_utterance_rewrite": "B: 我最近没有去打篮球"
}
```

## 4.3 训练Loss

```
loss = cosine_loss( dial_model.encode(dialogue), existing_model.encode(last_utterance_rewrite) )
```

dial_model就是要被训练的模型，本人是以stella-large-zh-v3-1792d作为base-model进行继续训练的

existing_model就是现有训练好的**通用编码模型**，本人使用的是stella-large-zh-v3-1792d

已开源dialogue-embedding的全量训练数据，理论上可以复现本模型效果。

Loss下降情况：

<div align="center">
<img src="dial_loss.png" alt="icon" width="2000px"/>
</div>

## 4.4 效果

目前还没有专门测试集，本人简单测试了下是有效果的，部分测试结果见文件`dial_retrieval_test.xlsx`。

# 5 后续TODO

1. 更多的dial-rewrite数据
2. 不同EmbeddingDimensions的编码模型

# 6 FAQ

Q: 为什么向量维度是1792？\
A: 最初考虑发布768、1024，768+768，1024+1024，1024+768维度，但是时间有限，先做了1792就只发布1792维度的模型。理论上维度越高效果越好。

Q: 如何复现CMTEB效果？\
A: SentenceTransformer加载后直接用官方评测脚本就行，注意对于Classification任务向量需要先normalize一下

Q: 复现的CMTEB效果和本文不一致？\
A: 聚类不一致正常，官方评测代码没有设定seed，其他不一致建议检查代码或联系本人。

Q: 如何选择向量模型？\
A: 没有免费的午餐，在自己测试集上试试，本人推荐bge、e5和stella.

Q: 长度为什么只有512，能否更长？\
A: 可以但没必要，长了效果普遍不好，这是当前训练方法和数据导致的，几乎无解，建议长文本还是走分块。

Q: 训练资源和算力？\
A: 亿级别的数据，单卡A100要一个月起步

{"tags": ["sentence-transformers", "feature-extraction", "sentence-similarity", "mteb"], "pipeline_tag": "sentence-similarity", "model-index": [{"name": "stella-large-zh-v3-1792d", "results": [{"task": {"type": "STS"}, "dataset": {"name": "MTEB AFQMC", "type": "C-MTEB/AFQMC", "config": "default", "split": "validation", "revision": "None"}, "metrics": [{"type": "cos_sim_pearson", "value": 54.48093298255762}, {"type": "cos_sim_spearman", "value": 59.105354109068685}, {"type": "euclidean_pearson", "value": 57.761189988643444}, {"type": "euclidean_spearman", "value": 59.10537421115596}, {"type": "manhattan_pearson", "value": 56.94359297051431}, {"type": "manhattan_spearman", "value": 58.37611109821567}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB ATEC", "type": "C-MTEB/ATEC", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "cos_sim_pearson", "value": 54.39711127600595}, {"type": "cos_sim_spearman", "value": 58.190191920824454}, {"type": "euclidean_pearson", "value": 61.80082379352729}, {"type": "euclidean_spearman", "value": 58.19018966860797}, {"type": "manhattan_pearson", "value": 60.927601060396206}, {"type": "manhattan_spearman", "value": 57.78832902694192}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB AmazonReviewsClassification (zh)", "type": "mteb/amazon_reviews_multi", "config": "zh", "split": "test", "revision": "1399c76144fd37290681b995c656ef9b2e06e26d"}, "metrics": [{"type": "accuracy", "value": 46.31600000000001}, {"type": "f1", "value": 44.45281663598873}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB BQ", "type": "C-MTEB/BQ", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "cos_sim_pearson", "value": 69.12211326097868}, {"type": "cos_sim_spearman", "value": 71.0741302039443}, {"type": "euclidean_pearson", "value": 69.89070483887852}, {"type": "euclidean_spearman", "value": 71.07413020351787}, {"type": "manhattan_pearson", "value": 69.62345441260962}, {"type": "manhattan_spearman", "value": 70.8517591280618}]}, {"task": {"type": "Clustering"}, "dataset": {"name": "MTEB CLSClusteringP2P", "type": "C-MTEB/CLSClusteringP2P", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "v_measure", "value": 41.937723608805314}]}, {"task": {"type": "Clustering"}, "dataset": {"name": "MTEB CLSClusteringS2S", "type": "C-MTEB/CLSClusteringS2S", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "v_measure", "value": 40.34373057675427}]}, {"task": {"type": "Reranking"}, "dataset": {"name": "MTEB CMedQAv1", "type": "C-MTEB/CMedQAv1-reranking", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "map", "value": 88.98896401788376}, {"type": "mrr", "value": 90.97119047619047}]}, {"task": {"type": "Reranking"}, "dataset": {"name": "MTEB CMedQAv2", "type": "C-MTEB/CMedQAv2-reranking", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "map", "value": 89.59718540244556}, {"type": "mrr", "value": 91.41246031746032}]}, {"task": {"type": "Retrieval"}, "dataset": {"name": "MTEB CmedqaRetrieval", "type": "C-MTEB/CmedqaRetrieval", "config": "default", "split": "dev", "revision": "None"}, "metrics": [{"type": "map_at_1", "value": 26.954}, {"type": "map_at_10", "value": 40.144999999999996}, {"type": "map_at_100", "value": 42.083999999999996}, {"type": "map_at_1000", "value": 42.181000000000004}, {"type": "map_at_3", "value": 35.709}, {"type": "map_at_5", "value": 38.141000000000005}, {"type": "mrr_at_1", "value": 40.71}, {"type": "mrr_at_10", "value": 48.93}, {"type": "mrr_at_100", "value": 49.921}, {"type": "mrr_at_1000", "value": 49.958999999999996}, {"type": "mrr_at_3", "value": 46.32}, {"type": "mrr_at_5", "value": 47.769}, {"type": "ndcg_at_1", "value": 40.71}, {"type": "ndcg_at_10", "value": 46.869}, {"type": "ndcg_at_100", "value": 54.234}, {"type": "ndcg_at_1000", "value": 55.854000000000006}, {"type": "ndcg_at_3", "value": 41.339}, {"type": "ndcg_at_5", "value": 43.594}, {"type": "precision_at_1", "value": 40.71}, {"type": "precision_at_10", "value": 10.408000000000001}, {"type": "precision_at_100", "value": 1.635}, {"type": "precision_at_1000", "value": 0.184}, {"type": "precision_at_3", "value": 23.348}, {"type": "precision_at_5", "value": 16.929}, {"type": "recall_at_1", "value": 26.954}, {"type": "recall_at_10", "value": 57.821999999999996}, {"type": "recall_at_100", "value": 88.08200000000001}, {"type": "recall_at_1000", "value": 98.83800000000001}, {"type": "recall_at_3", "value": 41.221999999999994}, {"type": "recall_at_5", "value": 48.241}]}, {"task": {"type": "PairClassification"}, "dataset": {"name": "MTEB Cmnli", "type": "C-MTEB/CMNLI", "config": "default", "split": "validation", "revision": "None"}, "metrics": [{"type": "cos_sim_accuracy", "value": 83.6680697534576}, {"type": "cos_sim_ap", "value": 90.77401562455269}, {"type": "cos_sim_f1", "value": 84.68266427450101}, {"type": "cos_sim_precision", "value": 81.36177547942253}, {"type": "cos_sim_recall", "value": 88.28618190320317}, {"type": "dot_accuracy", "value": 83.6680697534576}, {"type": "dot_ap", "value": 90.76429465198817}, {"type": "dot_f1", "value": 84.68266427450101}, {"type": "dot_precision", "value": 81.36177547942253}, {"type": "dot_recall", "value": 88.28618190320317}, {"type": "euclidean_accuracy", "value": 83.6680697534576}, {"type": "euclidean_ap", "value": 90.77401909305344}, {"type": "euclidean_f1", "value": 84.68266427450101}, {"type": "euclidean_precision", "value": 81.36177547942253}, {"type": "euclidean_recall", "value": 88.28618190320317}, {"type": "manhattan_accuracy", "value": 83.40348767288035}, {"type": "manhattan_ap", "value": 90.57002020310819}, {"type": "manhattan_f1", "value": 84.51526032315978}, {"type": "manhattan_precision", "value": 81.25134843581445}, {"type": "manhattan_recall", "value": 88.05237315875614}, {"type": "max_accuracy", "value": 83.6680697534576}, {"type": "max_ap", "value": 90.77401909305344}, {"type": "max_f1", "value": 84.68266427450101}]}, {"task": {"type": "Retrieval"}, "dataset": {"name": "MTEB CovidRetrieval", "type": "C-MTEB/CovidRetrieval", "config": "default", "split": "dev", "revision": "None"}, "metrics": [{"type": "map_at_1", "value": 69.705}, {"type": "map_at_10", "value": 78.648}, {"type": "map_at_100", "value": 78.888}, {"type": "map_at_1000", "value": 78.89399999999999}, {"type": "map_at_3", "value": 77.151}, {"type": "map_at_5", "value": 77.98}, {"type": "mrr_at_1", "value": 69.863}, {"type": "mrr_at_10", "value": 78.62599999999999}, {"type": "mrr_at_100", "value": 78.861}, {"type": "mrr_at_1000", "value": 78.867}, {"type": "mrr_at_3", "value": 77.204}, {"type": "mrr_at_5", "value": 78.005}, {"type": "ndcg_at_1", "value": 69.968}, {"type": "ndcg_at_10", "value": 82.44399999999999}, {"type": "ndcg_at_100", "value": 83.499}, {"type": "ndcg_at_1000", "value": 83.647}, {"type": "ndcg_at_3", "value": 79.393}, {"type": "ndcg_at_5", "value": 80.855}, {"type": "precision_at_1", "value": 69.968}, {"type": "precision_at_10", "value": 9.515}, {"type": "precision_at_100", "value": 0.9990000000000001}, {"type": "precision_at_1000", "value": 0.101}, {"type": "precision_at_3", "value": 28.802}, {"type": "precision_at_5", "value": 18.019}, {"type": "recall_at_1", "value": 69.705}, {"type": "recall_at_10", "value": 94.152}, {"type": "recall_at_100", "value": 98.84100000000001}, {"type": "recall_at_1000", "value": 100.0}, {"type": "recall_at_3", "value": 85.774}, {"type": "recall_at_5", "value": 89.252}]}, {"task": {"type": "Retrieval"}, "dataset": {"name": "MTEB DuRetrieval", "type": "C-MTEB/DuRetrieval", "config": "default", "split": "dev", "revision": "None"}, "metrics": [{"type": "map_at_1", "value": 25.88}, {"type": "map_at_10", "value": 79.857}, {"type": "map_at_100", "value": 82.636}, {"type": "map_at_1000", "value": 82.672}, {"type": "map_at_3", "value": 55.184}, {"type": "map_at_5", "value": 70.009}, {"type": "mrr_at_1", "value": 89.64999999999999}, {"type": "mrr_at_10", "value": 92.967}, {"type": "mrr_at_100", "value": 93.039}, {"type": "mrr_at_1000", "value": 93.041}, {"type": "mrr_at_3", "value": 92.65}, {"type": "mrr_at_5", "value": 92.86}, {"type": "ndcg_at_1", "value": 89.64999999999999}, {"type": "ndcg_at_10", "value": 87.126}, {"type": "ndcg_at_100", "value": 89.898}, {"type": "ndcg_at_1000", "value": 90.253}, {"type": "ndcg_at_3", "value": 86.012}, {"type": "ndcg_at_5", "value": 85.124}, {"type": "precision_at_1", "value": 89.64999999999999}, {"type": "precision_at_10", "value": 41.735}, {"type": "precision_at_100", "value": 4.797}, {"type": "precision_at_1000", "value": 0.488}, {"type": "precision_at_3", "value": 77.267}, {"type": "precision_at_5", "value": 65.48}, {"type": "recall_at_1", "value": 25.88}, {"type": "recall_at_10", "value": 88.28399999999999}, {"type": "recall_at_100", "value": 97.407}, {"type": "recall_at_1000", "value": 99.29299999999999}, {"type": "recall_at_3", "value": 57.38799999999999}, {"type": "recall_at_5", "value": 74.736}]}, {"task": {"type": "Retrieval"}, "dataset": {"name": "MTEB EcomRetrieval", "type": "C-MTEB/EcomRetrieval", "config": "default", "split": "dev", "revision": "None"}, "metrics": [{"type": "map_at_1", "value": 53.2}, {"type": "map_at_10", "value": 63.556000000000004}, {"type": "map_at_100", "value": 64.033}, {"type": "map_at_1000", "value": 64.044}, {"type": "map_at_3", "value": 60.983}, {"type": "map_at_5", "value": 62.588}, {"type": "mrr_at_1", "value": 53.2}, {"type": "mrr_at_10", "value": 63.556000000000004}, {"type": "mrr_at_100", "value": 64.033}, {"type": "mrr_at_1000", "value": 64.044}, {"type": "mrr_at_3", "value": 60.983}, {"type": "mrr_at_5", "value": 62.588}, {"type": "ndcg_at_1", "value": 53.2}, {"type": "ndcg_at_10", "value": 68.61699999999999}, {"type": "ndcg_at_100", "value": 70.88499999999999}, {"type": "ndcg_at_1000", "value": 71.15899999999999}, {"type": "ndcg_at_3", "value": 63.434000000000005}, {"type": "ndcg_at_5", "value": 66.301}, {"type": "precision_at_1", "value": 53.2}, {"type": "precision_at_10", "value": 8.450000000000001}, {"type": "precision_at_100", "value": 0.95}, {"type": "precision_at_1000", "value": 0.097}, {"type": "precision_at_3", "value": 23.5}, {"type": "precision_at_5", "value": 15.479999999999999}, {"type": "recall_at_1", "value": 53.2}, {"type": "recall_at_10", "value": 84.5}, {"type": "recall_at_100", "value": 95.0}, {"type": "recall_at_1000", "value": 97.1}, {"type": "recall_at_3", "value": 70.5}, {"type": "recall_at_5", "value": 77.4}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB IFlyTek", "type": "C-MTEB/IFlyTek-classification", "config": "default", "split": "validation", "revision": "None"}, "metrics": [{"type": "accuracy", "value": 50.63485956136976}, {"type": "f1", "value": 38.286307407751266}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB JDReview", "type": "C-MTEB/JDReview-classification", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "accuracy", "value": 86.11632270168855}, {"type": "ap", "value": 54.43932599806482}, {"type": "f1", "value": 80.85485110996076}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB LCQMC", "type": "C-MTEB/LCQMC", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "cos_sim_pearson", "value": 72.47315152994804}, {"type": "cos_sim_spearman", "value": 78.26531600908152}, {"type": "euclidean_pearson", "value": 77.8560788714531}, {"type": "euclidean_spearman", "value": 78.26531157334841}, {"type": "manhattan_pearson", "value": 77.70593783974188}, {"type": "manhattan_spearman", "value": 78.13880812439999}]}, {"task": {"type": "Reranking"}, "dataset": {"name": "MTEB MMarcoReranking", "type": "C-MTEB/Mmarco-reranking", "config": "default", "split": "dev", "revision": "None"}, "metrics": [{"type": "map", "value": 28.088177976572222}, {"type": "mrr", "value": 27.125}]}, {"task": {"type": "Retrieval"}, "dataset": {"name": "MTEB MMarcoRetrieval", "type": "C-MTEB/MMarcoRetrieval", "config": "default", "split": "dev", "revision": "None"}, "metrics": [{"type": "map_at_1", "value": 66.428}, {"type": "map_at_10", "value": 75.5}, {"type": "map_at_100", "value": 75.82600000000001}, {"type": "map_at_1000", "value": 75.837}, {"type": "map_at_3", "value": 73.74300000000001}, {"type": "map_at_5", "value": 74.87}, {"type": "mrr_at_1", "value": 68.754}, {"type": "mrr_at_10", "value": 76.145}, {"type": "mrr_at_100", "value": 76.432}, {"type": "mrr_at_1000", "value": 76.442}, {"type": "mrr_at_3", "value": 74.628}, {"type": "mrr_at_5", "value": 75.612}, {"type": "ndcg_at_1", "value": 68.754}, {"type": "ndcg_at_10", "value": 79.144}, {"type": "ndcg_at_100", "value": 80.60199999999999}, {"type": "ndcg_at_1000", "value": 80.886}, {"type": "ndcg_at_3", "value": 75.81599999999999}, {"type": "ndcg_at_5", "value": 77.729}, {"type": "precision_at_1", "value": 68.754}, {"type": "precision_at_10", "value": 9.544}, {"type": "precision_at_100", "value": 1.026}, {"type": "precision_at_1000", "value": 0.105}, {"type": "precision_at_3", "value": 28.534}, {"type": "precision_at_5", "value": 18.138}, {"type": "recall_at_1", "value": 66.428}, {"type": "recall_at_10", "value": 89.716}, {"type": "recall_at_100", "value": 96.313}, {"type": "recall_at_1000", "value": 98.541}, {"type": "recall_at_3", "value": 80.923}, {"type": "recall_at_5", "value": 85.48}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (zh-CN)", "type": "mteb/amazon_massive_intent", "config": "zh-CN", "split": "test", "revision": "31efe3c427b0bae9c22cbb560b8f15491cc6bed7"}, "metrics": [{"type": "accuracy", "value": 73.27841291190316}, {"type": "f1", "value": 70.65529957574735}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (zh-CN)", "type": "mteb/amazon_massive_scenario", "config": "zh-CN", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 76.30127774041695}, {"type": "f1", "value": 76.10358226518304}]}, {"task": {"type": "Retrieval"}, "dataset": {"name": "MTEB MedicalRetrieval", "type": "C-MTEB/MedicalRetrieval", "config": "default", "split": "dev", "revision": "None"}, "metrics": [{"type": "map_at_1", "value": 56.3}, {"type": "map_at_10", "value": 62.193}, {"type": "map_at_100", "value": 62.722}, {"type": "map_at_1000", "value": 62.765}, {"type": "map_at_3", "value": 60.633}, {"type": "map_at_5", "value": 61.617999999999995}, {"type": "mrr_at_1", "value": 56.3}, {"type": "mrr_at_10", "value": 62.193}, {"type": "mrr_at_100", "value": 62.722}, {"type": "mrr_at_1000", "value": 62.765}, {"type": "mrr_at_3", "value": 60.633}, {"type": "mrr_at_5", "value": 61.617999999999995}, {"type": "ndcg_at_1", "value": 56.3}, {"type": "ndcg_at_10", "value": 65.176}, {"type": "ndcg_at_100", "value": 67.989}, {"type": "ndcg_at_1000", "value": 69.219}, {"type": "ndcg_at_3", "value": 62.014}, {"type": "ndcg_at_5", "value": 63.766}, {"type": "precision_at_1", "value": 56.3}, {"type": "precision_at_10", "value": 7.46}, {"type": "precision_at_100", "value": 0.8829999999999999}, {"type": "precision_at_1000", "value": 0.098}, {"type": "precision_at_3", "value": 22.0}, {"type": "precision_at_5", "value": 14.04}, {"type": "recall_at_1", "value": 56.3}, {"type": "recall_at_10", "value": 74.6}, {"type": "recall_at_100", "value": 88.3}, {"type": "recall_at_1000", "value": 98.1}, {"type": "recall_at_3", "value": 66.0}, {"type": "recall_at_5", "value": 70.19999999999999}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MultilingualSentiment", "type": "C-MTEB/MultilingualSentiment-classification", "config": "default", "split": "validation", "revision": "None"}, "metrics": [{"type": "accuracy", "value": 76.44666666666666}, {"type": "f1", "value": 76.34548655475949}]}, {"task": {"type": "PairClassification"}, "dataset": {"name": "MTEB Ocnli", "type": "C-MTEB/OCNLI", "config": "default", "split": "validation", "revision": "None"}, "metrics": [{"type": "cos_sim_accuracy", "value": 82.34975636166757}, {"type": "cos_sim_ap", "value": 85.44149338593267}, {"type": "cos_sim_f1", "value": 83.68654509610647}, {"type": "cos_sim_precision", "value": 78.46580406654344}, {"type": "cos_sim_recall", "value": 89.65153115100317}, {"type": "dot_accuracy", "value": 82.34975636166757}, {"type": "dot_ap", "value": 85.4415701376729}, {"type": "dot_f1", "value": 83.68654509610647}, {"type": "dot_precision", "value": 78.46580406654344}, {"type": "dot_recall", "value": 89.65153115100317}, {"type": "euclidean_accuracy", "value": 82.34975636166757}, {"type": "euclidean_ap", "value": 85.4415701376729}, {"type": "euclidean_f1", "value": 83.68654509610647}, {"type": "euclidean_precision", "value": 78.46580406654344}, {"type": "euclidean_recall", "value": 89.65153115100317}, {"type": "manhattan_accuracy", "value": 81.97076340010828}, {"type": "manhattan_ap", "value": 84.83614660756733}, {"type": "manhattan_f1", "value": 83.34167083541772}, {"type": "manhattan_precision", "value": 79.18250950570342}, {"type": "manhattan_recall", "value": 87.96198521647307}, {"type": "max_accuracy", "value": 82.34975636166757}, {"type": "max_ap", "value": 85.4415701376729}, {"type": "max_f1", "value": 83.68654509610647}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB OnlineShopping", "type": "C-MTEB/OnlineShopping-classification", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "accuracy", "value": 93.24}, {"type": "ap", "value": 91.3586656455605}, {"type": "f1", "value": 93.22999314249503}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB PAWSX", "type": "C-MTEB/PAWSX", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "cos_sim_pearson", "value": 39.05676042449009}, {"type": "cos_sim_spearman", "value": 44.996534098358545}, {"type": "euclidean_pearson", "value": 44.42418609172825}, {"type": "euclidean_spearman", "value": 44.995941361058996}, {"type": "manhattan_pearson", "value": 43.98118203238076}, {"type": "manhattan_spearman", "value": 44.51414152788784}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB QBQTC", "type": "C-MTEB/QBQTC", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "cos_sim_pearson", "value": 36.694269474438045}, {"type": "cos_sim_spearman", "value": 38.686738967031616}, {"type": "euclidean_pearson", "value": 36.822540068407235}, {"type": "euclidean_spearman", "value": 38.68690745429757}, {"type": "manhattan_pearson", "value": 36.77180703308932}, {"type": "manhattan_spearman", "value": 38.45414914148094}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STS22 (zh)", "type": "mteb/sts22-crosslingual-sts", "config": "zh", "split": "test", "revision": "6d1ba47164174a496b7fa5d3569dae26a6813b80"}, "metrics": [{"type": "cos_sim_pearson", "value": 65.81209017614124}, {"type": "cos_sim_spearman", "value": 66.5255285833172}, {"type": "euclidean_pearson", "value": 66.01848701752732}, {"type": "euclidean_spearman", "value": 66.5255285833172}, {"type": "manhattan_pearson", "value": 66.66433676370542}, {"type": "manhattan_spearman", "value": 67.07086311480214}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STSB", "type": "C-MTEB/STSB", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "cos_sim_pearson", "value": 80.60785761283502}, {"type": "cos_sim_spearman", "value": 82.80278693241074}, {"type": "euclidean_pearson", "value": 82.47573315938638}, {"type": "euclidean_spearman", "value": 82.80290808593806}, {"type": "manhattan_pearson", "value": 82.49682028989669}, {"type": "manhattan_spearman", "value": 82.84565039346022}]}, {"task": {"type": "Reranking"}, "dataset": {"name": "MTEB T2Reranking", "type": "C-MTEB/T2Reranking", "config": "default", "split": "dev", "revision": "None"}, "metrics": [{"type": "map", "value": 66.37886004738723}, {"type": "mrr", "value": 76.08501655006394}]}, {"task": {"type": "Retrieval"}, "dataset": {"name": "MTEB T2Retrieval", "type": "C-MTEB/T2Retrieval", "config": "default", "split": "dev", "revision": "None"}, "metrics": [{"type": "map_at_1", "value": 28.102}, {"type": "map_at_10", "value": 78.071}, {"type": "map_at_100", "value": 81.71000000000001}, {"type": "map_at_1000", "value": 81.773}, {"type": "map_at_3", "value": 55.142}, {"type": "map_at_5", "value": 67.669}, {"type": "mrr_at_1", "value": 90.9}, {"type": "mrr_at_10", "value": 93.29499999999999}, {"type": "mrr_at_100", "value": 93.377}, {"type": "mrr_at_1000", "value": 93.379}, {"type": "mrr_at_3", "value": 92.901}, {"type": "mrr_at_5", "value": 93.152}, {"type": "ndcg_at_1", "value": 90.9}, {"type": "ndcg_at_10", "value": 85.564}, {"type": "ndcg_at_100", "value": 89.11200000000001}, {"type": "ndcg_at_1000", "value": 89.693}, {"type": "ndcg_at_3", "value": 87.024}, {"type": "ndcg_at_5", "value": 85.66}, {"type": "precision_at_1", "value": 90.9}, {"type": "precision_at_10", "value": 42.208}, {"type": "precision_at_100", "value": 5.027}, {"type": "precision_at_1000", "value": 0.517}, {"type": "precision_at_3", "value": 75.872}, {"type": "precision_at_5", "value": 63.566}, {"type": "recall_at_1", "value": 28.102}, {"type": "recall_at_10", "value": 84.44500000000001}, {"type": "recall_at_100", "value": 95.91300000000001}, {"type": "recall_at_1000", "value": 98.80799999999999}, {"type": "recall_at_3", "value": 56.772999999999996}, {"type": "recall_at_5", "value": 70.99499999999999}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB TNews", "type": "C-MTEB/TNews-classification", "config": "default", "split": "validation", "revision": "None"}, "metrics": [{"type": "accuracy", "value": 53.10599999999999}, {"type": "f1", "value": 51.40415523558322}]}, {"task": {"type": "Clustering"}, "dataset": {"name": "MTEB ThuNewsClusteringP2P", "type": "C-MTEB/ThuNewsClusteringP2P", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "v_measure", "value": 69.6145576098232}]}, {"task": {"type": "Clustering"}, "dataset": {"name": "MTEB ThuNewsClusteringS2S", "type": "C-MTEB/ThuNewsClusteringS2S", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "v_measure", "value": 63.7129548775017}]}, {"task": {"type": "Retrieval"}, "dataset": {"name": "MTEB VideoRetrieval", "type": "C-MTEB/VideoRetrieval", "config": "default", "split": "dev", "revision": "None"}, "metrics": [{"type": "map_at_1", "value": 60.199999999999996}, {"type": "map_at_10", "value": 69.724}, {"type": "map_at_100", "value": 70.185}, {"type": "map_at_1000", "value": 70.196}, {"type": "map_at_3", "value": 67.95}, {"type": "map_at_5", "value": 69.155}, {"type": "mrr_at_1", "value": 60.199999999999996}, {"type": "mrr_at_10", "value": 69.724}, {"type": "mrr_at_100", "value": 70.185}, {"type": "mrr_at_1000", "value": 70.196}, {"type": "mrr_at_3", "value": 67.95}, {"type": "mrr_at_5", "value": 69.155}, {"type": "ndcg_at_1", "value": 60.199999999999996}, {"type": "ndcg_at_10", "value": 73.888}, {"type": "ndcg_at_100", "value": 76.02799999999999}, {"type": "ndcg_at_1000", "value": 76.344}, {"type": "ndcg_at_3", "value": 70.384}, {"type": "ndcg_at_5", "value": 72.541}, {"type": "precision_at_1", "value": 60.199999999999996}, {"type": "precision_at_10", "value": 8.67}, {"type": "precision_at_100", "value": 0.9650000000000001}, {"type": "precision_at_1000", "value": 0.099}, {"type": "precision_at_3", "value": 25.8}, {"type": "precision_at_5", "value": 16.520000000000003}, {"type": "recall_at_1", "value": 60.199999999999996}, {"type": "recall_at_10", "value": 86.7}, {"type": "recall_at_100", "value": 96.5}, {"type": "recall_at_1000", "value": 99.0}, {"type": "recall_at_3", "value": 77.4}, {"type": "recall_at_5", "value": 82.6}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB Waimai", "type": "C-MTEB/waimai-classification", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "accuracy", "value": 88.08}, {"type": "ap", "value": 72.66435456846166}, {"type": "f1", "value": 86.55995793551286}]}]}]}
sentence-transformers pytorch bert feature-extraction sentence-similarity mteb model-index endpoints_compatible region:us