# xiaobu-embedding

模型：基于GTE模型[1]多任务微调。  
数据：闲聊类Query-Query、知识类Query-Doc、BGE开源Query-Doc[2]；清洗正例，挖掘中等难度负例；累计6M(质量更重要)。

## Usage (Sentence-Transformers)

```
pip install -U sentence-transformers
```
相似度计算：
```python
from sentence_transformers import SentenceTransformer
sentences_1 = ["样例数据-1", "样例数据-2"]
sentences_2 = ["样例数据-3", "样例数据-4"]
model = SentenceTransformer('lier007/xiaobu-embedding')
embeddings_1 = model.encode(sentences_1, normalize_embeddings=True)
embeddings_2 = model.encode(sentences_2, normalize_embeddings=True)
similarity = embeddings_1 @ embeddings_2.T
print(similarity)
```

## Evaluation
参考BGE中文CMTEB评估[2]

## Finetune
参考BGE微调模块[2]

## Reference
1. https://huggingface.co/thenlper/gte-large-zh
2. https://github.com/FlagOpen/FlagEmbedding
{"tags": ["mteb"], "model-index": [{"name": "xiaobu-embedding", "results": [{"task": {"type": "STS"}, "dataset": {"name": "MTEB AFQMC", "type": "C-MTEB/AFQMC", "config": "default", "split": "validation", "revision": "None"}, "metrics": [{"type": "cos_sim_pearson", "value": 49.37874132528482}, {"type": "cos_sim_spearman", "value": 54.84722470052176}, {"type": "euclidean_pearson", "value": 53.0495882931575}, {"type": "euclidean_spearman", "value": 54.847727301700665}, {"type": "manhattan_pearson", "value": 53.0632140838278}, {"type": "manhattan_spearman", "value": 54.8744258024692}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB ATEC", "type": "C-MTEB/ATEC", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "cos_sim_pearson", "value": 48.15992903013723}, {"type": "cos_sim_spearman", "value": 55.13198035464577}, {"type": "euclidean_pearson", "value": 55.435876753245715}, {"type": "euclidean_spearman", "value": 55.13215936702871}, {"type": "manhattan_pearson", "value": 55.41429518223402}, {"type": "manhattan_spearman", "value": 55.13363087679285}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB AmazonReviewsClassification (zh)", "type": "mteb/amazon_reviews_multi", "config": "zh", "split": "test", "revision": "1399c76144fd37290681b995c656ef9b2e06e26d"}, "metrics": [{"type": "accuracy", "value": 46.722}, {"type": "f1", "value": 45.039340641893205}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB BQ", "type": "C-MTEB/BQ", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "cos_sim_pearson", "value": 63.517830355554224}, {"type": "cos_sim_spearman", "value": 65.57007801018649}, {"type": "euclidean_pearson", "value": 64.05153340906585}, {"type": "euclidean_spearman", "value": 65.5696865661119}, {"type": "manhattan_pearson", "value": 63.95710619755406}, {"type": "manhattan_spearman", "value": 65.48565785379489}]}, {"task": {"type": "Clustering"}, "dataset": {"name": "MTEB CLSClusteringP2P", "type": "C-MTEB/CLSClusteringP2P", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "v_measure", "value": 43.24046498507819}]}, {"task": {"type": "Clustering"}, "dataset": {"name": "MTEB CLSClusteringS2S", "type": "C-MTEB/CLSClusteringS2S", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "v_measure", "value": 41.22618199372116}]}, {"task": {"type": "Reranking"}, "dataset": {"name": "MTEB CMedQAv1", "type": "C-MTEB/CMedQAv1-reranking", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "map", "value": 87.12213224673621}, {"type": "mrr", "value": 89.57150793650794}]}, {"task": {"type": "Reranking"}, "dataset": {"name": "MTEB CMedQAv2", "type": "C-MTEB/CMedQAv2-reranking", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "map", "value": 87.57290061886421}, {"type": "mrr", "value": 90.19202380952382}]}, {"task": {"type": "Retrieval"}, "dataset": {"name": "MTEB CmedqaRetrieval", "type": "C-MTEB/CmedqaRetrieval", "config": "default", "split": "dev", "revision": "None"}, "metrics": [{"type": "map_at_1", "value": 25.22}, {"type": "map_at_10", "value": 37.604}, {"type": "map_at_100", "value": 39.501}, {"type": "map_at_1000", "value": 39.614}, {"type": "map_at_3", "value": 33.378}, {"type": "map_at_5", "value": 35.774}, {"type": "mrr_at_1", "value": 38.385000000000005}, {"type": "mrr_at_10", "value": 46.487}, {"type": "mrr_at_100", "value": 47.504999999999995}, {"type": "mrr_at_1000", "value": 47.548}, {"type": "mrr_at_3", "value": 43.885999999999996}, {"type": "mrr_at_5", "value": 45.373000000000005}, {"type": "ndcg_at_1", "value": 38.385000000000005}, {"type": "ndcg_at_10", "value": 44.224999999999994}, {"type": "ndcg_at_100", "value": 51.637}, {"type": "ndcg_at_1000", "value": 53.55799999999999}, {"type": "ndcg_at_3", "value": 38.845}, {"type": "ndcg_at_5", "value": 41.163}, {"type": "precision_at_1", "value": 38.385000000000005}, {"type": "precision_at_10", "value": 9.812}, {"type": "precision_at_100", "value": 1.58}, {"type": "precision_at_1000", "value": 0.183}, {"type": "precision_at_3", "value": 21.88}, {"type": "precision_at_5", "value": 15.974}, {"type": "recall_at_1", "value": 25.22}, {"type": "recall_at_10", "value": 54.897}, {"type": "recall_at_100", "value": 85.469}, {"type": "recall_at_1000", "value": 98.18599999999999}, {"type": "recall_at_3", "value": 38.815}, {"type": "recall_at_5", "value": 45.885}]}, {"task": {"type": "PairClassification"}, "dataset": {"name": "MTEB Cmnli", "type": "C-MTEB/CMNLI", "config": "default", "split": "validation", "revision": "None"}, "metrics": [{"type": "cos_sim_accuracy", "value": 83.22309079975948}, {"type": "cos_sim_ap", "value": 89.94833400328307}, {"type": "cos_sim_f1", "value": 84.39319055464031}, {"type": "cos_sim_precision", "value": 79.5774647887324}, {"type": "cos_sim_recall", "value": 89.82931961655366}, {"type": "dot_accuracy", "value": 83.22309079975948}, {"type": "dot_ap", "value": 89.95618559578415}, {"type": "dot_f1", "value": 84.41173239591345}, {"type": "dot_precision", "value": 79.61044343141317}, {"type": "dot_recall", "value": 89.82931961655366}, {"type": "euclidean_accuracy", "value": 83.23511725796753}, {"type": "euclidean_ap", "value": 89.94836342787318}, {"type": "euclidean_f1", "value": 84.40550133096718}, {"type": "euclidean_precision", "value": 80.29120067524794}, {"type": "euclidean_recall", "value": 88.9642272620996}, {"type": "manhattan_accuracy", "value": 83.23511725796753}, {"type": "manhattan_ap", "value": 89.9450103956978}, {"type": "manhattan_f1", "value": 84.44444444444444}, {"type": "manhattan_precision", "value": 80.09647651006712}, {"type": "manhattan_recall", "value": 89.29155950432546}, {"type": "max_accuracy", "value": 83.23511725796753}, {"type": "max_ap", "value": 89.95618559578415}, {"type": "max_f1", "value": 84.44444444444444}]}, {"task": {"type": "Retrieval"}, "dataset": {"name": "MTEB CovidRetrieval", "type": "C-MTEB/CovidRetrieval", "config": "default", "split": "dev", "revision": "None"}, "metrics": [{"type": "map_at_1", "value": 76.87}, {"type": "map_at_10", "value": 84.502}, {"type": "map_at_100", "value": 84.615}, {"type": "map_at_1000", "value": 84.617}, {"type": "map_at_3", "value": 83.127}, {"type": "map_at_5", "value": 83.99600000000001}, {"type": "mrr_at_1", "value": 77.02799999999999}, {"type": "mrr_at_10", "value": 84.487}, {"type": "mrr_at_100", "value": 84.59299999999999}, {"type": "mrr_at_1000", "value": 84.59400000000001}, {"type": "mrr_at_3", "value": 83.193}, {"type": "mrr_at_5", "value": 83.994}, {"type": "ndcg_at_1", "value": 77.134}, {"type": "ndcg_at_10", "value": 87.68599999999999}, {"type": "ndcg_at_100", "value": 88.17099999999999}, {"type": "ndcg_at_1000", "value": 88.21}, {"type": "ndcg_at_3", "value": 84.993}, {"type": "ndcg_at_5", "value": 86.519}, {"type": "precision_at_1", "value": 77.134}, {"type": "precision_at_10", "value": 9.841999999999999}, {"type": "precision_at_100", "value": 1.006}, {"type": "precision_at_1000", "value": 0.101}, {"type": "precision_at_3", "value": 30.313000000000002}, {"type": "precision_at_5", "value": 18.945999999999998}, {"type": "recall_at_1", "value": 76.87}, {"type": "recall_at_10", "value": 97.418}, {"type": "recall_at_100", "value": 99.579}, {"type": "recall_at_1000", "value": 99.895}, {"type": "recall_at_3", "value": 90.227}, {"type": "recall_at_5", "value": 93.888}]}, {"task": {"type": "Retrieval"}, "dataset": {"name": "MTEB DuRetrieval", "type": "C-MTEB/DuRetrieval", "config": "default", "split": "dev", "revision": "None"}, "metrics": [{"type": "map_at_1", "value": 25.941}, {"type": "map_at_10", "value": 78.793}, {"type": "map_at_100", "value": 81.57799999999999}, {"type": "map_at_1000", "value": 81.626}, {"type": "map_at_3", "value": 54.749}, {"type": "map_at_5", "value": 69.16}, {"type": "mrr_at_1", "value": 90.45}, {"type": "mrr_at_10", "value": 93.406}, {"type": "mrr_at_100", "value": 93.453}, {"type": "mrr_at_1000", "value": 93.45700000000001}, {"type": "mrr_at_3", "value": 93.10000000000001}, {"type": "mrr_at_5", "value": 93.27499999999999}, {"type": "ndcg_at_1", "value": 90.45}, {"type": "ndcg_at_10", "value": 86.44500000000001}, {"type": "ndcg_at_100", "value": 89.28399999999999}, {"type": "ndcg_at_1000", "value": 89.739}, {"type": "ndcg_at_3", "value": 85.62100000000001}, {"type": "ndcg_at_5", "value": 84.441}, {"type": "precision_at_1", "value": 90.45}, {"type": "precision_at_10", "value": 41.19}, {"type": "precision_at_100", "value": 4.761}, {"type": "precision_at_1000", "value": 0.48700000000000004}, {"type": "precision_at_3", "value": 76.583}, {"type": "precision_at_5", "value": 64.68}, {"type": "recall_at_1", "value": 25.941}, {"type": "recall_at_10", "value": 87.443}, {"type": "recall_at_100", "value": 96.54}, {"type": "recall_at_1000", "value": 98.906}, {"type": "recall_at_3", "value": 56.947}, {"type": "recall_at_5", "value": 73.714}]}, {"task": {"type": "Retrieval"}, "dataset": {"name": "MTEB EcomRetrieval", "type": "C-MTEB/EcomRetrieval", "config": "default", "split": "dev", "revision": "None"}, "metrics": [{"type": "map_at_1", "value": 52.900000000000006}, {"type": "map_at_10", "value": 63.144}, {"type": "map_at_100", "value": 63.634}, {"type": "map_at_1000", "value": 63.644999999999996}, {"type": "map_at_3", "value": 60.817}, {"type": "map_at_5", "value": 62.202}, {"type": "mrr_at_1", "value": 52.900000000000006}, {"type": "mrr_at_10", "value": 63.144}, {"type": "mrr_at_100", "value": 63.634}, {"type": "mrr_at_1000", "value": 63.644999999999996}, {"type": "mrr_at_3", "value": 60.817}, {"type": "mrr_at_5", "value": 62.202}, {"type": "ndcg_at_1", "value": 52.900000000000006}, {"type": "ndcg_at_10", "value": 68.042}, {"type": "ndcg_at_100", "value": 70.417}, {"type": "ndcg_at_1000", "value": 70.722}, {"type": "ndcg_at_3", "value": 63.287000000000006}, {"type": "ndcg_at_5", "value": 65.77}, {"type": "precision_at_1", "value": 52.900000000000006}, {"type": "precision_at_10", "value": 8.34}, {"type": "precision_at_100", "value": 0.9450000000000001}, {"type": "precision_at_1000", "value": 0.097}, {"type": "precision_at_3", "value": 23.467}, {"type": "precision_at_5", "value": 15.28}, {"type": "recall_at_1", "value": 52.900000000000006}, {"type": "recall_at_10", "value": 83.39999999999999}, {"type": "recall_at_100", "value": 94.5}, {"type": "recall_at_1000", "value": 96.89999999999999}, {"type": "recall_at_3", "value": 70.39999999999999}, {"type": "recall_at_5", "value": 76.4}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB IFlyTek", "type": "C-MTEB/IFlyTek-classification", "config": "default", "split": "validation", "revision": "None"}, "metrics": [{"type": "accuracy", "value": 49.74220854174683}, {"type": "f1", "value": 38.01399980618159}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB JDReview", "type": "C-MTEB/JDReview-classification", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "accuracy", "value": 86.73545966228893}, {"type": "ap", "value": 55.72394235169542}, {"type": "f1", "value": 81.58550390953492}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB LCQMC", "type": "C-MTEB/LCQMC", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "cos_sim_pearson", "value": 69.96711977441642}, {"type": "cos_sim_spearman", "value": 75.54747609685569}, {"type": "euclidean_pearson", "value": 74.62663478056035}, {"type": "euclidean_spearman", "value": 75.54761576699639}, {"type": "manhattan_pearson", "value": 74.60983904582241}, {"type": "manhattan_spearman", "value": 75.52758938061503}]}, {"task": {"type": "Reranking"}, "dataset": {"name": "MTEB MMarcoReranking", "type": "C-MTEB/Mmarco-reranking", "config": "default", "split": "dev", "revision": "None"}, "metrics": [{"type": "map", "value": 28.076927649720986}, {"type": "mrr", "value": 26.98015873015873}]}, {"task": {"type": "Retrieval"}, "dataset": {"name": "MTEB MMarcoRetrieval", "type": "C-MTEB/MMarcoRetrieval", "config": "default", "split": "dev", "revision": "None"}, "metrics": [{"type": "map_at_1", "value": 65.58}, {"type": "map_at_10", "value": 74.763}, {"type": "map_at_100", "value": 75.077}, {"type": "map_at_1000", "value": 75.091}, {"type": "map_at_3", "value": 72.982}, {"type": "map_at_5", "value": 74.155}, {"type": "mrr_at_1", "value": 67.822}, {"type": "mrr_at_10", "value": 75.437}, {"type": "mrr_at_100", "value": 75.702}, {"type": "mrr_at_1000", "value": 75.715}, {"type": "mrr_at_3", "value": 73.91799999999999}, {"type": "mrr_at_5", "value": 74.909}, {"type": "ndcg_at_1", "value": 67.822}, {"type": "ndcg_at_10", "value": 78.472}, {"type": "ndcg_at_100", "value": 79.891}, {"type": "ndcg_at_1000", "value": 80.262}, {"type": "ndcg_at_3", "value": 75.138}, {"type": "ndcg_at_5", "value": 77.094}, {"type": "precision_at_1", "value": 67.822}, {"type": "precision_at_10", "value": 9.474}, {"type": "precision_at_100", "value": 1.019}, {"type": "precision_at_1000", "value": 0.105}, {"type": "precision_at_3", "value": 28.281}, {"type": "precision_at_5", "value": 18.017}, {"type": "recall_at_1", "value": 65.58}, {"type": "recall_at_10", "value": 89.18599999999999}, {"type": "recall_at_100", "value": 95.64399999999999}, {"type": "recall_at_1000", "value": 98.541}, {"type": "recall_at_3", "value": 80.455}, {"type": "recall_at_5", "value": 85.063}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (zh-CN)", "type": "mteb/amazon_massive_intent", "config": "zh-CN", "split": "test", "revision": "31efe3c427b0bae9c22cbb560b8f15491cc6bed7"}, "metrics": [{"type": "accuracy", "value": 72.86819098856758}, {"type": "f1", "value": 70.25369778283451}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (zh-CN)", "type": "mteb/amazon_massive_scenario", "config": "zh-CN", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 75.46738399462004}, {"type": "f1", "value": 75.02466838130249}]}, {"task": {"type": "Retrieval"}, "dataset": {"name": "MTEB MedicalRetrieval", "type": "C-MTEB/MedicalRetrieval", "config": "default", "split": "dev", "revision": "None"}, "metrics": [{"type": "map_at_1", "value": 53.300000000000004}, {"type": "map_at_10", "value": 60.072}, {"type": "map_at_100", "value": 60.618}, {"type": "map_at_1000", "value": 60.659}, {"type": "map_at_3", "value": 58.550000000000004}, {"type": "map_at_5", "value": 59.425}, {"type": "mrr_at_1", "value": 53.5}, {"type": "mrr_at_10", "value": 60.187999999999995}, {"type": "mrr_at_100", "value": 60.73499999999999}, {"type": "mrr_at_1000", "value": 60.775999999999996}, {"type": "mrr_at_3", "value": 58.667}, {"type": "mrr_at_5", "value": 59.541999999999994}, {"type": "ndcg_at_1", "value": 53.300000000000004}, {"type": "ndcg_at_10", "value": 63.376999999999995}, {"type": "ndcg_at_100", "value": 66.24600000000001}, {"type": "ndcg_at_1000", "value": 67.408}, {"type": "ndcg_at_3", "value": 60.211000000000006}, {"type": "ndcg_at_5", "value": 61.781}, {"type": "precision_at_1", "value": 53.300000000000004}, {"type": "precision_at_10", "value": 7.380000000000001}, {"type": "precision_at_100", "value": 0.877}, {"type": "precision_at_1000", "value": 0.097}, {"type": "precision_at_3", "value": 21.667}, {"type": "precision_at_5", "value": 13.76}, {"type": "recall_at_1", "value": 53.300000000000004}, {"type": "recall_at_10", "value": 73.8}, {"type": "recall_at_100", "value": 87.7}, {"type": "recall_at_1000", "value": 97.0}, {"type": "recall_at_3", "value": 65.0}, {"type": "recall_at_5", "value": 68.8}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MultilingualSentiment", "type": "C-MTEB/MultilingualSentiment-classification", "config": "default", "split": "validation", "revision": "None"}, "metrics": [{"type": "accuracy", "value": 76.27666666666667}, {"type": "f1", "value": 76.31280038435165}]}, {"task": {"type": "PairClassification"}, "dataset": {"name": "MTEB Ocnli", "type": "C-MTEB/OCNLI", "config": "default", "split": "validation", "revision": "None"}, "metrics": [{"type": "cos_sim_accuracy", "value": 78.72225230102869}, {"type": "cos_sim_ap", "value": 80.63941899467723}, {"type": "cos_sim_f1", "value": 80.52190121155638}, {"type": "cos_sim_precision", "value": 72.06005004170142}, {"type": "cos_sim_recall", "value": 91.23548046462513}, {"type": "dot_accuracy", "value": 78.72225230102869}, {"type": "dot_ap", "value": 80.63913939812744}, {"type": "dot_f1", "value": 80.51948051948052}, {"type": "dot_precision", "value": 71.7948717948718}, {"type": "dot_recall", "value": 91.65786694825766}, {"type": "euclidean_accuracy", "value": 78.72225230102869}, {"type": "euclidean_ap", "value": 80.64403797436798}, {"type": "euclidean_f1", "value": 80.52190121155638}, {"type": "euclidean_precision", "value": 72.06005004170142}, {"type": "euclidean_recall", "value": 91.23548046462513}, {"type": "manhattan_accuracy", "value": 78.18083378451544}, {"type": "manhattan_ap", "value": 80.5241189302444}, {"type": "manhattan_f1", "value": 80.43478260869566}, {"type": "manhattan_precision", "value": 72.7972626176219}, {"type": "manhattan_recall", "value": 89.86272439281943}, {"type": "max_accuracy", "value": 78.72225230102869}, {"type": "max_ap", "value": 80.64403797436798}, {"type": "max_f1", "value": 80.52190121155638}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB OnlineShopping", "type": "C-MTEB/OnlineShopping-classification", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "accuracy", "value": 92.49000000000001}, {"type": "ap", "value": 90.66330807324402}, {"type": "f1", "value": 92.48245049107115}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB PAWSX", "type": "C-MTEB/PAWSX", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "cos_sim_pearson", "value": 33.6275431596535}, {"type": "cos_sim_spearman", "value": 37.865700050451494}, {"type": "euclidean_pearson", "value": 38.1050665279388}, {"type": "euclidean_spearman", "value": 37.864125056066364}, {"type": "manhattan_pearson", "value": 38.11206873232881}, {"type": "manhattan_spearman", "value": 37.852977098473936}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB QBQTC", "type": "C-MTEB/QBQTC", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "cos_sim_pearson", "value": 32.137955501231104}, {"type": "cos_sim_spearman", "value": 33.68610910423116}, {"type": "euclidean_pearson", "value": 32.155444753547926}, {"type": "euclidean_spearman", "value": 33.685799252964124}, {"type": "manhattan_pearson", "value": 32.14490855334317}, {"type": "manhattan_spearman", "value": 33.656549820048554}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STS22 (zh)", "type": "mteb/sts22-crosslingual-sts", "config": "zh", "split": "test", "revision": "6d1ba47164174a496b7fa5d3569dae26a6813b80"}, "metrics": [{"type": "cos_sim_pearson", "value": 63.63884916818661}, {"type": "cos_sim_spearman", "value": 64.3217581571435}, {"type": "euclidean_pearson", "value": 63.475760085926055}, {"type": "euclidean_spearman", "value": 64.31638169371887}, {"type": "manhattan_pearson", "value": 64.39677572604752}, {"type": "manhattan_spearman", "value": 64.85585019406021}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STSB", "type": "C-MTEB/STSB", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "cos_sim_pearson", "value": 79.74698333415277}, {"type": "cos_sim_spearman", "value": 81.1850043859317}, {"type": "euclidean_pearson", "value": 80.94512578669881}, {"type": "euclidean_spearman", "value": 81.18825478390181}, {"type": "manhattan_pearson", "value": 80.88114336824758}, {"type": "manhattan_spearman", "value": 81.12266715583868}]}, {"task": {"type": "Reranking"}, "dataset": {"name": "MTEB T2Reranking", "type": "C-MTEB/T2Reranking", "config": "default", "split": "dev", "revision": "None"}, "metrics": [{"type": "map", "value": 66.59971552953814}, {"type": "mrr", "value": 76.42177408088038}]}, {"task": {"type": "Retrieval"}, "dataset": {"name": "MTEB T2Retrieval", "type": "C-MTEB/T2Retrieval", "config": "default", "split": "dev", "revision": "None"}, "metrics": [{"type": "map_at_1", "value": 28.825}, {"type": "map_at_10", "value": 77.48899999999999}, {"type": "map_at_100", "value": 81.144}, {"type": "map_at_1000", "value": 81.216}, {"type": "map_at_3", "value": 55.435}, {"type": "map_at_5", "value": 67.496}, {"type": "mrr_at_1", "value": 91.377}, {"type": "mrr_at_10", "value": 94.062}, {"type": "mrr_at_100", "value": 94.122}, {"type": "mrr_at_1000", "value": 94.123}, {"type": "mrr_at_3", "value": 93.709}, {"type": "mrr_at_5", "value": 93.932}, {"type": "ndcg_at_1", "value": 91.377}, {"type": "ndcg_at_10", "value": 85.44800000000001}, {"type": "ndcg_at_100", "value": 89.11099999999999}, {"type": "ndcg_at_1000", "value": 89.752}, {"type": "ndcg_at_3", "value": 87.262}, {"type": "ndcg_at_5", "value": 85.668}, {"type": "precision_at_1", "value": 91.377}, {"type": "precision_at_10", "value": 41.525}, {"type": "precision_at_100", "value": 4.989}, {"type": "precision_at_1000", "value": 0.516}, {"type": "precision_at_3", "value": 75.452}, {"type": "precision_at_5", "value": 62.785000000000004}, {"type": "recall_at_1", "value": 28.825}, {"type": "recall_at_10", "value": 84.202}, {"type": "recall_at_100", "value": 95.768}, {"type": "recall_at_1000", "value": 98.791}, {"type": "recall_at_3", "value": 57.284}, {"type": "recall_at_5", "value": 71.071}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB TNews", "type": "C-MTEB/TNews-classification", "config": "default", "split": "validation", "revision": "None"}, "metrics": [{"type": "accuracy", "value": 52.160000000000004}, {"type": "f1", "value": 50.49492950548829}]}, {"task": {"type": "Clustering"}, "dataset": {"name": "MTEB ThuNewsClusteringP2P", "type": "C-MTEB/ThuNewsClusteringP2P", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "v_measure", "value": 70.06019845009966}]}, {"task": {"type": "Clustering"}, "dataset": {"name": "MTEB ThuNewsClusteringS2S", "type": "C-MTEB/ThuNewsClusteringS2S", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "v_measure", "value": 63.9370959228245}]}, {"task": {"type": "Retrieval"}, "dataset": {"name": "MTEB VideoRetrieval", "type": "C-MTEB/VideoRetrieval", "config": "default", "split": "dev", "revision": "None"}, "metrics": [{"type": "map_at_1", "value": 60.0}, {"type": "map_at_10", "value": 69.362}, {"type": "map_at_100", "value": 69.819}, {"type": "map_at_1000", "value": 69.833}, {"type": "map_at_3", "value": 67.783}, {"type": "map_at_5", "value": 68.71300000000001}, {"type": "mrr_at_1", "value": 60.0}, {"type": "mrr_at_10", "value": 69.362}, {"type": "mrr_at_100", "value": 69.819}, {"type": "mrr_at_1000", "value": 69.833}, {"type": "mrr_at_3", "value": 67.783}, {"type": "mrr_at_5", "value": 68.71300000000001}, {"type": "ndcg_at_1", "value": 60.0}, {"type": "ndcg_at_10", "value": 73.59400000000001}, {"type": "ndcg_at_100", "value": 75.734}, {"type": "ndcg_at_1000", "value": 76.049}, {"type": "ndcg_at_3", "value": 70.33}, {"type": "ndcg_at_5", "value": 72.033}, {"type": "precision_at_1", "value": 60.0}, {"type": "precision_at_10", "value": 8.67}, {"type": "precision_at_100", "value": 0.9650000000000001}, {"type": "precision_at_1000", "value": 0.099}, {"type": "precision_at_3", "value": 25.900000000000002}, {"type": "precision_at_5", "value": 16.38}, {"type": "recall_at_1", "value": 60.0}, {"type": "recall_at_10", "value": 86.7}, {"type": "recall_at_100", "value": 96.5}, {"type": "recall_at_1000", "value": 98.9}, {"type": "recall_at_3", "value": 77.7}, {"type": "recall_at_5", "value": 81.89999999999999}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB Waimai", "type": "C-MTEB/waimai-classification", "config": "default", "split": "test", "revision": "None"}, "metrics": [{"type": "accuracy", "value": 88.36}, {"type": "ap", "value": 73.25144216855439}, {"type": "f1", "value": 86.75076261442027}]}]}]}
transformers pytorch bert feature-extraction mteb model-index endpoints_compatible region:us