Multilingual-E5-small-distill-base
This model is an attempt to distill intfloat/multilingual-e5-base (teacher) into intfloat/multilingual-e5-small (student),
as well as applying Matryoshka Representation Learning to it.
This was made by trying an L2 loss to teach the student model to match the same cosine similarity on text pairs as the teacher model.
The distillation dataset is composed of about 700k multilingual sentences pairs sampled for the following 3 datasets:
For code, see this github repository
Multilingual E5 Text Embeddings: A Technical Report. Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, Furu Wei, arXiv 2024
This model has 12 layers and the embedding size is 384.
Usage
Below is an example to encode queries and passages from the MS-MARCO passage ranking dataset.
import torch.nn.functional as F
from torch import Tensor
from transformers import AutoTokenizer, AutoModel
def average_pool(last_hidden_states: Tensor,
attention_mask: Tensor) -> Tensor:
last_hidden = last_hidden_states.masked_fill(~attention_mask[..., None].bool(), 0.0)
return last_hidden.sum(dim=1) / attention_mask.sum(dim=1)[..., None]
# Each input text should start with "query: " or "passage: ", even for non-English texts.
# For tasks other than retrieval, you can simply use the "query: " prefix.
input_texts = ['query: how much protein should a female eat',
'query: ๅ็็ๅฎถๅธธๅๆณ',
"passage: As a general guideline, the CDC's average requirement of protein for women ages 19 to 70 is 46 grams per day. But, as you can see from this chart, you'll need to increase that if you're expecting or training for a marathon. Check out the chart below to see how much protein you should be eating each day.",
"passage: 1.ๆธ
็ๅ็ไธ ๅๆ:ๅซฉๅ็ๅไธช ่ฐๆ:่ฑใ็ใ็ฝ็ณใ้ธก็ฒพ ๅๆณ: 1ใๅ็็จๅ่่็ๅๅป่กจ้ขไธๅฑ็ฎ,็จๅบๅญๅฎๅป็ค 2ใๆฆๆ็ปไธ(ๆฒกๆๆฆ่ๆฟๅฐฑ็จๅๆ
ขๆ
ขๅๆ็ปไธ) 3ใ้
็ง็ญๆพๆฒน,ๅ
ฅ่ฑ่ฑ็
ธๅบ้ฆๅณ 4ใๅ
ฅๅ็ไธๅฟซ้็ฟป็ไธๅ้ๅทฆๅณ,ๆพ็ใไธ็น็ฝ็ณๅ้ธก็ฒพ่ฐๅณๅบ้
2.้ฆ่ฑ็ๅ็ ๅๆ:ๅ็1ๅช ่ฐๆ:้ฆ่ฑใ่ๆซใๆฉๆฆๆฒนใ็ ๅๆณ: 1ใๅฐๅ็ๅป็ฎ,ๅๆ็ 2ใๆฒน้
8ๆ็ญๅ,ๅฐ่ๆซๆพๅ
ฅ็้ฆ 3ใ็้ฆๅ,ๅฐๅ็็ๆพๅ
ฅ,็ฟป็ 4ใๅจ็ฟป็็ๅๆถ,ๅฏไปฅไธๆถๅฐๅพ้
้ๅ ๆฐด,ไฝไธ่ฆๅคชๅค 5ใๆพๅ
ฅ็,็ๅ 6ใๅ็ๅทฎไธๅค่ฝฏๅ็ปตไบไนๅ,ๅฐฑๅฏไปฅๅ
ณ็ซ 7ใๆๅ
ฅ้ฆ่ฑ,ๅณๅฏๅบ้
"]
tokenizer = AutoTokenizer.from_pretrained('intfloat/multilingual-e5-small')
model = AutoModel.from_pretrained('intfloat/multilingual-e5-small')
# Tokenize the input texts
batch_dict = tokenizer(input_texts, max_length=512, padding=True, truncation=True, return_tensors='pt')
outputs = model(**batch_dict)
embeddings = average_pool(outputs.last_hidden_state, batch_dict['attention_mask'])
# normalize embeddings
embeddings = F.normalize(embeddings, p=2, dim=1)
scores = (embeddings[:2] @ embeddings[2:].T) * 100
print(scores.tolist())
Supported Languages
This model is initialized from microsoft/Multilingual-MiniLM-L12-H384 and continually trained on a mixture of multilingual datasets. It supports 100 languages from xlm-roberta, but low-resource languages may see performance degradation.
MTEB Benchmark Evaluation (Subset)
| intfloat/multilingual-e5-base | intfloat/multilingual-e5-large | intfloat/multilingual-e5-small | avditvs/multilingual-e5-small-distill-base-0.1 | |
|---|---|---|---|---|
| STS15 | 0.876 | 0.882 | 0.864 | 0.865 |
| BIOSSES | 0.870 | 0.863 | 0.857 | 0.863 |
| STS14 | 0.789 | 0.776 | 0.788 | 0.803 |
| STS12 | 0.858 | 0.873 | 0.854 | 0.856 |
| AskUbuntuDupQuestions | 0.571 | 0.577 | 0.568 | 0.574 |
| StackOverflowDupQuestions | 0.485 | 0.486 | 0.486 | 0.485 |
| AmazonReviewsClassification | 0.476 | 0.470 | 0.452 | 0.450 |
| ArguAna | 0.442 | 0.544 | 0.391 | 0.480 |
| ImdbClassification | 0.849 | 0.887 | 0.758 | 0.757 |
| STS13 | 0.756 | 0.751 | 0.764 | 0.785 |
| STSBenchmark | 0.832 | 0.836 | 0.809 | 0.818 |
| STS17 | 0.890 | 0.896 | 0.868 | 0.871 |
| SICK-R | 0.835 | 0.838 | 0.835 | 0.850 |
| STS22 | 0.645 | 0.675 | 0.640 | 0.648 |
| STS16 | 0.814 | 0.824 | 0.822 | 0.820 |
| Banking77Classification | 0.741 | 0.749 | 0.706 | 0.706 |
| average | 0.733 | 0.745 | 0.717 | 0.727 |
Support for Sentence Transformers
Below is an example for usage with sentence_transformers.
from sentence_transformers import SentenceTransformer
model = SentenceTransformer('avditvs/multilingual-e5-small-distill-base')
input_texts = [
'query: how much protein should a female eat',
'query: ๅ็็ๅฎถๅธธๅๆณ',
"passage: As a general guideline, the CDC's average requirement of protein for women ages 19 to 70 i s 46 grams per day. But, as you can see from this chart, you'll need to increase that if you're expecting or traini ng for a marathon. Check out the chart below to see how much protein you should be eating each day.",
"passage: 1.ๆธ
็ๅ็ไธ ๅๆ:ๅซฉๅ็ๅไธช ่ฐๆ:่ฑใ็ใ็ฝ็ณใ้ธก็ฒพ ๅๆณ: 1ใๅ็็จๅ่่็ๅๅป่กจ้ขไธๅฑ็ฎ ,็จๅบๅญๅฎๅป็ค 2ใๆฆๆ็ปไธ(ๆฒกๆๆฆ่ๆฟๅฐฑ็จๅๆ
ขๆ
ขๅๆ็ปไธ) 3ใ้
็ง็ญๆพๆฒน,ๅ
ฅ่ฑ่ฑ็
ธๅบ้ฆๅณ 4ใๅ
ฅๅ็ไธๅฟซ้็ฟป็ไธๅ้ๅทฆๅณ, ๆพ็ใไธ็น็ฝ็ณๅ้ธก็ฒพ่ฐๅณๅบ้
2.้ฆ่ฑ็ๅ็ ๅๆ:ๅ็1ๅช ่ฐๆ:้ฆ่ฑใ่ๆซใๆฉๆฆๆฒนใ็ ๅๆณ: 1ใๅฐๅ็ๅป็ฎ,ๅๆ็ 2ใๆฒน ้
8ๆ็ญๅ,ๅฐ่ๆซๆพๅ
ฅ็้ฆ 3ใ็้ฆๅ,ๅฐๅ็็ๆพๅ
ฅ,็ฟป็ 4ใๅจ็ฟป็็ๅๆถ,ๅฏไปฅไธๆถๅฐๅพ้
้ๅ ๆฐด,ไฝไธ่ฆๅคชๅค 5ใๆพๅ
ฅ็,็ๅ 6ใๅ็ๅทฎไธๅค่ฝฏๅ็ปตไบไนๅ,ๅฐฑๅฏไปฅๅ
ณ็ซ 7ใๆๅ
ฅ้ฆ่ฑ,ๅณๅฏๅบ้
"
]
embeddings = model.encode(input_texts, normalize_embeddings=True)
Package requirements
pip install sentence_transformers~=2.2.2
Contributors: michaelfeil
FAQ
1. Do I need to add the prefix "query: " and "passage: " to input texts?
Yes, this is how the model is trained, otherwise you will see a performance degradation.
Here are some rules of thumb:
Use "query: " and "passage: " correspondingly for asymmetric tasks such as passage retrieval in open QA, ad-hoc information retrieval.
Use "query: " prefix for symmetric tasks such as semantic similarity, bitext mining, paraphrase retrieval.
Use "query: " prefix if you want to use embeddings as features, such as linear probing classification, clustering.
2. Why are my reproduced results slightly different from reported in the model card?
Different versions of transformers and pytorch could cause negligible but non-zero performance differences.
3. Why does the cosine similarity scores distribute around 0.7 to 1.0?
This is a known and expected behavior as we use a low temperature 0.01 for InfoNCE contrastive loss.
For text embedding tasks like text retrieval or semantic similarity, what matters is the relative order of the scores instead of the absolute values, so this should not be an issue.
Citation
If you find our paper or models helpful, please consider cite as follows:
@article{wang2024multilingual,
title={Multilingual E5 Text Embeddings: A Technical Report},
author={Wang, Liang and Yang, Nan and Huang, Xiaolong and Yang, Linjun and Majumder, Rangan and Wei, Furu},
journal={arXiv preprint arXiv:2402.05672},
year={2024}
}
Limitations
Long texts will be truncated to at most 512 tokens.
- Downloads last month
- 64