
# Wikipedia (de) embedded with cohere.ai `multilingual-22-12` encoder

We encoded [Wikipedia (de)](https://de.wikipedia.org) using the [cohere.ai](https://txt.cohere.ai/multilingual/) `multilingual-22-12` embedding model.

To get an overview how this dataset was created and pre-processed, have a look at [Cohere/wikipedia-22-12](https://huggingface.co/datasets/Cohere/wikipedia-22-12).


## Embeddings
We compute for `title+" "+text` the embeddings using our `multilingual-22-12` embedding model, a state-of-the-art model that works for semantic search in 100 languages.  If you want to learn more about this model, have a look at [cohere.ai multilingual embedding model](https://txt.cohere.ai/multilingual/).

## Further languages
We provide embeddings of Wikipedia in many different languages:
[ar](https://huggingface.co/datasets/Cohere/wikipedia-22-12-ar-embeddings), [de](https://huggingface.co/datasets/Cohere/wikipedia-22-12-de-embeddings), [en](https://huggingface.co/datasets/Cohere/wikipedia-22-12-en-embeddings), [es](https://huggingface.co/datasets/Cohere/wikipedia-22-12-es-embeddings), [fr](https://huggingface.co/datasets/Cohere/wikipedia-22-12-fr-embeddings),  [hi](https://huggingface.co/datasets/Cohere/wikipedia-22-12-hi-embeddings), [it](https://huggingface.co/datasets/Cohere/wikipedia-22-12-it-embeddings), [ja](https://huggingface.co/datasets/Cohere/wikipedia-22-12-ja-embeddings), [ko](https://huggingface.co/datasets/Cohere/wikipedia-22-12-ko-embeddings), [simple english](https://huggingface.co/datasets/Cohere/wikipedia-22-12-simple-embeddings), [zh](https://huggingface.co/datasets/Cohere/wikipedia-22-12-zh-embeddings),

You can find the Wikipedia datasets without embeddings at [Cohere/wikipedia-22-12](https://huggingface.co/datasets/Cohere/wikipedia-22-12).


## Loading the dataset
You can either load the dataset like this:
```python
from datasets import load_dataset
docs = load_dataset(f"Cohere/wikipedia-22-12-de-embeddings", split="train")
```

Or you can also stream it without downloading it before:
```python
from datasets import load_dataset
docs = load_dataset(f"Cohere/wikipedia-22-12-de-embeddings", split="train", streaming=True)

for doc in docs:
	docid = doc['id']
	title = doc['title']
	text = doc['text']
	emb = doc['emb']
```

## Search
A full search example:
```python
#Run: pip install cohere datasets
from datasets import load_dataset
import torch
import cohere

co = cohere.Client(f"<<COHERE_API_KEY>>")  # Add your cohere API key from www.cohere.com

#Load at max 1000 documents + embeddings
max_docs = 1000
docs_stream = load_dataset(f"Cohere/wikipedia-22-12-de-embeddings", split="train", streaming=True)

docs = []
doc_embeddings = []

for doc in docs_stream:
    docs.append(doc)
    doc_embeddings.append(doc['emb'])
    if len(docs) >= max_docs:
        break

doc_embeddings = torch.tensor(doc_embeddings)

query = 'Who founded Youtube'
response = co.embed(texts=[query], model='multilingual-22-12')
query_embedding = response.embeddings 
query_embedding = torch.tensor(query_embedding)

# Compute dot score between query embedding and document embeddings
dot_scores = torch.mm(query_embedding, doc_embeddings.transpose(0, 1))
top_k = torch.topk(dot_scores, k=3)

# Print results
print("Query:", query)
for doc_id in top_k.indices[0].tolist():
    print(docs[doc_id]['title'])
    print(docs[doc_id]['text'], "\n")
```


## Performance
You can find performance on the MIRACL dataset (a semantic search evaluation dataset) here: [miracl-en-queries-22-12#performance](https://huggingface.co/datasets/Cohere/miracl-en-queries-22-12#performance) 
task_categories:text-retrieval task_ids:document-retrieval annotations_creators:expert-generated multilinguality:multilingual language:de license:apache-2.0 region:us
{"annotations_creators": ["expert-generated"], "language": ["de"], "license": ["apache-2.0"], "multilinguality": ["multilingual"], "size_categories": [], "source_datasets": [], "task_categories": ["text-retrieval"], "task_ids": ["document-retrieval"], "tags": []}
TAGS
#task_categories-text-retrieval #task_ids-document-retrieval #annotations_creators-expert-generated #multilinguality-multilingual #language-German #license-apache-2.0 #region-us 


# Wikipedia (de) embedded with URL 'multilingual-22-12' encoder

We encoded Wikipedia (de) using the URL 'multilingual-22-12' embedding model.

To get an overview how this dataset was created and pre-processed, have a look at Cohere/wikipedia-22-12.


## Embeddings
We compute for 'title+" "+text' the embeddings using our 'multilingual-22-12' embedding model, a state-of-the-art model that works for semantic search in 100 languages.  If you want to learn more about this model, have a look at URL multilingual embedding model.

## Further languages
We provide embeddings of Wikipedia in many different languages:
ar, de, en, es, fr,  hi, it, ja, ko, simple english, zh,

You can find the Wikipedia datasets without embeddings at Cohere/wikipedia-22-12.


## Loading the dataset
You can either load the dataset like this:


Or you can also stream it without downloading it before:


## Search
A full search example:



## Performance
You can find performance on the MIRACL dataset (a semantic search evaluation dataset) here: miracl-en-queries-22-12#performance 
# Wikipedia (de) embedded with URL 'multilingual-22-12' encoder

We encoded Wikipedia (de) using the URL 'multilingual-22-12' embedding model.

To get an overview how this dataset was created and pre-processed, have a look at Cohere/wikipedia-22-12. ## Embeddings
We compute for 'title+" "+text' the embeddings using our 'multilingual-22-12' embedding model, a state-of-the-art model that works for semantic search in 100 languages.  If you want to learn more about this model, have a look at URL multilingual embedding model. ## Further languages
We provide embeddings of Wikipedia in many different languages:
ar, de, en, es, fr,  hi, it, ja, ko, simple english, zh,

You can find the Wikipedia datasets without embeddings at Cohere/wikipedia-22-12. ## Loading the dataset
You can either load the dataset like this:


Or you can also stream it without downloading it before: ## Search
A full search example: ## Performance
You can find performance on the MIRACL dataset (a semantic search evaluation dataset) here: miracl-en-queries-22-12#performance
TAGS
#task_categories-text-retrieval #task_ids-document-retrieval #annotations_creators-expert-generated #multilinguality-multilingual #language-German #license-apache-2.0 #region-us 
 # Wikipedia (de) embedded with URL 'multilingual-22-12' encoder

We encoded Wikipedia (de) using the URL 'multilingual-22-12' embedding model.

To get an overview how this dataset was created and pre-processed, have a look at Cohere/wikipedia-22-12. ## Embeddings
We compute for 'title+" "+text' the embeddings using our 'multilingual-22-12' embedding model, a state-of-the-art model that works for semantic search in 100 languages.  If you want to learn more about this model, have a look at URL multilingual embedding model. ## Further languages
We provide embeddings of Wikipedia in many different languages:
ar, de, en, es, fr,  hi, it, ja, ko, simple english, zh,

You can find the Wikipedia datasets without embeddings at Cohere/wikipedia-22-12. ## Loading the dataset
You can either load the dataset like this:


Or you can also stream it without downloading it before: ## Search
A full search example: ## Performance
You can find performance on the MIRACL dataset (a semantic search evaluation dataset) here: miracl-en-queries-22-12#performance
passage: TAGS
#task_categories-text-retrieval #task_ids-document-retrieval #annotations_creators-expert-generated #multilinguality-multilingual #language-German #license-apache-2.0 #region-us 
# Wikipedia (de) embedded with URL 'multilingual-22-12' encoder

We encoded Wikipedia (de) using the URL 'multilingual-22-12' embedding model.

To get an overview how this dataset was created and pre-processed, have a look at Cohere/wikipedia-22-12.## Embeddings
We compute for 'title+" "+text' the embeddings using our 'multilingual-22-12' embedding model, a state-of-the-art model that works for semantic search in 100 languages.  If you want to learn more about this model, have a look at URL multilingual embedding model.## Further languages
We provide embeddings of Wikipedia in many different languages:
ar, de, en, es, fr,  hi, it, ja, ko, simple english, zh,

You can find the Wikipedia datasets without embeddings at Cohere/wikipedia-22-12.## Loading the dataset
You can either load the dataset like this:


Or you can also stream it without downloading it before:## Search
A full search example:## Performance
You can find performance on the MIRACL dataset (a semantic search evaluation dataset) here: miracl-en-queries-22-12#performance