
# abstracts-embeddings

This is the embeddings of the titles and abstracts of 95 million academic publications taken from the [OpenAlex](https://openalex.org) dataset as of May 5, 2023. The script that generated the embeddings is available on [Github](https://github.com/colonelwatch/abstracts-search/blob/master/build.py), but the general process is as follows:

1. Reconstruct the text of the abstract from the inverted index format
2. Construct a single document string in the format `title + ' ' + abstract` or just `abstract` if there is no title
3. Determine if the document string is in English using [fastText](https://fasttext.cc/docs/en/language-identification.html)
4. If it is in English, compute an embedding using the `all-MiniLM-L6-v2` model provided by [sentence-transformers](https://www.sbert.net/)

Though the OpenAlex dataset records 240 million works, not all of these works have abstracts or are in English. However, the `all-MiniLM-L6-v2` model was only trained on English texts, hence the filtering.

## Dataset Structure

In the future, this dataset might become a parquet in order to admit all the features offered by Hugging Face Datasets, but it consists only of a text file and a numpy memmap for now. The memmap is an array of many length-384 `np.float16` vectors, and the i-th row vector in this array corresponds with the i-th line in the text file. The text file is just a list of ids that can be used to get more information from the OpenAlex API.

```python
import numpy as np

with open('openalex_ids.txt', 'r') as f:
    idxs = f.read().splitlines()

embeddings = np.memmap('embeddings.memmap', dtype=np.float16, mode='r').reshape(-1, 384)
```

However, the memmap cannot be uploaded to Hugging Face as a single file, so it's split with the command `split -b 3221225472 -d --suffix-length=3 --additional-suffix=.memmap embeddings.memmap embeddings_`. It can be put back together with the command `cat embeddings_*.memmap > embeddings.memmap`.

task_categories:text-retrieval task_ids:document-retrieval size_categories:10M<n<100m language:en license:cc0-1.0 region:us
{"language": ["en"], "license": "cc0-1.0", "size_categories": ["10M<n<100m"], "task_categories": ["text-retrieval"], "task_ids": ["document-retrieval"]}
TAGS
#task_categories-text-retrieval #task_ids-document-retrieval #size_categories-10M<n<100m #language-English #license-cc0-1.0 #region-us 


# abstracts-embeddings

This is the embeddings of the titles and abstracts of 95 million academic publications taken from the OpenAlex dataset as of May 5, 2023. The script that generated the embeddings is available on Github, but the general process is as follows:

1. Reconstruct the text of the abstract from the inverted index format
2. Construct a single document string in the format 'title + ' ' + abstract' or just 'abstract' if there is no title
3. Determine if the document string is in English using fastText
4. If it is in English, compute an embedding using the 'all-MiniLM-L6-v2' model provided by sentence-transformers

Though the OpenAlex dataset records 240 million works, not all of these works have abstracts or are in English. However, the 'all-MiniLM-L6-v2' model was only trained on English texts, hence the filtering.

## Dataset Structure

In the future, this dataset might become a parquet in order to admit all the features offered by Hugging Face Datasets, but it consists only of a text file and a numpy memmap for now. The memmap is an array of many length-384 'np.float16' vectors, and the i-th row vector in this array corresponds with the i-th line in the text file. The text file is just a list of ids that can be used to get more information from the OpenAlex API.



However, the memmap cannot be uploaded to Hugging Face as a single file, so it's split with the command 'split -b 3221225472 -d --suffix-length=3 --additional-suffix=.memmap URL embeddings_'. It can be put back together with the command 'cat embeddings_*.memmap > URL'.

# abstracts-embeddings

This is the embeddings of the titles and abstracts of 95 million academic publications taken from the OpenAlex dataset as of May 5, 2023. The script that generated the embeddings is available on Github, but the general process is as follows:

1. Reconstruct the text of the abstract from the inverted index format
2. Construct a single document string in the format 'title + ' ' + abstract' or just 'abstract' if there is no title
3. Determine if the document string is in English using fastText
4. If it is in English, compute an embedding using the 'all-MiniLM-L6-v2' model provided by sentence-transformers

Though the OpenAlex dataset records 240 million works, not all of these works have abstracts or are in English. However, the 'all-MiniLM-L6-v2' model was only trained on English texts, hence the filtering. ## Dataset Structure

In the future, this dataset might become a parquet in order to admit all the features offered by Hugging Face Datasets, but it consists only of a text file and a numpy memmap for now. The memmap is an array of many length-384 'np.float16' vectors, and the i-th row vector in this array corresponds with the i-th line in the text file. The text file is just a list of ids that can be used to get more information from the OpenAlex API.



However, the memmap cannot be uploaded to Hugging Face as a single file, so it's split with the command 'split -b 3221225472 -d --suffix-length=3 --additional-suffix=.memmap URL embeddings_'. It can be put back together with the command 'cat embeddings_*.memmap > URL'.
TAGS
#task_categories-text-retrieval #task_ids-document-retrieval #size_categories-10M<n<100m #language-English #license-cc0-1.0 #region-us 
 # abstracts-embeddings

This is the embeddings of the titles and abstracts of 95 million academic publications taken from the OpenAlex dataset as of May 5, 2023. The script that generated the embeddings is available on Github, but the general process is as follows:

1. Reconstruct the text of the abstract from the inverted index format
2. Construct a single document string in the format 'title + ' ' + abstract' or just 'abstract' if there is no title
3. Determine if the document string is in English using fastText
4. If it is in English, compute an embedding using the 'all-MiniLM-L6-v2' model provided by sentence-transformers

Though the OpenAlex dataset records 240 million works, not all of these works have abstracts or are in English. However, the 'all-MiniLM-L6-v2' model was only trained on English texts, hence the filtering. ## Dataset Structure

In the future, this dataset might become a parquet in order to admit all the features offered by Hugging Face Datasets, but it consists only of a text file and a numpy memmap for now. The memmap is an array of many length-384 'np.float16' vectors, and the i-th row vector in this array corresponds with the i-th line in the text file. The text file is just a list of ids that can be used to get more information from the OpenAlex API.



However, the memmap cannot be uploaded to Hugging Face as a single file, so it's split with the command 'split -b 3221225472 -d --suffix-length=3 --additional-suffix=.memmap URL embeddings_'. It can be put back together with the command 'cat embeddings_*.memmap > URL'.