
# NFCorpus: 20 generated queries (BEIR Benchmark)

This HF dataset contains the top-20 synthetic queries generated for each passage in the above BEIR benchmark dataset.

- DocT5query model used: [BeIR/query-gen-msmarco-t5-base-v1](https://huggingface.co/BeIR/query-gen-msmarco-t5-base-v1)
- id (str): unique document id in NFCorpus in the BEIR benchmark (`corpus.jsonl`).
- Questions generated: 20
- Code used for generation: [evaluate_anserini_docT5query_parallel.py](https://github.com/beir-cellar/beir/blob/main/examples/retrieval/evaluation/sparse/evaluate_anserini_docT5query_parallel.py)


Below contains the old dataset card for the BEIR benchmark.


# Dataset Card for BEIR Benchmark

## Table of Contents
- [Dataset Description](#dataset-description)
  - [Dataset Summary](#dataset-summary)
  - [Supported Tasks and Leaderboards](#supported-tasks-and-leaderboards)
  - [Languages](#languages)
- [Dataset Structure](#dataset-structure)
  - [Data Instances](#data-instances)
  - [Data Fields](#data-fields)
  - [Data Splits](#data-splits)
- [Dataset Creation](#dataset-creation)
  - [Curation Rationale](#curation-rationale)
  - [Source Data](#source-data)
  - [Annotations](#annotations)
  - [Personal and Sensitive Information](#personal-and-sensitive-information)
- [Considerations for Using the Data](#considerations-for-using-the-data)
  - [Social Impact of Dataset](#social-impact-of-dataset)
  - [Discussion of Biases](#discussion-of-biases)
  - [Other Known Limitations](#other-known-limitations)
- [Additional Information](#additional-information)
  - [Dataset Curators](#dataset-curators)
  - [Licensing Information](#licensing-information)
  - [Citation Information](#citation-information)
  - [Contributions](#contributions)

## Dataset Description

- **Homepage:** https://github.com/UKPLab/beir
- **Repository:** https://github.com/UKPLab/beir
- **Paper:** https://openreview.net/forum?id=wCu6T5xFjeJ
- **Leaderboard:** https://docs.google.com/spreadsheets/d/1L8aACyPaXrL8iEelJLGqlMqXKPX2oSP_R10pZoy77Ns
- **Point of Contact:** nandan.thakur@uwaterloo.ca

### Dataset Summary

BEIR is a heterogeneous benchmark that has been built from 18 diverse datasets representing 9 information retrieval tasks:

- Fact-checking: [FEVER](http://fever.ai), [Climate-FEVER](http://climatefever.ai), [SciFact](https://github.com/allenai/scifact)
- Question-Answering: [NQ](https://ai.google.com/research/NaturalQuestions), [HotpotQA](https://hotpotqa.github.io), [FiQA-2018](https://sites.google.com/view/fiqa/)
- Bio-Medical IR: [TREC-COVID](https://ir.nist.gov/covidSubmit/index.html), [BioASQ](http://bioasq.org), [NFCorpus](https://www.cl.uni-heidelberg.de/statnlpgroup/nfcorpus/)
- News Retrieval: [TREC-NEWS](https://trec.nist.gov/data/news2019.html), [Robust04](https://trec.nist.gov/data/robust/04.guidelines.html)
- Argument Retrieval: [Touche-2020](https://webis.de/events/touche-20/shared-task-1.html), [ArguAna](tp://argumentation.bplaced.net/arguana/data)
- Duplicate Question Retrieval: [Quora](https://www.quora.com/q/quoradata/First-Quora-Dataset-Release-Question-Pairs), [CqaDupstack](http://nlp.cis.unimelb.edu.au/resources/cqadupstack/)
- Citation-Prediction: [SCIDOCS](https://allenai.org/data/scidocs)
- Tweet Retrieval: [Signal-1M](https://research.signal-ai.com/datasets/signal1m-tweetir.html)
- Entity Retrieval: [DBPedia](https://github.com/iai-group/DBpedia-Entity/)

All these datasets have been preprocessed and can be used for your experiments.


```python

```

### Supported Tasks and Leaderboards

The dataset supports a leaderboard that evaluates models against task-specific metrics such as F1 or EM, as well as their ability to retrieve supporting information from Wikipedia.

The current best performing models can be found [here](https://eval.ai/web/challenges/challenge-page/689/leaderboard/).

### Languages

All tasks are in English (`en`).

## Dataset Structure

All BEIR datasets must contain a corpus, queries and qrels (relevance judgments file). They must be in the following format:
- `corpus` file: a `.jsonl` file (jsonlines) that contains a list of dictionaries, each with three fields `_id` with unique document identifier, `title` with document title (optional) and `text` with document paragraph or passage. For example: `{"_id": "doc1", "title": "Albert Einstein", "text": "Albert Einstein was a German-born...."}`
- `queries` file: a `.jsonl` file (jsonlines) that contains a list of dictionaries, each with two fields `_id` with unique query identifier and `text` with query text. For example: `{"_id": "q1", "text": "Who developed the mass-energy equivalence formula?"}`
- `qrels` file: a `.tsv` file (tab-seperated) that contains three columns, i.e. the `query-id`, `corpus-id` and `score` in this order. Keep 1st row as header. For example: `q1 doc1 1`

### Data Instances

A high level example of any beir dataset:

```python
corpus = {
    "doc1" : {
        "title": "Albert Einstein", 
        "text": "Albert Einstein was a German-born theoretical physicist. who developed the theory of relativity, \
                 one of the two pillars of modern physics (alongside quantum mechanics). His work is also known for \
                 its influence on the philosophy of science. He is best known to the general public for his massâ€“energy \
                 equivalence formula E = mc2, which has been dubbed 'the world's most famous equation'. He received the 1921 \
                 Nobel Prize in Physics 'for his services to theoretical physics, and especially for his discovery of the law \
                 of the photoelectric effect', a pivotal step in the development of quantum theory."
        },
    "doc2" : {
        "title": "", # Keep title an empty string if not present
        "text": "Wheat beer is a top-fermented beer which is brewed with a large proportion of wheat relative to the amount of \
                 malted barley. The two main varieties are German WeiÃŸbier and Belgian witbier; other types include Lambic (made\
                 with wild yeast), Berliner Weisse (a cloudy, sour beer), and Gose (a sour, salty beer)."
    },
}

queries = {
    "q1" : "Who developed the mass-energy equivalence formula?",
    "q2" : "Which beer is brewed with a large proportion of wheat?"
}

qrels = {
    "q1" : {"doc1": 1},
    "q2" : {"doc2": 1},
}
```

### Data Fields

Examples from all configurations have the following features:

### Corpus
- `corpus`: a `dict` feature representing the document title and passage text, made up of:
    - `_id`: a `string` feature representing the unique document id
        - `title`: a `string` feature, denoting the title of the document.
        - `text`: a `string` feature, denoting the text of the document.

### Queries
- `queries`: a `dict` feature representing the query, made up of:
    - `_id`: a `string` feature representing the unique query id
    - `text`: a `string` feature, denoting the text of the query.

### Qrels
- `qrels`: a `dict` feature representing the query document relevance judgements, made up of:
    - `_id`: a `string` feature representing the query id
        - `_id`: a `string` feature, denoting the document id.
        - `score`: a `int32` feature, denoting the relevance judgement between query and document.


### Data Splits

| Dataset   | Website| BEIR-Name | Type | Queries  | Corpus | Rel D/Q | Down-load | md5 |
| -------- | -----| ---------| --------- | ----------- | ---------| ---------| :----------: | :------:|
| MSMARCO    | [Homepage](https://microsoft.github.io/msmarco/)| ``msmarco`` | ``train``<br>``dev``<br>``test``|  6,980   |  8.84M     |    1.1 | [Link](https://public.ukp.informatik.tu-darmstadt.de/thakur/BEIR/datasets/msmarco.zip) | ``444067daf65d982533ea17ebd59501e4`` |
| TREC-COVID |  [Homepage](https://ir.nist.gov/covidSubmit/index.html)| ``trec-covid``| ``test``| 50|  171K| 493.5 | [Link](https://public.ukp.informatik.tu-darmstadt.de/thakur/BEIR/datasets/trec-covid.zip) | ``ce62140cb23feb9becf6270d0d1fe6d1`` |
| NFCorpus   | [Homepage](https://www.cl.uni-heidelberg.de/statnlpgroup/nfcorpus/) | ``nfcorpus`` | ``train``<br>``dev``<br>``test``|  323     |  3.6K     |  38.2 | [Link](https://public.ukp.informatik.tu-darmstadt.de/thakur/BEIR/datasets/nfcorpus.zip) | ``a89dba18a62ef92f7d323ec890a0d38d`` |
| BioASQ     | [Homepage](http://bioasq.org) | ``bioasq``|  ``train``<br>``test`` | 500    |  14.91M    |  8.05 | No | [How to Reproduce?](https://github.com/UKPLab/beir/blob/main/examples/dataset#2-bioasq) |
| NQ         | [Homepage](https://ai.google.com/research/NaturalQuestions) | ``nq``| ``train``<br>``test``| 3,452   |  2.68M  |  1.2 | [Link](https://public.ukp.informatik.tu-darmstadt.de/thakur/BEIR/datasets/nq.zip) | ``d4d3d2e48787a744b6f6e691ff534307`` |
| HotpotQA   | [Homepage](https://hotpotqa.github.io) | ``hotpotqa``| ``train``<br>``dev``<br>``test``|  7,405   |  5.23M  |  2.0 | [Link](https://public.ukp.informatik.tu-darmstadt.de/thakur/BEIR/datasets/hotpotqa.zip)  | ``f412724f78b0d91183a0e86805e16114`` |
| FiQA-2018  | [Homepage](https://sites.google.com/view/fiqa/) | ``fiqa`` | ``train``<br>``dev``<br>``test``|  648     |  57K    |  2.6 | [Link](https://public.ukp.informatik.tu-darmstadt.de/thakur/BEIR/datasets/fiqa.zip)  | ``17918ed23cd04fb15047f73e6c3bd9d9`` |
| Signal-1M(RT) | [Homepage](https://research.signal-ai.com/datasets/signal1m-tweetir.html)| ``signal1m`` | ``test``| 97   |  2.86M  |  19.6 | No | [How to Reproduce?](https://github.com/UKPLab/beir/blob/main/examples/dataset#4-signal-1m) |
| TREC-NEWS  | [Homepage](https://trec.nist.gov/data/news2019.html) | ``trec-news``    | ``test``| 57    |  595K    |  19.6 | No | [How to Reproduce?](https://github.com/UKPLab/beir/blob/main/examples/dataset#1-trec-news) |
| ArguAna    | [Homepage](http://argumentation.bplaced.net/arguana/data) | ``arguana``| ``test`` | 1,406     |  8.67K    |  1.0 | [Link](https://public.ukp.informatik.tu-darmstadt.de/thakur/BEIR/datasets/arguana.zip)  | ``8ad3e3c2a5867cdced806d6503f29b99`` |
| Touche-2020| [Homepage](https://webis.de/events/touche-20/shared-task-1.html) | ``webis-touche2020``| ``test``| 49     |  382K    |  19.0 |  [Link](https://public.ukp.informatik.tu-darmstadt.de/thakur/BEIR/datasets/webis-touche2020.zip) | ``46f650ba5a527fc69e0a6521c5a23563`` |
| CQADupstack| [Homepage](http://nlp.cis.unimelb.edu.au/resources/cqadupstack/) | ``cqadupstack``| ``test``| 13,145 |  457K  |  1.4 |  [Link](https://public.ukp.informatik.tu-darmstadt.de/thakur/BEIR/datasets/cqadupstack.zip) | ``4e41456d7df8ee7760a7f866133bda78`` |
| Quora| [Homepage](https://www.quora.com/q/quoradata/First-Quora-Dataset-Release-Question-Pairs) | ``quora``| ``dev``<br>``test``| 10,000     |  523K    |  1.6 |  [Link](https://public.ukp.informatik.tu-darmstadt.de/thakur/BEIR/datasets/quora.zip) | ``18fb154900ba42a600f84b839c173167`` |
| DBPedia | [Homepage](https://github.com/iai-group/DBpedia-Entity/) | ``dbpedia-entity``| ``dev``<br>``test``| 400    |  4.63M    |  38.2 | [Link](https://public.ukp.informatik.tu-darmstadt.de/thakur/BEIR/datasets/dbpedia-entity.zip) | ``c2a39eb420a3164af735795df012ac2c`` |
| SCIDOCS| [Homepage](https://allenai.org/data/scidocs) | ``scidocs``| ``test``| 1,000     |  25K    |  4.9 |  [Link](https://public.ukp.informatik.tu-darmstadt.de/thakur/BEIR/datasets/scidocs.zip) | ``38121350fc3a4d2f48850f6aff52e4a9`` |
| FEVER | [Homepage](http://fever.ai) | ``fever``| ``train``<br>``dev``<br>``test``|  6,666     |  5.42M    |  1.2|  [Link](https://public.ukp.informatik.tu-darmstadt.de/thakur/BEIR/datasets/fever.zip)  | ``5a818580227bfb4b35bb6fa46d9b6c03`` |
| Climate-FEVER| [Homepage](http://climatefever.ai) | ``climate-fever``|``test``|  1,535     |  5.42M |  3.0 |  [Link](https://public.ukp.informatik.tu-darmstadt.de/thakur/BEIR/datasets/climate-fever.zip)  | ``8b66f0a9126c521bae2bde127b4dc99d`` |
| SciFact| [Homepage](https://github.com/allenai/scifact) | ``scifact``| ``train``<br>``test``|  300     |  5K    |  1.1 |  [Link](https://public.ukp.informatik.tu-darmstadt.de/thakur/BEIR/datasets/scifact.zip)  | ``5f7d1de60b170fc8027bb7898e2efca1`` |
| Robust04 | [Homepage](https://trec.nist.gov/data/robust/04.guidelines.html) | ``robust04``| ``test``| 249  |  528K  |  69.9 |  No  |  [How to Reproduce?](https://github.com/UKPLab/beir/blob/main/examples/dataset#3-robust04)  |


## Dataset Creation

### Curation Rationale

[Needs More Information]

### Source Data

#### Initial Data Collection and Normalization

[Needs More Information]

#### Who are the source language producers?

[Needs More Information]

### Annotations

#### Annotation process

[Needs More Information]

#### Who are the annotators?

[Needs More Information]

### Personal and Sensitive Information

[Needs More Information]

## Considerations for Using the Data

### Social Impact of Dataset

[Needs More Information]

### Discussion of Biases

[Needs More Information]

### Other Known Limitations

[Needs More Information]

## Additional Information

### Dataset Curators

[Needs More Information]

### Licensing Information

[Needs More Information]

### Citation Information

Cite as:
```
@inproceedings{
thakur2021beir,
title={{BEIR}: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models},
author={Nandan Thakur and Nils Reimers and Andreas R{\"u}ckl{\'e} and Abhishek Srivastava and Iryna Gurevych},
booktitle={Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2)},
year={2021},
url={https://openreview.net/forum?id=wCu6T5xFjeJ}
}
```

### Contributions

Thanks to [@Nthakur20](https://github.com/Nthakur20) for adding this dataset.Top-20 generated queries for every passage in NFCorpus 


# Dataset Card for BEIR Benchmark

## Table of Contents
- [Dataset Description](#dataset-description)
  - [Dataset Summary](#dataset-summary)
  - [Supported Tasks and Leaderboards](#supported-tasks-and-leaderboards)
  - [Languages](#languages)
- [Dataset Structure](#dataset-structure)
  - [Data Instances](#data-instances)
  - [Data Fields](#data-fields)
  - [Data Splits](#data-splits)
- [Dataset Creation](#dataset-creation)
  - [Curation Rationale](#curation-rationale)
  - [Source Data](#source-data)
  - [Annotations](#annotations)
  - [Personal and Sensitive Information](#personal-and-sensitive-information)
- [Considerations for Using the Data](#considerations-for-using-the-data)
  - [Social Impact of Dataset](#social-impact-of-dataset)
  - [Discussion of Biases](#discussion-of-biases)
  - [Other Known Limitations](#other-known-limitations)
- [Additional Information](#additional-information)
  - [Dataset Curators](#dataset-curators)
  - [Licensing Information](#licensing-information)
  - [Citation Information](#citation-information)
  - [Contributions](#contributions)

## Dataset Description

- **Homepage:** https://github.com/UKPLab/beir
- **Repository:** https://github.com/UKPLab/beir
- **Paper:** https://openreview.net/forum?id=wCu6T5xFjeJ
- **Leaderboard:** https://docs.google.com/spreadsheets/d/1L8aACyPaXrL8iEelJLGqlMqXKPX2oSP_R10pZoy77Ns
- **Point of Contact:** nandan.thakur@uwaterloo.ca

### Dataset Summary

BEIR is a heterogeneous benchmark that has been built from 18 diverse datasets representing 9 information retrieval tasks:

- Fact-checking: [FEVER](http://fever.ai), [Climate-FEVER](http://climatefever.ai), [SciFact](https://github.com/allenai/scifact)
- Question-Answering: [NQ](https://ai.google.com/research/NaturalQuestions), [HotpotQA](https://hotpotqa.github.io), [FiQA-2018](https://sites.google.com/view/fiqa/)
- Bio-Medical IR: [TREC-COVID](https://ir.nist.gov/covidSubmit/index.html), [BioASQ](http://bioasq.org), [NFCorpus](https://www.cl.uni-heidelberg.de/statnlpgroup/nfcorpus/)
- News Retrieval: [TREC-NEWS](https://trec.nist.gov/data/news2019.html), [Robust04](https://trec.nist.gov/data/robust/04.guidelines.html)
- Argument Retrieval: [Touche-2020](https://webis.de/events/touche-20/shared-task-1.html), [ArguAna](tp://argumentation.bplaced.net/arguana/data)
- Duplicate Question Retrieval: [Quora](https://www.quora.com/q/quoradata/First-Quora-Dataset-Release-Question-Pairs), [CqaDupstack](http://nlp.cis.unimelb.edu.au/resources/cqadupstack/)
- Citation-Prediction: [SCIDOCS](https://allenai.org/data/scidocs)
- Tweet Retrieval: [Signal-1M](https://research.signal-ai.com/datasets/signal1m-tweetir.html)
- Entity Retrieval: [DBPedia](https://github.com/iai-group/DBpedia-Entity/)

All these datasets have been preprocessed and can be used for your experiments.


```python

```

### Supported Tasks and Leaderboards

The dataset supports a leaderboard that evaluates models against task-specific metrics such as F1 or EM, as well as their ability to retrieve supporting information from Wikipedia.

The current best performing models can be found [here](https://eval.ai/web/challenges/challenge-page/689/leaderboard/).

### Languages

All tasks are in English (`en`).

## Dataset Structure

All BEIR datasets must contain a corpus, queries and qrels (relevance judgments file). They must be in the following format:
- `corpus` file: a `.jsonl` file (jsonlines) that contains a list of dictionaries, each with three fields `_id` with unique document identifier, `title` with document title (optional) and `text` with document paragraph or passage. For example: `{"_id": "doc1", "title": "Albert Einstein", "text": "Albert Einstein was a German-born...."}`
- `queries` file: a `.jsonl` file (jsonlines) that contains a list of dictionaries, each with two fields `_id` with unique query identifier and `text` with query text. For example: `{"_id": "q1", "text": "Who developed the mass-energy equivalence formula?"}`
- `qrels` file: a `.tsv` file (tab-seperated) that contains three columns, i.e. the `query-id`, `corpus-id` and `score` in this order. Keep 1st row as header. For example: `q1 doc1 1`

### Data Instances

A high level example of any beir dataset:

```python
corpus = {
    "doc1" : {
        "title": "Albert Einstein", 
        "text": "Albert Einstein was a German-born theoretical physicist. who developed the theory of relativity, \
                 one of the two pillars of modern physics (alongside quantum mechanics). His work is also known for \
                 its influence on the philosophy of science. He is best known to the general public for his massâ€“energy \
                 equivalence formula E = mc2, which has been dubbed 'the world's most famous equation'. He received the 1921 \
                 Nobel Prize in Physics 'for his services to theoretical physics, and especially for his discovery of the law \
                 of the photoelectric effect', a pivotal step in the development of quantum theory."
        },
    "doc2" : {
        "title": "", # Keep title an empty string if not present
        "text": "Wheat beer is a top-fermented beer which is brewed with a large proportion of wheat relative to the amount of \
                 malted barley. The two main varieties are German WeiÃŸbier and Belgian witbier; other types include Lambic (made\
                 with wild yeast), Berliner Weisse (a cloudy, sour beer), and Gose (a sour, salty beer)."
    },
}

queries = {
    "q1" : "Who developed the mass-energy equivalence formula?",
    "q2" : "Which beer is brewed with a large proportion of wheat?"
}

qrels = {
    "q1" : {"doc1": 1},
    "q2" : {"doc2": 1},
}
```

### Data Fields

Examples from all configurations have the following features:

### Corpus
- `corpus`: a `dict` feature representing the document title and passage text, made up of:
    - `_id`: a `string` feature representing the unique document id
        - `title`: a `string` feature, denoting the title of the document.
        - `text`: a `string` feature, denoting the text of the document.

### Queries
- `queries`: a `dict` feature representing the query, made up of:
    - `_id`: a `string` feature representing the unique query id
    - `text`: a `string` feature, denoting the text of the query.

### Qrels
- `qrels`: a `dict` feature representing the query document relevance judgements, made up of:
    - `_id`: a `string` feature representing the query id
        - `_id`: a `string` feature, denoting the document id.
        - `score`: a `int32` feature, denoting the relevance judgement between query and document.


### Data Splits

| Dataset   | Website| BEIR-Name | Type | Queries  | Corpus | Rel D/Q | Down-load | md5 |
| -------- | -----| ---------| --------- | ----------- | ---------| ---------| :----------: | :------:|
| MSMARCO    | [Homepage](https://microsoft.github.io/msmarco/)| ``msmarco`` | ``train``<br>``dev``<br>``test``|  6,980   |  8.84M     |    1.1 | [Link](https://public.ukp.informatik.tu-darmstadt.de/thakur/BEIR/datasets/msmarco.zip) | ``444067daf65d982533ea17ebd59501e4`` |
| TREC-COVID |  [Homepage](https://ir.nist.gov/covidSubmit/index.html)| ``trec-covid``| ``test``| 50|  171K| 493.5 | [Link](https://public.ukp.informatik.tu-darmstadt.de/thakur/BEIR/datasets/trec-covid.zip) | ``ce62140cb23feb9becf6270d0d1fe6d1`` |
| NFCorpus   | [Homepage](https://www.cl.uni-heidelberg.de/statnlpgroup/nfcorpus/) | ``nfcorpus`` | ``train``<br>``dev``<br>``test``|  323     |  3.6K     |  38.2 | [Link](https://public.ukp.informatik.tu-darmstadt.de/thakur/BEIR/datasets/nfcorpus.zip) | ``a89dba18a62ef92f7d323ec890a0d38d`` |
| BioASQ     | [Homepage](http://bioasq.org) | ``bioasq``|  ``train``<br>``test`` | 500    |  14.91M    |  8.05 | No | [How to Reproduce?](https://github.com/UKPLab/beir/blob/main/examples/dataset#2-bioasq) |
| NQ         | [Homepage](https://ai.google.com/research/NaturalQuestions) | ``nq``| ``train``<br>``test``| 3,452   |  2.68M  |  1.2 | [Link](https://public.ukp.informatik.tu-darmstadt.de/thakur/BEIR/datasets/nq.zip) | ``d4d3d2e48787a744b6f6e691ff534307`` |
| HotpotQA   | [Homepage](https://hotpotqa.github.io) | ``hotpotqa``| ``train``<br>``dev``<br>``test``|  7,405   |  5.23M  |  2.0 | [Link](https://public.ukp.informatik.tu-darmstadt.de/thakur/BEIR/datasets/hotpotqa.zip)  | ``f412724f78b0d91183a0e86805e16114`` |
| FiQA-2018  | [Homepage](https://sites.google.com/view/fiqa/) | ``fiqa`` | ``train``<br>``dev``<br>``test``|  648     |  57K    |  2.6 | [Link](https://public.ukp.informatik.tu-darmstadt.de/thakur/BEIR/datasets/fiqa.zip)  | ``17918ed23cd04fb15047f73e6c3bd9d9`` |
| Signal-1M(RT) | [Homepage](https://research.signal-ai.com/datasets/signal1m-tweetir.html)| ``signal1m`` | ``test``| 97   |  2.86M  |  19.6 | No | [How to Reproduce?](https://github.com/UKPLab/beir/blob/main/examples/dataset#4-signal-1m) |
| TREC-NEWS  | [Homepage](https://trec.nist.gov/data/news2019.html) | ``trec-news``    | ``test``| 57    |  595K    |  19.6 | No | [How to Reproduce?](https://github.com/UKPLab/beir/blob/main/examples/dataset#1-trec-news) |
| ArguAna    | [Homepage](http://argumentation.bplaced.net/arguana/data) | ``arguana``| ``test`` | 1,406     |  8.67K    |  1.0 | [Link](https://public.ukp.informatik.tu-darmstadt.de/thakur/BEIR/datasets/arguana.zip)  | ``8ad3e3c2a5867cdced806d6503f29b99`` |
| Touche-2020| [Homepage](https://webis.de/events/touche-20/shared-task-1.html) | ``webis-touche2020``| ``test``| 49     |  382K    |  19.0 |  [Link](https://public.ukp.informatik.tu-darmstadt.de/thakur/BEIR/datasets/webis-touche2020.zip) | ``46f650ba5a527fc69e0a6521c5a23563`` |
| CQADupstack| [Homepage](http://nlp.cis.unimelb.edu.au/resources/cqadupstack/) | ``cqadupstack``| ``test``| 13,145 |  457K  |  1.4 |  [Link](https://public.ukp.informatik.tu-darmstadt.de/thakur/BEIR/datasets/cqadupstack.zip) | ``4e41456d7df8ee7760a7f866133bda78`` |
| Quora| [Homepage](https://www.quora.com/q/quoradata/First-Quora-Dataset-Release-Question-Pairs) | ``quora``| ``dev``<br>``test``| 10,000     |  523K    |  1.6 |  [Link](https://public.ukp.informatik.tu-darmstadt.de/thakur/BEIR/datasets/quora.zip) | ``18fb154900ba42a600f84b839c173167`` |
| DBPedia | [Homepage](https://github.com/iai-group/DBpedia-Entity/) | ``dbpedia-entity``| ``dev``<br>``test``| 400    |  4.63M    |  38.2 | [Link](https://public.ukp.informatik.tu-darmstadt.de/thakur/BEIR/datasets/dbpedia-entity.zip) | ``c2a39eb420a3164af735795df012ac2c`` |
| SCIDOCS| [Homepage](https://allenai.org/data/scidocs) | ``scidocs``| ``test``| 1,000     |  25K    |  4.9 |  [Link](https://public.ukp.informatik.tu-darmstadt.de/thakur/BEIR/datasets/scidocs.zip) | ``38121350fc3a4d2f48850f6aff52e4a9`` |
| FEVER | [Homepage](http://fever.ai) | ``fever``| ``train``<br>``dev``<br>``test``|  6,666     |  5.42M    |  1.2|  [Link](https://public.ukp.informatik.tu-darmstadt.de/thakur/BEIR/datasets/fever.zip)  | ``5a818580227bfb4b35bb6fa46d9b6c03`` |
| Climate-FEVER| [Homepage](http://climatefever.ai) | ``climate-fever``|``test``|  1,535     |  5.42M |  3.0 |  [Link](https://public.ukp.informatik.tu-darmstadt.de/thakur/BEIR/datasets/climate-fever.zip)  | ``8b66f0a9126c521bae2bde127b4dc99d`` |
| SciFact| [Homepage](https://github.com/allenai/scifact) | ``scifact``| ``train``<br>``test``|  300     |  5K    |  1.1 |  [Link](https://public.ukp.informatik.tu-darmstadt.de/thakur/BEIR/datasets/scifact.zip)  | ``5f7d1de60b170fc8027bb7898e2efca1`` |
| Robust04 | [Homepage](https://trec.nist.gov/data/robust/04.guidelines.html) | ``robust04``| ``test``| 249  |  528K  |  69.9 |  No  |  [How to Reproduce?](https://github.com/UKPLab/beir/blob/main/examples/dataset#3-robust04)  |


## Dataset Creation

### Curation Rationale

[Needs More Information]

### Source Data

#### Initial Data Collection and Normalization

[Needs More Information]

#### Who are the source language producers?

[Needs More Information]

### Annotations

#### Annotation process

[Needs More Information]

#### Who are the annotators?

[Needs More Information]

### Personal and Sensitive Information

[Needs More Information]

## Considerations for Using the Data

### Social Impact of Dataset

[Needs More Information]

### Discussion of Biases

[Needs More Information]

### Other Known Limitations

[Needs More Information]

## Additional Information

### Dataset Curators

[Needs More Information]

### Licensing Information

[Needs More Information]

### Citation Information

Cite as:
```
@inproceedings{
thakur2021beir,
title={{BEIR}: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models},
author={Nandan Thakur and Nils Reimers and Andreas R{\"u}ckl{\'e} and Abhishek Srivastava and Iryna Gurevych},
booktitle={Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2)},
year={2021},
url={https://openreview.net/forum?id=wCu6T5xFjeJ}
}
```

### Contributions

Thanks to [@Nthakur20](https://github.com/Nthakur20) for adding this dataset.
task_categories:text-retrieval multilinguality:monolingual language:en license:cc-by-sa-4.0 region:us
{"annotations_creators": [], "language_creators": [], "language": ["en"], "license": ["cc-by-sa-4.0"], "multilinguality": ["monolingual"], "size_categories": {"msmarco": ["1M<n<10M"], "trec-covid": ["100k<n<1M"], "nfcorpus": ["1K<n<10K"], "nq": ["1M<n<10M"], "hotpotqa": ["1M<n<10M"], "fiqa": ["10K<n<100K"], "arguana": ["1K<n<10K"], "touche-2020": ["100K<n<1M"], "cqadupstack": ["100K<n<1M"], "quora": ["100K<n<1M"], "dbpedia": ["1M<n<10M"], "scidocs": ["10K<n<100K"], "fever": ["1M<n<10M"], "climate-fever": ["1M<n<10M"], "scifact": ["1K<n<10K"]}, "source_datasets": [], "task_categories": ["text-retrieval"], "paperswithcode_id": "beir", "pretty_name": "BEIR Benchmark"}
TAGS
#task_categories-text-retrieval #multilinguality-monolingual #language-English #license-cc-by-sa-4.0 #region-us 

NFCorpus: 20 generated queries (BEIR Benchmark)
===============================================


This HF dataset contains the top-20 synthetic queries generated for each passage in the above BEIR benchmark dataset.


* DocT5query model used: BeIR/query-gen-msmarco-t5-base-v1
* id (str): unique document id in NFCorpus in the BEIR benchmark ('URL').
* Questions generated: 20
* Code used for generation: evaluate\_anserini\_docT5query\_parallel.py


Below contains the old dataset card for the BEIR benchmark.


Dataset Card for BEIR Benchmark
===============================


Table of Contents
-----------------


* Dataset Description
	+ Dataset Summary
	+ Supported Tasks and Leaderboards
	+ Languages
* Dataset Structure
	+ Data Instances
	+ Data Fields
	+ Data Splits
* Dataset Creation
	+ Curation Rationale
	+ Source Data
	+ Annotations
	+ Personal and Sensitive Information
* Considerations for Using the Data
	+ Social Impact of Dataset
	+ Discussion of Biases
	+ Other Known Limitations
* Additional Information
	+ Dataset Curators
	+ Licensing Information
	+ Citation Information
	+ Contributions


Dataset Description
-------------------


* Homepage: URL
* Repository: URL
* Paper: URL
* Leaderboard: URL
* Point of Contact: URL@URL


### Dataset Summary


BEIR is a heterogeneous benchmark that has been built from 18 diverse datasets representing 9 information retrieval tasks:


* Fact-checking: FEVER, Climate-FEVER, SciFact
* Question-Answering: NQ, HotpotQA, FiQA-2018
* Bio-Medical IR: TREC-COVID, BioASQ, NFCorpus
* News Retrieval: TREC-NEWS, Robust04
* Argument Retrieval: Touche-2020, ArguAna
* Duplicate Question Retrieval: Quora, CqaDupstack
* Citation-Prediction: SCIDOCS
* Tweet Retrieval: Signal-1M
* Entity Retrieval: DBPedia


All these datasets have been preprocessed and can be used for your experiments.


### Supported Tasks and Leaderboards


The dataset supports a leaderboard that evaluates models against task-specific metrics such as F1 or EM, as well as their ability to retrieve supporting information from Wikipedia.


The current best performing models can be found here.


### Languages


All tasks are in English ('en').


Dataset Structure
-----------------


All BEIR datasets must contain a corpus, queries and qrels (relevance judgments file). They must be in the following format:


* 'corpus' file: a '.jsonl' file (jsonlines) that contains a list of dictionaries, each with three fields '\_id' with unique document identifier, 'title' with document title (optional) and 'text' with document paragraph or passage. For example: '{"\_id": "doc1", "title": "Albert Einstein", "text": "Albert Einstein was a German-born...."}'
* 'queries' file: a '.jsonl' file (jsonlines) that contains a list of dictionaries, each with two fields '\_id' with unique query identifier and 'text' with query text. For example: '{"\_id": "q1", "text": "Who developed the mass-energy equivalence formula?"}'
* 'qrels' file: a '.tsv' file (tab-seperated) that contains three columns, i.e. the 'query-id', 'corpus-id' and 'score' in this order. Keep 1st row as header. For example: 'q1 doc1 1'


### Data Instances


A high level example of any beir dataset:


### Data Fields


Examples from all configurations have the following features:


### Corpus


* 'corpus': a 'dict' feature representing the document title and passage text, made up of:
	+ '\_id': a 'string' feature representing the unique document id
		- 'title': a 'string' feature, denoting the title of the document.
		- 'text': a 'string' feature, denoting the text of the document.


### Queries


* 'queries': a 'dict' feature representing the query, made up of:
	+ '\_id': a 'string' feature representing the unique query id
	+ 'text': a 'string' feature, denoting the text of the query.


### Qrels


* 'qrels': a 'dict' feature representing the query document relevance judgements, made up of:
	+ '\_id': a 'string' feature representing the query id
		- '\_id': a 'string' feature, denoting the document id.
		- 'score': a 'int32' feature, denoting the relevance judgement between query and document.


### Data Splits



Dataset Creation
----------------


### Curation Rationale


### Source Data


#### Initial Data Collection and Normalization


#### Who are the source language producers?


### Annotations


#### Annotation process


#### Who are the annotators?


### Personal and Sensitive Information


Considerations for Using the Data
---------------------------------


### Social Impact of Dataset


### Discussion of Biases


### Other Known Limitations


Additional Information
----------------------


### Dataset Curators


### Licensing Information


Cite as:


### Contributions


Thanks to @Nthakur20 for adding this dataset.Top-20 generated queries for every passage in NFCorpus


Dataset Card for BEIR Benchmark
===============================


Table of Contents
-----------------


* Dataset Description
	+ Dataset Summary
	+ Supported Tasks and Leaderboards
	+ Languages
* Dataset Structure
	+ Data Instances
	+ Data Fields
	+ Data Splits
* Dataset Creation
	+ Curation Rationale
	+ Source Data
	+ Annotations
	+ Personal and Sensitive Information
* Considerations for Using the Data
	+ Social Impact of Dataset
	+ Discussion of Biases
	+ Other Known Limitations
* Additional Information
	+ Dataset Curators
	+ Licensing Information
	+ Citation Information
	+ Contributions


Dataset Description
-------------------


* Homepage: URL
* Repository: URL
* Paper: URL
* Leaderboard: URL
* Point of Contact: URL@URL


### Dataset Summary


BEIR is a heterogeneous benchmark that has been built from 18 diverse datasets representing 9 information retrieval tasks:


* Fact-checking: FEVER, Climate-FEVER, SciFact
* Question-Answering: NQ, HotpotQA, FiQA-2018
* Bio-Medical IR: TREC-COVID, BioASQ, NFCorpus
* News Retrieval: TREC-NEWS, Robust04
* Argument Retrieval: Touche-2020, ArguAna
* Duplicate Question Retrieval: Quora, CqaDupstack
* Citation-Prediction: SCIDOCS
* Tweet Retrieval: Signal-1M
* Entity Retrieval: DBPedia


All these datasets have been preprocessed and can be used for your experiments.


### Supported Tasks and Leaderboards


The dataset supports a leaderboard that evaluates models against task-specific metrics such as F1 or EM, as well as their ability to retrieve supporting information from Wikipedia.


The current best performing models can be found here.


### Languages


All tasks are in English ('en').


Dataset Structure
-----------------


All BEIR datasets must contain a corpus, queries and qrels (relevance judgments file). They must be in the following format:


* 'corpus' file: a '.jsonl' file (jsonlines) that contains a list of dictionaries, each with three fields '\_id' with unique document identifier, 'title' with document title (optional) and 'text' with document paragraph or passage. For example: '{"\_id": "doc1", "title": "Albert Einstein", "text": "Albert Einstein was a German-born...."}'
* 'queries' file: a '.jsonl' file (jsonlines) that contains a list of dictionaries, each with two fields '\_id' with unique query identifier and 'text' with query text. For example: '{"\_id": "q1", "text": "Who developed the mass-energy equivalence formula?"}'
* 'qrels' file: a '.tsv' file (tab-seperated) that contains three columns, i.e. the 'query-id', 'corpus-id' and 'score' in this order. Keep 1st row as header. For example: 'q1 doc1 1'


### Data Instances


A high level example of any beir dataset:


### Data Fields


Examples from all configurations have the following features:


### Corpus


* 'corpus': a 'dict' feature representing the document title and passage text, made up of:
	+ '\_id': a 'string' feature representing the unique document id
		- 'title': a 'string' feature, denoting the title of the document.
		- 'text': a 'string' feature, denoting the text of the document.


### Queries


* 'queries': a 'dict' feature representing the query, made up of:
	+ '\_id': a 'string' feature representing the unique query id
	+ 'text': a 'string' feature, denoting the text of the query.


### Qrels


* 'qrels': a 'dict' feature representing the query document relevance judgements, made up of:
	+ '\_id': a 'string' feature representing the query id
		- '\_id': a 'string' feature, denoting the document id.
		- 'score': a 'int32' feature, denoting the relevance judgement between query and document.


### Data Splits



Dataset Creation
----------------


### Curation Rationale


### Source Data


#### Initial Data Collection and Normalization


#### Who are the source language producers?


### Annotations


#### Annotation process


#### Who are the annotators?


### Personal and Sensitive Information


Considerations for Using the Data
---------------------------------


### Social Impact of Dataset


### Discussion of Biases


### Other Known Limitations


Additional Information
----------------------


### Dataset Curators


### Licensing Information


Cite as:


### Contributions


Thanks to @Nthakur20 for adding this dataset.



### Dataset Summary


BEIR is a heterogeneous benchmark that has been built from 18 diverse datasets representing 9 information retrieval tasks:


* Fact-checking: FEVER, Climate-FEVER, SciFact
* Question-Answering: NQ, HotpotQA, FiQA-2018
* Bio-Medical IR: TREC-COVID, BioASQ, NFCorpus
* News Retrieval: TREC-NEWS, Robust04
* Argument Retrieval: Touche-2020, ArguAna
* Duplicate Question Retrieval: Quora, CqaDupstack
* Citation-Prediction: SCIDOCS
* Tweet Retrieval: Signal-1M
* Entity Retrieval: DBPedia


All these datasets have been preprocessed and can be used for your experiments. ### Supported Tasks and Leaderboards


The dataset supports a leaderboard that evaluates models against task-specific metrics such as F1 or EM, as well as their ability to retrieve supporting information from Wikipedia.


The current best performing models can be found here. ### Languages


All tasks are in English ('en').


Dataset Structure
-----------------


All BEIR datasets must contain a corpus, queries and qrels (relevance judgments file). They must be in the following format:


* 'corpus' file: a '.jsonl' file (jsonlines) that contains a list of dictionaries, each with three fields '\_id' with unique document identifier, 'title' with document title (optional) and 'text' with document paragraph or passage. For example: '{"\_id": "doc1", "title": "Albert Einstein", "text": "Albert Einstein was a German-born...."}'
* 'queries' file: a '.jsonl' file (jsonlines) that contains a list of dictionaries, each with two fields '\_id' with unique query identifier and 'text' with query text. For example: '{"\_id": "q1", "text": "Who developed the mass-energy equivalence formula?"}'
* 'qrels' file: a '.tsv' file (tab-seperated) that contains three columns, i.e. the 'query-id', 'corpus-id' and 'score' in this order. Keep 1st row as header. For example: 'q1 doc1 1' ### Data Instances


A high level example of any beir dataset: ### Data Fields


Examples from all configurations have the following features: ### Corpus


* 'corpus': a 'dict' feature representing the document title and passage text, made up of:
	+ '\_id': a 'string' feature representing the unique document id
		- 'title': a 'string' feature, denoting the title of the document.
		- 'text': a 'string' feature, denoting the text of the document. ### Queries


* 'queries': a 'dict' feature representing the query, made up of:
	+ '\_id': a 'string' feature representing the unique query id
	+ 'text': a 'string' feature, denoting the text of the query. ### Qrels


* 'qrels': a 'dict' feature representing the query document relevance judgements, made up of:
	+ '\_id': a 'string' feature representing the query id
		- '\_id': a 'string' feature, denoting the document id.
		- 'score': a 'int32' feature, denoting the relevance judgement between query and document. ### Data Splits



Dataset Creation
---------------- ### Curation Rationale ### Source Data #### Initial Data Collection and Normalization #### Who are the source language producers? ### Annotations #### Annotation process #### Who are the annotators? ### Personal and Sensitive Information


Considerations for Using the Data
--------------------------------- ### Social Impact of Dataset ### Discussion of Biases ### Other Known Limitations


Additional Information
---------------------- ### Dataset Curators ### Licensing Information


Cite as: ### Contributions


Thanks to @Nthakur20 for adding this dataset.Top-20 generated queries for every passage in NFCorpus


Dataset Card for BEIR Benchmark
===============================


Table of Contents
-----------------


* Dataset Description
	+ Dataset Summary
	+ Supported Tasks and Leaderboards
	+ Languages
* Dataset Structure
	+ Data Instances
	+ Data Fields
	+ Data Splits
* Dataset Creation
	+ Curation Rationale
	+ Source Data
	+ Annotations
	+ Personal and Sensitive Information
* Considerations for Using the Data
	+ Social Impact of Dataset
	+ Discussion of Biases
	+ Other Known Limitations
* Additional Information
	+ Dataset Curators
	+ Licensing Information
	+ Citation Information
	+ Contributions


Dataset Description
-------------------


* Homepage: URL
* Repository: URL
* Paper: URL
* Leaderboard: URL
* Point of Contact: URL@URL ### Dataset Summary


BEIR is a heterogeneous benchmark that has been built from 18 diverse datasets representing 9 information retrieval tasks:


* Fact-checking: FEVER, Climate-FEVER, SciFact
* Question-Answering: NQ, HotpotQA, FiQA-2018
* Bio-Medical IR: TREC-COVID, BioASQ, NFCorpus
* News Retrieval: TREC-NEWS, Robust04
* Argument Retrieval: Touche-2020, ArguAna
* Duplicate Question Retrieval: Quora, CqaDupstack
* Citation-Prediction: SCIDOCS
* Tweet Retrieval: Signal-1M
* Entity Retrieval: DBPedia


All these datasets have been preprocessed and can be used for your experiments. ### Supported Tasks and Leaderboards


The dataset supports a leaderboard that evaluates models against task-specific metrics such as F1 or EM, as well as their ability to retrieve supporting information from Wikipedia.


The current best performing models can be found here. ### Languages


All tasks are in English ('en').


Dataset Structure
-----------------


All BEIR datasets must contain a corpus, queries and qrels (relevance judgments file). They must be in the following format:


* 'corpus' file: a '.jsonl' file (jsonlines) that contains a list of dictionaries, each with three fields '\_id' with unique document identifier, 'title' with document title (optional) and 'text' with document paragraph or passage. For example: '{"\_id": "doc1", "title": "Albert Einstein", "text": "Albert Einstein was a German-born...."}'
* 'queries' file: a '.jsonl' file (jsonlines) that contains a list of dictionaries, each with two fields '\_id' with unique query identifier and 'text' with query text. For example: '{"\_id": "q1", "text": "Who developed the mass-energy equivalence formula?"}'
* 'qrels' file: a '.tsv' file (tab-seperated) that contains three columns, i.e. the 'query-id', 'corpus-id' and 'score' in this order. Keep 1st row as header. For example: 'q1 doc1 1' ### Data Instances


A high level example of any beir dataset: ### Data Fields


Examples from all configurations have the following features: ### Corpus


* 'corpus': a 'dict' feature representing the document title and passage text, made up of:
	+ '\_id': a 'string' feature representing the unique document id
		- 'title': a 'string' feature, denoting the title of the document.
		- 'text': a 'string' feature, denoting the text of the document. ### Queries


* 'queries': a 'dict' feature representing the query, made up of:
	+ '\_id': a 'string' feature representing the unique query id
	+ 'text': a 'string' feature, denoting the text of the query. ### Qrels


* 'qrels': a 'dict' feature representing the query document relevance judgements, made up of:
	+ '\_id': a 'string' feature representing the query id
		- '\_id': a 'string' feature, denoting the document id.
		- 'score': a 'int32' feature, denoting the relevance judgement between query and document. ### Data Splits



Dataset Creation
---------------- ### Curation Rationale ### Source Data #### Initial Data Collection and Normalization #### Who are the source language producers? ### Annotations #### Annotation process #### Who are the annotators? ### Personal and Sensitive Information


Considerations for Using the Data
--------------------------------- ### Social Impact of Dataset ### Discussion of Biases ### Other Known Limitations


Additional Information
---------------------- ### Dataset Curators ### Licensing Information


Cite as: ### Contributions


Thanks to @Nthakur20 for adding this dataset.
TAGS
#task_categories-text-retrieval #multilinguality-monolingual #language-English #license-cc-by-sa-4.0 #region-us 
 ### Dataset Summary


BEIR is a heterogeneous benchmark that has been built from 18 diverse datasets representing 9 information retrieval tasks:


* Fact-checking: FEVER, Climate-FEVER, SciFact
* Question-Answering: NQ, HotpotQA, FiQA-2018
* Bio-Medical IR: TREC-COVID, BioASQ, NFCorpus
* News Retrieval: TREC-NEWS, Robust04
* Argument Retrieval: Touche-2020, ArguAna
* Duplicate Question Retrieval: Quora, CqaDupstack
* Citation-Prediction: SCIDOCS
* Tweet Retrieval: Signal-1M
* Entity Retrieval: DBPedia


All these datasets have been preprocessed and can be used for your experiments. ### Supported Tasks and Leaderboards


The dataset supports a leaderboard that evaluates models against task-specific metrics such as F1 or EM, as well as their ability to retrieve supporting information from Wikipedia.


The current best performing models can be found here. ### Languages


All tasks are in English ('en').


Dataset Structure
-----------------


All BEIR datasets must contain a corpus, queries and qrels (relevance judgments file). They must be in the following format:


* 'corpus' file: a '.jsonl' file (jsonlines) that contains a list of dictionaries, each with three fields '\_id' with unique document identifier, 'title' with document title (optional) and 'text' with document paragraph or passage. For example: '{"\_id": "doc1", "title": "Albert Einstein", "text": "Albert Einstein was a German-born...."}'
* 'queries' file: a '.jsonl' file (jsonlines) that contains a list of dictionaries, each with two fields '\_id' with unique query identifier and 'text' with query text. For example: '{"\_id": "q1", "text": "Who developed the mass-energy equivalence formula?"}'
* 'qrels' file: a '.tsv' file (tab-seperated) that contains three columns, i.e. the 'query-id', 'corpus-id' and 'score' in this order. Keep 1st row as header. For example: 'q1 doc1 1' ### Data Instances


A high level example of any beir dataset: ### Data Fields


Examples from all configurations have the following features: ### Corpus


* 'corpus': a 'dict' feature representing the document title and passage text, made up of:
	+ '\_id': a 'string' feature representing the unique document id
		- 'title': a 'string' feature, denoting the title of the document.
		- 'text': a 'string' feature, denoting the text of the document. ### Queries


* 'queries': a 'dict' feature representing the query, made up of:
	+ '\_id': a 'string' feature representing the unique query id
	+ 'text': a 'string' feature, denoting the text of the query. ### Qrels


* 'qrels': a 'dict' feature representing the query document relevance judgements, made up of:
	+ '\_id': a 'string' feature representing the query id
		- '\_id': a 'string' feature, denoting the document id.
		- 'score': a 'int32' feature, denoting the relevance judgement between query and document. ### Data Splits



Dataset Creation
---------------- ### Curation Rationale ### Source Data #### Initial Data Collection and Normalization #### Who are the source language producers? ### Annotations #### Annotation process #### Who are the annotators? ### Personal and Sensitive Information


Considerations for Using the Data
--------------------------------- ### Social Impact of Dataset ### Discussion of Biases ### Other Known Limitations


Additional Information
---------------------- ### Dataset Curators ### Licensing Information


Cite as: ### Contributions


Thanks to @Nthakur20 for adding this dataset.Top-20 generated queries for every passage in NFCorpus


Dataset Card for BEIR Benchmark
===============================


Table of Contents
-----------------


* Dataset Description
	+ Dataset Summary
	+ Supported Tasks and Leaderboards
	+ Languages
* Dataset Structure
	+ Data Instances
	+ Data Fields
	+ Data Splits
* Dataset Creation
	+ Curation Rationale
	+ Source Data
	+ Annotations
	+ Personal and Sensitive Information
* Considerations for Using the Data
	+ Social Impact of Dataset
	+ Discussion of Biases
	+ Other Known Limitations
* Additional Information
	+ Dataset Curators
	+ Licensing Information
	+ Citation Information
	+ Contributions


Dataset Description
-------------------


* Homepage: URL
* Repository: URL
* Paper: URL
* Leaderboard: URL
* Point of Contact: URL@URL ### Dataset Summary


BEIR is a heterogeneous benchmark that has been built from 18 diverse datasets representing 9 information retrieval tasks:


* Fact-checking: FEVER, Climate-FEVER, SciFact
* Question-Answering: NQ, HotpotQA, FiQA-2018
* Bio-Medical IR: TREC-COVID, BioASQ, NFCorpus
* News Retrieval: TREC-NEWS, Robust04
* Argument Retrieval: Touche-2020, ArguAna
* Duplicate Question Retrieval: Quora, CqaDupstack
* Citation-Prediction: SCIDOCS
* Tweet Retrieval: Signal-1M
* Entity Retrieval: DBPedia


All these datasets have been preprocessed and can be used for your experiments. ### Supported Tasks and Leaderboards


The dataset supports a leaderboard that evaluates models against task-specific metrics such as F1 or EM, as well as their ability to retrieve supporting information from Wikipedia.


The current best performing models can be found here. ### Languages


All tasks are in English ('en').


Dataset Structure
-----------------


All BEIR datasets must contain a corpus, queries and qrels (relevance judgments file). They must be in the following format:


* 'corpus' file: a '.jsonl' file (jsonlines) that contains a list of dictionaries, each with three fields '\_id' with unique document identifier, 'title' with document title (optional) and 'text' with document paragraph or passage. For example: '{"\_id": "doc1", "title": "Albert Einstein", "text": "Albert Einstein was a German-born...."}'
* 'queries' file: a '.jsonl' file (jsonlines) that contains a list of dictionaries, each with two fields '\_id' with unique query identifier and 'text' with query text. For example: '{"\_id": "q1", "text": "Who developed the mass-energy equivalence formula?"}'
* 'qrels' file: a '.tsv' file (tab-seperated) that contains three columns, i.e. the 'query-id', 'corpus-id' and 'score' in this order. Keep 1st row as header. For example: 'q1 doc1 1' ### Data Instances


A high level example of any beir dataset: ### Data Fields


Examples from all configurations have the following features: ### Corpus


* 'corpus': a 'dict' feature representing the document title and passage text, made up of:
	+ '\_id': a 'string' feature representing the unique document id
		- 'title': a 'string' feature, denoting the title of the document.
		- 'text': a 'string' feature, denoting the text of the document. ### Queries


* 'queries': a 'dict' feature representing the query, made up of:
	+ '\_id': a 'string' feature representing the unique query id
	+ 'text': a 'string' feature, denoting the text of the query. ### Qrels


* 'qrels': a 'dict' feature representing the query document relevance judgements, made up of:
	+ '\_id': a 'string' feature representing the query id
		- '\_id': a 'string' feature, denoting the document id.
		- 'score': a 'int32' feature, denoting the relevance judgement between query and document. ### Data Splits



Dataset Creation
---------------- ### Curation Rationale ### Source Data #### Initial Data Collection and Normalization #### Who are the source language producers? ### Annotations #### Annotation process #### Who are the annotators? ### Personal and Sensitive Information


Considerations for Using the Data
--------------------------------- ### Social Impact of Dataset ### Discussion of Biases ### Other Known Limitations


Additional Information
---------------------- ### Dataset Curators ### Licensing Information


Cite as: ### Contributions


Thanks to @Nthakur20 for adding this dataset.
passage: TAGS
#task_categories-text-retrieval #multilinguality-monolingual #language-English #license-cc-by-sa-4.0 #region-us 
### Dataset Summary


BEIR is a heterogeneous benchmark that has been built from 18 diverse datasets representing 9 information retrieval tasks:


* Fact-checking: FEVER, Climate-FEVER, SciFact
* Question-Answering: NQ, HotpotQA, FiQA-2018
* Bio-Medical IR: TREC-COVID, BioASQ, NFCorpus
* News Retrieval: TREC-NEWS, Robust04
* Argument Retrieval: Touche-2020, ArguAna
* Duplicate Question Retrieval: Quora, CqaDupstack
* Citation-Prediction: SCIDOCS
* Tweet Retrieval: Signal-1M
* Entity Retrieval: DBPedia


All these datasets have been preprocessed and can be used for your experiments.### Supported Tasks and Leaderboards


The dataset supports a leaderboard that evaluates models against task-specific metrics such as F1 or EM, as well as their ability to retrieve supporting information from Wikipedia.


The current best performing models can be found here. passage: ### Languages


All tasks are in English ('en').


Dataset Structure
-----------------


All BEIR datasets must contain a corpus, queries and qrels (relevance judgments file). They must be in the following format:


* 'corpus' file: a '.jsonl' file (jsonlines) that contains a list of dictionaries, each with three fields '\_id' with unique document identifier, 'title' with document title (optional) and 'text' with document paragraph or passage. For example: '{"\_id": "doc1", "title": "Albert Einstein", "text": "Albert Einstein was a German-born...."}'
* 'queries' file: a '.jsonl' file (jsonlines) that contains a list of dictionaries, each with two fields '\_id' with unique query identifier and 'text' with query text. For example: '{"\_id": "q1", "text": "Who developed the mass-energy equivalence formula?"}'
* 'qrels' file: a '.tsv' file (tab-seperated) that contains three columns, i.e. the 'query-id', 'corpus-id' and 'score' in this order. Keep 1st row as header. For example: 'q1 doc1 1'### Data Instances


A high level example of any beir dataset:### Data Fields


Examples from all configurations have the following features:### Corpus


* 'corpus': a 'dict' feature representing the document title and passage text, made up of:
	+ '\_id': a 'string' feature representing the unique document id
		- 'title': a 'string' feature, denoting the title of the document.
		- 'text': a 'string' feature, denoting the text of the document.### Queries


* 'queries': a 'dict' feature representing the query, made up of:
	+ '\_id': a 'string' feature representing the unique query id
	+ 'text': a 'string' feature, denoting the text of the query.### Qrels


* 'qrels': a 'dict' feature representing the query document relevance judgements, made up of:
	+ '\_id': a 'string' feature representing the query id
		- '\_id': a 'string' feature, denoting the document id.
		- 'score': a 'int32' feature, denoting the relevance judgement between query and document.### Data Splits



Dataset Creation
----------------### Curation Rationale### Source Data#### Initial Data Collection and Normalization#### Who are the source language producers?### Annotations#### Annotation process#### Who are the annotators?### Personal and Sensitive Information


Considerations for Using the Data
---------------------------------### Social Impact of Dataset### Discussion of Biases### Other Known Limitations


Additional Information
----------------------### Dataset Curators### Licensing Information


Cite as: passage: ### Contributions


Thanks to @Nthakur20 for adding this dataset.Top-20 generated queries for every passage in NFCorpus


Dataset Card for BEIR Benchmark
===============================


Table of Contents
-----------------


* Dataset Description
	+ Dataset Summary
	+ Supported Tasks and Leaderboards
	+ Languages
* Dataset Structure
	+ Data Instances
	+ Data Fields
	+ Data Splits
* Dataset Creation
	+ Curation Rationale
	+ Source Data
	+ Annotations
	+ Personal and Sensitive Information
* Considerations for Using the Data
	+ Social Impact of Dataset
	+ Discussion of Biases
	+ Other Known Limitations
* Additional Information
	+ Dataset Curators
	+ Licensing Information
	+ Citation Information
	+ Contributions


Dataset Description
-------------------


* Homepage: URL
* Repository: URL
* Paper: URL
* Leaderboard: URL
* Point of Contact: URL@URL### Dataset Summary


BEIR is a heterogeneous benchmark that has been built from 18 diverse datasets representing 9 information retrieval tasks:


* Fact-checking: FEVER, Climate-FEVER, SciFact
* Question-Answering: NQ, HotpotQA, FiQA-2018
* Bio-Medical IR: TREC-COVID, BioASQ, NFCorpus
* News Retrieval: TREC-NEWS, Robust04
* Argument Retrieval: Touche-2020, ArguAna
* Duplicate Question Retrieval: Quora, CqaDupstack
* Citation-Prediction: SCIDOCS
* Tweet Retrieval: Signal-1M
* Entity Retrieval: DBPedia


All these datasets have been preprocessed and can be used for your experiments.### Supported Tasks and Leaderboards


The dataset supports a leaderboard that evaluates models against task-specific metrics such as F1 or EM, as well as their ability to retrieve supporting information from Wikipedia.


The current best performing models can be found here.