
# Dataset Card for AIDA CoNLL-YAGO Wikidata

## Table of Contents
- [Dataset Description](#dataset-description)
  - [Dataset Summary](#dataset-summary)
  - [Supported Tasks](#supported-tasks)
  - [Languages](#languages)
- [Dataset Structure](#dataset-structure)
  - [Data Instances](#data-instances)
  - [Data Fields](#data-fields)
  - [Data Splits](#data-splits)
- [Additional Information](#additional-information)
  - [Licensing Information](#licensing-information)

## Dataset Description

- **Repository:** [AIDA CoNLL-YAGO Wikidata repository](https://github.com/cyanic-selkie/aida-conll-yago-wikidata)

### Dataset Summary

The AIDA CoNLL-YAGO Wikidata dataset is the same as the original [AIDA CoNLL-YAGO](https://www.mpi-inf.mpg.de/departments/databases-and-information-systems/research/ambiverse-nlu/aida/downloads) dataset, but with Wikidata QIDs instead of Wikipedia titles as entity identifiers. They are automatically generated (with a few manual corrections) from Wikidata and Wikipedia dumps (March 1, 2023).

The code for generating the dataset can be found [here](https://github.com/cyanic-selkie/aida-conll-yago-wikidata).

### Supported Tasks

- `named-entity-recognition`: The dataset can be used to train a model for Named Entity Recognition.
- `named-entity-linking`: The dataset can be used to train a model for Named Entity Linking.

### Languages

The text in the dataset is in English. The associated BCP-47 code is `en`.

## Dataset Structure

### Data Instances

A typical data point represents a document (news article).

The `text` field contains the original text in an NFC normalized, UTF-8 encoded string.

The `entities` field contains a list of entities, each represented by a struct with the inclusive starting byte `start` field, exclusive ending byte `end` field, a nullable `qid` field, and a nullable `pageid` field.

Additionally, each document has a unique `document_id` field.

An example from the AIDA CoNLL-YAGO Wikidata test set looks as follows:

```
{
  "document_id": 1214,
  "text": "RADIO ROMANIA AFTERNOON HEALINES AT 4 PM . BUCHAREST 1996-12-06 Radio Romania news headlines : * The Democratic Convention signed an agreement on government and parliamentary support with its coalition partners the Social Democratic Union and the Hungarian Democratic Union ( UDMR ) . The ceremony was attended by President Emil Constantinescu . * The three parties in the government coalition have committed themselves to a real reform of Romania 's economy , Constantinescu said after the ceremony . * The UDMR wants to contribute to social reform and economic revival in Romania , union leader Marko Bela said . * The international airport in Timisoara and the domestic airports in Arad , Oradea and Sibiu were closed due to fog . -- Bucharest Newsroom 40-1 3120264",
  "entities": [
    {
      "start": 0,
      "end": 13,
      "tag": "ORG",
      "pageid": null,
      "qid": null,
      "title": null
    },
    {
      "start": 43,
      "end": 52,
      "tag": "LOC",
      "pageid": 36877,
      "qid": 19660,
      "title": "Bucharest"
    },
    {
      "start": 64,
      "end": 77,
      "tag": "ORG",
      "pageid": null,
      "qid": null,
      "title": null
    },
    {
      "start": 101,
      "end": 122,
      "tag": "MISC",
      "pageid": null,
      "qid": null,
      "title": null
    },
    {
      "start": 215,
      "end": 238,
      "tag": "ORG",
      "pageid": null,
      "qid": null,
      "title": null
    },
    {
      "start": 247,
      "end": 273,
      "tag": "ORG",
      "pageid": null,
      "qid": null,
      "title": null
    },
    {
      "start": 276,
      "end": 280,
      "tag": "ORG",
      "pageid": 49749134,
      "qid": 266582,
      "title": "Democratic_Union_of_Hungarians_in_Romania"
    },
    {
      "start": 324,
      "end": 343,
      "tag": "PER",
      "pageid": 393370,
      "qid": 299152,
      "title": "Emil_Constantinescu"
    },
    {
      "start": 440,
      "end": 447,
      "tag": "LOC",
      "pageid": 25445,
      "qid": 218,
      "title": "Romania"
    },
    {
      "start": 461,
      "end": 475,
      "tag": "PER",
      "pageid": 393370,
      "qid": 299152,
      "title": "Emil_Constantinescu"
    },
    {
      "start": 508,
      "end": 512,
      "tag": "ORG",
      "pageid": 49749134,
      "qid": 266582,
      "title": "Democratic_Union_of_Hungarians_in_Romania"
    },
    {
      "start": 574,
      "end": 581,
      "tag": "LOC",
      "pageid": 25445,
      "qid": 218,
      "title": "Romania"
    },
    {
      "start": 597,
      "end": 607,
      "tag": "PER",
      "pageid": 1219345,
      "qid": 897108,
      "title": "Béla_Markó"
    },
    {
      "start": 646,
      "end": 655,
      "tag": "LOC",
      "pageid": 33693389,
      "qid": 83404,
      "title": "Timişoara"
    },
    {
      "start": 685,
      "end": 689,
      "tag": "LOC",
      "pageid": 22537901,
      "qid": 173591,
      "title": "Arad,_Romania"
    },
    {
      "start": 692,
      "end": 698,
      "tag": "LOC",
      "pageid": 2024606,
      "qid": 2102332,
      "title": "Oradea_International_Airport"
    },
    {
      "start": 703,
      "end": 708,
      "tag": "LOC",
      "pageid": 2384413,
      "qid": 946418,
      "title": "Sibiu_International_Airport"
    },
    {
      "start": 737,
      "end": 755,
      "tag": "ORG",
      "pageid": null,
      "qid": null,
      "title": null
    }
  ]
}
```

### Data Fields

- `document_id`: an integer that uniquely identifies the document this sentence belongs to
- `sentence_index`: an integer that uniquely identifies the position of the sentence in its original document
- `text`: an NFC normalized, UTF-8 encoded string representing the sentence
- `entities`: a list of structs representing entities, each entity has:
  - `start`: an integer representing the inclusive starting UTF-8 code point of the entity
  - `end`: an integer representing the exclusive ending UTF-8 code point of the entity
  - `tag`: a string representing the entity type (PER, LOC, ORG or MISC)
  - `qid`: an integer representing the Wikidata QID this entity refers to; it can be null if the entity didn't exist in Wikidata at the time of the creation of the original dataset
  - `pageid`: an integer representing the English Wikipedia's pageID this entity refers to; it can be null if the entity didn't exist in Wikipedia at the time of the creation of the original dataset
  - `title`: an NFC normalized, UTF-8 encoded string representing the English Wikipedia's title this entity refers to; it can be null if the entity didn't exist in Wikipedia at the time of the creation of the original dataset

### Data Splits

The data is split into training, validation and test sets; all of the sentences belonging to an article are in the same split. The final split sizes are as follows:

|                                               | Train     | Validation  | Test    |
| :-----                                        | :------:  | :-----:     | :----:  |
| AIDA CoNLL-YAGO Wikidata - documents          | 946       | 216         | 231     |
| AIDA CoNLL-YAGO Wikidata - entities           | 23,374    | 5,912       | 5,608   |
| AIDA CoNLL-YAGO Wikidata - entities with QIDs | 18,540    | 4,791       | 4,481   |

## Additional Information

### Licensing Information

The licensing status of the dataset is the same as the licensing status of the original [AIDA CoNLL-YAGO](https://www.mpi-inf.mpg.de/departments/databases-and-information-systems/research/ambiverse-nlu/aida/downloads) dataset which is under a [Creative Commons Attribution-ShareAlike 3.0 Unported License](http://creativecommons.org/licenses/by-sa/3.0/deed.en_US).
task_categories:token-classification size_categories:10K<n<100K language:en license:cc-by-sa-3.0 wikidata wikipedia named-entity-recognition named-entity-linking region:us
{"language": ["en"], "license": "cc-by-sa-3.0", "size_categories": ["10K<n<100K"], "task_categories": ["token-classification"], "pretty_name": "AIDA CoNLL-YAGO Wikidata", "tags": ["wikidata", "wikipedia", "named-entity-recognition", "named-entity-linking"]}
TAGS
#task_categories-token-classification #size_categories-10K<n<100K #language-English #license-cc-by-sa-3.0 #wikidata #wikipedia #named-entity-recognition #named-entity-linking #region-us 

Dataset Card for AIDA CoNLL-YAGO Wikidata
=========================================


Table of Contents
-----------------


* Dataset Description
	+ Dataset Summary
	+ Supported Tasks
	+ Languages
* Dataset Structure
	+ Data Instances
	+ Data Fields
	+ Data Splits
* Additional Information
	+ Licensing Information


Dataset Description
-------------------


* Repository: AIDA CoNLL-YAGO Wikidata repository


### Dataset Summary


The AIDA CoNLL-YAGO Wikidata dataset is the same as the original AIDA CoNLL-YAGO dataset, but with Wikidata QIDs instead of Wikipedia titles as entity identifiers. They are automatically generated (with a few manual corrections) from Wikidata and Wikipedia dumps (March 1, 2023).


The code for generating the dataset can be found here.


### Supported Tasks


* 'named-entity-recognition': The dataset can be used to train a model for Named Entity Recognition.
* 'named-entity-linking': The dataset can be used to train a model for Named Entity Linking.


### Languages


The text in the dataset is in English. The associated BCP-47 code is 'en'.


Dataset Structure
-----------------


### Data Instances


A typical data point represents a document (news article).


The 'text' field contains the original text in an NFC normalized, UTF-8 encoded string.


The 'entities' field contains a list of entities, each represented by a struct with the inclusive starting byte 'start' field, exclusive ending byte 'end' field, a nullable 'qid' field, and a nullable 'pageid' field.


Additionally, each document has a unique 'document\_id' field.


An example from the AIDA CoNLL-YAGO Wikidata test set looks as follows:


### Data Fields


* 'document\_id': an integer that uniquely identifies the document this sentence belongs to
* 'sentence\_index': an integer that uniquely identifies the position of the sentence in its original document
* 'text': an NFC normalized, UTF-8 encoded string representing the sentence
* 'entities': a list of structs representing entities, each entity has:
	+ 'start': an integer representing the inclusive starting UTF-8 code point of the entity
	+ 'end': an integer representing the exclusive ending UTF-8 code point of the entity
	+ 'tag': a string representing the entity type (PER, LOC, ORG or MISC)
	+ 'qid': an integer representing the Wikidata QID this entity refers to; it can be null if the entity didn't exist in Wikidata at the time of the creation of the original dataset
	+ 'pageid': an integer representing the English Wikipedia's pageID this entity refers to; it can be null if the entity didn't exist in Wikipedia at the time of the creation of the original dataset
	+ 'title': an NFC normalized, UTF-8 encoded string representing the English Wikipedia's title this entity refers to; it can be null if the entity didn't exist in Wikipedia at the time of the creation of the original dataset


### Data Splits


The data is split into training, validation and test sets; all of the sentences belonging to an article are in the same split. The final split sizes are as follows:



Additional Information
----------------------


### Licensing Information


The licensing status of the dataset is the same as the licensing status of the original AIDA CoNLL-YAGO dataset which is under a Creative Commons Attribution-ShareAlike 3.0 Unported License.



### Dataset Summary


The AIDA CoNLL-YAGO Wikidata dataset is the same as the original AIDA CoNLL-YAGO dataset, but with Wikidata QIDs instead of Wikipedia titles as entity identifiers. They are automatically generated (with a few manual corrections) from Wikidata and Wikipedia dumps (March 1, 2023).


The code for generating the dataset can be found here. ### Supported Tasks


* 'named-entity-recognition': The dataset can be used to train a model for Named Entity Recognition.
* 'named-entity-linking': The dataset can be used to train a model for Named Entity Linking. ### Languages


The text in the dataset is in English. The associated BCP-47 code is 'en'.


Dataset Structure
----------------- ### Data Instances


A typical data point represents a document (news article).


The 'text' field contains the original text in an NFC normalized, UTF-8 encoded string.


The 'entities' field contains a list of entities, each represented by a struct with the inclusive starting byte 'start' field, exclusive ending byte 'end' field, a nullable 'qid' field, and a nullable 'pageid' field.


Additionally, each document has a unique 'document\_id' field.


An example from the AIDA CoNLL-YAGO Wikidata test set looks as follows: ### Data Fields


* 'document\_id': an integer that uniquely identifies the document this sentence belongs to
* 'sentence\_index': an integer that uniquely identifies the position of the sentence in its original document
* 'text': an NFC normalized, UTF-8 encoded string representing the sentence
* 'entities': a list of structs representing entities, each entity has:
	+ 'start': an integer representing the inclusive starting UTF-8 code point of the entity
	+ 'end': an integer representing the exclusive ending UTF-8 code point of the entity
	+ 'tag': a string representing the entity type (PER, LOC, ORG or MISC)
	+ 'qid': an integer representing the Wikidata QID this entity refers to; it can be null if the entity didn't exist in Wikidata at the time of the creation of the original dataset
	+ 'pageid': an integer representing the English Wikipedia's pageID this entity refers to; it can be null if the entity didn't exist in Wikipedia at the time of the creation of the original dataset
	+ 'title': an NFC normalized, UTF-8 encoded string representing the English Wikipedia's title this entity refers to; it can be null if the entity didn't exist in Wikipedia at the time of the creation of the original dataset ### Data Splits


The data is split into training, validation and test sets; all of the sentences belonging to an article are in the same split. The final split sizes are as follows:



Additional Information
---------------------- ### Licensing Information


The licensing status of the dataset is the same as the licensing status of the original AIDA CoNLL-YAGO dataset which is under a Creative Commons Attribution-ShareAlike 3.0 Unported License.
TAGS
#task_categories-token-classification #size_categories-10K<n<100K #language-English #license-cc-by-sa-3.0 #wikidata #wikipedia #named-entity-recognition #named-entity-linking #region-us 
 ### Dataset Summary


The AIDA CoNLL-YAGO Wikidata dataset is the same as the original AIDA CoNLL-YAGO dataset, but with Wikidata QIDs instead of Wikipedia titles as entity identifiers. They are automatically generated (with a few manual corrections) from Wikidata and Wikipedia dumps (March 1, 2023).


The code for generating the dataset can be found here. ### Supported Tasks


* 'named-entity-recognition': The dataset can be used to train a model for Named Entity Recognition.
* 'named-entity-linking': The dataset can be used to train a model for Named Entity Linking. ### Languages


The text in the dataset is in English. The associated BCP-47 code is 'en'.


Dataset Structure
----------------- ### Data Instances


A typical data point represents a document (news article).


The 'text' field contains the original text in an NFC normalized, UTF-8 encoded string.


The 'entities' field contains a list of entities, each represented by a struct with the inclusive starting byte 'start' field, exclusive ending byte 'end' field, a nullable 'qid' field, and a nullable 'pageid' field.


Additionally, each document has a unique 'document\_id' field.


An example from the AIDA CoNLL-YAGO Wikidata test set looks as follows: ### Data Fields


* 'document\_id': an integer that uniquely identifies the document this sentence belongs to
* 'sentence\_index': an integer that uniquely identifies the position of the sentence in its original document
* 'text': an NFC normalized, UTF-8 encoded string representing the sentence
* 'entities': a list of structs representing entities, each entity has:
	+ 'start': an integer representing the inclusive starting UTF-8 code point of the entity
	+ 'end': an integer representing the exclusive ending UTF-8 code point of the entity
	+ 'tag': a string representing the entity type (PER, LOC, ORG or MISC)
	+ 'qid': an integer representing the Wikidata QID this entity refers to; it can be null if the entity didn't exist in Wikidata at the time of the creation of the original dataset
	+ 'pageid': an integer representing the English Wikipedia's pageID this entity refers to; it can be null if the entity didn't exist in Wikipedia at the time of the creation of the original dataset
	+ 'title': an NFC normalized, UTF-8 encoded string representing the English Wikipedia's title this entity refers to; it can be null if the entity didn't exist in Wikipedia at the time of the creation of the original dataset ### Data Splits


The data is split into training, validation and test sets; all of the sentences belonging to an article are in the same split. The final split sizes are as follows:



Additional Information
---------------------- ### Licensing Information


The licensing status of the dataset is the same as the licensing status of the original AIDA CoNLL-YAGO dataset which is under a Creative Commons Attribution-ShareAlike 3.0 Unported License.
passage: TAGS
#task_categories-token-classification #size_categories-10K<n<100K #language-English #license-cc-by-sa-3.0 #wikidata #wikipedia #named-entity-recognition #named-entity-linking #region-us 
### Dataset Summary


The AIDA CoNLL-YAGO Wikidata dataset is the same as the original AIDA CoNLL-YAGO dataset, but with Wikidata QIDs instead of Wikipedia titles as entity identifiers. They are automatically generated (with a few manual corrections) from Wikidata and Wikipedia dumps (March 1, 2023).


The code for generating the dataset can be found here.### Supported Tasks


* 'named-entity-recognition': The dataset can be used to train a model for Named Entity Recognition.
* 'named-entity-linking': The dataset can be used to train a model for Named Entity Linking.### Languages


The text in the dataset is in English. The associated BCP-47 code is 'en'.


Dataset Structure
-----------------### Data Instances


A typical data point represents a document (news article).


The 'text' field contains the original text in an NFC normalized, UTF-8 encoded string.


The 'entities' field contains a list of entities, each represented by a struct with the inclusive starting byte 'start' field, exclusive ending byte 'end' field, a nullable 'qid' field, and a nullable 'pageid' field.


Additionally, each document has a unique 'document\_id' field.


An example from the AIDA CoNLL-YAGO Wikidata test set looks as follows: