
# Dataset Card for WikiAnc HR

## Table of Contents
- [Dataset Description](#dataset-description)
  - [Dataset Summary](#dataset-summary)
  - [Supported Tasks](#supported-tasks)
  - [Languages](#languages)
- [Dataset Structure](#dataset-structure)
  - [Data Instances](#data-instances)
  - [Data Fields](#data-fields)
  - [Data Splits](#data-splits)
- [Additional Information](#additional-information)
  - [Licensing Information](#licensing-information)

## Dataset Description

- **Repository:** [WikiAnc repository](https://github.com/cyanic-selkie/wikianc)

### Dataset Summary

The WikiAnc HR datasets is an automatically generated dataset from Wikipedia (hr) and Wikidata dumps (March 1, 2023). 

The code for generating the dataset can be found [here](https://github.com/cyanic-selkie/wikianc).

### Supported Tasks

- `wikificiation`: The dataset can be used to train a model for Wikification.

### Languages

The text in the dataset is in Croatian. The associated BCP-47 code is `hr`.

You can find the English version [here](https://huggingface.co/datasets/cyanic-selkie/wikianc-en).

## Dataset Structure

### Data Instances

A typical data point represents a paragraph in a Wikipedia article.

The `paragraph_text` field contains the original text in an NFC normalized, UTF-8 encoded string.

The `paragraph_anchors` field contains a list of anchors, each represented by a struct with the inclusive starting UTF-8 code point `start` field, exclusive ending UTF-8 code point `end` field, a nullable `qid` field, a nullable `pageid` field, and an NFC normalized, UTF-8 encoded `title` (Wikipedia) field.

Additionally, each paragraph has `article_title`, `article_pageid`, and (nullable) `article_qid` fields referring to the article the paragraph came from.

There is also a nullable, NFC normalized, UTF-8 encoded `section_heading` field, and an integer `section_level` field referring to the heading (if it exists) of the article section, and the level in the section hierarchy that the paragraph came from.

The `qid` fields refers to Wikidata's QID identifiers, while the `pageid` and `title` fields refer to Wikipedia's pageID and title identifiers (there is a one-to-one mapping between pageIDs and titles).

**NOTE:** An anchor will always have a `title`, but that doesn't mean it has to have a `pageid`. This is because Wikipedia allows defining anchors to nonexistent articles.

An example from the WikiAnc HR test set looks as follows:

```
{
  "uuid": "8a9569ea-a398-4d14-8bce-76c263a8c0ac",
  "article_title": "Špiro_Dmitrović",
  "article_pageid": 70957,
  "article_qid": 16116278,
  "section_heading": null,
  "section_level": 0,
  "paragraph_text": "Špiro Dmitrović (Benkovac, 1803. – Zagreb, 6. veljače 1868.) hrvatski časnik i politički borac u doba ilirizma.",
  "paragraph_anchors": [
    {
      "start": 17,
      "end": 25,
      "qid": 397443,
      "pageid": 14426,
      "title": "Benkovac"
    },
    {
      "start": 27,
      "end": 32,
      "qid": 6887,
      "pageid": 1876,
      "title": "1803."
    },
    {
      "start": 35,
      "end": 41,
      "qid": 1435,
      "pageid": 5903,
      "title": "Zagreb"
    },
    {
      "start": 43,
      "end": 53,
      "qid": 2320,
      "pageid": 496,
      "title": "6._veljače"
    },
    {
      "start": 54,
      "end": 59,
      "qid": 7717,
      "pageid": 1811,
      "title": "1868."
    },
    {
      "start": 102,
      "end": 110,
      "qid": 680821,
      "pageid": 54622,
      "title": "Ilirizam"
    }
  ]
}
```

### Data Fields

- `uuid`: a UTF-8 encoded string representing a v4 UUID that uniquely identifies the example
- `article_title`: an NFC normalized, UTF-8 encoded Wikipedia title of the article; spaces are replaced with underscores 
- `article_pageid`: an integer representing the Wikipedia pageID of the article
- `article_qid`: an integer representing the Wikidata QID this article refers to; it can be null if the entity didn't exist in Wikidata at the time of the creation of the original dataset
- `section_heading`: a nullable, NFC normalized, UTF-8 encoded string representing the section heading
- `section_level`: an integer representing the level of the section in the section hierarchy
- `paragraph_text`: an NFC normalized, UTF-8 encoded string representing the paragraph
- `paragraph_anchors`: a list of structs representing anchors, each anchor has:
  - `start`: an integer representing the inclusive starting UTF-8 code point of the anchors
  - `end`: an integer representing the exclusive ending UTF-8 code point of the anchor
  - `qid`: a nullable integer representing the Wikidata QID this anchor refers to; it can be null if the entity didn't exist in Wikidata at the time of the creation of the original dataset
  - `pageid`: a nullable integer representing the Wikipedia pageID of the anchor; it can be null if the article didn't exist in Wikipedia at the time of the creation of the original dataset
  - `title`: an NFC normalized, UTF-8 encoded string representing the Wikipedia title of the anchor; spaces are replaced with underscores; can refer to a nonexistent Wikipedia article 

### Data Splits

The data is split into training, validation and test sets; paragraphs belonging to the same article aren't necessarily in the same split. The final split sizes are as follows:

|                                   | Train        | Validation | Test      |
| :-----                            | :------:     | :-----:    | :----:    |
| WikiAnc HR - articles             | 192,653      | 116,375    | 116,638   |
| WikiAnc HR - paragraphs           | 2,346,651    | 292,590    | 293,557   |
| WikiAnc HR - anchors              | 8,368,928    | 1,039,851  | 1,044,828 |
| WikiAnc HR - anchors with QIDs    | 7,160,367    | 891,959    | 896,414   |
| WikiAnc HR - anchors with pageIDs | 7,179,116    | 894,313    | 898,692   |

**NOTE:** The number of articles in the table above refers to the number of articles that have at least one paragraph belonging to the article appear in the split.

## Additional Information

### Licensing Information

The WikiAnc HR dataset is given under the [Creative Commons Attribution 4.0 International](https://creativecommons.org/licenses/by/4.0/) license.
task_categories:token-classification size_categories:1M<n<10M language:hr license:cc-by-sa-3.0 wikidata wikipedia wikification region:us
{"language": ["hr"], "license": "cc-by-sa-3.0", "size_categories": ["1M<n<10M"], "task_categories": ["token-classification"], "pretty_name": "WikiAnc HR", "tags": ["wikidata", "wikipedia", "wikification"]}
TAGS
#task_categories-token-classification #size_categories-1M<n<10M #language-Croatian #license-cc-by-sa-3.0 #wikidata #wikipedia #wikification #region-us 

Dataset Card for WikiAnc HR
===========================


Table of Contents
-----------------


* Dataset Description
	+ Dataset Summary
	+ Supported Tasks
	+ Languages
* Dataset Structure
	+ Data Instances
	+ Data Fields
	+ Data Splits
* Additional Information
	+ Licensing Information


Dataset Description
-------------------


* Repository: WikiAnc repository


### Dataset Summary


The WikiAnc HR datasets is an automatically generated dataset from Wikipedia (hr) and Wikidata dumps (March 1, 2023).


The code for generating the dataset can be found here.


### Supported Tasks


* 'wikificiation': The dataset can be used to train a model for Wikification.


### Languages


The text in the dataset is in Croatian. The associated BCP-47 code is 'hr'.


You can find the English version here.


Dataset Structure
-----------------


### Data Instances


A typical data point represents a paragraph in a Wikipedia article.


The 'paragraph\_text' field contains the original text in an NFC normalized, UTF-8 encoded string.


The 'paragraph\_anchors' field contains a list of anchors, each represented by a struct with the inclusive starting UTF-8 code point 'start' field, exclusive ending UTF-8 code point 'end' field, a nullable 'qid' field, a nullable 'pageid' field, and an NFC normalized, UTF-8 encoded 'title' (Wikipedia) field.


Additionally, each paragraph has 'article\_title', 'article\_pageid', and (nullable) 'article\_qid' fields referring to the article the paragraph came from.


There is also a nullable, NFC normalized, UTF-8 encoded 'section\_heading' field, and an integer 'section\_level' field referring to the heading (if it exists) of the article section, and the level in the section hierarchy that the paragraph came from.


The 'qid' fields refers to Wikidata's QID identifiers, while the 'pageid' and 'title' fields refer to Wikipedia's pageID and title identifiers (there is a one-to-one mapping between pageIDs and titles).


NOTE: An anchor will always have a 'title', but that doesn't mean it has to have a 'pageid'. This is because Wikipedia allows defining anchors to nonexistent articles.


An example from the WikiAnc HR test set looks as follows:


### Data Fields


* 'uuid': a UTF-8 encoded string representing a v4 UUID that uniquely identifies the example
* 'article\_title': an NFC normalized, UTF-8 encoded Wikipedia title of the article; spaces are replaced with underscores
* 'article\_pageid': an integer representing the Wikipedia pageID of the article
* 'article\_qid': an integer representing the Wikidata QID this article refers to; it can be null if the entity didn't exist in Wikidata at the time of the creation of the original dataset
* 'section\_heading': a nullable, NFC normalized, UTF-8 encoded string representing the section heading
* 'section\_level': an integer representing the level of the section in the section hierarchy
* 'paragraph\_text': an NFC normalized, UTF-8 encoded string representing the paragraph
* 'paragraph\_anchors': a list of structs representing anchors, each anchor has:
	+ 'start': an integer representing the inclusive starting UTF-8 code point of the anchors
	+ 'end': an integer representing the exclusive ending UTF-8 code point of the anchor
	+ 'qid': a nullable integer representing the Wikidata QID this anchor refers to; it can be null if the entity didn't exist in Wikidata at the time of the creation of the original dataset
	+ 'pageid': a nullable integer representing the Wikipedia pageID of the anchor; it can be null if the article didn't exist in Wikipedia at the time of the creation of the original dataset
	+ 'title': an NFC normalized, UTF-8 encoded string representing the Wikipedia title of the anchor; spaces are replaced with underscores; can refer to a nonexistent Wikipedia article


### Data Splits


The data is split into training, validation and test sets; paragraphs belonging to the same article aren't necessarily in the same split. The final split sizes are as follows:



NOTE: The number of articles in the table above refers to the number of articles that have at least one paragraph belonging to the article appear in the split.


Additional Information
----------------------


### Licensing Information


The WikiAnc HR dataset is given under the Creative Commons Attribution 4.0 International license.



### Dataset Summary


The WikiAnc HR datasets is an automatically generated dataset from Wikipedia (hr) and Wikidata dumps (March 1, 2023).


The code for generating the dataset can be found here. ### Supported Tasks


* 'wikificiation': The dataset can be used to train a model for Wikification. ### Languages


The text in the dataset is in Croatian. The associated BCP-47 code is 'hr'.


You can find the English version here.


Dataset Structure
----------------- ### Data Instances


A typical data point represents a paragraph in a Wikipedia article.


The 'paragraph\_text' field contains the original text in an NFC normalized, UTF-8 encoded string.


The 'paragraph\_anchors' field contains a list of anchors, each represented by a struct with the inclusive starting UTF-8 code point 'start' field, exclusive ending UTF-8 code point 'end' field, a nullable 'qid' field, a nullable 'pageid' field, and an NFC normalized, UTF-8 encoded 'title' (Wikipedia) field.


Additionally, each paragraph has 'article\_title', 'article\_pageid', and (nullable) 'article\_qid' fields referring to the article the paragraph came from.


There is also a nullable, NFC normalized, UTF-8 encoded 'section\_heading' field, and an integer 'section\_level' field referring to the heading (if it exists) of the article section, and the level in the section hierarchy that the paragraph came from.


The 'qid' fields refers to Wikidata's QID identifiers, while the 'pageid' and 'title' fields refer to Wikipedia's pageID and title identifiers (there is a one-to-one mapping between pageIDs and titles).


NOTE: An anchor will always have a 'title', but that doesn't mean it has to have a 'pageid'. This is because Wikipedia allows defining anchors to nonexistent articles.


An example from the WikiAnc HR test set looks as follows: ### Data Fields


* 'uuid': a UTF-8 encoded string representing a v4 UUID that uniquely identifies the example
* 'article\_title': an NFC normalized, UTF-8 encoded Wikipedia title of the article; spaces are replaced with underscores
* 'article\_pageid': an integer representing the Wikipedia pageID of the article
* 'article\_qid': an integer representing the Wikidata QID this article refers to; it can be null if the entity didn't exist in Wikidata at the time of the creation of the original dataset
* 'section\_heading': a nullable, NFC normalized, UTF-8 encoded string representing the section heading
* 'section\_level': an integer representing the level of the section in the section hierarchy
* 'paragraph\_text': an NFC normalized, UTF-8 encoded string representing the paragraph
* 'paragraph\_anchors': a list of structs representing anchors, each anchor has:
	+ 'start': an integer representing the inclusive starting UTF-8 code point of the anchors
	+ 'end': an integer representing the exclusive ending UTF-8 code point of the anchor
	+ 'qid': a nullable integer representing the Wikidata QID this anchor refers to; it can be null if the entity didn't exist in Wikidata at the time of the creation of the original dataset
	+ 'pageid': a nullable integer representing the Wikipedia pageID of the anchor; it can be null if the article didn't exist in Wikipedia at the time of the creation of the original dataset
	+ 'title': an NFC normalized, UTF-8 encoded string representing the Wikipedia title of the anchor; spaces are replaced with underscores; can refer to a nonexistent Wikipedia article ### Data Splits


The data is split into training, validation and test sets; paragraphs belonging to the same article aren't necessarily in the same split. The final split sizes are as follows:



NOTE: The number of articles in the table above refers to the number of articles that have at least one paragraph belonging to the article appear in the split.


Additional Information
---------------------- ### Licensing Information


The WikiAnc HR dataset is given under the Creative Commons Attribution 4.0 International license.
TAGS
#task_categories-token-classification #size_categories-1M<n<10M #language-Croatian #license-cc-by-sa-3.0 #wikidata #wikipedia #wikification #region-us 
 ### Dataset Summary


The WikiAnc HR datasets is an automatically generated dataset from Wikipedia (hr) and Wikidata dumps (March 1, 2023).


The code for generating the dataset can be found here. ### Supported Tasks


* 'wikificiation': The dataset can be used to train a model for Wikification. ### Languages


The text in the dataset is in Croatian. The associated BCP-47 code is 'hr'.


You can find the English version here.


Dataset Structure
----------------- ### Data Instances


A typical data point represents a paragraph in a Wikipedia article.


The 'paragraph\_text' field contains the original text in an NFC normalized, UTF-8 encoded string.


The 'paragraph\_anchors' field contains a list of anchors, each represented by a struct with the inclusive starting UTF-8 code point 'start' field, exclusive ending UTF-8 code point 'end' field, a nullable 'qid' field, a nullable 'pageid' field, and an NFC normalized, UTF-8 encoded 'title' (Wikipedia) field.


Additionally, each paragraph has 'article\_title', 'article\_pageid', and (nullable) 'article\_qid' fields referring to the article the paragraph came from.


There is also a nullable, NFC normalized, UTF-8 encoded 'section\_heading' field, and an integer 'section\_level' field referring to the heading (if it exists) of the article section, and the level in the section hierarchy that the paragraph came from.


The 'qid' fields refers to Wikidata's QID identifiers, while the 'pageid' and 'title' fields refer to Wikipedia's pageID and title identifiers (there is a one-to-one mapping between pageIDs and titles).


NOTE: An anchor will always have a 'title', but that doesn't mean it has to have a 'pageid'. This is because Wikipedia allows defining anchors to nonexistent articles.


An example from the WikiAnc HR test set looks as follows: ### Data Fields


* 'uuid': a UTF-8 encoded string representing a v4 UUID that uniquely identifies the example
* 'article\_title': an NFC normalized, UTF-8 encoded Wikipedia title of the article; spaces are replaced with underscores
* 'article\_pageid': an integer representing the Wikipedia pageID of the article
* 'article\_qid': an integer representing the Wikidata QID this article refers to; it can be null if the entity didn't exist in Wikidata at the time of the creation of the original dataset
* 'section\_heading': a nullable, NFC normalized, UTF-8 encoded string representing the section heading
* 'section\_level': an integer representing the level of the section in the section hierarchy
* 'paragraph\_text': an NFC normalized, UTF-8 encoded string representing the paragraph
* 'paragraph\_anchors': a list of structs representing anchors, each anchor has:
	+ 'start': an integer representing the inclusive starting UTF-8 code point of the anchors
	+ 'end': an integer representing the exclusive ending UTF-8 code point of the anchor
	+ 'qid': a nullable integer representing the Wikidata QID this anchor refers to; it can be null if the entity didn't exist in Wikidata at the time of the creation of the original dataset
	+ 'pageid': a nullable integer representing the Wikipedia pageID of the anchor; it can be null if the article didn't exist in Wikipedia at the time of the creation of the original dataset
	+ 'title': an NFC normalized, UTF-8 encoded string representing the Wikipedia title of the anchor; spaces are replaced with underscores; can refer to a nonexistent Wikipedia article ### Data Splits


The data is split into training, validation and test sets; paragraphs belonging to the same article aren't necessarily in the same split. The final split sizes are as follows:



NOTE: The number of articles in the table above refers to the number of articles that have at least one paragraph belonging to the article appear in the split.


Additional Information
---------------------- ### Licensing Information


The WikiAnc HR dataset is given under the Creative Commons Attribution 4.0 International license.
passage: TAGS
#task_categories-token-classification #size_categories-1M<n<10M #language-Croatian #license-cc-by-sa-3.0 #wikidata #wikipedia #wikification #region-us 
### Dataset Summary


The WikiAnc HR datasets is an automatically generated dataset from Wikipedia (hr) and Wikidata dumps (March 1, 2023).


The code for generating the dataset can be found here.### Supported Tasks


* 'wikificiation': The dataset can be used to train a model for Wikification.### Languages


The text in the dataset is in Croatian. The associated BCP-47 code is 'hr'.


You can find the English version here.


Dataset Structure
----------------- passage: ### Data Instances


A typical data point represents a paragraph in a Wikipedia article.


The 'paragraph\_text' field contains the original text in an NFC normalized, UTF-8 encoded string.


The 'paragraph\_anchors' field contains a list of anchors, each represented by a struct with the inclusive starting UTF-8 code point 'start' field, exclusive ending UTF-8 code point 'end' field, a nullable 'qid' field, a nullable 'pageid' field, and an NFC normalized, UTF-8 encoded 'title' (Wikipedia) field.


Additionally, each paragraph has 'article\_title', 'article\_pageid', and (nullable) 'article\_qid' fields referring to the article the paragraph came from.


There is also a nullable, NFC normalized, UTF-8 encoded 'section\_heading' field, and an integer 'section\_level' field referring to the heading (if it exists) of the article section, and the level in the section hierarchy that the paragraph came from.


The 'qid' fields refers to Wikidata's QID identifiers, while the 'pageid' and 'title' fields refer to Wikipedia's pageID and title identifiers (there is a one-to-one mapping between pageIDs and titles).


NOTE: An anchor will always have a 'title', but that doesn't mean it has to have a 'pageid'. This is because Wikipedia allows defining anchors to nonexistent articles.


An example from the WikiAnc HR test set looks as follows:### Data Fields


* 'uuid': a UTF-8 encoded string representing a v4 UUID that uniquely identifies the example
* 'article\_title': an NFC normalized, UTF-8 encoded Wikipedia title of the article; spaces are replaced with underscores
* 'article\_pageid': an integer representing the Wikipedia pageID of the article
* 'article\_qid': an integer representing the Wikidata QID this article refers to; it can be null if the entity didn't exist in Wikidata at the time of the creation of the original dataset
* 'section\_heading': a nullable, NFC normalized, UTF-8 encoded string representing the section heading
* 'section\_level': an integer representing the level of the section in the section hierarchy
* 'paragraph\_text': an NFC normalized, UTF-8 encoded string representing the paragraph
* 'paragraph\_anchors': a list of structs representing anchors, each anchor has:
	+ 'start': an integer representing the inclusive starting UTF-8 code point of the anchors
	+ 'end': an integer representing the exclusive ending UTF-8 code point of the anchor
	+ 'qid': a nullable integer representing the Wikidata QID this anchor refers to; it can be null if the entity didn't exist in Wikidata at the time of the creation of the original dataset
	+ 'pageid': a nullable integer representing the Wikipedia pageID of the anchor; it can be null if the article didn't exist in Wikipedia at the time of the creation of the original dataset
	+ 'title': an NFC normalized, UTF-8 encoded string representing the Wikipedia title of the anchor; spaces are replaced with underscores; can refer to a nonexistent Wikipedia article