
# Dataset Card for "answerable-tydiqa"


## Dataset Description

- **Homepage:** [https://github.com/google-research-datasets/tydiqa](https://github.com/google-research-datasets/tydiqa)
- **Paper:** [Paper](https://aclanthology.org/2020.tacl-1.30/)
- **Size of downloaded dataset files:** 75.43 MB
- **Size of the generated dataset:** 131.78 MB
- **Total amount of disk used:** 207.21 MB

### Dataset Summary

[TyDi QA](https://huggingface.co/datasets/tydiqa) is a question answering dataset covering 11 typologically diverse languages. 
Answerable TyDi QA is an extension of the GoldP subtask of the original TyDi QA dataset to also include unanswertable questions.

## Dataset Structure

The dataset contains a train and a validation set, with 116067 and 13325 examples, respectively. Access them with

```py
from datasets import load_dataset
dataset = load_dataset("copenlu/answerable_tydiqa")
train_set = dataset["train"]
validation_set = dataset["validation"]
```

### Data Instances

Here is an example of an instance of the dataset:

```
{'question_text': 'dimanakah  Dr. Ernest François Eugène Douwes Dekker meninggal?',
 'document_title': 'Ernest Douwes Dekker',
 'language': 'indonesian',
 'annotations': 
             {'answer_start': [45],
              'answer_text': ['28 Agustus 1950']
              },
 'document_plaintext': 'Ernest Douwes Dekker wafat dini hari tanggal 28 Agustus 1950 (tertulis di batu nisannya; 29 Agustus 1950 versi van der Veur, 2006) dan dimakamkan di TMP Cikutra, Bandung.',
 'document_url': 'https://id.wikipedia.org/wiki/Ernest%20Douwes%20Dekker'}
```

Description of the dataset columns:

| Column name                  | type        |  Description                                                                                                     |
| -----------                  | ----------- | -----------                                                                                                      |
| document_title               | str         | The title of the Wikipedia article from which the data instance was generated                                    |
| document_url                 | str         | The URL of said article                                                                                          |
| language                     | str         | The language of the data instance                                                                                |
| question_text                | str         | The question to answer                                                                                           |
| document_plaintext           | str         | The context, a Wikipedia paragraph that might or might not contain the answer to the question                    | 
| annotations["answer_start"]  | list[int]   | The char index in 'document_plaintext' where the answer starts. If the question is unanswerable - [-1]  |
| annotations["answer_text"]   | list[str]   | The answer, a span of text from 'document_plaintext'. If the question is unanswerable - ['']            |


**Notice:** If the question is *answerable*, annotations["answer_start"] and annotations["answer_text"] contain a list of length 1  
(In some variations of the dataset the lists might be longer, e.g. if more than one person annotated the instance, but not in our case). 
If the question is *unanswerable*, annotations["answer_start"] will have "-1", while annotations["answer_text"] contain a list with an empty sring.


## Useful stuff

Check out the [datasets ducumentations](https://huggingface.co/docs/datasets/quickstart) to learn how to manipulate and use the dataset. Specifically, you might find the following functions useful:

`dataset.filter`, for filtering out data (useful for keeping instances of specific languages, for example).

`dataset.map`, for manipulating the dataset.

`dataset.to_pandas`, to convert the dataset into a pandas.DataFrame format.


```
@article{tydiqa,
title   = {TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languages},
author  = {Jonathan H. Clark and Eunsol Choi and Michael Collins and Dan Garrette and Tom Kwiatkowski and Vitaly Nikolaev and Jennimaria Palomaki}
year    = {2020},
journal = {Transactions of the Association for Computational Linguistics}
}

```


### Contributions

Thanks to [@thomwolf](https://github.com/thomwolf), [@albertvillanova](https://github.com/albertvillanova), [@lewtun](https://github.com/lewtun), [@patrickvonplaten](https://github.com/patrickvonplaten) for adding this dataset.
task_categories:question-answering task_ids:extractive-qa annotations_creators:crowdsourced language_creators:crowdsourced multilinguality:multilingual source_datasets:extended|wikipedia language:en language:ar language:bn language:fi language:id language:ja language:sw language:ko language:ru language:te language:th license:apache-2.0 region:us
{"annotations_creators": ["crowdsourced"], "language_creators": ["crowdsourced"], "language": ["en", "ar", "bn", "fi", "id", "ja", "sw", "ko", "ru", "te", "th"], "license": ["apache-2.0"], "multilinguality": ["multilingual"], "size_categories": [["100K<n<1M"]], "source_datasets": ["extended|wikipedia"], "task_categories": ["question-answering"], "task_ids": ["extractive-qa"], "pretty_name": "Answerable TyDi QA"}
TAGS
#task_categories-question-answering #task_ids-extractive-qa #annotations_creators-crowdsourced #language_creators-crowdsourced #multilinguality-multilingual #source_datasets-extended|wikipedia #language-English #language-Arabic #language-Bengali #language-Finnish #language-Indonesian #language-Japanese #language-Swahili (macrolanguage) #language-Korean #language-Russian #language-Telugu #language-Thai #license-apache-2.0 #region-us 

Dataset Card for "answerable-tydiqa"
====================================


Dataset Description
-------------------


* Homepage: URL
* Paper: Paper
* Size of downloaded dataset files: 75.43 MB
* Size of the generated dataset: 131.78 MB
* Total amount of disk used: 207.21 MB


### Dataset Summary


TyDi QA is a question answering dataset covering 11 typologically diverse languages.
Answerable TyDi QA is an extension of the GoldP subtask of the original TyDi QA dataset to also include unanswertable questions.


Dataset Structure
-----------------


The dataset contains a train and a validation set, with 116067 and 13325 examples, respectively. Access them with


### Data Instances


Here is an example of an instance of the dataset:


Description of the dataset columns:


Column name: document\_title, type: str, Description: The title of the Wikipedia article from which the data instance was generated
Column name: document\_url, type: str, Description: The URL of said article
Column name: language, type: str, Description: The language of the data instance
Column name: question\_text, type: str, Description: The question to answer
Column name: document\_plaintext, type: str, Description: The context, a Wikipedia paragraph that might or might not contain the answer to the question
Column name: annotations["answer\_start"], type: list[int], Description: The char index in 'document\_plaintext' where the answer starts. If the question is unanswerable - [-1]
Column name: annotations["answer\_text"], type: list[str], Description: The answer, a span of text from 'document\_plaintext'. If the question is unanswerable - ['']


Notice: If the question is *answerable*, annotations["answer\_start"] and annotations["answer\_text"] contain a list of length 1  

(In some variations of the dataset the lists might be longer, e.g. if more than one person annotated the instance, but not in our case).
If the question is *unanswerable*, annotations["answer\_start"] will have "-1", while annotations["answer\_text"] contain a list with an empty sring.


Useful stuff
------------


Check out the datasets ducumentations to learn how to manipulate and use the dataset. Specifically, you might find the following functions useful:


'URL', for filtering out data (useful for keeping instances of specific languages, for example).


'URL', for manipulating the dataset.


'dataset.to\_pandas', to convert the dataset into a pandas.DataFrame format.


### Contributions


Thanks to @thomwolf, @albertvillanova, @lewtun, @patrickvonplaten for adding this dataset.



### Dataset Summary


TyDi QA is a question answering dataset covering 11 typologically diverse languages.
Answerable TyDi QA is an extension of the GoldP subtask of the original TyDi QA dataset to also include unanswertable questions.


Dataset Structure
-----------------


The dataset contains a train and a validation set, with 116067 and 13325 examples, respectively. Access them with ### Data Instances


Here is an example of an instance of the dataset:


Description of the dataset columns:


Column name: document\_title, type: str, Description: The title of the Wikipedia article from which the data instance was generated
Column name: document\_url, type: str, Description: The URL of said article
Column name: language, type: str, Description: The language of the data instance
Column name: question\_text, type: str, Description: The question to answer
Column name: document\_plaintext, type: str, Description: The context, a Wikipedia paragraph that might or might not contain the answer to the question
Column name: annotations["answer\_start"], type: list[int], Description: The char index in 'document\_plaintext' where the answer starts. If the question is unanswerable - [-1]
Column name: annotations["answer\_text"], type: list[str], Description: The answer, a span of text from 'document\_plaintext'. If the question is unanswerable - ['']


Notice: If the question is *answerable*, annotations["answer\_start"] and annotations["answer\_text"] contain a list of length 1  

(In some variations of the dataset the lists might be longer, e.g. if more than one person annotated the instance, but not in our case).
If the question is *unanswerable*, annotations["answer\_start"] will have "-1", while annotations["answer\_text"] contain a list with an empty sring.


Useful stuff
------------


Check out the datasets ducumentations to learn how to manipulate and use the dataset. Specifically, you might find the following functions useful:


'URL', for filtering out data (useful for keeping instances of specific languages, for example).


'URL', for manipulating the dataset.


'dataset.to\_pandas', to convert the dataset into a pandas.DataFrame format. ### Contributions


Thanks to @thomwolf, @albertvillanova, @lewtun, @patrickvonplaten for adding this dataset.
TAGS
#task_categories-question-answering #task_ids-extractive-qa #annotations_creators-crowdsourced #language_creators-crowdsourced #multilinguality-multilingual #source_datasets-extended|wikipedia #language-English #language-Arabic #language-Bengali #language-Finnish #language-Indonesian #language-Japanese #language-Swahili (macrolanguage) #language-Korean #language-Russian #language-Telugu #language-Thai #license-apache-2.0 #region-us 
 ### Dataset Summary


TyDi QA is a question answering dataset covering 11 typologically diverse languages.
Answerable TyDi QA is an extension of the GoldP subtask of the original TyDi QA dataset to also include unanswertable questions.


Dataset Structure
-----------------


The dataset contains a train and a validation set, with 116067 and 13325 examples, respectively. Access them with ### Data Instances


Here is an example of an instance of the dataset:


Description of the dataset columns:


Column name: document\_title, type: str, Description: The title of the Wikipedia article from which the data instance was generated
Column name: document\_url, type: str, Description: The URL of said article
Column name: language, type: str, Description: The language of the data instance
Column name: question\_text, type: str, Description: The question to answer
Column name: document\_plaintext, type: str, Description: The context, a Wikipedia paragraph that might or might not contain the answer to the question
Column name: annotations["answer\_start"], type: list[int], Description: The char index in 'document\_plaintext' where the answer starts. If the question is unanswerable - [-1]
Column name: annotations["answer\_text"], type: list[str], Description: The answer, a span of text from 'document\_plaintext'. If the question is unanswerable - ['']


Notice: If the question is *answerable*, annotations["answer\_start"] and annotations["answer\_text"] contain a list of length 1  

(In some variations of the dataset the lists might be longer, e.g. if more than one person annotated the instance, but not in our case).
If the question is *unanswerable*, annotations["answer\_start"] will have "-1", while annotations["answer\_text"] contain a list with an empty sring.


Useful stuff
------------


Check out the datasets ducumentations to learn how to manipulate and use the dataset. Specifically, you might find the following functions useful:


'URL', for filtering out data (useful for keeping instances of specific languages, for example).


'URL', for manipulating the dataset.


'dataset.to\_pandas', to convert the dataset into a pandas.DataFrame format. ### Contributions


Thanks to @thomwolf, @albertvillanova, @lewtun, @patrickvonplaten for adding this dataset.
passage: TAGS
#task_categories-question-answering #task_ids-extractive-qa #annotations_creators-crowdsourced #language_creators-crowdsourced #multilinguality-multilingual #source_datasets-extended|wikipedia #language-English #language-Arabic #language-Bengali #language-Finnish #language-Indonesian #language-Japanese #language-Swahili (macrolanguage) #language-Korean #language-Russian #language-Telugu #language-Thai #license-apache-2.0 #region-us 
### Dataset Summary


TyDi QA is a question answering dataset covering 11 typologically diverse languages.
Answerable TyDi QA is an extension of the GoldP subtask of the original TyDi QA dataset to also include unanswertable questions.


Dataset Structure
-----------------


The dataset contains a train and a validation set, with 116067 and 13325 examples, respectively. Access them with