region:us
# TyDiQA-GoldP-Th
This dataset contains a removed Thai TyDiQA dataset obtained from [Khalidalt's TyDiQA Dataset](https://huggingface.co/datasets/khalidalt/tydiqa-goldp).
This dataset version does the following additional preprocessing to the dataset
1. Convert byte-level index into character-level index
2. Fix any mismatch text between answer span and actual text
3. Re-split train/development set such that there's no leakage in context passage
4. Deduplicate questions from the same context passage

## Dataset Format
The dataset is formatted to make it compatible to [XTREME benchmark](https://github.com/google-research/xtreme) format. The data is formatted as the following pattern:
```json
{
  "version": "TyDiQA-GoldP-1.1-for-SQuAD-1.1",
  "data": [
    {
      "paragrahs": [{
        "context": [PASSAGE CONTEXT HERE],
        "qas": [{
          "answers": [{
            "answer_start": [CONTEXT START CHAR INDEX OF ANSWER],
            "text": [TEXT SPAN FROM CONTEXT],
          }],
          "question": [QUESTION],
          "id": [ID]
        }]
      }],
    },
    ...
  ]
}
```

## Author
Chompakorn Chaksangchaichot