
# Dataset Card for KQA Pro

## Table of Contents
- [Table of Contents](#table-of-contents)
- [Dataset Description](#dataset-description)
  - [Dataset Summary](#dataset-summary)
  - [Supported Tasks and Leaderboards](#supported-tasks-and-leaderboards)
  - [Languages](#languages)
- [Dataset Structure](#dataset-structure)
  - [Data Configs](#data-configs)
  - [Data Splits](#data-splits)
- [Additional Information](#additional-information)
  - [How to run SPARQLs and programs](#how-to-run-sparqls-and-programs)
  - [Knowledge Graph File](#knowledge-graph-file)
  - [How to Submit to Leaderboard](#how-to-submit-results-of-test-set)
  - [Licensing Information](#licensing-information)
  - [Citation Information](#citation-information)
  - [Contributions](#contributions)

## Dataset Description

- **Homepage:** http://thukeg.gitee.io/kqa-pro/
- **Repository:** https://github.com/shijx12/KQAPro_Baselines
- **Paper:** [KQA Pro: A Dataset with Explicit Compositional Programs for Complex Question Answering over Knowledge Base](https://aclanthology.org/2022.acl-long.422/)
- **Leaderboard:** http://thukeg.gitee.io/kqa-pro/leaderboard.html
- **Point of Contact:** shijx12 at gmail dot com

### Dataset Summary

KQA Pro is a large-scale dataset of complex question answering over knowledge base. The questions are very diverse and challenging, requiring multiple reasoning capabilities including compositional reasoning, multi-hop reasoning, quantitative comparison, set operations, and etc. Strong supervisions of SPARQL and program are provided for each question.

### Supported Tasks and Leaderboards

It supports knowlege graph based question answering. Specifically, it provides SPARQL and *program* for each question.

### Languages

English

## Dataset Structure

**train.json/val.json**
```
[
    {
        'question': str,
        'sparql': str, # executable in our virtuoso engine
        'program': 
        [
            {
                'function': str,  # function name
                'dependencies': [int],  # functional inputs, representing indices of the preceding functions
                'inputs': [str],  # textual inputs
            }
        ],
        'choices': [str],  # 10 answer choices
        'answer': str,  # golden answer
    }
]
```

**test.json**
```
[
    {
        'question': str,
        'choices': [str],  # 10 answer choices
    }
]
```

### Data Configs

This dataset has two configs: `train_val` and `test` because they have different available fields. Please specify this like `load_dataset('drt/kqa_pro', 'train_val')`.

### Data Splits

train, val, test


## Additional Information

### Knowledge Graph File

You can find the knowledge graph file `kb.json` in the original github repository. It comes with the format:

```json
{
    'concepts':
    {
        '<id>':
        {
            'name': str,
            'instanceOf': ['<id>', '<id>'], # ids of parent concept
        }
    },
    'entities': # excluding concepts
    {
        '<id>': 
        {
            'name': str,
            'instanceOf': ['<id>', '<id>'], # ids of parent concept
            'attributes':
            [
                {
                    'key': str, # attribute key
                    'value':  # attribute value
                    {
                        'type': 'string'/'quantity'/'date'/'year',
                        'value': float/int/str, # float or int for quantity, int for year, 'yyyy/mm/dd' for date
                        'unit': str,  # for quantity
                    },
                    'qualifiers':
                    {
                        '<qk>':  # qualifier key, one key may have multiple corresponding qualifier values
                        [
                            {
                                'type': 'string'/'quantity'/'date'/'year',
                                'value': float/int/str,
                                'unit': str,
                            }, # the format of qualifier value is similar to attribute value
                        ]
                    }
                },
            ]
            'relations':
            [
                {
                    'predicate': str,
                    'object': '<id>', # NOTE: it may be a concept id
                    'direction': 'forward'/'backward',
                    'qualifiers':
                    {
                        '<qk>':  # qualifier key, one key may have multiple corresponding qualifier values
                        [
                            {
                                'type': 'string'/'quantity'/'date'/'year',
                                'value': float/int/str,
                                'unit': str,
                            }, # the format of qualifier value is similar to attribute value
                        ]
                    }
                },
            ]
        }
    }
}
```



### How to run SPARQLs and programs

We implement multiple baselines in our [codebase](https://github.com/shijx12/KQAPro_Baselines), which includes a supervised SPARQL parser and program parser.

In the SPARQL parser, we implement a query engine based on [Virtuoso](https://github.com/openlink/virtuoso-opensource.git).
You can install the engine based on our [instructions](https://github.com/shijx12/KQAPro_Baselines/blob/master/SPARQL/README.md), and then feed your predicted SPARQL to get the answer.

In the program parser, we implement a rule-based program executor, which receives a predicted program and returns the answer.
Detailed introductions of our functions can be found in our [paper](https://arxiv.org/abs/2007.03875).

### How to submit results of test set
You need to predict answers for all questions of test set and write them in a text file **in order**, one per line.
Here is an example:
```
Tron: Legacy
Palm Beach County
1937-03-01
The Queen
...
```

Then you need to send the prediction file to us by email <caosl19@mails.tsinghua.edu.cn>, we will reply to you with the performance as soon as possible.
To appear in the learderboard, you need to also provide following information:

- model name
- affiliation
- open-ended or multiple-choice
- whether use the supervision of SPARQL in your model or not
- whether use the supervision of program in your model or not
- single model or ensemble model
- (optional) paper link
- (optional) code link

### Licensing Information

MIT License

### Citation Information

If you find our dataset is helpful in your work, please cite us by

```
@inproceedings{KQAPro,
  title={{KQA P}ro: A Large Diagnostic Dataset for Complex Question Answering over Knowledge Base},
  author={Cao, Shulin and Shi, Jiaxin and Pan, Liangming and Nie, Lunyiu and Xiang, Yutong and Hou, Lei and Li, Juanzi and He, Bin and Zhang, Hanwang},
  booktitle={ACL'22},
  year={2022}
}
```

### Contributions

Thanks to [@happen2me](https://github.com/happen2me) for adding this dataset.

task_categories:question-answering task_ids:open-domain-qa annotations_creators:machine-generated annotations_creators:expert-generated language_creators:found multilinguality:monolingual size_categories:10K<n<100K source_datasets:original language:en license:mit knowledge graph freebase arxiv:2007.03875 region:us
{"annotations_creators": ["machine-generated", "expert-generated"], "language_creators": ["found"], "language": ["en"], "license": ["mit"], "multilinguality": ["monolingual"], "size_categories": ["10K<n<100K"], "source_datasets": ["original"], "task_categories": ["question-answering"], "task_ids": ["open-domain-qa"], "pretty_name": "KQA-Pro", "tags": ["knowledge graph", "freebase"]}
TAGS
#task_categories-question-answering #task_ids-open-domain-qa #annotations_creators-machine-generated #annotations_creators-expert-generated #language_creators-found #multilinguality-monolingual #size_categories-10K<n<100K #source_datasets-original #language-English #license-mit #knowledge graph #freebase #arxiv-2007.03875 #region-us 


# Dataset Card for KQA Pro

## Table of Contents
- Table of Contents
- Dataset Description
  - Dataset Summary
  - Supported Tasks and Leaderboards
  - Languages
- Dataset Structure
  - Data Configs
  - Data Splits
- Additional Information
  - How to run SPARQLs and programs
  - Knowledge Graph File
  - How to Submit to Leaderboard
  - Licensing Information
  - Citation Information
  - Contributions

## Dataset Description

- Homepage: URL
- Repository: URL
- Paper: KQA Pro: A Dataset with Explicit Compositional Programs for Complex Question Answering over Knowledge Base
- Leaderboard: URL
- Point of Contact: shijx12 at gmail dot com

### Dataset Summary

KQA Pro is a large-scale dataset of complex question answering over knowledge base. The questions are very diverse and challenging, requiring multiple reasoning capabilities including compositional reasoning, multi-hop reasoning, quantitative comparison, set operations, and etc. Strong supervisions of SPARQL and program are provided for each question.

### Supported Tasks and Leaderboards

It supports knowlege graph based question answering. Specifically, it provides SPARQL and *program* for each question.

### Languages

English

## Dataset Structure

URL


URL


### Data Configs

This dataset has two configs: 'train_val' and 'test' because they have different available fields. Please specify this like 'load_dataset('drt/kqa_pro', 'train_val')'.

### Data Splits

train, val, test


## Additional Information

### Knowledge Graph File

You can find the knowledge graph file 'URL' in the original github repository. It comes with the format:





### How to run SPARQLs and programs

We implement multiple baselines in our codebase, which includes a supervised SPARQL parser and program parser.

In the SPARQL parser, we implement a query engine based on Virtuoso.
You can install the engine based on our instructions, and then feed your predicted SPARQL to get the answer.

In the program parser, we implement a rule-based program executor, which receives a predicted program and returns the answer.
Detailed introductions of our functions can be found in our paper.

### How to submit results of test set
You need to predict answers for all questions of test set and write them in a text file in order, one per line.
Here is an example:


Then you need to send the prediction file to us by email <caosl19@URL>, we will reply to you with the performance as soon as possible.
To appear in the learderboard, you need to also provide following information:

- model name
- affiliation
- open-ended or multiple-choice
- whether use the supervision of SPARQL in your model or not
- whether use the supervision of program in your model or not
- single model or ensemble model
- (optional) paper link
- (optional) code link

### Licensing Information

MIT License



If you find our dataset is helpful in your work, please cite us by



### Contributions

Thanks to @happen2me for adding this dataset.

# Dataset Card for KQA Pro ## Table of Contents
- Table of Contents
- Dataset Description
  - Dataset Summary
  - Supported Tasks and Leaderboards
  - Languages
- Dataset Structure
  - Data Configs
  - Data Splits
- Additional Information
  - How to run SPARQLs and programs
  - Knowledge Graph File
  - How to Submit to Leaderboard
  - Licensing Information
  - Citation Information
  - Contributions ## Dataset Description

- Homepage: URL
- Repository: URL
- Paper: KQA Pro: A Dataset with Explicit Compositional Programs for Complex Question Answering over Knowledge Base
- Leaderboard: URL
- Point of Contact: shijx12 at gmail dot com ### Dataset Summary

KQA Pro is a large-scale dataset of complex question answering over knowledge base. The questions are very diverse and challenging, requiring multiple reasoning capabilities including compositional reasoning, multi-hop reasoning, quantitative comparison, set operations, and etc. Strong supervisions of SPARQL and program are provided for each question. ### Supported Tasks and Leaderboards

It supports knowlege graph based question answering. Specifically, it provides SPARQL and *program* for each question. ### Languages

English ## Dataset Structure

URL


URL ### Data Configs

This dataset has two configs: 'train_val' and 'test' because they have different available fields. Please specify this like 'load_dataset('drt/kqa_pro', 'train_val')'. ### Data Splits

train, val, test ## Additional Information ### Knowledge Graph File

You can find the knowledge graph file 'URL' in the original github repository. It comes with the format: ### How to run SPARQLs and programs

We implement multiple baselines in our codebase, which includes a supervised SPARQL parser and program parser.

In the SPARQL parser, we implement a query engine based on Virtuoso.
You can install the engine based on our instructions, and then feed your predicted SPARQL to get the answer.

In the program parser, we implement a rule-based program executor, which receives a predicted program and returns the answer.
Detailed introductions of our functions can be found in our paper. ### How to submit results of test set
You need to predict answers for all questions of test set and write them in a text file in order, one per line.
Here is an example:


Then you need to send the prediction file to us by email <caosl19@URL>, we will reply to you with the performance as soon as possible.
To appear in the learderboard, you need to also provide following information:

- model name
- affiliation
- open-ended or multiple-choice
- whether use the supervision of SPARQL in your model or not
- whether use the supervision of program in your model or not
- single model or ensemble model
- (optional) paper link
- (optional) code link ### Licensing Information

MIT License



If you find our dataset is helpful in your work, please cite us by ### Contributions

Thanks to @happen2me for adding this dataset.
TAGS
#task_categories-question-answering #task_ids-open-domain-qa #annotations_creators-machine-generated #annotations_creators-expert-generated #language_creators-found #multilinguality-monolingual #size_categories-10K<n<100K #source_datasets-original #language-English #license-mit #knowledge graph #freebase #arxiv-2007.03875 #region-us 
 # Dataset Card for KQA Pro ## Table of Contents
- Table of Contents
- Dataset Description
  - Dataset Summary
  - Supported Tasks and Leaderboards
  - Languages
- Dataset Structure
  - Data Configs
  - Data Splits
- Additional Information
  - How to run SPARQLs and programs
  - Knowledge Graph File
  - How to Submit to Leaderboard
  - Licensing Information
  - Citation Information
  - Contributions ## Dataset Description

- Homepage: URL
- Repository: URL
- Paper: KQA Pro: A Dataset with Explicit Compositional Programs for Complex Question Answering over Knowledge Base
- Leaderboard: URL
- Point of Contact: shijx12 at gmail dot com ### Dataset Summary

KQA Pro is a large-scale dataset of complex question answering over knowledge base. The questions are very diverse and challenging, requiring multiple reasoning capabilities including compositional reasoning, multi-hop reasoning, quantitative comparison, set operations, and etc. Strong supervisions of SPARQL and program are provided for each question. ### Supported Tasks and Leaderboards

It supports knowlege graph based question answering. Specifically, it provides SPARQL and *program* for each question. ### Languages

English ## Dataset Structure

URL


URL ### Data Configs

This dataset has two configs: 'train_val' and 'test' because they have different available fields. Please specify this like 'load_dataset('drt/kqa_pro', 'train_val')'. ### Data Splits

train, val, test ## Additional Information ### Knowledge Graph File

You can find the knowledge graph file 'URL' in the original github repository. It comes with the format: ### How to run SPARQLs and programs

We implement multiple baselines in our codebase, which includes a supervised SPARQL parser and program parser.

In the SPARQL parser, we implement a query engine based on Virtuoso.
You can install the engine based on our instructions, and then feed your predicted SPARQL to get the answer.

In the program parser, we implement a rule-based program executor, which receives a predicted program and returns the answer.
Detailed introductions of our functions can be found in our paper. ### How to submit results of test set
You need to predict answers for all questions of test set and write them in a text file in order, one per line.
Here is an example:


Then you need to send the prediction file to us by email <caosl19@URL>, we will reply to you with the performance as soon as possible.
To appear in the learderboard, you need to also provide following information:

- model name
- affiliation
- open-ended or multiple-choice
- whether use the supervision of SPARQL in your model or not
- whether use the supervision of program in your model or not
- single model or ensemble model
- (optional) paper link
- (optional) code link ### Licensing Information

MIT License



If you find our dataset is helpful in your work, please cite us by ### Contributions

Thanks to @happen2me for adding this dataset.