
# Dataset Card for Calc-aqua_rat


## Summary

This dataset is an instance of [AQuA-RAT](https://huggingface.co/datasets/aqua_rat) dataset extended with in-context calls of a sympy calculator.


## Supported Tasks

The dataset is intended for training Chain-of-Thought reasoning models able to use external tools to enhance the factuality of their responses.
This dataset presents in-context scenarios where models can outsource the computations in the reasoning chain to a calculator.


## Construction Process

The dataset was constructed automatically by evaluating all candidate calls to a `sympy` library that were extracted from the originally annotated 
*rationale*s. The selection of candidates is pivoted by the matching of equals ('=') symbols in the chain, where the left-hand side of the equation is evaluated,
and accepted as a correct gadget call, if the result occurs closely on the right-hand side. 
Therefore, the extraction of calculator calls may inhibit false negatives (where the calculator could have been used but was not), but not any known
false positives.

We also perform in-dataset and cross-dataset data-leak detection within the [Calc-X collection](https://huggingface.co/collections/MU-NLPC/calc-x-652fee9a6b838fd820055483). Specifically for AQuA-RAT, we removed a few percent of the train split that were near-duplicates with some of the test or validation examples.

A full description of the extraction process can be found in the [corresponding parse script](https://github.com/prompteus/calc-x/blob/7799a7841940b15593d4667219424ee71c74327e/gadgets/aqua.py#L19),

**If you find an issue in the dataset or in the fresh version of the parsing script, we'd be happy if you report it, or create a PR.**


## Data splits

The dataset with the near-duplicates removed can be loaded in the default config using:

```python
datasets.load_dataset("MU-NLPC/calc-aqua_rat")
```

If you want the unfiltered version, you can use:

```python
datasets.load_dataset("MU-NLPC/calc-aqua_rat", "original-splits")
```


## Attributes

- **id**: an id of the example
- **question**: A natural language definition of the problem to solve, including the options to choose from
- **chain**: A natural language step-by-step solution with automatically inserted calculator calls and outputs of the sympy calculator
- **result**: The correct option (one of A...E)
- **options**: a dictionary with 5 possible options (A, B, C, D and E), among which one is correct
- **question_without_options**: same as **question** but without the options inserted

Attributes **id**, **question**, **chain**, and **result** are present in all datasets in [Calc-X collection](https://huggingface.co/collections/MU-NLPC/calc-x-652fee9a6b838fd820055483).


## Related work

This dataset was created as a part of a larger effort in training models capable of using a calculator during inference, which we call Calcformers.

- [**Calc-X collection**](https://huggingface.co/collections/MU-NLPC/calc-x-652fee9a6b838fd820055483) - datasets for training Calcformers
- [**Calcformers collection**](https://huggingface.co/collections/MU-NLPC/calcformers-65367392badc497807b3caf5) - calculator-using models we trained and published on HF
- [**Calc-X and Calcformers paper**](https://arxiv.org/abs/2305.15017)
- [**Calc-X and Calcformers repo**](https://github.com/prompteus/calc-x)

Here are links to the original dataset:

- [**original AQuA-RAT dataset**](https://huggingface.co/datasets/aqua_rat)
- [**original AQuA-RAT paper**](https://arxiv.org/pdf/1705.04146.pdf)
- [**original AQuA-RAT repo**](https://github.com/google-deepmind/AQuA)


## License

Apache-2.0, consistently with the original aqua-rat dataset.


## Cite

If you use this dataset in research, please cite the original [AQuA-RAT paper](https://arxiv.org/pdf/1705.04146.pdf), and [Calc-X paper](https://arxiv.org/abs/2305.15017) as follows:

```bibtex
@inproceedings{kadlcik-etal-2023-soft,
    title = "Calc-X and Calcformers: Empowering Arithmetical Chain-of-Thought through Interaction with Symbolic Systems",
    author = "Marek Kadlčík and Michal Štefánik and Ondřej Sotolář and Vlastimil Martinek",
    booktitle = "Proceedings of the The 2023 Conference on Empirical Methods in Natural Language Processing: Main track",
    month = dec,
    year = "2023",
    address = "Singapore, Singapore",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/2305.15017",
}
```
task_categories:question-answering size_categories:10K<n<100K language:en license:apache-2.0 arxiv:2305.15017 arxiv:1705.04146 region:us
{"language": ["en"], "license": "apache-2.0", "size_categories": ["10K<n<100K"], "task_categories": ["question-answering"], "pretty_name": "AQuA-RAT with Calculator", "dataset_info": [{"config_name": "default", "features": [{"name": "id", "dtype": "string"}, {"name": "question", "dtype": "string"}, {"name": "chain", "dtype": "string"}, {"name": "result", "dtype": "string"}, {"name": "options", "struct": [{"name": "A", "dtype": "string"}, {"name": "B", "dtype": "string"}, {"name": "C", "dtype": "string"}, {"name": "D", "dtype": "string"}, {"name": "E", "dtype": "string"}]}, {"name": "question_without_options", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 72917721, "num_examples": 94760}, {"name": "validation", "num_bytes": 212928, "num_examples": 254}, {"name": "test", "num_bytes": 206180, "num_examples": 254}], "download_size": 42057527, "dataset_size": 73336829}, {"config_name": "original-splits", "features": [{"name": "id", "dtype": "string"}, {"name": "question", "dtype": "string"}, {"name": "chain", "dtype": "string"}, {"name": "result", "dtype": "string"}, {"name": "options", "struct": [{"name": "A", "dtype": "string"}, {"name": "B", "dtype": "string"}, {"name": "C", "dtype": "string"}, {"name": "D", "dtype": "string"}, {"name": "E", "dtype": "string"}]}, {"name": "question_without_options", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 74265737, "num_examples": 97467}, {"name": "validation", "num_bytes": 212928, "num_examples": 254}, {"name": "test", "num_bytes": 206180, "num_examples": 254}], "download_size": 42873590, "dataset_size": 74684845}], "configs": [{"config_name": "default", "data_files": [{"split": "train", "path": "data/train-*"}, {"split": "validation", "path": "data/validation-*"}, {"split": "test", "path": "data/test-*"}]}, {"config_name": "original-splits", "data_files": [{"split": "train", "path": "original-splits/train-*"}, {"split": "validation", "path": "original-splits/validation-*"}, {"split": "test", "path": "original-splits/test-*"}]}]}
TAGS
#task_categories-question-answering #size_categories-10K<n<100K #language-English #license-apache-2.0 #arxiv-2305.15017 #arxiv-1705.04146 #region-us 


# Dataset Card for Calc-aqua_rat


## Summary

This dataset is an instance of AQuA-RAT dataset extended with in-context calls of a sympy calculator.


## Supported Tasks

The dataset is intended for training Chain-of-Thought reasoning models able to use external tools to enhance the factuality of their responses.
This dataset presents in-context scenarios where models can outsource the computations in the reasoning chain to a calculator.


## Construction Process

The dataset was constructed automatically by evaluating all candidate calls to a 'sympy' library that were extracted from the originally annotated 
*rationale*s. The selection of candidates is pivoted by the matching of equals ('=') symbols in the chain, where the left-hand side of the equation is evaluated,
and accepted as a correct gadget call, if the result occurs closely on the right-hand side. 
Therefore, the extraction of calculator calls may inhibit false negatives (where the calculator could have been used but was not), but not any known
false positives.

We also perform in-dataset and cross-dataset data-leak detection within the Calc-X collection. Specifically for AQuA-RAT, we removed a few percent of the train split that were near-duplicates with some of the test or validation examples.

A full description of the extraction process can be found in the corresponding parse script,

If you find an issue in the dataset or in the fresh version of the parsing script, we'd be happy if you report it, or create a PR.


## Data splits

The dataset with the near-duplicates removed can be loaded in the default config using:



If you want the unfiltered version, you can use:




## Attributes

- id: an id of the example
- question: A natural language definition of the problem to solve, including the options to choose from
- chain: A natural language step-by-step solution with automatically inserted calculator calls and outputs of the sympy calculator
- result: The correct option (one of A...E)
- options: a dictionary with 5 possible options (A, B, C, D and E), among which one is correct
- question_without_options: same as question but without the options inserted

Attributes id, question, chain, and result are present in all datasets in Calc-X collection.


## Related work

This dataset was created as a part of a larger effort in training models capable of using a calculator during inference, which we call Calcformers.

- Calc-X collection - datasets for training Calcformers
- Calcformers collection - calculator-using models we trained and published on HF
- Calc-X and Calcformers paper
- Calc-X and Calcformers repo

Here are links to the original dataset:

- original AQuA-RAT dataset
- original AQuA-RAT paper
- original AQuA-RAT repo


## License

Apache-2.0, consistently with the original aqua-rat dataset.


## Cite

If you use this dataset in research, please cite the original AQuA-RAT paper, and Calc-X paper as follows:


# Dataset Card for Calc-aqua_rat ## Summary

This dataset is an instance of AQuA-RAT dataset extended with in-context calls of a sympy calculator. ## Supported Tasks

The dataset is intended for training Chain-of-Thought reasoning models able to use external tools to enhance the factuality of their responses.
This dataset presents in-context scenarios where models can outsource the computations in the reasoning chain to a calculator. ## Construction Process

The dataset was constructed automatically by evaluating all candidate calls to a 'sympy' library that were extracted from the originally annotated 
*rationale*s. The selection of candidates is pivoted by the matching of equals ('=') symbols in the chain, where the left-hand side of the equation is evaluated,
and accepted as a correct gadget call, if the result occurs closely on the right-hand side. 
Therefore, the extraction of calculator calls may inhibit false negatives (where the calculator could have been used but was not), but not any known
false positives.

We also perform in-dataset and cross-dataset data-leak detection within the Calc-X collection. Specifically for AQuA-RAT, we removed a few percent of the train split that were near-duplicates with some of the test or validation examples.

A full description of the extraction process can be found in the corresponding parse script,

If you find an issue in the dataset or in the fresh version of the parsing script, we'd be happy if you report it, or create a PR. ## Data splits

The dataset with the near-duplicates removed can be loaded in the default config using:



If you want the unfiltered version, you can use: ## Attributes

- id: an id of the example
- question: A natural language definition of the problem to solve, including the options to choose from
- chain: A natural language step-by-step solution with automatically inserted calculator calls and outputs of the sympy calculator
- result: The correct option (one of A...E)
- options: a dictionary with 5 possible options (A, B, C, D and E), among which one is correct
- question_without_options: same as question but without the options inserted

Attributes id, question, chain, and result are present in all datasets in Calc-X collection. ## Related work

This dataset was created as a part of a larger effort in training models capable of using a calculator during inference, which we call Calcformers.

- Calc-X collection - datasets for training Calcformers
- Calcformers collection - calculator-using models we trained and published on HF
- Calc-X and Calcformers paper
- Calc-X and Calcformers repo

Here are links to the original dataset:

- original AQuA-RAT dataset
- original AQuA-RAT paper
- original AQuA-RAT repo ## License

Apache-2.0, consistently with the original aqua-rat dataset. ## Cite

If you use this dataset in research, please cite the original AQuA-RAT paper, and Calc-X paper as follows:
TAGS
#task_categories-question-answering #size_categories-10K<n<100K #language-English #license-apache-2.0 #arxiv-2305.15017 #arxiv-1705.04146 #region-us 
 # Dataset Card for Calc-aqua_rat ## Summary

This dataset is an instance of AQuA-RAT dataset extended with in-context calls of a sympy calculator. ## Supported Tasks

The dataset is intended for training Chain-of-Thought reasoning models able to use external tools to enhance the factuality of their responses.
This dataset presents in-context scenarios where models can outsource the computations in the reasoning chain to a calculator. ## Construction Process

The dataset was constructed automatically by evaluating all candidate calls to a 'sympy' library that were extracted from the originally annotated 
*rationale*s. The selection of candidates is pivoted by the matching of equals ('=') symbols in the chain, where the left-hand side of the equation is evaluated,
and accepted as a correct gadget call, if the result occurs closely on the right-hand side. 
Therefore, the extraction of calculator calls may inhibit false negatives (where the calculator could have been used but was not), but not any known
false positives.

We also perform in-dataset and cross-dataset data-leak detection within the Calc-X collection. Specifically for AQuA-RAT, we removed a few percent of the train split that were near-duplicates with some of the test or validation examples.

A full description of the extraction process can be found in the corresponding parse script,

If you find an issue in the dataset or in the fresh version of the parsing script, we'd be happy if you report it, or create a PR. ## Data splits

The dataset with the near-duplicates removed can be loaded in the default config using:



If you want the unfiltered version, you can use: ## Attributes

- id: an id of the example
- question: A natural language definition of the problem to solve, including the options to choose from
- chain: A natural language step-by-step solution with automatically inserted calculator calls and outputs of the sympy calculator
- result: The correct option (one of A...E)
- options: a dictionary with 5 possible options (A, B, C, D and E), among which one is correct
- question_without_options: same as question but without the options inserted

Attributes id, question, chain, and result are present in all datasets in Calc-X collection. ## Related work

This dataset was created as a part of a larger effort in training models capable of using a calculator during inference, which we call Calcformers.

- Calc-X collection - datasets for training Calcformers
- Calcformers collection - calculator-using models we trained and published on HF
- Calc-X and Calcformers paper
- Calc-X and Calcformers repo

Here are links to the original dataset:

- original AQuA-RAT dataset
- original AQuA-RAT paper
- original AQuA-RAT repo ## License

Apache-2.0, consistently with the original aqua-rat dataset. ## Cite

If you use this dataset in research, please cite the original AQuA-RAT paper, and Calc-X paper as follows: