
# Statcan Dialogue Dataset

<div align="center">

[**💻Code**](https://github.com/mcGill-NLP/statcan-dialogue-dataset) | [**📄Paper**](https://arxiv.org/abs/2304.01412) | [**🌐Homepage**](https://mcgill-nlp.github.io/statcan-dialogue-dataset) | [**🤗Huggingface**](https://huggingface.co/datasets/McGill-NLP/statcan-dialogue-dataset) | [**🐦Tweets**](https://twitter.com/xhluca/status/1648728708142727180) | [**📺Video**](https://aclanthology.org/2023.eacl-main.206.mp4) |
| :--: | :--: | :--: | :--: | :--: | :--: |

[**The StatCan Dialogue Dataset: Retrieving Data Tables through Conversations with Genuine Intents**](https://arxiv.org/abs/2304.01412)\
[*Xing Han Lu*](https://xinghanlu.com), [*Siva Reddy*](https://sivareddy.in), [*Harm de Vries*](https://www.harmdevries.com/)\
EACL 2023

![Banner Image showing a sample conversation between a user and an agent](https://mcgill-nlp.github.io/statcan-dialogue-dataset/assets/images/banner.svg)

</div>


## Access

To access this dataset, you must read and accept the following terms of use and restrictions, then request access with your academic or professional email. 

We will manually review each request. To ensure your request is not rejected, make sure that:

- Your huggingface account is linked to your professional/research website, which we may review to ensure the dataset will be used for the intended purpose
- Your request is made with an academic (e.g. `.edu`) or professional email (e.g. `@servicenow.com`). To do this, your have to set your primary email to your academic/professional email, or create a new Huggingface account.

If your academic institution does not end with `.edu`, or you are part of a professional group that does not have an email address, please contact us (see email in paper).

### Terms of use

Researchers must agree to the following terms: 

1. These data represent anonymized (de-identified) data from individuals. Best efforts have been implemented to ensure that all directly and indirectly identifiable information has been removed. Researchers who download this dataset must agree to notify Graeme Gilmour (`graeme.gilmour <at> statcan.gc.ca`) and Harm de Vries (`harm.devries <at> servicenow.com`) if any inadvertently remaining identifiable information is discovered during the process of re-using this dataset. Researchers must agree to destroy any version of this dataset containing identifiable information.
2. The terms of this dataset require that reusers give credit to the creators. It allows reusers to distribute, remix, adapt, and build upon the material in any medium or format, even for commercial purposes.
3. Have read and acknowledged the Appendix B (Dataset Card) of the latest version of the paper prior to using the dataset.

### Restrictions

Downloaders cannot:

1. obtain information from the dataset that results in the researcher or any third party(ies) directly or indirectly identifying any participant with the aid of other information acquired elsewhere;
2. produce connections or links among or between the information included in the dataset and other third-party information that could be used to identify any individuals; and
3. extract information from the dataset that could aid researchers (downloaders) in gaining knowledge about or obtaining any means of contacting any individuals already known to the downloader/researcher

## Quickstart

Quickstart code is available in the Readme and on the user guide (see [documentation](https://mcgill-nlp.github.io/statcan-dialogue-dataset/docs)).

## Dataset Card

Please refer to Appendix B of the manuscript.

## Usage on Huggingface `datasets`

It is recommended to use the `statcan-dialogue-dataset` library to access the dataset, which you can install with `pip install statcan-dialogue-dataset` and learn about in the [documentation](https://mcgill-nlp.github.io/statcan-dialogue-dataset/docs). 

However, it is possible to load certain files directly on Huggingface `datasets` (however, for other files, you will need to use the `statcan-dialogue-dataset` library):

```python
from datasets import load_dataset

# Load retrieval task data (without bm25 hard negatives)
ds_ret = load_dataset("McGill-NLP/statcan-dialogue-dataset", data_dir="retrieval")

# Load generation task data (without retrieval augmentations)
ds_gen = load_dataset("McGill-NLP/statcan-dialogue-dataset", data_dir="generation")

# Load french version of datasets
ds_ret_fr = load_dataset("McGill-NLP/statcan-dialogue-dataset", data_dir="retrieval_fr")
ds_gen_fr = load_dataset("McGill-NLP/statcan-dialogue-dataset", data_dir="generation_fr")
```

> **IMPORTANT NOTE**: Do not download the content of this repository into `~/.statcan_dialogue_dataset/` as this will cause conflicts with the `statcan-dialogue-dataset` library. As you have noticed, the file names and path are different - the files and directories here have been modified from the original files located in `task_data.zip`. If you need to cache the files, please use the default Huggingface cache directory.


## Citation

If you use our dataset, please cite as follows:

```bibtex
@inproceedings{lu-etal-2023-statcan,
    title = "The {S}tat{C}an Dialogue Dataset: Retrieving Data Tables through Conversations with Genuine Intents",
    author = "Lu, Xing Han  and
      Reddy, Siva  and
      de Vries, Harm",
    booktitle = "Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics",
    month = may,
    year = "2023",
    address = "Dubrovnik, Croatia",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/2304.01412",
    pages = "2799--2829",
}
```

task_categories:conversational task_categories:table-question-answering size_categories:1K<n<10K language:en language:fr arxiv:2304.01412 region:us
{"language": ["en", "fr"], "size_categories": ["1K<n<10K"], "task_categories": ["conversational", "table-question-answering"], "pretty_name": "Statcan Dialogue Dataset", "extra_gated_prompt": "You agree to not attempt to determine the identity of individuals in this dataset", "extra_gated_fields": {"Full Name": "text", "Affiliation": "text", "Country": "text", "Academic/Work Email Address": "text", "I agree to follow the terms of use": "checkbox", "I have read and will respect the restrictions": "checkbox"}}
TAGS
#task_categories-conversational #task_categories-table-question-answering #size_categories-1K<n<10K #language-English #language-French #arxiv-2304.01412 #region-us 

Statcan Dialogue Dataset
========================




The StatCan Dialogue Dataset: Retrieving Data Tables through Conversations with Genuine Intents  

*Xing Han Lu*, *Siva Reddy*, *Harm de Vries*  

EACL 2023


!Banner Image showing a sample conversation between a user and an agent



Access
------


To access this dataset, you must read and accept the following terms of use and restrictions, then request access with your academic or professional email.


We will manually review each request. To ensure your request is not rejected, make sure that:


* Your huggingface account is linked to your professional/research website, which we may review to ensure the dataset will be used for the intended purpose
* Your request is made with an academic (e.g. '.edu') or professional email (e.g. '@URL'). To do this, your have to set your primary email to your academic/professional email, or create a new Huggingface account.


If your academic institution does not end with '.edu', or you are part of a professional group that does not have an email address, please contact us (see email in paper).


### Terms of use


Researchers must agree to the following terms:


1. These data represent anonymized (de-identified) data from individuals. Best efforts have been implemented to ensure that all directly and indirectly identifiable information has been removed. Researchers who download this dataset must agree to notify Graeme Gilmour ('graeme.gilmour  URL') and Harm de Vries ('harm.devries  URL') if any inadvertently remaining identifiable information is discovered during the process of re-using this dataset. Researchers must agree to destroy any version of this dataset containing identifiable information.
2. The terms of this dataset require that reusers give credit to the creators. It allows reusers to distribute, remix, adapt, and build upon the material in any medium or format, even for commercial purposes.
3. Have read and acknowledged the Appendix B (Dataset Card) of the latest version of the paper prior to using the dataset.


### Restrictions


Downloaders cannot:


1. obtain information from the dataset that results in the researcher or any third party(ies) directly or indirectly identifying any participant with the aid of other information acquired elsewhere;
2. produce connections or links among or between the information included in the dataset and other third-party information that could be used to identify any individuals; and
3. extract information from the dataset that could aid researchers (downloaders) in gaining knowledge about or obtaining any means of contacting any individuals already known to the downloader/researcher


Quickstart
----------


Quickstart code is available in the Readme and on the user guide (see documentation).


Dataset Card
------------


Please refer to Appendix B of the manuscript.


Usage on Huggingface 'datasets'
-------------------------------


It is recommended to use the 'statcan-dialogue-dataset' library to access the dataset, which you can install with 'pip install statcan-dialogue-dataset' and learn about in the documentation.


However, it is possible to load certain files directly on Huggingface 'datasets' (however, for other files, you will need to use the 'statcan-dialogue-dataset' library):



> 
> IMPORTANT NOTE: Do not download the content of this repository into '~/.statcan\_dialogue\_dataset/' as this will cause conflicts with the 'statcan-dialogue-dataset' library. As you have noticed, the file names and path are different - the files and directories here have been modified from the original files located in 'task\_data.zip'. If you need to cache the files, please use the default Huggingface cache directory.
> 
> 
> 


If you use our dataset, please cite as follows:



### Terms of use


Researchers must agree to the following terms:


1. These data represent anonymized (de-identified) data from individuals. Best efforts have been implemented to ensure that all directly and indirectly identifiable information has been removed. Researchers who download this dataset must agree to notify Graeme Gilmour ('graeme.gilmour  URL') and Harm de Vries ('harm.devries  URL') if any inadvertently remaining identifiable information is discovered during the process of re-using this dataset. Researchers must agree to destroy any version of this dataset containing identifiable information.
2. The terms of this dataset require that reusers give credit to the creators. It allows reusers to distribute, remix, adapt, and build upon the material in any medium or format, even for commercial purposes.
3. Have read and acknowledged the Appendix B (Dataset Card) of the latest version of the paper prior to using the dataset. ### Restrictions


Downloaders cannot:


1. obtain information from the dataset that results in the researcher or any third party(ies) directly or indirectly identifying any participant with the aid of other information acquired elsewhere;
2. produce connections or links among or between the information included in the dataset and other third-party information that could be used to identify any individuals; and
3. extract information from the dataset that could aid researchers (downloaders) in gaining knowledge about or obtaining any means of contacting any individuals already known to the downloader/researcher


Quickstart
----------


Quickstart code is available in the Readme and on the user guide (see documentation).


Dataset Card
------------


Please refer to Appendix B of the manuscript.


Usage on Huggingface 'datasets'
-------------------------------


It is recommended to use the 'statcan-dialogue-dataset' library to access the dataset, which you can install with 'pip install statcan-dialogue-dataset' and learn about in the documentation.


However, it is possible to load certain files directly on Huggingface 'datasets' (however, for other files, you will need to use the 'statcan-dialogue-dataset' library):



> 
> IMPORTANT NOTE: Do not download the content of this repository into '~/.statcan\_dialogue\_dataset/' as this will cause conflicts with the 'statcan-dialogue-dataset' library. As you have noticed, the file names and path are different - the files and directories here have been modified from the original files located in 'task\_data.zip'. If you need to cache the files, please use the default Huggingface cache directory.
> 
> 
> 


If you use our dataset, please cite as follows:
TAGS
#task_categories-conversational #task_categories-table-question-answering #size_categories-1K<n<10K #language-English #language-French #arxiv-2304.01412 #region-us 
 ### Terms of use


Researchers must agree to the following terms:


1. These data represent anonymized (de-identified) data from individuals. Best efforts have been implemented to ensure that all directly and indirectly identifiable information has been removed. Researchers who download this dataset must agree to notify Graeme Gilmour ('graeme.gilmour  URL') and Harm de Vries ('harm.devries  URL') if any inadvertently remaining identifiable information is discovered during the process of re-using this dataset. Researchers must agree to destroy any version of this dataset containing identifiable information.
2. The terms of this dataset require that reusers give credit to the creators. It allows reusers to distribute, remix, adapt, and build upon the material in any medium or format, even for commercial purposes.
3. Have read and acknowledged the Appendix B (Dataset Card) of the latest version of the paper prior to using the dataset. ### Restrictions


Downloaders cannot:


1. obtain information from the dataset that results in the researcher or any third party(ies) directly or indirectly identifying any participant with the aid of other information acquired elsewhere;
2. produce connections or links among or between the information included in the dataset and other third-party information that could be used to identify any individuals; and
3. extract information from the dataset that could aid researchers (downloaders) in gaining knowledge about or obtaining any means of contacting any individuals already known to the downloader/researcher


Quickstart
----------


Quickstart code is available in the Readme and on the user guide (see documentation).


Dataset Card
------------


Please refer to Appendix B of the manuscript.


Usage on Huggingface 'datasets'
-------------------------------


It is recommended to use the 'statcan-dialogue-dataset' library to access the dataset, which you can install with 'pip install statcan-dialogue-dataset' and learn about in the documentation.


However, it is possible to load certain files directly on Huggingface 'datasets' (however, for other files, you will need to use the 'statcan-dialogue-dataset' library):



> 
> IMPORTANT NOTE: Do not download the content of this repository into '~/.statcan\_dialogue\_dataset/' as this will cause conflicts with the 'statcan-dialogue-dataset' library. As you have noticed, the file names and path are different - the files and directories here have been modified from the original files located in 'task\_data.zip'. If you need to cache the files, please use the default Huggingface cache directory.
> 
> 
> 


If you use our dataset, please cite as follows:
passage: TAGS
#task_categories-conversational #task_categories-table-question-answering #size_categories-1K<n<10K #language-English #language-French #arxiv-2304.01412 #region-us 
### Terms of use


Researchers must agree to the following terms:


1. These data represent anonymized (de-identified) data from individuals. Best efforts have been implemented to ensure that all directly and indirectly identifiable information has been removed. Researchers who download this dataset must agree to notify Graeme Gilmour ('graeme.gilmour  URL') and Harm de Vries ('harm.devries  URL') if any inadvertently remaining identifiable information is discovered during the process of re-using this dataset. Researchers must agree to destroy any version of this dataset containing identifiable information.
2. The terms of this dataset require that reusers give credit to the creators. It allows reusers to distribute, remix, adapt, and build upon the material in any medium or format, even for commercial purposes.
3. Have read and acknowledged the Appendix B (Dataset Card) of the latest version of the paper prior to using the dataset.