

<img src="https://huggingface.co/datasets/Salesforce/dialogstudio/resolve/main/logo.png"
alt="drawing" width="510"/>

# DialogStudio: Unified Dialog Datasets and Instruction-Aware Models for Conversational AI
**Author**: [Jianguo Zhang](https://github.com/jianguoz), [Kun Qian](https://github.com/qbetterk)

[Paper](https://arxiv.org/pdf/2307.10172.pdf)|[Github](https://github.com/salesforce/DialogStudio)|[GDrive]


<img src="https://huggingface.co/datasets/Salesforce/dialogstudio/resolve/main/DialogStudio_Stats.jpg"
alt="drawing" width="800"/>



**Follow the [DialogStudio](https://github.com/salesforce/DialogStudio) GitHub repository for latest information.**

### Datasets



### Load dataset
The datasets are split into several categories in HuggingFace 
```
Datasets/
├── Knowledge-Grounded-Dialogues
├── Natural-Language-Understanding
├── Open-Domain-Dialogues
├── Task-Oriented-Dialogues
├── Dialogue-Summarization
├── Conversational-Recommendation-Dialogs
```


You can load any dataset in the DialogStudio from the [HuggingFace hub](https://huggingface.co/datasets/Salesforce/dialogstudio) by claiming the `{dataset_name}`, which is exactly the dataset folder name. All available datasets are described in [dataset content](https://github.com/salesforce/DialogStudio/blob/main/Dataset_Stats.csv). For easier reference, [available dataset names](#Available Datasets) are also listed below.

Below is one example to load the [MULTIWOZ2_2](https://huggingface.co/datasets/Salesforce/dialogstudio/blob/main/task_oriented/MULTIWOZ2_2.zip) dataset under the [task-oriented-dialogues](https://huggingface.co/datasets/Salesforce/dialogstudio/tree/main/task_oriented) category:


Load the dataset
```python
from datasets import load_dataset

dataset = load_dataset('Salesforce/dialogstudio', 'MULTIWOZ2_2')
```
Here is the output structure of MultiWOZ 2.2
```python
DatasetDict({
    train: Dataset({
        features: ['original dialog id', 'new dialog id', 'dialog index', 'original dialog info', 'log', 'prompt', 'external knowledge non-flat', 'external knowledge', 'dst knowledge', 'intent knowledge'],
        num_rows: 8437
    })
    validation: Dataset({
        features: ['original dialog id', 'new dialog id', 'dialog index', 'original dialog info', 'log', 'prompt', 'external knowledge non-flat', 'external knowledge', 'dst knowledge', 'intent knowledge'],
        num_rows: 1000
    })
    test: Dataset({
        features: ['original dialog id', 'new dialog id', 'dialog index', 'original dialog info', 'log', 'prompt', 'external knowledge non-flat', 'external knowledge', 'dst knowledge', 'intent knowledge'],
        num_rows: 1000
    })
})
```


### Available Datasets

The ``data_name`` for  ``load_dataset("Salesforce/dialogstudio", data_name)`` can be found below. More detailed information for each dataset can be found in out [github](https://github.com/salesforce/DialogStudio/blob/main/Dataset_Stats.csv).
```python
    "natural_language_understanding": [
                        "ATIS", "ATIS-NER", "BANKING77", "BANKING77-OOS", "CLINC-Single-Domain-OOS-banking", 
                        "CLINC-Single-Domain-OOS-credit_cards", "CLINC150", "DSTC8-SGD", "HWU64", "MIT-Movie",
                        "MIT-Restaurant", "RESTAURANTS8K", "SNIPS", "SNIPS-NER", "TOP", "TOP-NER"
                    ],  
    "task_oriented": [
                        "ABCD", "AirDialogue", "BiTOD", "CaSiNo", "CraigslistBargains", 
                        "Disambiguation", "DSTC2-Clean", "FRAMES", "GECOR", "HDSA-Dialog", 
                        "KETOD", "KVRET", "MetaLWOZ", "MS-DC", "MuDoCo", 
                        "MulDoGO", "MultiWOZ_2.1", "MULTIWOZ2_2", "SGD", "SimJointGEN", 
                        "SimJointMovie", "SimJointRestaurant", "STAR", "Taskmaster1", "Taskmaster2", 
                        "Taskmaster3", "WOZ2_0"
                    ],
    "dialogue_summarization": [
                        "AMI", "CRD3", "DialogSum", "ECTSum", "ICSI", 
                        "MediaSum", "QMSum", "SAMSum", "TweetSumm", "ConvoSumm", 
                        "SummScreen_ForeverDreaming", "SummScreen_TVMegaSite"
                    ],
    "conversational_recommendation": [
                        "Redial", "DuRecDial-2.0", "OpenDialKG", "SalesBot",
                    ],
    "open_domain": [
                        "chitchat-dataset", "ConvAI2", "AntiScam", "Empathetic", "HH-RLHF", 
                        "PLACES3.5", "Prosocial", "SODA", "ShareGPT"
                    ],
    "knowledge_grounded": [
                        "CompWebQ", "CoQA", "CoSQL", "DART", "FeTaQA", 
                        "GrailQA", "HybridQA", "MTOP", "MultiModalQA", "SParC", 
                        "Spider", "SQA", "ToTTo", "WebQSP", "WikiSQL", 
                        "WikiTQ", "wizard_of_internet", "wizard_of_wikipedia"
                    ],  
```

# License

Our project follows the following structure with respect to licensing:

1. For all the modified datasets in DialogStudio: 
   - A portion of these datasets is under the [Apache License 2.0](https://github.com/salesforce/DialogStudio/blob/main/LICENSE.txt).
   - Some retain their original licenses even after modification.
   - For a few datasets that lacked a license, we have cited the relevant papers.
2. Original dataset licenses: For reference, we also put the original avaliable licenses for each dataset into their respective dataset folders.
3. Code: Our codebase is under the [Apache License 2.0](https://github.com/salesforce/DialogStudio/blob/main/LICENSE.txt).

For detailed licensing information, please refer to the specific licenses accompanying the datasets. If you utilize datasets from DialogStudio, we kindly request that you cite our work.


# Citation
The data and code in this repository is mostly developed for or derived from the paper below. If you utilize datasets from DialogStudio, we kindly request that you cite both the original work and our own (Accepted by EACL 2024 Findings as a long paper).

```
@article{zhang2023dialogstudio,
  title={DialogStudio: Towards Richest and Most Diverse Unified Dataset Collection for Conversational AI},
  author={Zhang, Jianguo and Qian, Kun and Liu, Zhiwei and Heinecke, Shelby and Meng, Rui and Liu, Ye and Yu, Zhou and Savarese, Silvio and Xiong, Caiming},
  journal={arXiv preprint arXiv:2307.10172},
  year={2023}
}
```
task_categories:conversational task_categories:question-answering task_categories:summarization task_categories:text-generation language:en license:apache-2.0 arxiv:2307.10172 region:us
{"language": ["en"], "license": "apache-2.0", "task_categories": ["conversational", "question-answering", "summarization", "text-generation"], "pretty_name": "Dialog Studio", "extra_gated_heading": "Acknowledge to follow corresponding dataset licenses to access the repository", "extra_gated_button_content": "Agree and access repository"}
TAGS
#task_categories-conversational #task_categories-question-answering #task_categories-summarization #task_categories-text-generation #language-English #license-apache-2.0 #arxiv-2307.10172 #region-us 



<img src="URL
alt="drawing" width="510"/>

# DialogStudio: Unified Dialog Datasets and Instruction-Aware Models for Conversational AI
Author: Jianguo Zhang, Kun Qian

Paper|Github|[GDrive]


<img src="URL
alt="drawing" width="800"/>



Follow the DialogStudio GitHub repository for latest information.

### Datasets



### Load dataset
The datasets are split into several categories in HuggingFace 



You can load any dataset in the DialogStudio from the HuggingFace hub by claiming the '{dataset_name}', which is exactly the dataset folder name. All available datasets are described in dataset content. For easier reference, available dataset names are also listed below.

Below is one example to load the MULTIWOZ2_2 dataset under the task-oriented-dialogues category:


Load the dataset

Here is the output structure of MultiWOZ 2.2



### Available Datasets

The ''data_name'' for  ''load_dataset("Salesforce/dialogstudio", data_name)'' can be found below. More detailed information for each dataset can be found in out github.


# License

Our project follows the following structure with respect to licensing:

1. For all the modified datasets in DialogStudio: 
   - A portion of these datasets is under the Apache License 2.0.
   - Some retain their original licenses even after modification.
   - For a few datasets that lacked a license, we have cited the relevant papers.
2. Original dataset licenses: For reference, we also put the original avaliable licenses for each dataset into their respective dataset folders.
3. Code: Our codebase is under the Apache License 2.0.

For detailed licensing information, please refer to the specific licenses accompanying the datasets. If you utilize datasets from DialogStudio, we kindly request that you cite our work.


The data and code in this repository is mostly developed for or derived from the paper below. If you utilize datasets from DialogStudio, we kindly request that you cite both the original work and our own (Accepted by EACL 2024 Findings as a long paper).


# DialogStudio: Unified Dialog Datasets and Instruction-Aware Models for Conversational AI
Author: Jianguo Zhang, Kun Qian

Paper|Github|[GDrive]


<img src="URL
alt="drawing" width="800"/>



Follow the DialogStudio GitHub repository for latest information. ### Datasets ### Load dataset
The datasets are split into several categories in HuggingFace 



You can load any dataset in the DialogStudio from the HuggingFace hub by claiming the '{dataset_name}', which is exactly the dataset folder name. All available datasets are described in dataset content. For easier reference, available dataset names are also listed below.

Below is one example to load the MULTIWOZ2_2 dataset under the task-oriented-dialogues category:


Load the dataset

Here is the output structure of MultiWOZ 2.2 ### Available Datasets

The ''data_name'' for  ''load_dataset("Salesforce/dialogstudio", data_name)'' can be found below. More detailed information for each dataset can be found in out github. # License

Our project follows the following structure with respect to licensing:

1. For all the modified datasets in DialogStudio: 
   - A portion of these datasets is under the Apache License 2.0.
   - Some retain their original licenses even after modification.
   - For a few datasets that lacked a license, we have cited the relevant papers.
2. Original dataset licenses: For reference, we also put the original avaliable licenses for each dataset into their respective dataset folders.
3. Code: Our codebase is under the Apache License 2.0.

For detailed licensing information, please refer to the specific licenses accompanying the datasets. If you utilize datasets from DialogStudio, we kindly request that you cite our work.


The data and code in this repository is mostly developed for or derived from the paper below. If you utilize datasets from DialogStudio, we kindly request that you cite both the original work and our own (Accepted by EACL 2024 Findings as a long paper).
TAGS
#task_categories-conversational #task_categories-question-answering #task_categories-summarization #task_categories-text-generation #language-English #license-apache-2.0 #arxiv-2307.10172 #region-us 
 # DialogStudio: Unified Dialog Datasets and Instruction-Aware Models for Conversational AI
Author: Jianguo Zhang, Kun Qian

Paper|Github|[GDrive]


<img src="URL
alt="drawing" width="800"/>



Follow the DialogStudio GitHub repository for latest information. ### Datasets ### Load dataset
The datasets are split into several categories in HuggingFace 



You can load any dataset in the DialogStudio from the HuggingFace hub by claiming the '{dataset_name}', which is exactly the dataset folder name. All available datasets are described in dataset content. For easier reference, available dataset names are also listed below.

Below is one example to load the MULTIWOZ2_2 dataset under the task-oriented-dialogues category:


Load the dataset

Here is the output structure of MultiWOZ 2.2 ### Available Datasets

The ''data_name'' for  ''load_dataset("Salesforce/dialogstudio", data_name)'' can be found below. More detailed information for each dataset can be found in out github. # License

Our project follows the following structure with respect to licensing:

1. For all the modified datasets in DialogStudio: 
   - A portion of these datasets is under the Apache License 2.0.
   - Some retain their original licenses even after modification.
   - For a few datasets that lacked a license, we have cited the relevant papers.
2. Original dataset licenses: For reference, we also put the original avaliable licenses for each dataset into their respective dataset folders.
3. Code: Our codebase is under the Apache License 2.0.

For detailed licensing information, please refer to the specific licenses accompanying the datasets. If you utilize datasets from DialogStudio, we kindly request that you cite our work.


The data and code in this repository is mostly developed for or derived from the paper below. If you utilize datasets from DialogStudio, we kindly request that you cite both the original work and our own (Accepted by EACL 2024 Findings as a long paper).
passage: TAGS
#task_categories-conversational #task_categories-question-answering #task_categories-summarization #task_categories-text-generation #language-English #license-apache-2.0 #arxiv-2307.10172 #region-us 
# DialogStudio: Unified Dialog Datasets and Instruction-Aware Models for Conversational AI
Author: Jianguo Zhang, Kun Qian

Paper|Github|[GDrive]


<img src="URL
alt="drawing" width="800"/>



Follow the DialogStudio GitHub repository for latest information.### Datasets### Load dataset
The datasets are split into several categories in HuggingFace 



You can load any dataset in the DialogStudio from the HuggingFace hub by claiming the '{dataset_name}', which is exactly the dataset folder name. All available datasets are described in dataset content. For easier reference, available dataset names are also listed below.

Below is one example to load the MULTIWOZ2_2 dataset under the task-oriented-dialogues category:


Load the dataset

Here is the output structure of MultiWOZ 2.2### Available Datasets

The ''data_name'' for  ''load_dataset("Salesforce/dialogstudio", data_name)'' can be found below. More detailed information for each dataset can be found in out github.