## Overview
Original dataset available [here](https://wellecks.github.io/dialogue_nli/).


## Dataset curation
Original `label` column is renamed `original_label`. The original classes are renamed as follows

```
{"positive": "entailment", "negative": "contradiction", "neutral": "neutral"})
```

and encoded with the following mapping

```
{"entailment": 0, "neutral": 1, "contradiction": 2}
```

and stored in the newly created column `label`.


The following splits and the corresponding columns are present in the original files

```
train {'dtype', 'id', 'sentence1', 'sentence2', 'original_label', 'label', 'triple2', 'triple1'}
dev {'dtype', 'id', 'sentence1', 'sentence2', 'original_label', 'label', 'triple2', 'triple1'}
test {'dtype', 'id', 'sentence1', 'sentence2', 'original_label', 'label', 'triple2', 'triple1'}
verified_test {'dtype', 'annotation3', 'sentence1', 'sentence2', 'annotation1', 'annotation2', 'original_label', 'label', 'triple2', 'triple1'}
extra_test {'dtype', 'id', 'sentence1', 'sentence2', 'original_label', 'label', 'triple2', 'triple1'}
extra_dev {'dtype', 'id', 'sentence1', 'sentence2', 'original_label', 'label', 'triple2', 'triple1'}
extra_train {'dtype', 'id', 'sentence1', 'sentence2', 'original_label', 'label', 'triple2', 'triple1'}
valid_havenot {'dtype', 'id', 'sentence1', 'sentence2', 'original_label', 'label', 'triple2', 'triple1'}
valid_attributes {'dtype', 'id', 'sentence1', 'sentence2', 'original_label', 'label', 'triple2', 'triple1'}
valid_likedislike {'dtype', 'id', 'sentence1', 'sentence2', 'original_label', 'label', 'triple2', 'triple1'}
```

Note that I only keep the common columns, which means that I drop "annotation{1, 2, 3}" from `verified_test`.
Note that there are some splits with the same instances, as found by matching on "original_label", "sentence1", "sentence2".


## Code to create dataset
```python
import pandas as pd
from pathlib import Path
import json
from datasets import Features, Value, ClassLabel, Dataset, DatasetDict, Sequence


# load data
ds = {}
for path in Path(".").rglob("<path to folder>/*.jsonl"):
    print(path, flush=True)
    with path.open("r") as fl:
        data = fl.read()
    try:
        d = json.loads(data, encoding="utf-8")
    except json.JSONDecodeError as error:
        print(error)
    
    df = pd.DataFrame(d)

    # encode labels
    df["original_label"] = df["label"]
    df["label"] = df["label"].map({"positive": "entailment", "negative": "contradiction", "neutral": "neutral"})
    df["label"] = df["label"].map({"entailment": 0, "neutral": 1, "contradiction": 2})

    ds[path.name.split(".")[0]] = df
    
    
# prettify names of data splits
datasets = {
    k.replace("dialogue_nli_", "").replace("uu_", "").lower(): v
    for k, v in ds.items()
}
datasets.keys()
#> dict_keys(['train', 'dev', 'test', 'verified_test', 'extra_test', 'extra_dev', 'extra_train', 'valid_havenot', 'valid_attributes', 'valid_likedislike'])


# cast to datasets using only common columns
features = Features({
    "label": ClassLabel(num_classes=3, names=["entailment", "neutral", "contradiction"]),
    "sentence1": Value(dtype="string", id=None),
    "sentence2": Value(dtype="string", id=None),
    "triple1": Sequence(feature=Value(dtype="string", id=None), length=3),
    "triple2": Sequence(feature=Value(dtype="string", id=None), length=3),
    "dtype": Value(dtype="string", id=None),
    "id": Value(dtype="string", id=None),
    "original_label": Value(dtype="string", id=None),
})

ds = {}
for name, df in datasets.items():
    if "id" not in df.columns:
        df["id"] = ""
    ds[name] = Dataset.from_pandas(df.loc[:, list(features.keys())], features=features)
ds = DatasetDict(ds)
ds.push_to_hub("dialogue_nli", token="<token>")


# check overlap between splits
from itertools import combinations
for i, j in combinations(ds.keys(), 2):
    print(
        f"{i} - {j}: ",
        pd.merge(
            ds[i].to_pandas(), 
            ds[j].to_pandas(), 
            on=["original_label", "sentence1", "sentence2"], 
            how="inner",
        ).shape[0],
    )
#> train - dev:  58
#> train - test:  98
#> train - verified_test:  90
#> train - extra_test:  0
#> train - extra_dev:  0
#> train - extra_train:  0
#> train - valid_havenot:  0
#> train - valid_attributes:  0
#> train - valid_likedislike:  0
#> dev - test:  19
#> dev - verified_test:  19
#> dev - extra_test:  0
#> dev - extra_dev:  75
#> dev - extra_train:  75
#> dev - valid_havenot:  75
#> dev - valid_attributes:  75
#> dev - valid_likedislike:  75
#> test - verified_test:  12524
#> test - extra_test:  34
#> test - extra_dev:  0
#> test - extra_train:  0
#> test - valid_havenot:  0
#> test - valid_attributes:  0
#> test - valid_likedislike:  0
#> verified_test - extra_test:  29
#> verified_test - extra_dev:  0
#> verified_test - extra_train:  0
#> verified_test - valid_havenot:  0
#> verified_test - valid_attributes:  0
#> verified_test - valid_likedislike:  0
#> extra_test - extra_dev:  0
#> extra_test - extra_train:  0
#> extra_test - valid_havenot:  0
#> extra_test - valid_attributes:  0
#> extra_test - valid_likedislike:  0
#> extra_dev - extra_train:  250946
#> extra_dev - valid_havenot:  250946
#> extra_dev - valid_attributes:  250946
#> extra_dev - valid_likedislike:  250946
#> extra_train - valid_havenot:  250946
#> extra_train - valid_attributes:  250946
#> extra_train - valid_likedislike:  250946
#> valid_havenot - valid_attributes:  250946
#> valid_havenot - valid_likedislike:  250946
#> valid_attributes - valid_likedislike:  250946
```
region:us
{}
TAGS
#region-us 

## Overview
Original dataset available here.


## Dataset curation
Original 'label' column is renamed 'original_label'. The original classes are renamed as follows



and encoded with the following mapping



and stored in the newly created column 'label'.


The following splits and the corresponding columns are present in the original files



Note that I only keep the common columns, which means that I drop "annotation{1, 2, 3}" from 'verified_test'.
Note that there are some splits with the same instances, as found by matching on "original_label", "sentence1", "sentence2".


## Code to create dataset

## Overview
Original dataset available here. ## Dataset curation
Original 'label' column is renamed 'original_label'. The original classes are renamed as follows



and encoded with the following mapping



and stored in the newly created column 'label'.


The following splits and the corresponding columns are present in the original files



Note that I only keep the common columns, which means that I drop "annotation{1, 2, 3}" from 'verified_test'.
Note that there are some splits with the same instances, as found by matching on "original_label", "sentence1", "sentence2". ## Code to create dataset
TAGS
#region-us 
 ## Overview
Original dataset available here. ## Dataset curation
Original 'label' column is renamed 'original_label'. The original classes are renamed as follows



and encoded with the following mapping



and stored in the newly created column 'label'.


The following splits and the corresponding columns are present in the original files



Note that I only keep the common columns, which means that I drop "annotation{1, 2, 3}" from 'verified_test'.
Note that there are some splits with the same instances, as found by matching on "original_label", "sentence1", "sentence2". ## Code to create dataset