# Universal Text Classification Dataset (UTCD)

## Load dataset 
```python
from datasets import load_dataset
dataset = load_dataset('claritylab/utcd', name='in-domain')
```

## Description

UTCD is a curated compilation of 18 datasets revised for Zero-shot Text Classification spanning 3 aspect categories of Sentiment, Intent/Dialogue, and Topic classification. UTCD focuses on the task of zero-shot text classification where the candidate labels are descriptive of the text being classified. TUTCD consists of ~ 6M/800K train/test examples.

UTCD was introduced in the Findings of ACL'23 Paper **Label Agnostic Pre-training for Zero-shot Text Classification** by ***Christopher Clarke, Yuzhao Heng, Yiping Kang, Krisztian Flautner, Lingjia Tang and Jason Mars***. [Project Homepage](https://github.com/ChrisIsKing/zero-shot-text-classification/tree/master). 

UTCD Datasets & Principles:

In order to make NLP models more broadly useful, zero-shot techniques need to be capable of label, domain \& aspect transfer. As such, in the construction of UTCD we enforce the following principles: 

- **Textual labels**: In UTCD, we mandate the use of textual labels. While numerical label values are often used in classification tasks, descriptive textual labels such as those present in the datasets across UTCD enable the development of techniques that can leverage the class name which is instrumental in providing zero-shot support. As such, for each of the compiled datasets, labels are standardized such that the labels are descriptive of the text in natural language. 
- **Diverse domains and Sequence lengths**: In addition to broad coverage of aspects, UTCD compiles diverse data across several domains such as Banking, Finance, Legal, etc each comprising varied length sequences (long and short). The datasets are listed above.


- Sentiment
    - GoEmotions introduced in [GoEmotions: A Dataset of Fine-Grained Emotions](https://arxiv.org/pdf/2005.00547v2.pdf)
    - TweetEval introduced in [TWEETEVAL: Unified Benchmark and Comparative Evaluation for Tweet Classification](https://arxiv.org/pdf/2010.12421v2.pdf) (Sentiment subset)
    - Emotion introduced in [CARER: Contextualized Affect Representations for Emotion Recognition](https://aclanthology.org/D18-1404.pdf)
    - Amazon Polarity introduced in [Character-level Convolutional Networks for Text Classification](https://arxiv.org/pdf/1509.01626.pdf)
    - Finance Phrasebank introduced in [Good debt or bad debt: Detecting semantic orientations in economic texts](https://arxiv.org/pdf/1307.5336.pdf)
    - Yelp introduced in [Character-level Convolutional Networks for Text Classification](https://arxiv.org/pdf/1509.01626.pdf)
- Intent/Dialogue
    - Schema-Guided Dialogue introduced in [Towards Scalable Multi-Domain Conversational Agents: The Schema-Guided Dialogue Dataset](https://arxiv.org/pdf/1909.05855v2.pdf)
    - Clinc-150 introduced in [An Evaluation Dataset for Intent Classification and Out-of-Scope Prediction](https://arxiv.org/pdf/1909.02027v1.pdf)
    - SLURP SLU introduced in [SLURP: A Spoken Language Understanding Resource Package](https://arxiv.org/pdf/2011.13205.pdf)
    - Banking77 introduced in [Efficient Intent Detection with Dual Sentence Encoders](https://arxiv.org/abs/2003.04807](https://arxiv.org/pdf/2003.04807.pdf)
    - Snips introduced in [Snips Voice Platform: an embedded Spoken Language Understanding system for private-by-design voice interfaces](https://arxiv.org/pdf/1805.10190.pdf)
    - NLU Evaluation introduced in [Benchmarking Natural Language Understanding Services for building Conversational Agents](https://arxiv.org/pdf/1903.05566.pdf)
- Topic
    - AG News introduced in [Character-level Convolutional Networks for Text Classification](https://arxiv.org/pdf/1509.01626.pdf)
    - DBpedia 14 introduced in [DBpedia: A Nucleus for a Web of Open Data](https://link.springer.com/chapter/10.1007/978-3-540-76298-0_52)
    - Yahoo Answer Topics introduced in [Character-level Convolutional Networks for Text Classification](https://arxiv.org/pdf/1509.01626.pdf)
    - MultiEurlex introduced in [MultiEURLEX -- A multi-lingual and multi-label legal document classification dataset for zero-shot cross-lingual transfer](https://aclanthology.org/2021.emnlp-main.559v2.pdf)
    - BigPatent introduced in [BIGPATENT: A Large-Scale Dataset for Abstractive and Coherent Summarization](https://aclanthology.org/P19-1212.pdf)
    - Consumer Finance introduced in [Consumer Complaint Database](https://www.consumerfinance.gov/data-research/consumer-complaints/)




## Structure

### Data Samples

Each dataset sample contains the text, the label encoded as an integer, and the dataset name encoded as an integer. 

```python
{
	'text': "My favourite food is anything I didn't have to cook myself.",
    'labels': [215],
    'dataset_name': 0
}
```

### Datasets Contained

The UTCD dataset contains 18 datasets, 9 `in-domain`, 9 `out-of-domain`, spanning 3 aspects: `sentiment`, `intent` and `topic`. 

Below are statistics on the datasets. 



**In-Domain Datasets**

| Dataset    | Aspect    | #Samples in Train/Test | #labels | average #token in text in Train/Test |
| ---------- | --------- | ---------------------- | ------- | ------------------------------------ |
| GoEmotions | sentiment | 43K/5.4K               | 28      | 12/12                                |
| TweetEval  | sentiment | 45K/12K                | 3       | 19/14                                |
| Emotion    | sentiment | 16K/2K                 | 6       | 17/17                                |
| SGD        | intent    | 16K/4.2K               | 26      | 8/9                                  |
| Clinc-150  | intent    | 15K/4.5K               | 150     | 8/8                                  |
| SLURP      | intent    | 12K/2.6K               | 75      | 7/7                                  |
| AG News    | topic     | 120K7.6K               | 4       | 38/37                                |
| DBpedia    | topic     | 560K/70K               | 14      | 45/45                                |
| Yahoo      | topic     | 1.4M/60K               | 10      | 10/10                                |

**Out-of-Domain Datasets**

| Dataset               | Aspect    | #Samples in Train/Test | #labels | average #token in text |
| --------------------- | --------- | ---------------------- | ------- | ---------------------- |
| Amazon Polarity       | sentiment | 3.6M/400K              | 2       | 71/71                  |
| Financial Phrase Bank | sentiment | 1.8K/453               | 3       | 19/19                  |
| Yelp                  | sentiment | 650K/50K               | 3       | 128/128                |
| Banking77             | intent    | 10K/3.1K               | 77      | 11/10                  |
| SNIPS                 | intent    | 14K/697                | 7       | 8/8                    |
| NLU Eval              | intent    | 21K/5.2K               | 68      | 7/7                    |
| MultiEURLEX           | topic     | 55K/5K                 | 21      | 1198/1853              |
| Big Patent            | topic     | 25K/5K                 | 9       | 2872/2892              |
| Consumer Finance      | topic     | 630K/160K              | 18      | 190/189                |



### Configurations

The `in-domain` and `out-of-domain` configurations has 2 splits: `train` and `test`. 

The aspect-normalized configurations (`aspect-normalized-in-domain`, `aspect-normalized-out-of-domain`) has 3 splits: `train`, `validation` and `test`. 



Below are statistics on the configuration splits. 

**In-Domain Configuration**

| Split | #samples  |
| ----- | --------- |
| Train | 2,192,703 |
| Test  | 168,365   |

**Out-of-Domain Configuration**

| Split | #samples  |
| ----- | --------- |
| Train | 4,996,673 |
| Test  | 625,911   |

**Aspect-Normalized In-Domain Configuration**

| Split      | #samples |
| ---------- | -------- |
| Train      | 115,127  |
| Validation | 12,806   |
| Test       | 168,365  |

**Aspect-Normalized Out-of-Domain Configuration**

| Split      | #samples |
| ---------- | -------- |
| Train      | 119,167  |
| Validation | 13,263   |
| Test       | 625,911  |

task_categories:text-classification annotations_creators:no-annotation multilinguality:monolingual size_categories:1M<n<10M language:en license:mit arxiv:2005.00547 arxiv:2010.12421 arxiv:1509.01626 arxiv:1307.5336 arxiv:1909.05855 arxiv:1909.02027 arxiv:2011.13205 arxiv:2003.04807 arxiv:1805.10190 arxiv:1903.05566 region:us
{"annotations_creators": ["no-annotation"], "language": ["en"], "license": "mit", "multilinguality": ["monolingual"], "size_categories": ["1M<n<10M"], "task_categories": ["text-classification"], "pretty_name": "UTCD", "dataset_info": [{"config_name": "in-domain", "features": [{"name": "text", "dtype": "string"}, {"name": "labels", "sequence": {"class_label": {"names": {"0": "Add Alarm", "1": "Album", "2": "Animal", "3": "Artist", "4": "Athlete", "5": "Book Appointment", "6": "Book House", "7": "Building", "8": "Business", "9": "Business & Finance", "10": "Buy Bus Ticket", "11": "Buy Event Tickets", "12": "Buy Movie Tickets", "13": "Check Balance", "14": "Company", "15": "Computers & Internet", "16": "Education & Reference", "17": "Educational Institution", "18": "Entertainment & Music", "19": "Family & Relationships", "20": "Film", "21": "Find Apartment", "22": "Find Attractions", "23": "Find Bus", "24": "Find Events", "25": "Find Home By Area", "26": "Find Movies", "27": "Find Provider", "28": "Find Restaurants", "29": "Find Trains", "30": "Get Alarms", "31": "Get Available Time", "32": "Get Cars Available", "33": "Get Event Dates", "34": "Get Events", "35": "Get Ride", "36": "Get Times For Movie", "37": "Get Weather", "38": "Health", "39": "Lookup Music", "40": "Lookup Song", "41": "Make Payment", "42": "Mean Of Transportation", "43": "Natural Place", "44": "Office Holder", "45": "Plant", "46": "Play Media", "47": "Play Movie", "48": "Play Song", "49": "Politics & Government", "50": "Request Payment", "51": "Reserve Car", "52": "Reserve Hotel", "53": "Reserve One way Flight", "54": "Reserve Restaurant", "55": "Reserve Round trip Flights", "56": "Schedule Visit", "57": "Science & Mathematics", "58": "Science & Technology", "59": "Search Hotel", "60": "Search House", "61": "Search One way Flight", "62": "Search Round trip Flights", "63": "Society & Culture", "64": "Sports", "65": "Transfer Money", "66": "Village", "67": "World News", "68": "Written Work", "69": "accept reservations", "70": "account blocked", "71": "add contact", "72": "admiration", "73": "alarm", "74": "alarm query", "75": "alarm remove", "76": "alarm set", "77": "amusement", "78": "anger", "79": "annoyance", "80": "application status", "81": "approval", "82": "apr", "83": "are you a bot", "84": "audio volume down", "85": "audio volume mute", "86": "audio volume other", "87": "audio volume up", "88": "balance", "89": "bill balance", "90": "bill due", "91": "book flight", "92": "book hotel", "93": "calculator", "94": "calendar", "95": "calendar query", "96": "calendar remove", "97": "calendar set", "98": "calendar update", "99": "calories", "100": "cancel", "101": "cancel reservation", "102": "car rental", "103": "card declined", "104": "caring", "105": "carry on", "106": "change accent", "107": "change ai name", "108": "change language", "109": "change speed", "110": "change user name", "111": "change volume", "112": "cleaning", "113": "coffee", "114": "confirm reservation", "115": "confusion", "116": "convert", "117": "cook time", "118": "cooking query", "119": "cooking recipe", "120": "create or add", "121": "credit limit", "122": "credit limit change", "123": "credit score", "124": "curiosity", "125": "currency", "126": "current location", "127": "damaged card", "128": "date", "129": "date time convert", "130": "date time query", "131": "definition", "132": "desire", "133": "direct deposit", "134": "directions", "135": "disappointment", "136": "disapproval", "137": "disgust", "138": "distance", "139": "do you have pets", "140": "email add contact", "141": "email query", "142": "email query contact", "143": "email send email", "144": "embarrassment", "145": "events", "146": "exchange rate", "147": "excitement", "148": "expiration date", "149": "factoid", "150": "fear", "151": "find phone", "152": "flight status", "153": "flip coin", "154": "food last", "155": "freeze account", "156": "fun fact", "157": "game", "158": "gas", "159": "gas type", "160": "general greet", "161": "general joke", "162": "general quirky", "163": "goodbye", "164": "gratitude", "165": "greet", "166": "greeting", "167": "grief", "168": "how busy", "169": "how old are you", "170": "hue light dim", "171": "hue light off", "172": "hue light up", "173": "improve credit score", "174": "income", "175": "ingredient substitution", "176": "ingredients list", "177": "insurance", "178": "insurance change", "179": "interest rate", "180": "international fees", "181": "international visa", "182": "iot cleaning", "183": "iot coffee", "184": "iot hue light change", "185": "iot hue light dim", "186": "iot hue light off", "187": "iot hue light on", "188": "iot hue light up", "189": "iot wemo on", "190": "iot wemo plug off", "191": "joke", "192": "joy", "193": "jump start", "194": "last maintenance", "195": "lists create or add", "196": "lists query", "197": "lists remove", "198": "lost luggage", "199": "love", "200": "make call", "201": "maybe", "202": "meal suggestion", "203": "meaning of life", "204": "measurement conversion", "205": "meeting schedule", "206": "min payment", "207": "mpg", "208": "music", "209": "music dislike ness", "210": "music likeness", "211": "music query", "212": "music settings", "213": "negative", "214": "nervousness", "215": "neutral", "216": "new card", "217": "news query", "218": "next holiday", "219": "next song", "220": "no", "221": "nutrition info", "222": "oil change how", "223": "oil change when", "224": "optimism", "225": "order", "226": "order checks", "227": "order status", "228": "paid time off request status", "229": "paid time off used", "230": "pay bill", "231": "payday", "232": "pin change", "233": "play audiobook", "234": "play game", "235": "play music", "236": "play podcasts", "237": "play radio", "238": "plug type", "239": "podcasts", "240": "positive", "241": "post", "242": "pride", "243": "pto balance", "244": "pto request", "245": "qa currency", "246": "qa definition", "247": "qa factoid", "248": "qa maths", "249": "qa stock", "250": "query", "251": "query contact", "252": "quirky", "253": "radio", "254": "realization", "255": "recipe", "256": "recommendation events", "257": "recommendation locations", "258": "recommendation movies", "259": "redeem rewards", "260": "relief", "261": "reminder", "262": "reminder update", "263": "remorse", "264": "remove", "265": "repeat", "266": "replacement card duration", "267": "report fraud", "268": "report lost card", "269": "reset settings", "270": "restaurant reservation", "271": "restaurant reviews", "272": "restaurant suggestion", "273": "rewards balance", "274": "roll dice", "275": "rollover 401k", "276": "routing", "277": "sadness", "278": "schedule maintenance", "279": "schedule meeting", "280": "send email", "281": "set", "282": "settings", "283": "share location", "284": "shopping list", "285": "shopping list update", "286": "smart home", "287": "social post", "288": "social query", "289": "spelling", "290": "spending history", "291": "surprise", "292": "sync device", "293": "take away order", "294": "take away query", "295": "taxes", "296": "tell joke", "297": "text", "298": "thank you", "299": "ticket", "300": "time", "301": "timer", "302": "timezone", "303": "tire change", "304": "tire pressure", "305": "todo list", "306": "todo list update", "307": "traffic", "308": "transactions", "309": "transfer", "310": "translate", "311": "transport query", "312": "transport taxi", "313": "transport ticket", "314": "transport traffic", "315": "travel alert", "316": "travel notification", "317": "travel suggestion", "318": "uber", "319": "update playlist", "320": "user name", "321": "vaccines", "322": "volume other", "323": "w2 wage and tax statement", "324": "weather", "325": "weather query", "326": "wemo off", "327": "wemo plug on", "328": "what are your hobbies", "329": "what can i ask you", "330": "what is your name", "331": "what song", "332": "where are you from", "333": "whisper mode", "334": "who do you work for", "335": "who made you", "336": "yes"}}}}, {"name": "dataset_name", "dtype": {"class_label": {"names": {"0": "go_emotion", "1": "sentiment_tweets_2020", "2": "emotion", "3": "sgd", "4": "clinc_150", "5": "slurp", "6": "ag_news", "7": "dbpedia", "8": "yahoo"}}}}], "splits": [{"name": "train", "num_bytes": 347382307, "num_examples": 2192703}, {"name": "test", "num_bytes": 36063588, "num_examples": 168365}], "download_size": 1744258165, "dataset_size": 383445895}, {"config_name": "aspect-normalized-in-domain", "features": [{"name": "text", "dtype": "string"}, {"name": "labels", "sequence": {"class_label": {"names": {"0": "Add Alarm", "1": "Album", "2": "Animal", "3": "Artist", "4": "Athlete", "5": "Book Appointment", "6": "Book House", "7": "Building", "8": "Business", "9": "Business & Finance", "10": "Buy Bus Ticket", "11": "Buy Event Tickets", "12": "Buy Movie Tickets", "13": "Check Balance", "14": "Company", "15": "Computers & Internet", "16": "Education & Reference", "17": "Educational Institution", "18": "Entertainment & Music", "19": "Family & Relationships", "20": "Film", "21": "Find Apartment", "22": "Find Attractions", "23": "Find Bus", "24": "Find Events", "25": "Find Home By Area", "26": "Find Movies", "27": "Find Provider", "28": "Find Restaurants", "29": "Find Trains", "30": "Get Alarms", "31": "Get Available Time", "32": "Get Cars Available", "33": "Get Event Dates", "34": "Get Events", "35": "Get Ride", "36": "Get Times For Movie", "37": "Get Weather", "38": "Health", "39": "Lookup Music", "40": "Lookup Song", "41": "Make Payment", "42": "Mean Of Transportation", "43": "Natural Place", "44": "Office Holder", "45": "Plant", "46": "Play Media", "47": "Play Movie", "48": "Play Song", "49": "Politics & Government", "50": "Request Payment", "51": "Reserve Car", "52": "Reserve Hotel", "53": "Reserve One way Flight", "54": "Reserve Restaurant", "55": "Reserve Round trip Flights", "56": "Schedule Visit", "57": "Science & Mathematics", "58": "Science & Technology", "59": "Search Hotel", "60": "Search House", "61": "Search One way Flight", "62": "Search Round trip Flights", "63": "Society & Culture", "64": "Sports", "65": "Transfer Money", "66": "Village", "67": "World News", "68": "Written Work", "69": "accept reservations", "70": "account blocked", "71": "add contact", "72": "admiration", "73": "alarm", "74": "alarm query", "75": "alarm remove", "76": "alarm set", "77": "amusement", "78": "anger", "79": "annoyance", "80": "application status", "81": "approval", "82": "apr", "83": "are you a bot", "84": "audio volume down", "85": "audio volume mute", "86": "audio volume other", "87": "audio volume up", "88": "balance", "89": "bill balance", "90": "bill due", "91": "book flight", "92": "book hotel", "93": "calculator", "94": "calendar", "95": "calendar query", "96": "calendar remove", "97": "calendar set", "98": "calendar update", "99": "calories", "100": "cancel", "101": "cancel reservation", "102": "car rental", "103": "card declined", "104": "caring", "105": "carry on", "106": "change accent", "107": "change ai name", "108": "change language", "109": "change speed", "110": "change user name", "111": "change volume", "112": "cleaning", "113": "coffee", "114": "confirm reservation", "115": "confusion", "116": "convert", "117": "cook time", "118": "cooking query", "119": "cooking recipe", "120": "create or add", "121": "credit limit", "122": "credit limit change", "123": "credit score", "124": "curiosity", "125": "currency", "126": "current location", "127": "damaged card", "128": "date", "129": "date time convert", "130": "date time query", "131": "definition", "132": "desire", "133": "direct deposit", "134": "directions", "135": "disappointment", "136": "disapproval", "137": "disgust", "138": "distance", "139": "do you have pets", "140": "email add contact", "141": "email query", "142": "email query contact", "143": "email send email", "144": "embarrassment", "145": "events", "146": "exchange rate", "147": "excitement", "148": "expiration date", "149": "factoid", "150": "fear", "151": "find phone", "152": "flight status", "153": "flip coin", "154": "food last", "155": "freeze account", "156": "fun fact", "157": "game", "158": "gas", "159": "gas type", "160": "general greet", "161": "general joke", "162": "general quirky", "163": "goodbye", "164": "gratitude", "165": "greet", "166": "greeting", "167": "grief", "168": "how busy", "169": "how old are you", "170": "hue light dim", "171": "hue light off", "172": "hue light up", "173": "improve credit score", "174": "income", "175": "ingredient substitution", "176": "ingredients list", "177": "insurance", "178": "insurance change", "179": "interest rate", "180": "international fees", "181": "international visa", "182": "iot cleaning", "183": "iot coffee", "184": "iot hue light change", "185": "iot hue light dim", "186": "iot hue light off", "187": "iot hue light on", "188": "iot hue light up", "189": "iot wemo on", "190": "iot wemo plug off", "191": "joke", "192": "joy", "193": "jump start", "194": "last maintenance", "195": "lists create or add", "196": "lists query", "197": "lists remove", "198": "lost luggage", "199": "love", "200": "make call", "201": "maybe", "202": "meal suggestion", "203": "meaning of life", "204": "measurement conversion", "205": "meeting schedule", "206": "min payment", "207": "mpg", "208": "music", "209": "music dislike ness", "210": "music likeness", "211": "music query", "212": "music settings", "213": "negative", "214": "nervousness", "215": "neutral", "216": "new card", "217": "news query", "218": "next holiday", "219": "next song", "220": "no", "221": "nutrition info", "222": "oil change how", "223": "oil change when", "224": "optimism", "225": "order", "226": "order checks", "227": "order status", "228": "paid time off request status", "229": "paid time off used", "230": "pay bill", "231": "payday", "232": "pin change", "233": "play audiobook", "234": "play game", "235": "play music", "236": "play podcasts", "237": "play radio", "238": "plug type", "239": "podcasts", "240": "positive", "241": "post", "242": "pride", "243": "pto balance", "244": "pto request", "245": "qa currency", "246": "qa definition", "247": "qa factoid", "248": "qa maths", "249": "qa stock", "250": "query", "251": "query contact", "252": "quirky", "253": "radio", "254": "realization", "255": "recipe", "256": "recommendation events", "257": "recommendation locations", "258": "recommendation movies", "259": "redeem rewards", "260": "relief", "261": "reminder", "262": "reminder update", "263": "remorse", "264": "remove", "265": "repeat", "266": "replacement card duration", "267": "report fraud", "268": "report lost card", "269": "reset settings", "270": "restaurant reservation", "271": "restaurant reviews", "272": "restaurant suggestion", "273": "rewards balance", "274": "roll dice", "275": "rollover 401k", "276": "routing", "277": "sadness", "278": "schedule maintenance", "279": "schedule meeting", "280": "send email", "281": "set", "282": "settings", "283": "share location", "284": "shopping list", "285": "shopping list update", "286": "smart home", "287": "social post", "288": "social query", "289": "spelling", "290": "spending history", "291": "surprise", "292": "sync device", "293": "take away order", "294": "take away query", "295": "taxes", "296": "tell joke", "297": "text", "298": "thank you", "299": "ticket", "300": "time", "301": "timer", "302": "timezone", "303": "tire change", "304": "tire pressure", "305": "todo list", "306": "todo list update", "307": "traffic", "308": "transactions", "309": "transfer", "310": "translate", "311": "transport query", "312": "transport taxi", "313": "transport ticket", "314": "transport traffic", "315": "travel alert", "316": "travel notification", "317": "travel suggestion", "318": "uber", "319": "update playlist", "320": "user name", "321": "vaccines", "322": "volume other", "323": "w2 wage and tax statement", "324": "weather", "325": "weather query", "326": "wemo off", "327": "wemo plug on", "328": "what are your hobbies", "329": "what can i ask you", "330": "what is your name", "331": "what song", "332": "where are you from", "333": "whisper mode", "334": "who do you work for", "335": "who made you", "336": "yes"}}}}, {"name": "dataset_name", "dtype": {"class_label": {"names": {"0": "go_emotion", "1": "sentiment_tweets_2020", "2": "emotion", "3": "sgd", "4": "clinc_150", "5": "slurp", "6": "ag_news", "7": "dbpedia", "8": "yahoo"}}}}], "splits": [{"name": "train", "num_bytes": 28974188, "num_examples": 115127}, {"name": "validation", "num_bytes": 3213586, "num_examples": 12806}, {"name": "test", "num_bytes": 36063590, "num_examples": 168365}], "download_size": 1744258165, "dataset_size": 68251364}, {"config_name": "out-of-domain", "features": [{"name": "text", "dtype": "string"}, {"name": "labels", "sequence": {"class_label": {"names": {"0": "Add To Playlist", "1": "Bank account or service", "2": "Book Restaurant", "3": "Checking or savings account", "4": "Chemistry; Metallurgy", "5": "Consumer Loan", "6": "Credit card", "7": "Credit card or prepaid card", "8": "Credit reporting", "9": "Credit reporting, credit repair services, or other personal consumer reports", "10": "Debt collection", "11": "EUROPEAN UNION", "12": "Electricity", "13": "Fixed Constructions", "14": "General tagging of new or cross-sectional technology", "15": "Get Weather", "16": "Human Necessities", "17": "Mechanical Engineering; Lightning; Heating; Weapons; Blasting", "18": "Money transfer, virtual currency, or money service", "19": "Money transfers", "20": "Mortgage", "21": "Other financial service", "22": "Payday loan", "23": "Payday loan, title loan, or personal loan", "24": "Performing Operations; Transporting", "25": "Physics", "26": "Play Music", "27": "Prepaid card", "28": "Rate Book", "29": "Refund not showing up", "30": "Search Creative Work", "31": "Search Screening Event", "32": "Student loan", "33": "Textiles; Paper", "34": "Vehicle loan or lease", "35": "Virtual currency", "36": "activate my card", "37": "age limit", "38": "agri-foodstuffs", "39": "agriculture, forestry and fisheries", "40": "alarm query", "41": "alarm remove", "42": "alarm set", "43": "apple pay or google pay", "44": "atm support", "45": "audio volume down", "46": "audio volume mute", "47": "audio volume other", "48": "audio volume up", "49": "automatic top up", "50": "balance not updated after bank transfer", "51": "balance not updated after cheque or cash deposit", "52": "beneficiary not allowed", "53": "business and competition", "54": "calendar query", "55": "calendar remove", "56": "calendar set", "57": "cancel transfer", "58": "card about to expire", "59": "card acceptance", "60": "card arrival", "61": "card delivery estimate", "62": "card linking", "63": "card not working", "64": "card payment fee charged", "65": "card payment not recognised", "66": "card payment wrong exchange rate", "67": "card swallowed", "68": "cash withdrawal charge", "69": "cash withdrawal not recognised", "70": "change pin", "71": "compromised card", "72": "contactless not working", "73": "cooking query", "74": "cooking recipe", "75": "country support", "76": "datetime convert", "77": "datetime query", "78": "declined card payment", "79": "declined cash withdrawal", "80": "declined transfer", "81": "direct debit payment not recognised", "82": "disposable card limits", "83": "economics", "84": "edit personal details", "85": "education and communications", "86": "email addcontact", "87": "email query", "88": "email querycontact", "89": "email sendemail", "90": "employment and working conditions", "91": "energy", "92": "environment", "93": "exchange charge", "94": "exchange rate", "95": "exchange via app", "96": "extra charge on statement", "97": "failed transfer", "98": "fiat currency support", "99": "finance", "100": "general affirm", "101": "general commandstop", "102": "general confirm", "103": "general dontcare", "104": "general explain", "105": "general greet", "106": "general joke", "107": "general negate", "108": "general praise", "109": "general quirky", "110": "general repeat", "111": "geography", "112": "get disposable virtual card", "113": "get physical card", "114": "getting spare card", "115": "getting virtual card", "116": "industry", "117": "international organisations", "118": "international relations", "119": "iot cleaning", "120": "iot coffee", "121": "iot hue lightchange", "122": "iot hue lightdim", "123": "iot hue lightoff", "124": "iot hue lighton", "125": "iot hue lightup", "126": "iot wemo off", "127": "iot wemo on", "128": "law", "129": "lists createoradd", "130": "lists query", "131": "lists remove", "132": "lost or stolen card", "133": "lost or stolen phone", "134": "music dislikeness", "135": "music likeness", "136": "music query", "137": "music settings", "138": "negative", "139": "neutral", "140": "news query", "141": "order physical card", "142": "passcode forgotten", "143": "pending card payment", "144": "pending cash withdrawal", "145": "pending top up", "146": "pending transfer", "147": "pin blocked", "148": "play audiobook", "149": "play game", "150": "play music", "151": "play podcasts", "152": "play radio", "153": "politics", "154": "positive", "155": "production, technology and research", "156": "qa currency", "157": "qa definition", "158": "qa factoid", "159": "qa maths", "160": "qa stock", "161": "receiving money", "162": "recommendation events", "163": "recommendation locations", "164": "recommendation movies", "165": "request refund", "166": "reverted card payment?", "167": "science", "168": "social post", "169": "social query", "170": "social questions", "171": "supported cards and currencies", "172": "takeaway order", "173": "takeaway query", "174": "terminate account", "175": "top up by bank transfer charge", "176": "top up by card charge", "177": "top up by cash or cheque", "178": "top up failed", "179": "top up limits", "180": "top up reverted", "181": "topping up by card", "182": "trade", "183": "transaction charged twice", "184": "transfer fee charged", "185": "transfer into account", "186": "transfer not received by recipient", "187": "transfer timing", "188": "transport", "189": "transport query", "190": "transport taxi", "191": "transport ticket", "192": "transport traffic", "193": "unable to verify identity", "194": "verify my identity", "195": "verify source of funds", "196": "verify top up", "197": "virtual card not working", "198": "visa or mastercard", "199": "weather query", "200": "why verify identity", "201": "wrong amount of cash received", "202": "wrong exchange rate for cash withdrawal"}}}}, {"name": "dataset_name", "dtype": {"class_label": {"names": {"0": "amazon_polarity", "1": "finance_sentiment", "2": "yelp", "3": "banking77", "4": "snips", "5": "nlu_evaluation", "6": "multi_eurlex", "7": "patent", "8": "consumer_finance"}}}}], "splits": [{"name": "train", "num_bytes": 3608196895, "num_examples": 4996673}, {"name": "test", "num_bytes": 541174753, "num_examples": 625911}], "download_size": 1744258165, "dataset_size": 4149371648}, {"config_name": "aspect-normalized-out-of-domain", "features": [{"name": "text", "dtype": "string"}, {"name": "labels", "sequence": {"class_label": {"names": {"0": "Add To Playlist", "1": "Bank account or service", "2": "Book Restaurant", "3": "Checking or savings account", "4": "Chemistry; Metallurgy", "5": "Consumer Loan", "6": "Credit card", "7": "Credit card or prepaid card", "8": "Credit reporting", "9": "Credit reporting, credit repair services, or other personal consumer reports", "10": "Debt collection", "11": "EUROPEAN UNION", "12": "Electricity", "13": "Fixed Constructions", "14": "General tagging of new or cross-sectional technology", "15": "Get Weather", "16": "Human Necessities", "17": "Mechanical Engineering; Lightning; Heating; Weapons; Blasting", "18": "Money transfer, virtual currency, or money service", "19": "Money transfers", "20": "Mortgage", "21": "Other financial service", "22": "Payday loan", "23": "Payday loan, title loan, or personal loan", "24": "Performing Operations; Transporting", "25": "Physics", "26": "Play Music", "27": "Prepaid card", "28": "Rate Book", "29": "Refund not showing up", "30": "Search Creative Work", "31": "Search Screening Event", "32": "Student loan", "33": "Textiles; Paper", "34": "Vehicle loan or lease", "35": "Virtual currency", "36": "activate my card", "37": "age limit", "38": "agri-foodstuffs", "39": "agriculture, forestry and fisheries", "40": "alarm query", "41": "alarm remove", "42": "alarm set", "43": "apple pay or google pay", "44": "atm support", "45": "audio volume down", "46": "audio volume mute", "47": "audio volume other", "48": "audio volume up", "49": "automatic top up", "50": "balance not updated after bank transfer", "51": "balance not updated after cheque or cash deposit", "52": "beneficiary not allowed", "53": "business and competition", "54": "calendar query", "55": "calendar remove", "56": "calendar set", "57": "cancel transfer", "58": "card about to expire", "59": "card acceptance", "60": "card arrival", "61": "card delivery estimate", "62": "card linking", "63": "card not working", "64": "card payment fee charged", "65": "card payment not recognised", "66": "card payment wrong exchange rate", "67": "card swallowed", "68": "cash withdrawal charge", "69": "cash withdrawal not recognised", "70": "change pin", "71": "compromised card", "72": "contactless not working", "73": "cooking query", "74": "cooking recipe", "75": "country support", "76": "datetime convert", "77": "datetime query", "78": "declined card payment", "79": "declined cash withdrawal", "80": "declined transfer", "81": "direct debit payment not recognised", "82": "disposable card limits", "83": "economics", "84": "edit personal details", "85": "education and communications", "86": "email addcontact", "87": "email query", "88": "email querycontact", "89": "email sendemail", "90": "employment and working conditions", "91": "energy", "92": "environment", "93": "exchange charge", "94": "exchange rate", "95": "exchange via app", "96": "extra charge on statement", "97": "failed transfer", "98": "fiat currency support", "99": "finance", "100": "general affirm", "101": "general commandstop", "102": "general confirm", "103": "general dontcare", "104": "general explain", "105": "general greet", "106": "general joke", "107": "general negate", "108": "general praise", "109": "general quirky", "110": "general repeat", "111": "geography", "112": "get disposable virtual card", "113": "get physical card", "114": "getting spare card", "115": "getting virtual card", "116": "industry", "117": "international organisations", "118": "international relations", "119": "iot cleaning", "120": "iot coffee", "121": "iot hue lightchange", "122": "iot hue lightdim", "123": "iot hue lightoff", "124": "iot hue lighton", "125": "iot hue lightup", "126": "iot wemo off", "127": "iot wemo on", "128": "law", "129": "lists createoradd", "130": "lists query", "131": "lists remove", "132": "lost or stolen card", "133": "lost or stolen phone", "134": "music dislikeness", "135": "music likeness", "136": "music query", "137": "music settings", "138": "negative", "139": "neutral", "140": "news query", "141": "order physical card", "142": "passcode forgotten", "143": "pending card payment", "144": "pending cash withdrawal", "145": "pending top up", "146": "pending transfer", "147": "pin blocked", "148": "play audiobook", "149": "play game", "150": "play music", "151": "play podcasts", "152": "play radio", "153": "politics", "154": "positive", "155": "production, technology and research", "156": "qa currency", "157": "qa definition", "158": "qa factoid", "159": "qa maths", "160": "qa stock", "161": "receiving money", "162": "recommendation events", "163": "recommendation locations", "164": "recommendation movies", "165": "request refund", "166": "reverted card payment?", "167": "science", "168": "social post", "169": "social query", "170": "social questions", "171": "supported cards and currencies", "172": "takeaway order", "173": "takeaway query", "174": "terminate account", "175": "top up by bank transfer charge", "176": "top up by card charge", "177": "top up by cash or cheque", "178": "top up failed", "179": "top up limits", "180": "top up reverted", "181": "topping up by card", "182": "trade", "183": "transaction charged twice", "184": "transfer fee charged", "185": "transfer into account", "186": "transfer not received by recipient", "187": "transfer timing", "188": "transport", "189": "transport query", "190": "transport taxi", "191": "transport ticket", "192": "transport traffic", "193": "unable to verify identity", "194": "verify my identity", "195": "verify source of funds", "196": "verify top up", "197": "virtual card not working", "198": "visa or mastercard", "199": "weather query", "200": "why verify identity", "201": "wrong amount of cash received", "202": "wrong exchange rate for cash withdrawal"}}}}, {"name": "dataset_name", "dtype": {"class_label": {"names": {"0": "amazon_polarity", "1": "finance_sentiment", "2": "yelp", "3": "banking77", "4": "snips", "5": "nlu_evaluation", "6": "multi_eurlex", "7": "patent", "8": "consumer_finance"}}}}], "splits": [{"name": "train", "num_bytes": 109566474, "num_examples": 119167}, {"name": "validation", "num_bytes": 12432497, "num_examples": 13263}, {"name": "test", "num_bytes": 541174753, "num_examples": 625911}], "download_size": 1744258165, "dataset_size": 663173724}]}
TAGS
#task_categories-text-classification #annotations_creators-no-annotation #multilinguality-monolingual #size_categories-1M<n<10M #language-English #license-mit #arxiv-2005.00547 #arxiv-2010.12421 #arxiv-1509.01626 #arxiv-1307.5336 #arxiv-1909.05855 #arxiv-1909.02027 #arxiv-2011.13205 #arxiv-2003.04807 #arxiv-1805.10190 #arxiv-1903.05566 #region-us 

Universal Text Classification Dataset (UTCD)
============================================


Load dataset
------------


Description
-----------


UTCD is a curated compilation of 18 datasets revised for Zero-shot Text Classification spanning 3 aspect categories of Sentiment, Intent/Dialogue, and Topic classification. UTCD focuses on the task of zero-shot text classification where the candidate labels are descriptive of the text being classified. TUTCD consists of ~ 6M/800K train/test examples.


UTCD was introduced in the Findings of ACL'23 Paper Label Agnostic Pre-training for Zero-shot Text Classification by *Christopher Clarke, Yuzhao Heng, Yiping Kang, Krisztian Flautner, Lingjia Tang and Jason Mars*. Project Homepage.


UTCD Datasets & Principles:


In order to make NLP models more broadly useful, zero-shot techniques need to be capable of label, domain & aspect transfer. As such, in the construction of UTCD we enforce the following principles:


* Textual labels: In UTCD, we mandate the use of textual labels. While numerical label values are often used in classification tasks, descriptive textual labels such as those present in the datasets across UTCD enable the development of techniques that can leverage the class name which is instrumental in providing zero-shot support. As such, for each of the compiled datasets, labels are standardized such that the labels are descriptive of the text in natural language.
* Diverse domains and Sequence lengths: In addition to broad coverage of aspects, UTCD compiles diverse data across several domains such as Banking, Finance, Legal, etc each comprising varied length sequences (long and short). The datasets are listed above.
* Sentiment


	+ GoEmotions introduced in GoEmotions: A Dataset of Fine-Grained Emotions
	+ TweetEval introduced in TWEETEVAL: Unified Benchmark and Comparative Evaluation for Tweet Classification (Sentiment subset)
	+ Emotion introduced in CARER: Contextualized Affect Representations for Emotion Recognition
	+ Amazon Polarity introduced in Character-level Convolutional Networks for Text Classification
	+ Finance Phrasebank introduced in Good debt or bad debt: Detecting semantic orientations in economic texts
	+ Yelp introduced in Character-level Convolutional Networks for Text Classification
* Intent/Dialogue


	+ Schema-Guided Dialogue introduced in Towards Scalable Multi-Domain Conversational Agents: The Schema-Guided Dialogue Dataset
	+ Clinc-150 introduced in An Evaluation Dataset for Intent Classification and Out-of-Scope Prediction
	+ SLURP SLU introduced in SLURP: A Spoken Language Understanding Resource Package
	+ Banking77 introduced in Efficient Intent Detection with Dual Sentence Encoders
	+ Snips introduced in Snips Voice Platform: an embedded Spoken Language Understanding system for private-by-design voice interfaces
	+ NLU Evaluation introduced in Benchmarking Natural Language Understanding Services for building Conversational Agents
* Topic


	+ AG News introduced in Character-level Convolutional Networks for Text Classification
	+ DBpedia 14 introduced in DBpedia: A Nucleus for a Web of Open Data
	+ Yahoo Answer Topics introduced in Character-level Convolutional Networks for Text Classification
	+ MultiEurlex introduced in MultiEURLEX -- A multi-lingual and multi-label legal document classification dataset for zero-shot cross-lingual transfer
	+ BigPatent introduced in BIGPATENT: A Large-Scale Dataset for Abstractive and Coherent Summarization
	+ Consumer Finance introduced in Consumer Complaint Database


Structure
---------


### Data Samples


Each dataset sample contains the text, the label encoded as an integer, and the dataset name encoded as an integer.


### Datasets Contained


The UTCD dataset contains 18 datasets, 9 'in-domain', 9 'out-of-domain', spanning 3 aspects: 'sentiment', 'intent' and 'topic'.


Below are statistics on the datasets.


In-Domain Datasets



Out-of-Domain Datasets



### Configurations


The 'in-domain' and 'out-of-domain' configurations has 2 splits: 'train' and 'test'.


The aspect-normalized configurations ('aspect-normalized-in-domain', 'aspect-normalized-out-of-domain') has 3 splits: 'train', 'validation' and 'test'.


Below are statistics on the configuration splits.


In-Domain Configuration



Out-of-Domain Configuration



Aspect-Normalized In-Domain Configuration



Aspect-Normalized Out-of-Domain Configuration




### Data Samples


Each dataset sample contains the text, the label encoded as an integer, and the dataset name encoded as an integer. ### Datasets Contained


The UTCD dataset contains 18 datasets, 9 'in-domain', 9 'out-of-domain', spanning 3 aspects: 'sentiment', 'intent' and 'topic'.


Below are statistics on the datasets.


In-Domain Datasets



Out-of-Domain Datasets ### Configurations


The 'in-domain' and 'out-of-domain' configurations has 2 splits: 'train' and 'test'.


The aspect-normalized configurations ('aspect-normalized-in-domain', 'aspect-normalized-out-of-domain') has 3 splits: 'train', 'validation' and 'test'.


Below are statistics on the configuration splits.


In-Domain Configuration



Out-of-Domain Configuration



Aspect-Normalized In-Domain Configuration



Aspect-Normalized Out-of-Domain Configuration
TAGS
#task_categories-text-classification #annotations_creators-no-annotation #multilinguality-monolingual #size_categories-1M<n<10M #language-English #license-mit #arxiv-2005.00547 #arxiv-2010.12421 #arxiv-1509.01626 #arxiv-1307.5336 #arxiv-1909.05855 #arxiv-1909.02027 #arxiv-2011.13205 #arxiv-2003.04807 #arxiv-1805.10190 #arxiv-1903.05566 #region-us 
 ### Data Samples


Each dataset sample contains the text, the label encoded as an integer, and the dataset name encoded as an integer. ### Datasets Contained


The UTCD dataset contains 18 datasets, 9 'in-domain', 9 'out-of-domain', spanning 3 aspects: 'sentiment', 'intent' and 'topic'.


Below are statistics on the datasets.


In-Domain Datasets



Out-of-Domain Datasets ### Configurations


The 'in-domain' and 'out-of-domain' configurations has 2 splits: 'train' and 'test'.


The aspect-normalized configurations ('aspect-normalized-in-domain', 'aspect-normalized-out-of-domain') has 3 splits: 'train', 'validation' and 'test'.


Below are statistics on the configuration splits.


In-Domain Configuration



Out-of-Domain Configuration



Aspect-Normalized In-Domain Configuration



Aspect-Normalized Out-of-Domain Configuration
passage: TAGS
#task_categories-text-classification #annotations_creators-no-annotation #multilinguality-monolingual #size_categories-1M<n<10M #language-English #license-mit #arxiv-2005.00547 #arxiv-2010.12421 #arxiv-1509.01626 #arxiv-1307.5336 #arxiv-1909.05855 #arxiv-1909.02027 #arxiv-2011.13205 #arxiv-2003.04807 #arxiv-1805.10190 #arxiv-1903.05566 #region-us 
### Data Samples


Each dataset sample contains the text, the label encoded as an integer, and the dataset name encoded as an integer.### Datasets Contained


The UTCD dataset contains 18 datasets, 9 'in-domain', 9 'out-of-domain', spanning 3 aspects: 'sentiment', 'intent' and 'topic'.


Below are statistics on the datasets.


In-Domain Datasets



Out-of-Domain Datasets### Configurations


The 'in-domain' and 'out-of-domain' configurations has 2 splits: 'train' and 'test'.


The aspect-normalized configurations ('aspect-normalized-in-domain', 'aspect-normalized-out-of-domain') has 3 splits: 'train', 'validation' and 'test'.


Below are statistics on the configuration splits.


In-Domain Configuration



Out-of-Domain Configuration



Aspect-Normalized In-Domain Configuration



Aspect-Normalized Out-of-Domain Configuration