

# SuperTweetEval

# Dataset Card for "super_tweeteval"

### Dataset Summary
This is the oficial repository for SuperTweetEval, a unified benchmark of 12 heterogeneous NLP tasks. 
More details on the task and an evaluation of language models can be found on the [reference paper](https://arxiv.org/abs/2310.14757), published in EMNLP 2023 (Findings).


### Data Splits

All tasks provide custom training, validation and test splits.

| **task**                   | **dataset**    | **load dataset** | **description**                    | **number of instances** |
|----------------------------|----------------|------------------|------------------------------------|-------------------------|
| Topic Classification       | TweetTopic     | tweet_topic      | multi-label classification         | 4,585 / 573 / 1,679     |
| NER                        | TweetNER7      | tweet_ner7       | sequence labeling                  | 4,616 / 576 / 2,807     |
| Question Answering         | TweettQA        | tweet_qa         | generation                         | 9,489 / 1,086 / 1,203   |
| Question Generation        | TweetQG        | tweet_qg         | generation                         | 9,489 / 1,086 / 1,203   |
| Intimacy Analysis          | TweetIntimacy  | tweet_intimacy   | regression on a single text        | 1,191 / 396 / 396       |
| Tweet Similarity           | TweetSIM       | tweet_similarity | regression on two texts            | 450 / 100 / 450         |
| Meaning Shift Detection    | TempoWIC       | tempo_wic        | binary classification on two texts | 1,427 / 395 / 1,472     |
| Hate Speech Detection      | TweetHate      | tweet_hate       | multi-class classification         | 5,019 / 716 / 1,433     |
| Emoji Classification       | TweetEmoji100  | tweet_emoji      | multi-class classification         | 50,000 / 5,000 / 50,000 |
| Sentiment Classification   | TweetSentiment | tweet_sentiment  | ABSA on a five-pointscale          | 26,632 / 4,000 / 12,379 |
| Name Entity Disambiguation | TweetNERD      | tweet_nerd       | binary classification              | 20,164 / 4,100 / 20,075 |
| Emotion Classification     | TweetEmotion   | tweet_emotion    | multi-label classification         | 6,838 / 886 / 3,259     |

## Dataset Structure
### Data Fields

The data fields are unified among all splits. 
In the following we present the information contained in each of the datasets.

#### tweet_topic
- `text`: a `string` feature.
- `gold_label_list`: a list of  `string` feature.
- `date`: a `string` feature.

#### tweet_ner7
- `text`: a `string` feature.
- `text_tokenized`: a list of `string` feature.
- `gold_label_sequence`: a list of `string` feature.
- `date`: a `string` feature.
- `entities`: a list of `dictionary` feature containing `{"entity": "string", "type": "string"}`.

#### tweet_qa
- `text`: a `string` feature.
- `gold_label_str`: a `string` feature.
- `context`: a `string` feature.

#### tweet_qg
- `text`: a `string` feature.
- `gold_label_str`: a `string` feature.
- `context`: a `string` feature.

#### tweet_intimacy
- `text`: a `string` feature.
- `gold_score`: a `float` feature.

#### tweet_similarity
- `text_1`: a `string` feature.
- `text_2`: a `string` feature.
- `gold_score`: a `float` feature.

#### tempo_wic
- `gold_label_binary`: a `int` feature.
- `target`: a `string` feature.
- `text_1`: a `string` feature.
- `text_tokenized_1`: a list of `string` feature.
- `token_idx_1`: a `int` feature.
- `date_1`: a `string` feature.
- `text_2`: a `string` feature.
- `text_tokenized_2`: a list of `string` feature.
- `token_idx_2`: a `int` feature.
- `date_2`: a `string` feature.

#### tweet_hate
- `gold_label`: a `int` feature.
- `text`: a `string` feature.

#### tweet_emoji
- `gold_label`: a `int` feature.
- `text`: a `string` feature.
- `date`: a `string` feature.

#### tweet_sentiment
- `gold_label`: a `int` feature.
- `text`: a `string` feature.
- `target`: a `string` feature.

#### tweet_nerd
- `gold_label_binary`: a `int` feature.
- `target`: a `string` feature.
- `text`: a `string` feature.
- `definition`: a `string` feature.
- `text_start`: a `int` feature.
- `text_end`: a `int` feature.
- `date`: a `string` feature.

#### tweet_emotion
- `text`: a `string` feature.
- `gold_label_list`: a list of  `string` feature.



## Evaluation metrics & Models



| **dataset**    	| **evaluation metric**                                                                                          	| **gold label**                                                                                                                                                                                                                                                                                                                                                                                                     	| **model card** 	|
|-------------------|-------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------|
| TweetTopic     	| ```macro-F1```                                                                                                 	| _arts\_&\_culture, business\_&\_entrepreneurs, celebrity\_&\_pop\_culture,  <br />diaries\_&\_daily\_life, family, fashion\_&\_style,  <br />film\_tv\_&\_video, fitness\_&\_health, food\_&\_dining,  <br />gaming, learning\_&\_educational, music,  <br />news\_&\_social\_concern, other\_hobbies, relationships,  <br />science\_&\_technology, sports, travel\_&\_adventure,  <br />youth\_&\_student\_life_ 	| [twitter-roberta-base-topic-latest (base)](https://huggingface.co/cardiffnlp/twitter-roberta-base-latest-tweet-topic) <br> [twitter-roberta-large-topic-latest (large)](https://huggingface.co/cardiffnlp/twitter-roberta-large-latest-tweet-topic)              	|
| TweetNER7      	| ```macro-F1```                                                                                                 	| _B-corporation, B-creative_work, B-event,  <br />B-group, B-location, B-person,  <br />B-product, I-corporation, I-creative_work,  <br />I-event, I-group, I-location,  <br />I-person, I-product, O_                                                                                                                                                                                                              	| [twitter-roberta-base-ner7-latest (base)](https://huggingface.co/cardiffnlp/twitter-roberta-base-latest-tweet-ner7) <br> TBA              	|
| TweettQA       	| ```answer-F1```                                                                                                	| -                                                                                                                                                                                                                                                                                                                                                                                                                  	|  [flan-t5-small-tweet-qa (small)](https://huggingface.co/cardiffnlp/flan-t5-small-tweet-qa) <br> [flan-t5-base-tweet-qa (base)](https://huggingface.co/cardiffnlp/flan-t5-base-tweet-qa)              	|
| TweetQG        	| ```METEOR```                                                                                                   	| -                                                                                                                                                                                                                                                                                                                                                                                                                  	|   [flan-t5-small-tweet-qg (small)](https://huggingface.co/cardiffnlp/flan-t5-small-tweet-qg) <br> [flan-t5-base-tweet-qg (base)](https://huggingface.co/cardiffnlp/flan-t5-base-tweet-qg)              	|
| TweetIntimacy  	| ```spearman correlation```                                                                                     	| _[1 - 5]_                                                                                                                                                                                                                                                                                                                                                                                                          	|  [twitter-roberta-base-intimacy-latest (base)](https://huggingface.co/cardiffnlp/twitter-roberta-base-latest-tweet-intimacy) <br> [twitter-roberta-large-intimacy-latest (large)](https://huggingface.co/cardiffnlp/twitter-roberta-large-latest-tweet-intimacy)              	|
| TweetSIM       	| ```spearman correlation```                                                                                     	| _[0 - 5]_                                                                                                                                                                                                                                                                                                                                                                                                          	|   [twitter-roberta-base-similarity-latest (base)](https://huggingface.co/cardiffnlp/twitter-roberta-base-latest-tweet-similarity) <br> [twitter-roberta-large-similarity-latest (large)](https://huggingface.co/cardiffnlp/twitter-roberta-large-latest-tweet-similarity)              	|
| TempoWIC       	| ```accuracy```                                                                                                 	| _no,  yes_                                                                                                                                                                                                                                                                                                                                                                                                         	|   [twitter-roberta-base-tempo-wic-latest (base)](https://huggingface.co/cardiffnlp/twitter-roberta-base-latest-tempo-wic) <br> [twitter-roberta-large-tempo-wic-latest (large)](https://huggingface.co/cardiffnlp/twitter-roberta-large-latest-tempo-wic)              	|
| TweetHate      	| ```combined-F1```<br /> ```(micro-F1 for hate/not-hate &```<br /> ```  macro-F1 for hate speech subclasses)``` 	| _hate_gender, hate_race, hate_sexuality, hate_religion, hate_origin,  <br />hate_disability, hate_age, not_hate_                                                                                                                                                                                                                                                                                                   	|   [twitter-roberta-base-hate-latest-st (base)](https://huggingface.co/cardiffnlp/twitter-roberta-base-latest-tweet-hate) <br> [twitter-roberta-large-hate-latest (large)](https://huggingface.co/cardiffnlp/twitter-roberta-large-latest-tweet-hate)              	|
| TweetEmoji100  	| ```accuracy at top 5```                                                                                        	| _Full emoji list: ./data/tweet_emoji/map.txt_                                                                                                                                                                                                                                                                                                                                                                      	|  [twitter-roberta-base-emoji-latest (base)](https://huggingface.co/cardiffnlp/twitter-roberta-base-latest-tweet-emoji) <br> [twitter-roberta-large-emoji-latest (large)](https://huggingface.co/cardiffnlp/twitter-roberta-large-latest-tweet-emoji)              	|
| TweetSentiment 	| ```1 - MAE^M``` <br /> ```(MAE^M : Macro Averaged Mean Absolute Error)```                                      	| _'strongly negative' , 'negative', 'negative or neutral',  <br /> 'positive', 'strongly positive'_                                                                                                                                                                                                                                                                                                                 	|   [twitter-roberta-base-topic-sentiment-latest (base)](https://huggingface.co/cardiffnlp/twitter-roberta-base-latest-tweet-sentiment) <br> [twitter-roberta-large-topic-sentiment-latest (large)](https://huggingface.co/cardiffnlp/twitter-roberta-large-latest-tweet-sentiment)              	|
| TweetNERD      	| ```accuracy```                                                                                                 	| _no, yes_                                                                                                                                                                                                                                                                                                                                                                                                          	|    [twitter-roberta-base-nerd-latest (base)](https://huggingface.co/cardiffnlp/twitter-roberta-base-latest-tweet-nerd) <br> [twitter-roberta-large-nerd-latest (large)](https://huggingface.co/cardiffnlp/twitter-roberta-large-latest-tweet-nerd)              	|
| TweetEmotion   	| ```macro-F1```                                                                                                 	| _anger, anticipation, disgust, fear, joy, love, optimism,  <br />pessimism, sadness, surprise, trust_                                                                                                                                                                                                                                                                                                              	|    [twitter-roberta-base-emotion-latest (base)](https://huggingface.co/cardiffnlp/twitter-roberta-base-latest-tweet-emotion) <br> [twitter-roberta-large-emotion-latest (large)](https://huggingface.co/cardiffnlp/twitter-roberta-large-latest-tweet-emotion)              	|





## Citation Information

### Main reference paper

Please cite the [reference paper](https://arxiv.org/abs/2310.14757) if you use this benchmark.

```bibtex
@inproceedings{antypas2023supertweeteval,
  title={SuperTweetEval: A Challenging, Unified and Heterogeneous Benchmark for Social Media NLP Research},
  author={Dimosthenis Antypas and Asahi Ushio and Francesco Barbieri and Leonardo Neves and Kiamehr Rezaee and Luis Espinosa-Anke and Jiaxin Pei and Jose Camacho-Collados},
  booktitle={Findings of the Association for Computational Linguistics: EMNLP 2023},
  year={2023}
}
```

### References of individual datasets

In addition to the main reference paper, please cite the individual task datasets included in SuperTweetEval if you use them.


- TweetTopic
```
@inproceedings{antypas-etal-2022-twitter,
    title = "{T}witter Topic Classification",
    author = "Antypas, Dimosthenis  and
      Ushio, Asahi  and
      Camacho-Collados, Jose  and
      Silva, Vitor  and
      Neves, Leonardo  and
      Barbieri, Francesco",
    booktitle = "Proceedings of the 29th International Conference on Computational Linguistics",
    month = oct,
    year = "2022",
    address = "Gyeongju, Republic of Korea",
    publisher = "International Committee on Computational Linguistics",
    url = "https://aclanthology.org/2022.coling-1.299",
    pages = "3386--3400",
    abstract = "Social media platforms host discussions about a wide variety of topics that arise everyday. Making sense of all the content and organising it into categories is an arduous task. A common way to deal with this issue is relying on topic modeling, but topics discovered using this technique are difficult to interpret and can differ from corpus to corpus. In this paper, we present a new task based on tweet topic classification and release two associated datasets. Given a wide range of topics covering the most important discussion points in social media, we provide training and testing data from recent time periods that can be used to evaluate tweet classification models. Moreover, we perform a quantitative evaluation and analysis of current general- and domain-specific language models on the task, which provide more insights on the challenges and nature of the task.",
}
```

- TweetNER7
```
@inproceedings{ushio-etal-2022-named,
    title = "Named Entity Recognition in {T}witter: A Dataset and Analysis on Short-Term Temporal Shifts",
    author = "Ushio, Asahi  and
      Barbieri, Francesco  and
      Sousa, Vitor  and
      Neves, Leonardo  and
      Camacho-Collados, Jose",
    booktitle = "Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)",
    month = nov,
    year = "2022",
    address = "Online only",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2022.aacl-main.25",
    pages = "309--319",
    abstract = "Recent progress in language model pre-training has led to important improvements in Named Entity Recognition (NER). Nonetheless, this progress has been mainly tested in well-formatted documents such as news, Wikipedia, or scientific articles. In social media the landscape is different, in which it adds another layer of complexity due to its noisy and dynamic nature. In this paper, we focus on NER in Twitter, one of the largest social media platforms, and construct a new NER dataset, TweetNER7, which contains seven entity types annotated over 11,382 tweets from September 2019 to August 2021. The dataset was constructed by carefully distributing the tweets over time and taking representative trends as a basis. Along with the dataset, we provide a set of language model baselines and perform an analysis on the language model performance on the task, especially analyzing the impact of different time periods. In particular, we focus on three important temporal aspects in our analysis: short-term degradation of NER models over time, strategies to fine-tune a language model over different periods, and self-labeling as an alternative to lack of recently-labeled data. TweetNER7 is released publicly (https://huggingface.co/datasets/tner/tweetner7) along with the models fine-tuned on it (NER models have been integrated into TweetNLP and can be found at https://github.com/asahi417/tner/tree/master/examples/tweetner7{\_}paper).",
}
```
- TweetQA
```
@inproceedings{xiong2019tweetqa,
  title={TweetQA: A Social Media Focused Question Answering Dataset},
  author={Xiong, Wenhan and Wu, Jiawei and Wang, Hong and Kulkarni, Vivek and Yu, Mo and Guo, Xiaoxiao and Chang, Shiyu and Wang, William Yang},
  booktitle={Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics},
  year={2019}
}
```

- TweetIntimacy
```
@misc{pei2023semeval,
      title={SemEval 2023 Task 9: Multilingual Tweet Intimacy Analysis}, 
      author={Jiaxin Pei and Vítor Silva and Maarten Bos and Yozon Liu and Leonardo Neves and David Jurgens and Francesco Barbieri},
      year={2023},
      eprint={2210.01108},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}
```

- Tweet Similarity
```
N/A
```

- TempoWiC
```
@inproceedings{loureiro-etal-2022-tempowic,
    title = "{T}empo{W}i{C}: An Evaluation Benchmark for Detecting Meaning Shift in Social Media",
    author = "Loureiro, Daniel  and
      D{'}Souza, Aminette  and
      Muhajab, Areej Nasser  and
      White, Isabella A.  and
      Wong, Gabriel  and
      Espinosa-Anke, Luis  and
      Neves, Leonardo  and
      Barbieri, Francesco  and
      Camacho-Collados, Jose",
    booktitle = "Proceedings of the 29th International Conference on Computational Linguistics",
    month = oct,
    year = "2022",
    address = "Gyeongju, Republic of Korea",
    publisher = "International Committee on Computational Linguistics",
    url = "https://aclanthology.org/2022.coling-1.296",
    pages = "3353--3359",
    abstract = "Language evolves over time, and word meaning changes accordingly. This is especially true in social media, since its dynamic nature leads to faster semantic shifts, making it challenging for NLP models to deal with new content and trends. However, the number of datasets and models that specifically address the dynamic nature of these social platforms is scarce. To bridge this gap, we present TempoWiC, a new benchmark especially aimed at accelerating research in social media-based meaning shift. Our results show that TempoWiC is a challenging benchmark, even for recently-released language models specialized in social media.",
}
```

- TweetHate
```
@inproceedings{sachdeva-etal-2022-measuring,
    title = "The Measuring Hate Speech Corpus: Leveraging Rasch Measurement Theory for Data Perspectivism",
    author = "Sachdeva, Pratik  and
      Barreto, Renata  and
      Bacon, Geoff  and
      Sahn, Alexander  and
      von Vacano, Claudia  and
      Kennedy, Chris",
    booktitle = "Proceedings of the 1st Workshop on Perspectivist Approaches to NLP @LREC2022",
    month = jun,
    year = "2022",
    address = "Marseille, France",
    publisher = "European Language Resources Association",
    url = "https://aclanthology.org/2022.nlperspectives-1.11",
    pages = "83--94",
    abstract = "We introduce the Measuring Hate Speech corpus, a dataset created to measure hate speech while adjusting for annotators{'} perspectives. It consists of 50,070 social media comments spanning YouTube, Reddit, and Twitter, labeled by 11,143 annotators recruited from Amazon Mechanical Turk. Each observation includes 10 ordinal labels: sentiment, disrespect, insult, attacking/defending, humiliation, inferior/superior status, dehumanization, violence, genocide, and a 3-valued hate speech benchmark label. The labels are aggregated using faceted Rasch measurement theory (RMT) into a continuous score that measures each comment{'}s location on a hate speech spectrum. The annotation experimental design assigned comments to multiple annotators in order to yield a linked network, allowing annotator disagreement (perspective) to be statistically summarized. Annotators{'} labeling strictness was estimated during the RMT scaling, projecting their perspective onto a linear measure that was adjusted for the hate speech score. Models that incorporate this annotator perspective parameter as an auxiliary input can generate label- and score-level predictions conditional on annotator perspective. The corpus includes the identity group targets of each comment (8 groups, 42 subgroups) and annotator demographics (6 groups, 40 subgroups), facilitating analyses of interactions between annotator- and comment-level identities, i.e. identity-related annotator perspective.",
}
```

- TweetEmoji
```
N/A
```

- TweetSentiment
```
@inproceedings{rosenthal-etal-2017-semeval,
    title = "{S}em{E}val-2017 Task 4: Sentiment Analysis in {T}witter",
    author = "Rosenthal, Sara  and
      Farra, Noura  and
      Nakov, Preslav",
    booktitle = "Proceedings of the 11th International Workshop on Semantic Evaluation ({S}em{E}val-2017)",
    month = aug,
    year = "2017",
    address = "Vancouver, Canada",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/S17-2088",
    doi = "10.18653/v1/S17-2088",
    pages = "502--518",
    abstract = "This paper describes the fifth year of the Sentiment Analysis in Twitter task. SemEval-2017 Task 4 continues with a rerun of the subtasks of SemEval-2016 Task 4, which include identifying the overall sentiment of the tweet, sentiment towards a topic with classification on a two-point and on a five-point ordinal scale, and quantification of the distribution of sentiment towards a topic across a number of tweets: again on a two-point and on a five-point ordinal scale. Compared to 2016, we made two changes: (i) we introduced a new language, Arabic, for all subtasks, and (ii) we made available information from the profiles of the Twitter users who posted the target tweets. The task continues to be very popular, with a total of 48 teams participating this year.",
}
```

- TweetNERD
```
@article{mishra2022tweetnerd,
  title={TweetNERD--End to End Entity Linking Benchmark for Tweets},
  author={Mishra, Shubhanshu and Saini, Aman and Makki, Raheleh and Mehta, Sneha and Haghighi, Aria and Mollahosseini, Ali},
  journal={arXiv preprint arXiv:2210.08129},
  year={2022}
}
```

- TweetEmotion
```
@inproceedings{mohammad-etal-2018-semeval,
    title = "{S}em{E}val-2018 Task 1: Affect in Tweets",
    author = "Mohammad, Saif  and
      Bravo-Marquez, Felipe  and
      Salameh, Mohammad  and
      Kiritchenko, Svetlana",
    booktitle = "Proceedings of the 12th International Workshop on Semantic Evaluation",
    month = jun,
    year = "2018",
    address = "New Orleans, Louisiana",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/S18-1001",
    doi = "10.18653/v1/S18-1001",
    pages = "1--17",
    abstract = "We present the SemEval-2018 Task 1: Affect in Tweets, which includes an array of subtasks on inferring the affectual state of a person from their tweet. For each task, we created labeled data from English, Arabic, and Spanish tweets. The individual tasks are: 1. emotion intensity regression, 2. emotion intensity ordinal classification, 3. valence (sentiment) regression, 4. valence ordinal classification, and 5. emotion classification. Seventy-five teams (about 200 team members) participated in the shared task. We summarize the methods, resources, and tools used by the participating teams, with a focus on the techniques and resources that are particularly useful. We also analyze systems for consistent bias towards a particular race or gender. The data is made freely available to further improve our understanding of how people convey emotions through language.",
}
```
task_categories:text-classification task_categories:token-classification task_categories:question-answering task_categories:other task_ids:topic-classification task_ids:named-entity-recognition task_ids:abstractive-qa annotations_creators:expert-generated multilinguality:monolingual size_categories:n<50K source_datasets:extended|other language:en license:unknown super_tweet_eval tweet_eval natural language understanding arxiv:2310.14757 arxiv:2210.01108 region:us
{"annotations_creators": ["expert-generated"], "language": ["en"], "license": ["unknown"], "multilinguality": ["monolingual"], "size_categories": ["n<50K"], "source_datasets": ["extended|other"], "task_categories": ["text-classification", "token-classification", "question-answering", "other"], "task_ids": ["topic-classification", "named-entity-recognition", "abstractive-qa"], "pretty_name": "SuperTweetEval", "tags": ["super_tweet_eval", "tweet_eval", "natural language understanding"]}
TAGS
#task_categories-text-classification #task_categories-token-classification #task_categories-question-answering #task_categories-other #task_ids-topic-classification #task_ids-named-entity-recognition #task_ids-abstractive-qa #annotations_creators-expert-generated #multilinguality-monolingual #size_categories-n<50K #source_datasets-extended|other #language-English #license-unknown #super_tweet_eval #tweet_eval #natural language understanding #arxiv-2310.14757 #arxiv-2210.01108 #region-us 

SuperTweetEval
==============


Dataset Card for "super\_tweeteval"
===================================


### Dataset Summary


This is the oficial repository for SuperTweetEval, a unified benchmark of 12 heterogeneous NLP tasks.
More details on the task and an evaluation of language models can be found on the reference paper, published in EMNLP 2023 (Findings).


### Data Splits


All tasks provide custom training, validation and test splits.



Dataset Structure
-----------------


### Data Fields


The data fields are unified among all splits.
In the following we present the information contained in each of the datasets.


#### tweet\_topic


* 'text': a 'string' feature.
* 'gold\_label\_list': a list of 'string' feature.
* 'date': a 'string' feature.


#### tweet\_ner7


* 'text': a 'string' feature.
* 'text\_tokenized': a list of 'string' feature.
* 'gold\_label\_sequence': a list of 'string' feature.
* 'date': a 'string' feature.
* 'entities': a list of 'dictionary' feature containing '{"entity": "string", "type": "string"}'.


#### tweet\_qa


* 'text': a 'string' feature.
* 'gold\_label\_str': a 'string' feature.
* 'context': a 'string' feature.


#### tweet\_qg


* 'text': a 'string' feature.
* 'gold\_label\_str': a 'string' feature.
* 'context': a 'string' feature.


#### tweet\_intimacy


* 'text': a 'string' feature.
* 'gold\_score': a 'float' feature.


#### tweet\_similarity


* 'text\_1': a 'string' feature.
* 'text\_2': a 'string' feature.
* 'gold\_score': a 'float' feature.


#### tempo\_wic


* 'gold\_label\_binary': a 'int' feature.
* 'target': a 'string' feature.
* 'text\_1': a 'string' feature.
* 'text\_tokenized\_1': a list of 'string' feature.
* 'token\_idx\_1': a 'int' feature.
* 'date\_1': a 'string' feature.
* 'text\_2': a 'string' feature.
* 'text\_tokenized\_2': a list of 'string' feature.
* 'token\_idx\_2': a 'int' feature.
* 'date\_2': a 'string' feature.


#### tweet\_hate


* 'gold\_label': a 'int' feature.
* 'text': a 'string' feature.


#### tweet\_emoji


* 'gold\_label': a 'int' feature.
* 'text': a 'string' feature.
* 'date': a 'string' feature.


#### tweet\_sentiment


* 'gold\_label': a 'int' feature.
* 'text': a 'string' feature.
* 'target': a 'string' feature.


#### tweet\_nerd


* 'gold\_label\_binary': a 'int' feature.
* 'target': a 'string' feature.
* 'text': a 'string' feature.
* 'definition': a 'string' feature.
* 'text\_start': a 'int' feature.
* 'text\_end': a 'int' feature.
* 'date': a 'string' feature.


#### tweet\_emotion


* 'text': a 'string' feature.
* 'gold\_label\_list': a list of 'string' feature.


Evaluation metrics & Models
---------------------------



### Main reference paper


Please cite the reference paper if you use this benchmark.


### References of individual datasets


In addition to the main reference paper, please cite the individual task datasets included in SuperTweetEval if you use them.


* TweetTopic
* TweetNER7
* TweetQA
* TweetIntimacy
* Tweet Similarity
* TempoWiC
* TweetHate
* TweetEmoji
* TweetSentiment
* TweetNERD
* TweetEmotion



### Dataset Summary


This is the oficial repository for SuperTweetEval, a unified benchmark of 12 heterogeneous NLP tasks.
More details on the task and an evaluation of language models can be found on the reference paper, published in EMNLP 2023 (Findings). ### Data Splits


All tasks provide custom training, validation and test splits.



Dataset Structure
----------------- ### Data Fields


The data fields are unified among all splits.
In the following we present the information contained in each of the datasets. #### tweet\_topic


* 'text': a 'string' feature.
* 'gold\_label\_list': a list of 'string' feature.
* 'date': a 'string' feature. #### tweet\_ner7


* 'text': a 'string' feature.
* 'text\_tokenized': a list of 'string' feature.
* 'gold\_label\_sequence': a list of 'string' feature.
* 'date': a 'string' feature.
* 'entities': a list of 'dictionary' feature containing '{"entity": "string", "type": "string"}'. #### tweet\_qa


* 'text': a 'string' feature.
* 'gold\_label\_str': a 'string' feature.
* 'context': a 'string' feature. #### tweet\_qg


* 'text': a 'string' feature.
* 'gold\_label\_str': a 'string' feature.
* 'context': a 'string' feature. #### tweet\_intimacy


* 'text': a 'string' feature.
* 'gold\_score': a 'float' feature. #### tweet\_similarity


* 'text\_1': a 'string' feature.
* 'text\_2': a 'string' feature.
* 'gold\_score': a 'float' feature. #### tempo\_wic


* 'gold\_label\_binary': a 'int' feature.
* 'target': a 'string' feature.
* 'text\_1': a 'string' feature.
* 'text\_tokenized\_1': a list of 'string' feature.
* 'token\_idx\_1': a 'int' feature.
* 'date\_1': a 'string' feature.
* 'text\_2': a 'string' feature.
* 'text\_tokenized\_2': a list of 'string' feature.
* 'token\_idx\_2': a 'int' feature.
* 'date\_2': a 'string' feature. #### tweet\_hate


* 'gold\_label': a 'int' feature.
* 'text': a 'string' feature. #### tweet\_emoji


* 'gold\_label': a 'int' feature.
* 'text': a 'string' feature.
* 'date': a 'string' feature. #### tweet\_sentiment


* 'gold\_label': a 'int' feature.
* 'text': a 'string' feature.
* 'target': a 'string' feature. #### tweet\_nerd


* 'gold\_label\_binary': a 'int' feature.
* 'target': a 'string' feature.
* 'text': a 'string' feature.
* 'definition': a 'string' feature.
* 'text\_start': a 'int' feature.
* 'text\_end': a 'int' feature.
* 'date': a 'string' feature. #### tweet\_emotion


* 'text': a 'string' feature.
* 'gold\_label\_list': a list of 'string' feature.


Evaluation metrics & Models
--------------------------- ### Main reference paper


Please cite the reference paper if you use this benchmark. ### References of individual datasets


In addition to the main reference paper, please cite the individual task datasets included in SuperTweetEval if you use them.


* TweetTopic
* TweetNER7
* TweetQA
* TweetIntimacy
* Tweet Similarity
* TempoWiC
* TweetHate
* TweetEmoji
* TweetSentiment
* TweetNERD
* TweetEmotion
TAGS
#task_categories-text-classification #task_categories-token-classification #task_categories-question-answering #task_categories-other #task_ids-topic-classification #task_ids-named-entity-recognition #task_ids-abstractive-qa #annotations_creators-expert-generated #multilinguality-monolingual #size_categories-n<50K #source_datasets-extended|other #language-English #license-unknown #super_tweet_eval #tweet_eval #natural language understanding #arxiv-2310.14757 #arxiv-2210.01108 #region-us 
 ### Dataset Summary


This is the oficial repository for SuperTweetEval, a unified benchmark of 12 heterogeneous NLP tasks.
More details on the task and an evaluation of language models can be found on the reference paper, published in EMNLP 2023 (Findings). ### Data Splits


All tasks provide custom training, validation and test splits.



Dataset Structure
----------------- ### Data Fields


The data fields are unified among all splits.
In the following we present the information contained in each of the datasets. #### tweet\_topic


* 'text': a 'string' feature.
* 'gold\_label\_list': a list of 'string' feature.
* 'date': a 'string' feature. #### tweet\_ner7


* 'text': a 'string' feature.
* 'text\_tokenized': a list of 'string' feature.
* 'gold\_label\_sequence': a list of 'string' feature.
* 'date': a 'string' feature.
* 'entities': a list of 'dictionary' feature containing '{"entity": "string", "type": "string"}'. #### tweet\_qa


* 'text': a 'string' feature.
* 'gold\_label\_str': a 'string' feature.
* 'context': a 'string' feature. #### tweet\_qg


* 'text': a 'string' feature.
* 'gold\_label\_str': a 'string' feature.
* 'context': a 'string' feature. #### tweet\_intimacy


* 'text': a 'string' feature.
* 'gold\_score': a 'float' feature. #### tweet\_similarity


* 'text\_1': a 'string' feature.
* 'text\_2': a 'string' feature.
* 'gold\_score': a 'float' feature. #### tempo\_wic


* 'gold\_label\_binary': a 'int' feature.
* 'target': a 'string' feature.
* 'text\_1': a 'string' feature.
* 'text\_tokenized\_1': a list of 'string' feature.
* 'token\_idx\_1': a 'int' feature.
* 'date\_1': a 'string' feature.
* 'text\_2': a 'string' feature.
* 'text\_tokenized\_2': a list of 'string' feature.
* 'token\_idx\_2': a 'int' feature.
* 'date\_2': a 'string' feature. #### tweet\_hate


* 'gold\_label': a 'int' feature.
* 'text': a 'string' feature. #### tweet\_emoji


* 'gold\_label': a 'int' feature.
* 'text': a 'string' feature.
* 'date': a 'string' feature. #### tweet\_sentiment


* 'gold\_label': a 'int' feature.
* 'text': a 'string' feature.
* 'target': a 'string' feature. #### tweet\_nerd


* 'gold\_label\_binary': a 'int' feature.
* 'target': a 'string' feature.
* 'text': a 'string' feature.
* 'definition': a 'string' feature.
* 'text\_start': a 'int' feature.
* 'text\_end': a 'int' feature.
* 'date': a 'string' feature. #### tweet\_emotion


* 'text': a 'string' feature.
* 'gold\_label\_list': a list of 'string' feature.


Evaluation metrics & Models
--------------------------- ### Main reference paper


Please cite the reference paper if you use this benchmark. ### References of individual datasets


In addition to the main reference paper, please cite the individual task datasets included in SuperTweetEval if you use them.


* TweetTopic
* TweetNER7
* TweetQA
* TweetIntimacy
* Tweet Similarity
* TempoWiC
* TweetHate
* TweetEmoji
* TweetSentiment
* TweetNERD
* TweetEmotion