task_categories:text-classification task_ids:sentiment-classification task_ids:hate-speech-detection size_categories:10K<n<100K language:en region:us
---
task_categories:
- text-classification
language:
- en
task_ids:
- sentiment-classification
- hate-speech-detection
size_categories:
- 10K<n<100K
---

# Tweet Annotations in Five Experimental Conditions 

## Description

The dataset contains tweet data annotations of **hate speech** (HS) and **offensive language** (OL) in five experimental conditions. The tweet data was sampled from the corpus created by [Davidson et al. (2017)](https://ojs.aaai.org/index.php/ICWSM/article/view/14955). We selected 3,000 Tweets for our annotation. We developed five experimental conditions that varied the annotation task structure, as shown in the following figure. All tweets were annotated in each condition.

- **<font color= #871F78>Condition A</font>** presented the tweet and three options on a single screen: hate speech, offensive language, or neither. Annotators could select one or both of hate speech, offensive language, or indicate that neither applied.

- Conditions B and C split the annotation of a single tweet across two screens.
  + For **<font color= Blue>Condition B</font>**, the first screen prompted the annotator to indicate whether the tweet contained hate speech. On the following screen, they were shown the tweet again and asked whether it contained offensive language.
  + **<font color= red>Condition C</font>** was similar to Condition B, but flipped the order of hate speech and offensive language for each tweet. 

- In Conditions D and E, the two tasks are treated independently with annotators being asked to first annotate all tweets for one task, followed by annotating all tweets again for the second task.
  + Annotators assigned **<font color=green>Condition D</font>** were first asked to annotate hate speech for all their assigned tweets, and then asked to annotate offensive language for the same set of tweets.
  + **Condition E** worked the same way, but started with the offensive language annotation task followed by the hate speech annotation task.

We recruited US-based annotators from the crowdsourcing platform [Prolific](https://www.prolific.com/) during November and December 2022. Each annotator annotated up to 50 tweets. The dataset also contains demographic information about the annotators. Annotators received a fixed hourly wage in excess of the US federal minimum wage after completing the task.

<img src="https://raw.githubusercontent.com/chkern/tweet-annotation-sensitivity/main/fig/exp_conditions.png" width = "300" height = "200" alt="" align=center />


## Dataset Structure

| Column Name     | Description       | Type       |
| -------------- | ------------------ |---------------- |
| case_id  | case ID | integer |
| duration_seconds  | duration of connection to task in seconds | integer |
| last_screen  | last question answered  | factor |
| device  | device type  | factor |
| ethn_hispanic  | Hispanic race/ethnicity  | binary |
| ethn_white  | White race/ethnicity  | binary |
| ethn_afr_american | African-American race/ethnicity  | binary |
| ethn_asian | Asian race/ethnicity  | binary |
| ethn_sth_else | race/ethnicity something else  | binary |
| ethn_prefer_not | race/ethnicity prefer not to say  | binary |
| age | age  | integer |
| education | education attainment  <br>1: Less than high school <br>2: High school <br>3: Some college <br>4: College graduate <br>5: Master's degree or professional degree (law, medicine, MPH, etc.) <br>6: Doctoral degree (PhD, DPH, EdD, etc.)| factor |
| english_fl | English as first language   | binary |
| twitter_use | Twitter use frequency  <br>1: Most days <br>2: Most weeks, but not every day <br>3: A few times a month <br>4: A few times a year <br>5: Less often <br>6: Never  | factor |
| socmedia_use | social media use frequency  <br>1: Most days <br>2: Most weeks, but not every day <br>3: A few times a month <br>4: A few times a year <br>5: Less often <br>6: Never  | factor |
| prolific_hours  | workload on the platform prolific in hours in the last month | integer |
| task_fun  | task perception: fun | binary |
| task_interesting  | task perception: interesting | binary |
| task_boring  | task perception: boring | binary |
| task_repetitive  | task perception: repetitive | binary |
| task_important  | task perception: important | binary |
| task_depressing  | task perception: depressing | binary |
| task_offensive  | task perception: offensive | binary |
| repeat_tweet_coding  | likelihood for another tweet task <br>1: Not at all likely <br>2: Somewhat likely <br>3: Very likely | factor |
| repeat_hs_coding  | likelihood for another hate speech task <br>1: Not at all likely <br>2: Somewhat likely <br>3: Very likely | factor |
| target_online_harassment  | targeted by hateful online behavior | binary |
| target_other_harassment  | targeted by other hateful behavior | binary |
| party_affiliation  | party identification <br>1: Republican <br>2: Democrat <br>3: Independent  | factor |
| societal_relevance_hs  | relevance perception of hate speech  <br>1: Not at all likely <br>2: Somewhat likely <br>3: Very likely | factor |
| annotator_id  | annotator ID | integer |
| condition  | experimental conditions (A-E)  | factor |
| tweet_batch  | tweet ID in batch  | factor |
| hate_speech  | hate speech annotation  | logical |
| offensive_language  | offensive language annotation  | logical |
| tweet_id  | tweet ID  | integer |
| orig_label_hs  | number of persons who annotated the tweet as hate speech in the original dataset from [Davidson et al. (2017)](https://ojs.aaai.org/index.php/ICWSM/article/view/14955)  | integer |
| orig_label_ol  | number of persons who annotated the tweet as offensive language in the original dataset from [Davidson et al. (2017)](https://ojs.aaai.org/index.php/ICWSM/article/view/14955)  | integer |
| orig_label_ne  | number of persons who annotated the tweet as neither in the original dataset from [Davidson et al. (2017)](https://ojs.aaai.org/index.php/ICWSM/article/view/14955)  | integer |
| tweet_hashed  | tweet with usernames hashed  | character |




## Citation

If you find the dataset useful, please cite:
```
@inproceedings{kern-et-al-2023-annotation,
    title = "Annotation Sensitivity: Training Data Collection Methods Affect Model Performance",
    author = "Christoph Kern and Stephanie Eckman and Jacob Beck and Rob Chew and Bolei Ma and Frauke Kreuter",
    booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2023",
    month = "dec",
    year = "2023",
    address = "Singapore",
    publisher = "Association for Computational Linguistics"
}
```

