offenseval
84843035
SemEval 2019 Task 6: Identifying and Categorizing Offensive Language in Social Media (OffensEval)
We present the results and the main findings of SemEval 2019 Task 6 on Identifying and Categorizing Offensive Language in Social Media (OffensEval) The task was based on a new dataset, the Offensive Language Identification Dataset (OLID) which contains over 14,000 English tweets, and it featured three sub tasks. In sub task A, systems were asked to discriminate between offensive and non offensive posts. In sub task B, systems had to identify the type of offensive content in the post. Finally, in sub task C, systems had to detect the target of the offensive posts. OffensEval attracted a large number of participants and it was one of the most popular tasks in SemEval 2019. In total, nearly 800 teams signed up to participate in the task and 115 of them submitted results, which are presented and analyzed in this report.
*SEMEVAL
Marcos Zampieri Shervin Malmasi Preslav Nakov Sara Rosenthal Noura Farra Ritesh Kumar
81978051
OffensEval at SemEval 2019 Task 6: Okham's Razor on Identifying and Categorizing Offensive Language in Social Media
This document describes our approach to building an Offensive Language Classifier. More specifically, the OffensEval 2019(Zampieri et al. 2019b) competition required us to build three classifiers with slightly different goals: 1. Offensive language identification: would classify a tweet as offensive or not. 2. Automatic categorization of offense types: would recognize if the target of the offense was an individual or not. 3. Offense target identification: would identify the target of the offense between an individual, group or other. In this report, we will discuss the different architectures, algorithms and pre processing strategies we tried, together with a detailed description of the designs of our final classifiers and the reasons we choose them over others. We evaluated our classifiers on the official test set provided for the OffenseEval 2019 competition(Zampieri et al. 2019a) obtaining a macro averaged F1 score of 0.7189 for Task A, 0.6708 on Task B and 0.5442 on Task C.
Silvia Sapora Bogdan Lazarescu Christo Lolov
203111720
OffensEval at SemEval 2018 Task 6: Identifying and Categorizing Offensive Language in Social Media
This document describes our approach to building an Offensive Language Classifier. More specifically, the OffensEval 2019 competition required us to build three classifiers with slightly different goals: Offensive language identification: would classify a tweet as offensive or not. Automatic categorization of offense types: would recognize if the target of the offense was an individual or not. Offense target identification: would identify the target of the offense between an individual, group or other. In this report, we will discuss the different architectures, algorithms and pre processing strategies we tried, together with a detailed description of the designs of our final classifiers and the reasons we choose them over others. We evaluated our classifiers on the official test set provided for the OffenseEval 2019 competition, obtaining a macro averaged F1 score of 0.7189 for Task A, 0.6708 on Task B and 0.5442 on Task C.
Silvia Sapora Bogdan Lazarescu Christo Lolov
52967399
BERT: Pre training of Deep Bidirectional Transformers for Language Understanding
We introduce a new language representation model called BERT, which stands for Bidirectional Encoder Representations from Transformers. Unlike recent language representation models (Peters et al. 2018a; Radford et al. 2018) BERT is designed to pre train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers. As a result, the pre trained BERT model can be fine tuned with just one additional output layer to create state of the art models for a wide range of tasks, such as question answering and language inference, without substantial task specific architecture modifications. BERT is conceptually simple and empirically powerful. It obtains new state of the art results on eleven natural language processing tasks, including pushing the GLUE score to 80.5 (7.7 point absolute improvement) MultiNLI accuracy to 86.7% (4.6% absolute improvement) SQuAD v1.1 question answering Test F1 to 93.2 (1.5 point absolute improvement) and SQuAD v2.0 Test F1 to 83.1 (5.1 point absolute improvement)
NAACL
Jacob Devlin Ming-Wei Chang Kenton Lee Kristina Toutanova
3626819
Deep Contextualized Word Representations
We introduce a new type of deep contextualized word representation that models both (1) complex characteristics of word use (e.g. syntax and semantics) and (2) how these uses vary across linguistic contexts (i.e. to model polysemy) Our word vectors are learned functions of the internal states of a deep bidirectional language model (biLM) which is pre trained on a large text corpus. We show that these representations can be easily added to existing models and significantly improve the state of the art across six challenging NLP problems, including question answering, textual entailment and sentiment analysis. We also present an analysis showing that exposing the deep internals of the pre trained network is crucial, allowing downstream models to mix different types of semi supervision signals.
NAACL
Matthew E Peters Mark Neumann Mohit Iyyer Matt Gardner Christopher Clark Kenton Lee Luke Zettlemoyer
67856299
Predicting the Type and Target of Offensive Posts in Social Media
As offensive content has become pervasive in social media, there has been much research in identifying potentially offensive messages. However, previous work on this topic did not consider the problem as a whole, but rather focused on detecting very specific types of offensive content, e.g. hate speech, cyberbulling, or cyber aggression. In contrast, here we target several different kinds of offensive content. In particular, we model the task hierarchically, identifying the type and the target of offensive messages in social media. For this purpose, we complied the Offensive Language Identification Dataset (OLID) a new dataset with tweets annotated for offensive content using a fine grained three layer annotation scheme, which we make publicly available. We discuss the main similarities and differences between OLID and pre existing datasets for hate speech identification, aggression detection, and similar tasks. We further experiment with and we compare the performance of different machine learning models on OLID.
NAACL
Marcos Zampieri Shervin Malmasi Preslav Nakov Sara Rosenthal Noura Farra Ritesh Kumar
184483123
SemEval 2019 Task 5: Multilingual Detection of Hate Speech Against Immigrants and Women in Twitter
The paper describes the organization of the SemEval 2019 Task 5 about the detection of hate speech against immigrants and women in Spanish and English messages extracted from Twitter. The task is organized in two related classification subtasks: a main binary subtask for detecting the presence of hate speech, and a finer grained one devoted to identifying further features in hateful contents such as the aggressive attitude and the target harassed, to distinguish if the incitement is against an individual rather than a group. HatEval has been one of the most popular tasks in SemEval 2019 with a total of 108 submitted runs for Subtask A and 70 runs for Subtask B, from a total of 74 different teams. Data provided for the task are described by showing how they have been collected and annotated. Moreover, the paper provides an analysis and discussion about the participant systems and the results they achieved in both subtasks.
*SEMEVAL
Valerio Basile Cristina Bosco Elisabetta Fersini Debora Nozza Viviana Patti Francisco Manuel Rangel Pardo Paolo Rosso Manuela Sanguinetti
209393462
Overview of the HASOC track at FIRE 2019: Hate Speech and Offensive Content Identification in Indo European Languages
The identification of Hate Speech in Social Media is of great importance and receives much attention in the text classification community. There is a huge demand for research for languages other than English. The HASOC track intends to stimulate development in Hate Speech for Hindi, German and English. Three datasets were developed from Twitter and Facebook and made available. Binary classification and more fine grained subclasses were offered in 3 subtasks. For all subtasks, 321 experiments were submitted. The approaches used most often were LSTM networks processing word embedding input. The performance of the best system for identification of Hate Speech for English, Hindi, and German was a Marco F1 score of 0.78, 0.81 and 0.61, respectively.
FIRE
Thomas Mandl Sandip Modha Prasenjit Majumder Daksh Patel Mohana Dave Chintak Mandalia Aditya Patel
174799974
Detection of Abusive Language: the Problem of Biased Datasets
We discuss the impact of data bias on abusive language detection. We show that classification scores on popular datasets reported in previous work are much lower under realistic settings in which this bias is reduced. Such biases are most notably observed on datasets that are created by focused sampling instead of random sampling. Datasets with a higher proportion of implicit abuse are more affected than datasets with a lower proportion.
NAACL
Michael Wiegand Josef Ruppenhofer Thomas Kleinbauer
207917760
Overview of the GermEval 2018 Shared Task on the Identification of Offensive Language
We present the pilot edition of the GermEval Shared Task on the Identification of Offensive Language. This shared task deals with the classification of German tweets from Twitter. It comprises two tasks, a coarse grained binary classification task and a fine grained multi class classification task. The shared task had 20 participants submitting 51 runs for the coarse grained task and 25 runs for the fine grained task. Since this is a pilot task, we describe the process of extracting the raw data for the data collection and the annotation schema. We evaluate the results of the systems submitted to the shared task. The shared task homepage can be found at https:/projects.cai. fbi.h da.de/iggsa/
Michael Wiegand Melanie Siegel