@inproceedings{varadarajan-etal-2023-transfer,
    title = "Transfer and Active Learning for Dissonance Detection: Addressing the Rare-Class Challenge",
    author = "Varadarajan, Vasudha  and
      Juhng, Swanie  and
      Mahwish, Syeda  and
      Liu, Xiaoran  and
      Luby, Jonah  and
      Luhmann, Christian  and
      Schwartz, H. Andrew",
    editor = "Rogers, Anna  and
      Boyd-Graber, Jordan  and
      Okazaki, Naoaki",
    booktitle = "Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2023",
    address = "Toronto, Canada",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.acl-long.665",
    doi = "10.18653/v1/2023.acl-long.665",
    pages = "11923--11936",
    abstract = "While transformer-based systems have enabled greater accuracies with fewer training examples, data acquisition obstacles still persist for rare-class tasks {--} when the class label is very infrequent (e.g. {\textless} 5{\%} of samples). Active learning has in general been proposed to alleviate such challenges, but choice of selection strategy, the criteria by which rare-class examples are chosen, has not been systematically evaluated. Further, transformers enable iterative transfer-learning approaches. We propose and investigate transfer- and active learning solutions to the rare class problem of dissonance detection through utilizing models trained on closely related tasks and the evaluation of acquisition strategies, including a proposed probability-of-rare-class (PRC) approach. We perform these experiments for a specific rare-class problem: collecting language samples of cognitive dissonance from social media. We find that PRC is a simple and effective strategy to guide annotations and ultimately improve model accuracy while transfer-learning in a specific order can improve the cold-start performance of the learner but does not benefit iterations of active learning.",
}

While transformer-based systems have enabled greater accuracies with fewer training examples, data acquisition obstacles still persist for rare-class tasks {--} when the class label is very infrequent (e.g. {\textless} 5{\%} of samples). Active learning has in general been proposed to alleviate such challenges, but choice of selection strategy, the criteria by which rare-class examples are chosen, has not been systematically evaluated. Further, transformers enable iterative transfer-learning approaches. We propose and investigate transfer- and active learning solutions to the rare class problem of dissonance detection through utilizing models trained on closely related tasks and the evaluation of acquisition strategies, including a proposed probability-of-rare-class (PRC) approach. We perform these experiments for a specific rare-class problem: collecting language samples of cognitive dissonance from social media. We find that PRC is a simple and effective strategy to guide annotations and ultimately improve model accuracy while transfer-learning in a specific order can improve the cold-start performance of the learner but does not benefit iterations of active learning.
Varadarajan, Vasudha Juhng, Swanie Mahwish, Syeda Liu, Xiaoran Luby, Jonah Luhmann, Christian Schwartz, H. Andrew
Transfer and Active Learning for Dissonance Detection: Addressing the Rare-Class Challenge