@inproceedings{ghanbarzadeh-etal-2023-gender,
    title = "Gender-tuning: Empowering Fine-tuning for Debiasing Pre-trained Language Models",
    author = "Ghanbarzadeh, Somayeh  and
      Huang, Yan  and
      Palangi, Hamid  and
      Cruz Moreno, Radames  and
      Khanpour, Hamed",
    editor = "Rogers, Anna  and
      Boyd-Graber, Jordan  and
      Okazaki, Naoaki",
    booktitle = "Findings of the Association for Computational Linguistics: ACL 2023",
    month = jul,
    year = "2023",
    address = "Toronto, Canada",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.findings-acl.336",
    doi = "10.18653/v1/2023.findings-acl.336",
    pages = "5448--5458",
    abstract = "Recent studies have revealed that the widely-used Pre-trained Language Models (PLMs) propagate societal biases from the large unmoderated pre-training corpora. Existing solutions require debiasing training processes and datasets for debiasing, which are resource-intensive and costly. Furthermore, these methods hurt the PLMs{'} performance on downstream tasks. In this study, we propose Gender-tuning, which debiases the PLMs through fine-tuning on downstream tasks{'} datasets. For this aim, Gender-tuning integrates Masked Language Modeling (MLM) training objectives into fine-tuning{'}s training process. Comprehensive experiments show that Gender-tuning outperforms the state-of-the-art baselines in terms of average gender bias scores in PLMs while improving PLMs{'} performance on downstream tasks solely using the downstream tasks{'} dataset. Also, Gender-tuning is a deployable debiasing tool for any PLM that works with original fine-tuning.",
}

Recent studies have revealed that the widely-used Pre-trained Language Models (PLMs) propagate societal biases from the large unmoderated pre-training corpora. Existing solutions require debiasing training processes and datasets for debiasing, which are resource-intensive and costly. Furthermore, these methods hurt the PLMs{'} performance on downstream tasks. In this study, we propose Gender-tuning, which debiases the PLMs through fine-tuning on downstream tasks{'} datasets. For this aim, Gender-tuning integrates Masked Language Modeling (MLM) training objectives into fine-tuning{'}s training process. Comprehensive experiments show that Gender-tuning outperforms the state-of-the-art baselines in terms of average gender bias scores in PLMs while improving PLMs{'} performance on downstream tasks solely using the downstream tasks{'} dataset. Also, Gender-tuning is a deployable debiasing tool for any PLM that works with original fine-tuning.
Ghanbarzadeh, Somayeh Huang, Yan Palangi, Hamid Cruz Moreno, Radames Khanpour, Hamed
Gender-tuning: Empowering Fine-tuning for Debiasing Pre-trained Language Models