@inproceedings{ghosh-etal-2023-radling,
    title = "{R}ad{L}ing: Towards Efficient Radiology Report Understanding",
    author = "Ghosh, Rikhiya  and
      Farri, Oladimeji  and
      Karn, Sanjeev Kumar  and
      Danu, Manuela  and
      Vunikili, Ramya  and
      Micu, Larisa",
    editor = "Sitaram, Sunayana  and
      Beigman Klebanov, Beata  and
      Williams, Jason D",
    booktitle = "Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 5: Industry Track)",
    month = jul,
    year = "2023",
    address = "Toronto, Canada",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.acl-industry.61",
    doi = "10.18653/v1/2023.acl-industry.61",
    pages = "640--651",
    abstract = "Most natural language tasks in the radiology domain use language models pre-trained on biomedical corpus. There are few pretrained language models trained specifically for radiology, and fewer still that have been trained in a low data setting and gone on to produce comparable results in fine-tuning tasks. We present RadLing, a continuously pretrained language model using ELECTRA-small architecture, trained using over 500K radiology reports that can compete with state-of-the-art results for fine tuning tasks in radiology domain. Our main contribution in this paper is knowledge-aware masking which is an taxonomic knowledge-assisted pre-training task that dynamically masks tokens to inject knowledge during pretraining. In addition, we also introduce an knowledge base-aided vocabulary extension to adapt the general tokenization vocabulary to radiology domain.",
}

Most natural language tasks in the radiology domain use language models pre-trained on biomedical corpus. There are few pretrained language models trained specifically for radiology, and fewer still that have been trained in a low data setting and gone on to produce comparable results in fine-tuning tasks. We present RadLing, a continuously pretrained language model using ELECTRA-small architecture, trained using over 500K radiology reports that can compete with state-of-the-art results for fine tuning tasks in radiology domain. Our main contribution in this paper is knowledge-aware masking which is an taxonomic knowledge-assisted pre-training task that dynamically masks tokens to inject knowledge during pretraining. In addition, we also introduce an knowledge base-aided vocabulary extension to adapt the general tokenization vocabulary to radiology domain.
Ghosh, Rikhiya Farri, Oladimeji Karn, Sanjeev Kumar Danu, Manuela Vunikili, Ramya Micu, Larisa
RadLing: Towards Efficient Radiology Report Understanding