@inproceedings{sileo-moens-2023-probing,
    title = "Probing neural language models for understanding of words of estimative probability",
    author = "Sileo, Damien  and
      Moens, Marie-francine",
    editor = "Palmer, Alexis  and
      Camacho-collados, Jose",
    booktitle = "Proceedings of the 12th Joint Conference on Lexical and Computational Semantics (*SEM 2023)",
    month = jul,
    year = "2023",
    address = "Toronto, Canada",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.starsem-1.41",
    doi = "10.18653/v1/2023.starsem-1.41",
    pages = "469--476",
    abstract = "Words of Estimative Probability (WEP) are phrases used to express the plausibility of a statement. Examples include terms like {\textbackslash}textit{probably, maybe, likely, doubt, unlikely}, and {\textbackslash}textit{impossible}. Surveys have shown that human evaluators tend to agree when assigning numerical probability levels to these WEPs. For instance, the term {\textbackslash}textit{highly likely} equates to a median probability of {\$}0.90{{\textbackslash}pm}0.08{\$} according to a survey by {\textbackslash}citet{fagen-ulmschneider}.In this study, our focus is to gauge the competency of neural language processing models in accurately capturing the consensual probability level associated with each WEP. Our first approach is utilizing the UNLI dataset {\textbackslash}cite{chen-etal-2020-uncertain}, which links premises and hypotheses with their perceived joint probability {\$}p{\$}. From this, we craft prompts in the form: ''[{\textbackslash}textsc{Premise}]. [{\textbackslash}textsc{Wep}], [{\textbackslash}textsc{Hypothesis}].{''} This allows us to evaluate whether language models can predict if the consensual probability level of a WEP aligns closely with {\$}p{\$}.In our second approach, we develop a dataset based on WEP-focused probabilistic reasoning to assess if language models can logically process WEP compositions. For example, given the prompt ''[{\textbackslash}textsc{EventA}] {\textbackslash}textit{is likely}. [{\textbackslash}textsc{EventB}] {\textbackslash}textit{is impossible}.{''}, a well-functioning language model should not conclude that [{\textbackslash}textsc{EventA{\$}{\textbackslash}{\&}amp;{\$}B}] is likely. Through our study, we observe that both tasks present challenges to out-of-the-box English language models. However, we also demonstrate that fine-tuning these models can lead to significant and transferable improvements.",
}

Words of Estimative Probability (WEP) are phrases used to express the plausibility of a statement. Examples include terms like {\textbackslash}textit{probably, maybe, likely, doubt, unlikely}, and {\textbackslash}textit{impossible}. Surveys have shown that human evaluators tend to agree when assigning numerical probability levels to these WEPs. For instance, the term {\textbackslash}textit{highly likely} equates to a median probability of {\$}0.90{{\textbackslash}pm}0.08{\$} according to a survey by {\textbackslash}citet{fagen-ulmschneider}.In this study, our focus is to gauge the competency of neural language processing models in accurately capturing the consensual probability level associated with each WEP. Our first approach is utilizing the UNLI dataset {\textbackslash}cite{chen-etal-2020-uncertain}, which links premises and hypotheses with their perceived joint probability {\$}p{\$}. From this, we craft prompts in the form: ''[{\textbackslash}textsc{Premise}]. [{\textbackslash}textsc{Wep}], [{\textbackslash}textsc{Hypothesis}].{''} This allows us to evaluate whether language models can predict if the consensual probability level of a WEP aligns closely with {\$}p{\$}.In our second approach, we develop a dataset based on WEP-focused probabilistic reasoning to assess if language models can logically process WEP compositions. For example, given the prompt ''[{\textbackslash}textsc{EventA}] {\textbackslash}textit{is likely}. [{\textbackslash}textsc{EventB}] {\textbackslash}textit{is impossible}.{''}, a well-functioning language model should not conclude that [{\textbackslash}textsc{EventA{\$}{\textbackslash}{\&}amp;{\$}B}] is likely. Through our study, we observe that both tasks present challenges to out-of-the-box English language models. However, we also demonstrate that fine-tuning these models can lead to significant and transferable improvements.
Sileo, Damien Moens, Marie-francine
Probing neural language models for understanding of words of estimative probability