@inproceedings{yu-etal-2023-alert,
    title = "{ALERT}: Adapt Language Models to Reasoning Tasks",
    author = "Yu, Ping  and
      Wang, Tianlu  and
      Golovneva, Olga  and
      AlKhamissi, Badr  and
      Verma, Siddharth  and
      Jin, Zhijing  and
      Ghosh, Gargi  and
      Diab, Mona  and
      Celikyilmaz, Asli",
    editor = "Rogers, Anna  and
      Boyd-Graber, Jordan  and
      Okazaki, Naoaki",
    booktitle = "Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2023",
    address = "Toronto, Canada",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.acl-long.60",
    doi = "10.18653/v1/2023.acl-long.60",
    pages = "1055--1081",
    abstract = "Recent advancements in large language models have enabled them to perform well on complex tasks that require step-by-step reasoning with few-shot learning. However, it is unclear whether these models are applying reasoning skills they have learnt during pre-training , or if they are simply memorizing their training corpus at finer granularity and have learnt to better understand their context. To address this question, we introduce {pasted macro {`}OUR{'}}model, a benchmark and suite of analyses for evaluating reasoning skills of language models. {pasted macro {`}OUR{'}}model enables comparing pre-trained and finetuned models on complex tasks that require reasoning skills to solve. Our benchmark provides a test bed to asses any language model on fine-grained reasoning skills, which spans over 20 datasets and covers 10 different reasoning skills. By using {pasted macro {`}OUR{'}}model we further investigate \textit{the role of finetuning}. Our extensive empirical analysis shows that language models learn more reasoning skills such as textual entailment, abductive reasoning, and analogical reasoning during the finetuning stage compared to pretraining stage. However, we also find that when language models are finetuned they tend to overfit to the prompt template, which hurts the robustness of models causing generalization problems.",
}

Recent advancements in large language models have enabled them to perform well on complex tasks that require step-by-step reasoning with few-shot learning. However, it is unclear whether these models are applying reasoning skills they have learnt during pre-training , or if they are simply memorizing their training corpus at finer granularity and have learnt to better understand their context. To address this question, we introduce {pasted macro {`}OUR{'}}model, a benchmark and suite of analyses for evaluating reasoning skills of language models. {pasted macro {`}OUR{'}}model enables comparing pre-trained and finetuned models on complex tasks that require reasoning skills to solve. Our benchmark provides a test bed to asses any language model on fine-grained reasoning skills, which spans over 20 datasets and covers 10 different reasoning skills. By using {pasted macro {`}OUR{'}}model we further investigate \textit{the role of finetuning}. Our extensive empirical analysis shows that language models learn more reasoning skills such as textual entailment, abductive reasoning, and analogical reasoning during the finetuning stage compared to pretraining stage. However, we also find that when language models are finetuned they tend to overfit to the prompt template, which hurts the robustness of models causing generalization problems.
Yu, Ping Wang, Tianlu Golovneva, Olga AlKhamissi, Badr Verma, Siddharth Jin, Zhijing Ghosh, Gargi Diab, Mona Celikyilmaz, Asli
ALERT: Adapt Language Models to Reasoning Tasks