@inproceedings{wu-etal-2023-hence,
    title = "Hence, Socrates is mortal: A Benchmark for Natural Language Syllogistic Reasoning",
    author = "Wu, Yongkang  and
      Han, Meng  and
      Zhu, Yutao  and
      Li, Lei  and
      Zhang, Xinyu  and
      Lai, Ruofei  and
      Li, Xiaoguang  and
      Ren, Yuanhang  and
      Dou, Zhicheng  and
      Cao, Zhao",
    editor = "Rogers, Anna  and
      Boyd-Graber, Jordan  and
      Okazaki, Naoaki",
    booktitle = "Findings of the Association for Computational Linguistics: ACL 2023",
    month = jul,
    year = "2023",
    address = "Toronto, Canada",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.findings-acl.148",
    doi = "10.18653/v1/2023.findings-acl.148",
    pages = "2347--2367",
    abstract = "Syllogistic reasoning, a typical form of deductive reasoning, is a critical capability widely required in natural language understanding tasks, such as text entailment and question answering. To better facilitate research on syllogistic reasoning, we develop a benchmark called SylloBase that differs from existing syllogistic datasets in three aspects: (1) Covering a complete taxonomy of syllogism reasoning patterns; (2) Containing both automatically and manually constructed samples; and (3) Involving both the generation and understanding tasks. We automatically construct 50k template-based syllogism samples by mining syllogism patterns from Wikidata and ConceptNet. To improve our dataset{'}s naturalness and challenge, we apply GPT-3 to paraphrase the template-based data and further manually rewrite 1,000 samples as the test set. State-of-the-art pre-trained language models can achieve the best generation ROUGE-L of 38.72 by T5 and the best multi-choice accuracy of 72.77{\%} by RoBERTa on SylloBase, which indicates the great challenge of learning diverse syllogistic reasoning types on SylloBase. Our datasets are released at \url{https://github.com/casually-PYlearner/SYLLOBASE}.",
}

Syllogistic reasoning, a typical form of deductive reasoning, is a critical capability widely required in natural language understanding tasks, such as text entailment and question answering. To better facilitate research on syllogistic reasoning, we develop a benchmark called SylloBase that differs from existing syllogistic datasets in three aspects: (1) Covering a complete taxonomy of syllogism reasoning patterns; (2) Containing both automatically and manually constructed samples; and (3) Involving both the generation and understanding tasks. We automatically construct 50k template-based syllogism samples by mining syllogism patterns from Wikidata and ConceptNet. To improve our dataset{'}s naturalness and challenge, we apply GPT-3 to paraphrase the template-based data and further manually rewrite 1,000 samples as the test set. State-of-the-art pre-trained language models can achieve the best generation ROUGE-L of 38.72 by T5 and the best multi-choice accuracy of 72.77{\%} by RoBERTa on SylloBase, which indicates the great challenge of learning diverse syllogistic reasoning types on SylloBase. Our datasets are released at \url{https://github.com/casually-PYlearner/SYLLOBASE}.
Wu, Yongkang Han, Meng Zhu, Yutao Li, Lei Zhang, Xinyu Lai, Ruofei Li, Xiaoguang Ren, Yuanhang Dou, Zhicheng Cao, Zhao
Hence, Socrates is mortal: A Benchmark for Natural Language Syllogistic Reasoning