@inproceedings{wijesiriwardene-etal-2023-analogical,
    title = "{ANALOGICAL} - A Novel Benchmark for Long Text Analogy Evaluation in Large Language Models",
    author = "Wijesiriwardene, Thilini  and
      Wickramarachchi, Ruwan  and
      Gajera, Bimal  and
      Gowaikar, Shreeyash  and
      Gupta, Chandan  and
      Chadha, Aman  and
      Reganti, Aishwarya Naresh  and
      Sheth, Amit  and
      Das, Amitava",
    editor = "Rogers, Anna  and
      Boyd-Graber, Jordan  and
      Okazaki, Naoaki",
    booktitle = "Findings of the Association for Computational Linguistics: ACL 2023",
    month = jul,
    year = "2023",
    address = "Toronto, Canada",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.findings-acl.218",
    doi = "10.18653/v1/2023.findings-acl.218",
    pages = "3534--3549",
    abstract = "Over the past decade, analogies, in the form of word-level analogies, have played a significant role as an intrinsic measure of evaluating the quality of word embedding methods such as word2vec. Modern large language models (LLMs), however, are primarily evaluated on extrinsic measures based on benchmarks such as GLUE and SuperGLUE, and there are only a few investigations on whether LLMs can draw analogies between long texts. In this paper, we present ANALOGICAL, a new benchmark to intrinsically evaluate LLMs across a taxonomy of analogies of long text with six levels of complexity {--} (i) word, (ii) word vs. sentence, (iii) syntactic, (iv) negation, (v) entailment, and (vi) metaphor. Using thirteen datasets and three different distance measures, we evaluate the abilities of eight LLMs in identifying analogical pairs in the semantic vector space. Our evaluation finds that it is increasingly challenging for LLMs to identify analogies when going up the analogy taxonomy.",
}

Over the past decade, analogies, in the form of word-level analogies, have played a significant role as an intrinsic measure of evaluating the quality of word embedding methods such as word2vec. Modern large language models (LLMs), however, are primarily evaluated on extrinsic measures based on benchmarks such as GLUE and SuperGLUE, and there are only a few investigations on whether LLMs can draw analogies between long texts. In this paper, we present ANALOGICAL, a new benchmark to intrinsically evaluate LLMs across a taxonomy of analogies of long text with six levels of complexity {--} (i) word, (ii) word vs. sentence, (iii) syntactic, (iv) negation, (v) entailment, and (vi) metaphor. Using thirteen datasets and three different distance measures, we evaluate the abilities of eight LLMs in identifying analogical pairs in the semantic vector space. Our evaluation finds that it is increasingly challenging for LLMs to identify analogies when going up the analogy taxonomy.
Wijesiriwardene, Thilini Wickramarachchi, Ruwan Gajera, Bimal Gowaikar, Shreeyash Gupta, Ch an Chadha, Aman Reganti, Aishwarya Naresh Sheth, Amit Das, Amitava
ANALOGICAL - A Novel Benchmark for Long Text Analogy Evaluation in Large Language Models