@inproceedings{nagasawa-etal-2023-lms,
    title = "Can {LM}s Store and Retrieve 1-to-N Relational Knowledge?",
    author = "Nagasawa, Haruki  and
      Heinzerling, Benjamin  and
      Kokuta, Kazuma  and
      Inui, Kentaro",
    editor = "Padmakumar, Vishakh  and
      Vallejo, Gisela  and
      Fu, Yao",
    booktitle = "Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop)",
    month = jul,
    year = "2023",
    address = "Toronto, Canada",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.acl-srw.22",
    doi = "10.18653/v1/2023.acl-srw.22",
    pages = "130--138",
    abstract = "It has been suggested that pretrained language models can be viewed as knowledge bases. One of the prerequisites for using language models as knowledge bases is how accurately they can store and retrieve world knowledge. It is already revealed that language models can store much 1-to-1 relational knowledge, such as {''}country and its capital,{''} with high memorization accuracy. On the other hand, world knowledge includes not only 1-to-1 but also 1-to-N relational knowledge, such as {''}parent and children.{''}However, it is not clear how accurately language models can handle 1-to-N relational knowledge. To investigate language models{'} abilities toward 1-to-N relational knowledge, we start by designing the problem settings. Specifically, we organize the character of 1-to-N relational knowledge and define two essential skills: (i) memorizing multiple objects individually and (ii) retrieving multiple stored objects without excesses or deficiencies at once. We inspect LMs{'} ability to handle 1-to-N relational knowledge on the controlled synthesized data. As a result, we report that it is possible to memorize multiple objects with high accuracy, but generalizing the retrieval ability (expressly, enumeration) is challenging.",
}

It has been suggested that pretrained language models can be viewed as knowledge bases. One of the prerequisites for using language models as knowledge bases is how accurately they can store and retrieve world knowledge. It is already revealed that language models can store much 1-to-1 relational knowledge, such as {''}country and its capital,{''} with high memorization accuracy. On the other hand, world knowledge includes not only 1-to-1 but also 1-to-N relational knowledge, such as {''}parent and children.{''}However, it is not clear how accurately language models can handle 1-to-N relational knowledge. To investigate language models{'} abilities toward 1-to-N relational knowledge, we start by designing the problem settings. Specifically, we organize the character of 1-to-N relational knowledge and define two essential skills: (i) memorizing multiple objects individually and (ii) retrieving multiple stored objects without excesses or deficiencies at once. We inspect LMs{'} ability to handle 1-to-N relational knowledge on the controlled synthesized data. As a result, we report that it is possible to memorize multiple objects with high accuracy, but generalizing the retrieval ability (expressly, enumeration) is challenging.
Nagasawa, Haruki Heinzerling, Benjamin Kokuta, Kazuma Inui, Kentaro
Can LMs Store and Retrieve 1-to-N Relational Knowledge?