@inproceedings{singhania-etal-2023-extracting,
    title = "Extracting Multi-valued Relations from Language Models",
    author = "Singhania, Sneha  and
      Razniewski, Simon  and
      Weikum, Gerhard",
    editor = "Can, Burcu  and
      Mozes, Maximilian  and
      Cahyawijaya, Samuel  and
      Saphra, Naomi  and
      Kassner, Nora  and
      Ravfogel, Shauli  and
      Ravichander, Abhilasha  and
      Zhao, Chen  and
      Augenstein, Isabelle  and
      Rogers, Anna  and
      Cho, Kyunghyun  and
      Grefenstette, Edward  and
      Voita, Lena",
    booktitle = "Proceedings of the 8th Workshop on Representation Learning for NLP (RepL4NLP 2023)",
    month = jul,
    year = "2023",
    address = "Toronto, Canada",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.repl4nlp-1.12",
    doi = "10.18653/v1/2023.repl4nlp-1.12",
    pages = "139--154",
    abstract = "The widespread usage of latent language representations via pre-trained language models (LMs) suggests that they are a promising source of structured knowledge. However, existing methods focus only on a single object per subject-relation pair, even though often multiple objects are correct. To overcome this limitation, we analyze these representations for their potential to yield materialized multi-object relational knowledge. We formulate the problem as a rank-then-select task. For ranking candidate objects, we evaluate existing prompting techniques and propose new ones incorporating domain knowledge. Among the selection methods, we find that choosing objects with a likelihood above a learned relation-specific threshold gives a 49.5{\%} F1 score. Our results highlight the difficulty of employing LMs for the multi-valued slot-filling task, and pave the way for further research on extracting relational knowledge from latent language representations.",
}

The widespread usage of latent language representations via pre-trained language models (LMs) suggests that they are a promising source of structured knowledge. However, existing methods focus only on a single object per subject-relation pair, even though often multiple objects are correct. To overcome this limitation, we analyze these representations for their potential to yield materialized multi-object relational knowledge. We formulate the problem as a rank-then-select task. For ranking candidate objects, we evaluate existing prompting techniques and propose new ones incorporating domain knowledge. Among the selection methods, we find that choosing objects with a likelihood above a learned relation-specific threshold gives a 49.5{\%} F1 score. Our results highlight the difficulty of employing LMs for the multi-valued slot-filling task, and pave the way for further research on extracting relational knowledge from latent language representations.
Singhania, Sneha Razniewski, Simon Weikum, Gerhard
Extracting Multi-valued Relations from Language Models