
# OntoLAMA: LAnguage Model Analysis for Ontology Subsumption Inference


### Dataset Summary

OntoLAMA is a set of language model (LM) probing datasets for ontology subsumption inference. 
The work follows the "LMs-as-KBs" literature but focuses on conceptualised knowledge extracted from formalised KBs 
such as the OWL ontologies. Specifically, the subsumption inference (SI) task is introduced and formulated in the
Natural Language Inference (NLI) style, where the sub-concept and the super-concept involved in a subsumption 
axiom are verbalised and fitted into a template to form the premise and hypothesis, respectively. 
The sampled axioms are verified through ontology reasoning. The SI task is further divided into Atomic SI and 
Complex SI where the former involves only atomic named concepts and the latter involves both atomic and complex concepts. 
Real-world ontologies of different scales and domains are used for constructing OntoLAMA and in total there are four Atomic 
SI datasets and two Complex SI datasets.

See dataset specifications on [DeepOnto](https://krr-oxford.github.io/DeepOnto/ontolama/).

See the published paper on [Arxiv](https://arxiv.org/abs/2302.06761) or [ACL Anthology](https://aclanthology.org/2023.findings-acl.213/).

### Languages

The text in the dataset is in English, as used in the source ontologies. The associated BCP-47 code is `en`.

## Dataset Structure

### Data Instances


An example in the **Atomic SI** dataset created from the Gene Ontology (GO) is as follows:
```
{
    'v_sub_concept': 'ctpase activity',
    'v_super_concept': 'ribonucleoside triphosphate phosphatase activity',
    'label': 1,
    'axiom': 'SubClassOf(<http://purl.obolibrary.org/obo/GO_0043273> <http://purl.obolibrary.org/obo/GO_0017111>)'
}
```

An example in the **Complex SI** dataset created from the Food Ontology (FoodOn) is as follows:
```
{
    'v_sub_concept': 'ham and cheese sandwich that derives from some lima bean (whole)',
    'v_super_concept': 'lima bean substance',
    'label': 0,
    'axiom': 'SubClassOf(ObjectIntersectionOf(<http://purl.obolibrary.org/obo/FOODON_03307824> ObjectSomeValuesFrom(<http://purl.obolibrary.org/obo/RO_0001000> <http://purl.obolibrary.org/obo/FOODON_03302053>)) <http://purl.obolibrary.org/obo/FOODON_00002776>)',
    'anchor_axiom': 'EquivalentClasses(<http://purl.obolibrary.org/obo/FOODON_00002776> ObjectIntersectionOf(<http://purl.obolibrary.org/obo/FOODON_00002000> ObjectSomeValuesFrom(<http://purl.obolibrary.org/obo/RO_0001000> <http://purl.obolibrary.org/obo/FOODON_03302053>)) )'
}
```


An example in the **biMNLI** dataset created from the MNLI dataset is as follows:
```
{
    'premise': 'At the turn of the 19th century Los Angeles and Salt Lake City were among the burgeoning metropolises of the new American West.',
    'hypothesis': 'Salt Lake City was booming in the early 19th century.',
    'label': 1
}
```

### Data Fields

#### SI Data Fields
- `v_sub_concept`: verbalised sub-concept expression.
- `v_super_concept`: verbalised super-concept expression.
- `label`: a binary class label indicating whether two concepts really form a subsumption relationship (`1` means yes).
- `axiom`: a string representation of the original subsumption axiom which is useful for tracing back to the ontology.
- `anchor_axiom`: (for complex SI only) a string representation of the anchor equivalence axiom used for sampling the `axiom`.

#### biMNLI Data Fields
- `premise`: inheritated from the MNLI dataset.
- `hypothesis`: inheritated from the MNLI dataset.
- `label`: a binary class label indicating `contradiction` (`0`) or `entailment` (`1`).

### Data Splits

| Source     | #NamedConcepts | #EquivAxioms | #Dataset (Train/Dev/Test)                                              |
|------------|----------------|--------------|------------------------------------------------------------------------|
| Schema.org | 894            | -            | Atomic SI: 808/404/2,830                                               |
| DOID       | 11,157         | -            | Atomic SI: 90,500/11,312/11,314                                        |
| FoodOn     | 30,995         | 2,383        | Atomic SI: 768,486/96,060/96,062 <br /> Complex SI: 3,754/1,850/13,080 |
| GO         | 43,303         | 11,456       | Atomic SI: 772,870/96,608/96,610 <br /> Complex SI: 72,318/9,040/9,040 |
| MNLI       | -              | -            | biMNLI: 235,622/26,180/12,906                                          |


### Licensing Information

Creative Commons Attribution 4.0 International

### Citation Information

The relevant paper has been accepted at Findings of ACL 2023.

```
@inproceedings{he-etal-2023-language,
    title = "Language Model Analysis for Ontology Subsumption Inference",
    author = "He, Yuan  and
    Chen, Jiaoyan  and
    Jimenez-Ruiz, Ernesto  and
    Dong, Hang  and
    Horrocks, Ian",
    booktitle = "Findings of the Association for Computational Linguistics: ACL 2023",
    month = jul,
    year = "2023",
    address = "Toronto, Canada",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.findings-acl.213",
    doi = "10.18653/v1/2023.findings-acl.213",
    pages = "3439--3453"
}
```
task_categories:text-classification size_categories:1M<n<10M language:en license:apache-2.0 Ontologies Subsumption Inference Natural Language Inference Conceptual Knowledge LMs-as-KBs arxiv:2302.06761 region:us
{"language": ["en"], "license": "apache-2.0", "size_categories": ["1M<n<10M"], "task_categories": ["text-classification"], "pretty_name": "OntoLAMA", "tags": ["Ontologies", "Subsumption Inference", "Natural Language Inference", "Conceptual Knowledge", "LMs-as-KBs"], "dataset_info": [{"config_name": "schemaorg-atomic-SI", "features": [{"name": "v_sub_concept", "dtype": "string"}, {"name": "v_super_concept", "dtype": "string"}, {"name": "label", "dtype": {"class_label": {"names": {"0": "negative_subsumption", "1": "positive_subsumption"}}}}, {"name": "axiom", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 103485, "num_examples": 808}, {"name": "validation", "num_bytes": 51523, "num_examples": 404}, {"name": "test", "num_bytes": 361200, "num_examples": 2830}], "download_size": 82558, "dataset_size": 516208}, {"config_name": "doid-atomic-SI", "features": [{"name": "v_sub_concept", "dtype": "string"}, {"name": "v_super_concept", "dtype": "string"}, {"name": "label", "dtype": {"class_label": {"names": {"0": "negative_subsumption", "1": "positive_subsumption"}}}}, {"name": "axiom", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 15803053, "num_examples": 90500}, {"name": "validation", "num_bytes": 1978584, "num_examples": 11312}, {"name": "test", "num_bytes": 1977582, "num_examples": 11314}], "download_size": 3184028, "dataset_size": 19759219}, {"config_name": "foodon-atomic-SI", "features": [{"name": "v_sub_concept", "dtype": "string"}, {"name": "v_super_concept", "dtype": "string"}, {"name": "label", "dtype": {"class_label": {"names": {"0": "negative_subsumption", "1": "positive_subsumption"}}}}, {"name": "axiom", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 128737404, "num_examples": 768486}, {"name": "validation", "num_bytes": 16090857, "num_examples": 96060}, {"name": "test", "num_bytes": 16098373, "num_examples": 96062}], "download_size": 28499028, "dataset_size": 160926634}, {"config_name": "go-atomic-SI", "features": [{"name": "v_sub_concept", "dtype": "string"}, {"name": "v_super_concept", "dtype": "string"}, {"name": "label", "dtype": {"class_label": {"names": {"0": "negative_subsumption", "1": "positive_subsumption"}}}}, {"name": "axiom", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 152537233, "num_examples": 772870}, {"name": "validation", "num_bytes": 19060490, "num_examples": 96608}, {"name": "test", "num_bytes": 19069265, "num_examples": 96610}], "download_size": 32379717, "dataset_size": 190666988}, {"config_name": "bimnli", "features": [{"name": "premise", "dtype": "string"}, {"name": "hypothesis", "dtype": "string"}, {"name": "label", "dtype": {"class_label": {"names": {"0": "contradiction", "1": "entailment"}}}}], "splits": [{"name": "train", "num_bytes": 43363266, "num_examples": 235622}, {"name": "validation", "num_bytes": 4818648, "num_examples": 26180}, {"name": "test", "num_bytes": 2420273, "num_examples": 12906}], "download_size": 19264134, "dataset_size": 50602187}, {"config_name": "foodon-complex-SI", "features": [{"name": "v_sub_concept", "dtype": "string"}, {"name": "v_super_concept", "dtype": "string"}, {"name": "label", "dtype": {"class_label": {"names": {"0": "negative_subsumption", "1": "positive_subsumption"}}}}, {"name": "axiom", "dtype": "string"}, {"name": "anchor_axiom", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 2553731, "num_examples": 3754}, {"name": "validation", "num_bytes": 1271721, "num_examples": 1850}, {"name": "test", "num_bytes": 8926305, "num_examples": 13080}], "download_size": 1064602, "dataset_size": 12751757}, {"config_name": "go-complex-SI", "features": [{"name": "v_sub_concept", "dtype": "string"}, {"name": "v_super_concept", "dtype": "string"}, {"name": "label", "dtype": {"class_label": {"names": {"0": "negative_subsumption", "1": "positive_subsumption"}}}}, {"name": "axiom", "dtype": "string"}, {"name": "anchor_axiom", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 45328802, "num_examples": 72318}, {"name": "validation", "num_bytes": 5671713, "num_examples": 9040}, {"name": "test", "num_bytes": 5667069, "num_examples": 9040}], "download_size": 5059364, "dataset_size": 56667584}]}
TAGS
#task_categories-text-classification #size_categories-1M<n<10M #language-English #license-apache-2.0 #Ontologies #Subsumption Inference #Natural Language Inference #Conceptual Knowledge #LMs-as-KBs #arxiv-2302.06761 #region-us 

OntoLAMA: LAnguage Model Analysis for Ontology Subsumption Inference
====================================================================


### Dataset Summary


OntoLAMA is a set of language model (LM) probing datasets for ontology subsumption inference.
The work follows the "LMs-as-KBs" literature but focuses on conceptualised knowledge extracted from formalised KBs
such as the OWL ontologies. Specifically, the subsumption inference (SI) task is introduced and formulated in the
Natural Language Inference (NLI) style, where the sub-concept and the super-concept involved in a subsumption
axiom are verbalised and fitted into a template to form the premise and hypothesis, respectively.
The sampled axioms are verified through ontology reasoning. The SI task is further divided into Atomic SI and
Complex SI where the former involves only atomic named concepts and the latter involves both atomic and complex concepts.
Real-world ontologies of different scales and domains are used for constructing OntoLAMA and in total there are four Atomic
SI datasets and two Complex SI datasets.


See dataset specifications on DeepOnto.


See the published paper on Arxiv or ACL Anthology.


### Languages


The text in the dataset is in English, as used in the source ontologies. The associated BCP-47 code is 'en'.


Dataset Structure
-----------------


### Data Instances


An example in the Atomic SI dataset created from the Gene Ontology (GO) is as follows:


An example in the Complex SI dataset created from the Food Ontology (FoodOn) is as follows:


An example in the biMNLI dataset created from the MNLI dataset is as follows:


### Data Fields


#### SI Data Fields


* 'v\_sub\_concept': verbalised sub-concept expression.
* 'v\_super\_concept': verbalised super-concept expression.
* 'label': a binary class label indicating whether two concepts really form a subsumption relationship ('1' means yes).
* 'axiom': a string representation of the original subsumption axiom which is useful for tracing back to the ontology.
* 'anchor\_axiom': (for complex SI only) a string representation of the anchor equivalence axiom used for sampling the 'axiom'.


#### biMNLI Data Fields


* 'premise': inheritated from the MNLI dataset.
* 'hypothesis': inheritated from the MNLI dataset.
* 'label': a binary class label indicating 'contradiction' ('0') or 'entailment' ('1').


### Data Splits



### Licensing Information


Creative Commons Attribution 4.0 International


The relevant paper has been accepted at Findings of ACL 2023.



### Dataset Summary


OntoLAMA is a set of language model (LM) probing datasets for ontology subsumption inference.
The work follows the "LMs-as-KBs" literature but focuses on conceptualised knowledge extracted from formalised KBs
such as the OWL ontologies. Specifically, the subsumption inference (SI) task is introduced and formulated in the
Natural Language Inference (NLI) style, where the sub-concept and the super-concept involved in a subsumption
axiom are verbalised and fitted into a template to form the premise and hypothesis, respectively.
The sampled axioms are verified through ontology reasoning. The SI task is further divided into Atomic SI and
Complex SI where the former involves only atomic named concepts and the latter involves both atomic and complex concepts.
Real-world ontologies of different scales and domains are used for constructing OntoLAMA and in total there are four Atomic
SI datasets and two Complex SI datasets.


See dataset specifications on DeepOnto.


See the published paper on Arxiv or ACL Anthology. ### Languages


The text in the dataset is in English, as used in the source ontologies. The associated BCP-47 code is 'en'.


Dataset Structure
----------------- ### Data Instances


An example in the Atomic SI dataset created from the Gene Ontology (GO) is as follows:


An example in the Complex SI dataset created from the Food Ontology (FoodOn) is as follows:


An example in the biMNLI dataset created from the MNLI dataset is as follows: ### Data Fields #### SI Data Fields


* 'v\_sub\_concept': verbalised sub-concept expression.
* 'v\_super\_concept': verbalised super-concept expression.
* 'label': a binary class label indicating whether two concepts really form a subsumption relationship ('1' means yes).
* 'axiom': a string representation of the original subsumption axiom which is useful for tracing back to the ontology.
* 'anchor\_axiom': (for complex SI only) a string representation of the anchor equivalence axiom used for sampling the 'axiom'. #### biMNLI Data Fields


* 'premise': inheritated from the MNLI dataset.
* 'hypothesis': inheritated from the MNLI dataset.
* 'label': a binary class label indicating 'contradiction' ('0') or 'entailment' ('1'). ### Data Splits ### Licensing Information


Creative Commons Attribution 4.0 International


The relevant paper has been accepted at Findings of ACL 2023.
TAGS
#task_categories-text-classification #size_categories-1M<n<10M #language-English #license-apache-2.0 #Ontologies #Subsumption Inference #Natural Language Inference #Conceptual Knowledge #LMs-as-KBs #arxiv-2302.06761 #region-us 
 ### Dataset Summary


OntoLAMA is a set of language model (LM) probing datasets for ontology subsumption inference.
The work follows the "LMs-as-KBs" literature but focuses on conceptualised knowledge extracted from formalised KBs
such as the OWL ontologies. Specifically, the subsumption inference (SI) task is introduced and formulated in the
Natural Language Inference (NLI) style, where the sub-concept and the super-concept involved in a subsumption
axiom are verbalised and fitted into a template to form the premise and hypothesis, respectively.
The sampled axioms are verified through ontology reasoning. The SI task is further divided into Atomic SI and
Complex SI where the former involves only atomic named concepts and the latter involves both atomic and complex concepts.
Real-world ontologies of different scales and domains are used for constructing OntoLAMA and in total there are four Atomic
SI datasets and two Complex SI datasets.


See dataset specifications on DeepOnto.


See the published paper on Arxiv or ACL Anthology. ### Languages


The text in the dataset is in English, as used in the source ontologies. The associated BCP-47 code is 'en'.


Dataset Structure
----------------- ### Data Instances


An example in the Atomic SI dataset created from the Gene Ontology (GO) is as follows:


An example in the Complex SI dataset created from the Food Ontology (FoodOn) is as follows:


An example in the biMNLI dataset created from the MNLI dataset is as follows: ### Data Fields #### SI Data Fields


* 'v\_sub\_concept': verbalised sub-concept expression.
* 'v\_super\_concept': verbalised super-concept expression.
* 'label': a binary class label indicating whether two concepts really form a subsumption relationship ('1' means yes).
* 'axiom': a string representation of the original subsumption axiom which is useful for tracing back to the ontology.
* 'anchor\_axiom': (for complex SI only) a string representation of the anchor equivalence axiom used for sampling the 'axiom'. #### biMNLI Data Fields


* 'premise': inheritated from the MNLI dataset.
* 'hypothesis': inheritated from the MNLI dataset.
* 'label': a binary class label indicating 'contradiction' ('0') or 'entailment' ('1'). ### Data Splits ### Licensing Information


Creative Commons Attribution 4.0 International


The relevant paper has been accepted at Findings of ACL 2023.