
# Dataset accompanying the "Probing neural language models for understanding of words of estimative probability" article

This dataset tests the capabilities of language models to correctly capture the meaning of words denoting probabilities (WEP, also called verbal probabilities), e.g. words like "probably", "maybe", "surely", "impossible".

We used probabilitic soft logic to combine probabilistic statements expressed with WEP (WEP-Reasoning) and we also used the UNLI dataset (https://nlp.jhu.edu/unli/) to directly check whether models can detect the WEP matching human-annotated probabilities according to [Fagen-Ulmschneider, 2018](https://github.com/wadefagen/datasets/tree/master/Perception-of-Probability-Words).
The dataset can be used as natural language inference data (context, premise, label) or multiple choice question answering (context,valid_hypothesis, invalid_hypothesis).

Code : [colab](https://colab.research.google.com/drive/10ILEWY2-J6Q1hT97cCB3eoHJwGSflKHp?usp=sharing)

# Citation
https://arxiv.org/abs/2211.03358
```bib
@inproceedings{sileo-moens-2023-probing,
    title = "Probing neural language models for understanding of words of estimative probability",
    author = "Sileo, Damien  and
      Moens, Marie-francine",
    booktitle = "Proceedings of the 12th Joint Conference on Lexical and Computational Semantics (*SEM 2023)",
    month = jul,
    year = "2023",
    address = "Toronto, Canada",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.starsem-1.41",
    doi = "10.18653/v1/2023.starsem-1.41",
    pages = "469--476",
}
```

task_categories:text-classification task_categories:multiple-choice task_categories:question-answering task_ids:open-domain-qa task_ids:multiple-choice-qa task_ids:natural-language-inference task_ids:multi-input-text-classification annotations_creators:expert-generated language_creators:crowdsourced multilinguality:monolingual size_categories:1K<n<10K source_datasets:original language:en license:apache-2.0 wep words of estimative probability probability logical reasoning soft logic nli verbal probabilities natural-language-inference reasoning logic arxiv:2211.03358 region:us
{"annotations_creators": ["expert-generated"], "language_creators": ["crowdsourced"], "language": ["en"], "license": ["apache-2.0"], "multilinguality": ["monolingual"], "size_categories": ["1K<n<10K"], "source_datasets": ["original"], "task_categories": ["text-classification", "multiple-choice", "question-answering"], "task_ids": ["open-domain-qa", "multiple-choice-qa", "natural-language-inference", "multi-input-text-classification"], "pretty_name": "probability_words_nli", "paperwithcoode_id": "probability-words-nli", "tags": ["wep", "words of estimative probability", "probability", "logical reasoning", "soft logic", "nli", "verbal probabilities", "natural-language-inference", "reasoning", "logic"], "train-eval-index": [{"config": "usnli", "task": "text-classification", "task_id": "multi-class-classification", "splits": {"train_split": "train", "eval_split": "validation"}, "col_mapping": {"sentence1": "context", "sentence2": "hypothesis", "label": "label"}, "metrics": [{"type": "accuracy", "name": "Accuracy"}, {"type": "f1", "name": "F1 binary"}]}, {"config": "reasoning-1hop", "task": "text-classification", "task_id": "multi-class-classification", "splits": {"train_split": "train", "eval_split": "validation"}, "col_mapping": {"sentence1": "context", "sentence2": "hypothesis", "label": "label"}, "metrics": [{"type": "accuracy", "name": "Accuracy"}, {"type": "f1", "name": "F1 binary"}]}, {"config": "reasoning-2hop", "task": "text-classification", "task_id": "multi-class-classification", "splits": {"train_split": "train", "eval_split": "validation"}, "col_mapping": {"sentence1": "context", "sentence2": "hypothesis", "label": "label"}, "metrics": [{"type": "accuracy", "name": "Accuracy"}, {"type": "f1", "name": "F1 binary"}]}]}
TAGS
#task_categories-text-classification #task_categories-multiple-choice #task_categories-question-answering #task_ids-open-domain-qa #task_ids-multiple-choice-qa #task_ids-natural-language-inference #task_ids-multi-input-text-classification #annotations_creators-expert-generated #language_creators-crowdsourced #multilinguality-monolingual #size_categories-1K<n<10K #source_datasets-original #language-English #license-apache-2.0 #wep #words of estimative probability #probability #logical reasoning #soft logic #nli #verbal probabilities #natural-language-inference #reasoning #logic #arxiv-2211.03358 #region-us 


# Dataset accompanying the "Probing neural language models for understanding of words of estimative probability" article

This dataset tests the capabilities of language models to correctly capture the meaning of words denoting probabilities (WEP, also called verbal probabilities), e.g. words like "probably", "maybe", "surely", "impossible".

We used probabilitic soft logic to combine probabilistic statements expressed with WEP (WEP-Reasoning) and we also used the UNLI dataset (URL to directly check whether models can detect the WEP matching human-annotated probabilities according to Fagen-Ulmschneider, 2018.
The dataset can be used as natural language inference data (context, premise, label) or multiple choice question answering (context,valid_hypothesis, invalid_hypothesis).

Code : colab

URL


# Dataset accompanying the "Probing neural language models for understanding of words of estimative probability" article

This dataset tests the capabilities of language models to correctly capture the meaning of words denoting probabilities (WEP, also called verbal probabilities), e.g. words like "probably", "maybe", "surely", "impossible".

We used probabilitic soft logic to combine probabilistic statements expressed with WEP (WEP-Reasoning) and we also used the UNLI dataset (URL to directly check whether models can detect the WEP matching human-annotated probabilities according to Fagen-Ulmschneider, 2018.
The dataset can be used as natural language inference data (context, premise, label) or multiple choice question answering (context,valid_hypothesis, invalid_hypothesis).

Code : colab

URL
TAGS
#task_categories-text-classification #task_categories-multiple-choice #task_categories-question-answering #task_ids-open-domain-qa #task_ids-multiple-choice-qa #task_ids-natural-language-inference #task_ids-multi-input-text-classification #annotations_creators-expert-generated #language_creators-crowdsourced #multilinguality-monolingual #size_categories-1K<n<10K #source_datasets-original #language-English #license-apache-2.0 #wep #words of estimative probability #probability #logical reasoning #soft logic #nli #verbal probabilities #natural-language-inference #reasoning #logic #arxiv-2211.03358 #region-us 
 # Dataset accompanying the "Probing neural language models for understanding of words of estimative probability" article

This dataset tests the capabilities of language models to correctly capture the meaning of words denoting probabilities (WEP, also called verbal probabilities), e.g. words like "probably", "maybe", "surely", "impossible".

We used probabilitic soft logic to combine probabilistic statements expressed with WEP (WEP-Reasoning) and we also used the UNLI dataset (URL to directly check whether models can detect the WEP matching human-annotated probabilities according to Fagen-Ulmschneider, 2018.
The dataset can be used as natural language inference data (context, premise, label) or multiple choice question answering (context,valid_hypothesis, invalid_hypothesis).

Code : colab

URL