

# Dataset for evaluation of (zero-shot) recommendation with language models

We showed that pretrained large language models can act as a recommender system, and compare few-shot learning results to matrix factorization baselines.
This is the BIG-Bench version of our language-based movie recommendation dataset.

<https://github.com/google/BIG-bench/tree/main/bigbench/benchmark_tasks/movie_recommendation>

GPT-2 has a 48.8% accuracy, chance is 25%.
Human accuracy is 60.4%.

# Citation
```
@InProceedings{sileodreclm22,
    author="Sileo, Damien
    and Vossen, Wout
    and Raymaekers, Robbe",
    editor="Hagen, Matthias
    and Verberne, Suzan
    and Macdonald, Craig
    and Seifert, Christin
    and Balog, Krisztian
    and N{\o}rv{\aa}g, Kjetil
    and Setty, Vinay",
    title="Zero-Shot Recommendation as Language Modeling",
    booktitle="Advances in Information Retrieval",
    year="2022",
    publisher="Springer International Publishing",
    address="Cham",
    pages="223--230",
    isbn="978-3-030-99739-7"
}
```
task_categories:multiple-choice task_categories:question-answering task_ids:multiple-choice-qa task_ids:open-domain-qa annotations_creators:expert-generated language_creators:crowdsourced multilinguality:monolingual size_categories:n<1K source_datasets:original language:en license:apache-2.0 movie-recommendation collaborative-filtering movielens film doi:10.57967/hf/0257 region:us
{"annotations_creators": ["expert-generated"], "language_creators": ["crowdsourced"], "language": ["en"], "license": ["apache-2.0"], "multilinguality": ["monolingual"], "size_categories": ["n<1K"], "source_datasets": ["original"], "task_categories": ["multiple-choice", "question-answering"], "task_ids": ["multiple-choice-qa", "open-domain-qa"], "pretty_name": "movie_recommendation", "tags": ["movie-recommendation", "collaborative-filtering", "movielens", "film"]}
TAGS
#task_categories-multiple-choice #task_categories-question-answering #task_ids-multiple-choice-qa #task_ids-open-domain-qa #annotations_creators-expert-generated #language_creators-crowdsourced #multilinguality-monolingual #size_categories-n<1K #source_datasets-original #language-English #license-apache-2.0 #movie-recommendation #collaborative-filtering #movielens #film #doi-10.57967/hf/0257 #region-us 



# Dataset for evaluation of (zero-shot) recommendation with language models

We showed that pretrained large language models can act as a recommender system, and compare few-shot learning results to matrix factorization baselines.
This is the BIG-Bench version of our language-based movie recommendation dataset.

<URL

GPT-2 has a 48.8% accuracy, chance is 25%.
Human accuracy is 60.4%.


# Dataset for evaluation of (zero-shot) recommendation with language models

We showed that pretrained large language models can act as a recommender system, and compare few-shot learning results to matrix factorization baselines.
This is the BIG-Bench version of our language-based movie recommendation dataset.

<URL

GPT-2 has a 48.8% accuracy, chance is 25%.
Human accuracy is 60.4%.
TAGS
#task_categories-multiple-choice #task_categories-question-answering #task_ids-multiple-choice-qa #task_ids-open-domain-qa #annotations_creators-expert-generated #language_creators-crowdsourced #multilinguality-monolingual #size_categories-n<1K #source_datasets-original #language-English #license-apache-2.0 #movie-recommendation #collaborative-filtering #movielens #film #doi-10.57967/hf/0257 #region-us 
 # Dataset for evaluation of (zero-shot) recommendation with language models

We showed that pretrained large language models can act as a recommender system, and compare few-shot learning results to matrix factorization baselines.
This is the BIG-Bench version of our language-based movie recommendation dataset.

<URL

GPT-2 has a 48.8% accuracy, chance is 25%.
Human accuracy is 60.4%.