@inproceedings{hardalov-etal-2023-bgglue,
    title = "bg{GLUE}: A {B}ulgarian General Language Understanding Evaluation Benchmark",
    author = "Hardalov, Momchil  and
      Atanasova, Pepa  and
      Mihaylov, Todor  and
      Angelova, Galia  and
      Simov, Kiril  and
      Osenova, Petya  and
      Stoyanov, Veselin  and
      Koychev, Ivan  and
      Nakov, Preslav  and
      Radev, Dragomir",
    editor = "Rogers, Anna  and
      Boyd-Graber, Jordan  and
      Okazaki, Naoaki",
    booktitle = "Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2023",
    address = "Toronto, Canada",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.acl-long.487",
    doi = "10.18653/v1/2023.acl-long.487",
    pages = "8733--8759",
    abstract = "We present bgGLUE (Bulgarian General Language Understanding Evaluation), a benchmark for evaluating language models on Natural Language Understanding (NLU) tasks in Bulgarian. Our benchmark includes NLU tasks targeting a variety of NLP problems (e.g., natural language inference, fact-checking, named entity recognition, sentiment analysis, question answering, etc.) and machine learning tasks (sequence labeling, document-level classification, and regression). We run the first systematic evaluation of pre-trained language models for Bulgarian, comparing and contrasting results across the nine tasks in the benchmark. The evaluation results show strong performance on sequence labeling tasks, but there is a lot of room for improvement for tasks that require more complex reasoning. We make bgGLUE publicly available together with the fine-tuning and the evaluation code, as well as a public leaderboard at \url{https://bgglue.github.io}, and we hope that it will enable further advancements in developing NLU models for Bulgarian.",
}

We present bgGLUE (Bulgarian General Language Understanding Evaluation), a benchmark for evaluating language models on Natural Language Understanding (NLU) tasks in Bulgarian. Our benchmark includes NLU tasks targeting a variety of NLP problems (e.g., natural language inference, fact-checking, named entity recognition, sentiment analysis, question answering, etc.) and machine learning tasks (sequence labeling, document-level classification, and regression). We run the first systematic evaluation of pre-trained language models for Bulgarian, comparing and contrasting results across the nine tasks in the benchmark. The evaluation results show strong performance on sequence labeling tasks, but there is a lot of room for improvement for tasks that require more complex reasoning. We make bgGLUE publicly available together with the fine-tuning and the evaluation code, as well as a public leaderboard at \url{https://bgglue.github.io}, and we hope that it will enable further advancements in developing NLU models for Bulgarian.
Hardalov, Momchil Atanasova, Pepa Mihaylov, Todor Angelova, Galia Simov, Kiril Osenova, Petya Stoyanov, Veselin Koychev, Ivan Nakov, Preslav Radev, Dragomir
bgGLUE: A Bulgarian General Language Understanding Evaluation Benchmark