@inproceedings{alghamdi-etal-2023-aramus,
    title = "{A}ra{MUS}: Pushing the Limits of Data and Model Scale for {A}rabic Natural Language Processing",
    author = "Alghamdi, Asaad  and
      Duan, Xinyu  and
      Jiang, Wei  and
      Wang, Zhenhai  and
      Wu, Yimeng  and
      Xia, Qingrong  and
      Wang, Zhefeng  and
      Zheng, Yi  and
      Rezagholizadeh, Mehdi  and
      Huai, Baoxing  and
      Cheng, Peilun  and
      Ghaddar, Abbas",
    editor = "Rogers, Anna  and
      Boyd-Graber, Jordan  and
      Okazaki, Naoaki",
    booktitle = "Findings of the Association for Computational Linguistics: ACL 2023",
    month = jul,
    year = "2023",
    address = "Toronto, Canada",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.findings-acl.181",
    doi = "10.18653/v1/2023.findings-acl.181",
    pages = "2883--2894",
    abstract = "Developing monolingual large Pre-trained Language Models (PLMs) is shown to be very successful in handling different tasks in Natural Language Processing (NLP). In this work, we present AraMUS, the largest Arabic PLM with 11B parameters trained on 529GB of high-quality Arabic textual data. AraMUS achieves state-of-the-art performances on a diverse set of Arabic classification and generative tasks. Moreover, AraMUS shows impressive few-shot learning abilities compared with the best existing Arabic PLMs.",
}

Developing monolingual large Pre-trained Language Models (PLMs) is shown to be very successful in handling different tasks in Natural Language Processing (NLP). In this work, we present AraMUS, the largest Arabic PLM with 11B parameters trained on 529GB of high-quality Arabic textual data. AraMUS achieves state-of-the-art performances on a diverse set of Arabic classification and generative tasks. Moreover, AraMUS shows impressive few-shot learning abilities compared with the best existing Arabic PLMs.
Alghamdi, Asaad Duan, Xinyu Jiang, Wei Wang, Zhenhai Wu, Yimeng Xia, Qingrong Wang, Zhefeng Zheng, Yi Rezagholizadeh, Mehdi Huai, Baoxing Cheng, Peilun Ghaddar, Abbas
AraMUS: Pushing the Limits of Data and Model Scale for Arabic Natural Language Processing