language:btk language:bew language:bug language:jav language:mad language:mak language:min language:mui language:rej language:sun topic-modeling region:us
Democratizing access to natural language processing (NLP) technology is crucial, especially for underrepresented and extremely low-resource languages. Previous research has focused on developing labeled and unlabeled corpora for these languages through online scraping and document translation. While these methods have proven effective and cost-efficient, we have identified limitations in the resulting corpora, including a lack of lexical diversity and cultural relevance to local communities. To address this gap, we conduct a case study on Indonesian local languages. We compare the effectiveness of online scraping, human translation, and paragraph writing by native speakers in constructing datasets. Our findings demonstrate that datasets generated through paragraph writing by native speakers exhibit superior quality in terms of lexical diversity and cultural content. In addition, we present the NusaWrites benchmark, encompassing 12 underrepresented and extremely low-resource languages spoken by millions of individuals in Indonesia. Our empirical experiment results using existing multilingual large language models conclude the need to extend these models to more underrepresented languages.
We introduce a novel high quality human curated corpora, i.e., NusaMenulis, which covers 12 languages spoken in Indonesia. The resource extend the coverage of languages to 5 new languages, i.e., Ambon (abs), Bima (bhp), Makassarese (mak), Palembang / Musi (mui), and Rejang (rej).
For the topic modeling task, we cover 8 topics, i.e., food \& beverages, sports, leisure, religion, culture \& heritage, a slice of life, technology, and business.
@unpublished{anonymous2023nusawrites:,        
    title={NusaWrites: Constructing High-Quality Corpora for Underrepresented and Extremely Low-Resource Languages},        
    author={Anonymous},        
    journal={OpenReview Preprint},        
    year={2023},        
    note={anonymous preprint under review}    
}
---
tags:
- topic-modeling
language:
- btk
- bew
- bug
- jav
- mad
- mak
- min
- mui
- rej
- sun
---

# nusaparagraph_topic

Democratizing access to natural language processing (NLP) technology is crucial, especially for underrepresented and extremely low-resource languages. Previous research has focused on developing labeled and unlabeled corpora for these languages through online scraping and document translation. While these methods have proven effective and cost-efficient, we have identified limitations in the resulting corpora, including a lack of lexical diversity and cultural relevance to local communities. To address this gap, we conduct a case study on Indonesian local languages. We compare the effectiveness of online scraping, human translation, and paragraph writing by native speakers in constructing datasets. Our findings demonstrate that datasets generated through paragraph writing by native speakers exhibit superior quality in terms of lexical diversity and cultural content. In addition, we present the NusaWrites benchmark, encompassing 12 underrepresented and extremely low-resource languages spoken by millions of individuals in Indonesia. Our empirical experiment results using existing multilingual large language models conclude the need to extend these models to more underrepresented languages.

We introduce a novel high quality human curated corpora, i.e., NusaMenulis, which covers 12 languages spoken in Indonesia. The resource extend the coverage of languages to 5 new languages, i.e., Ambon (abs), Bima (bhp), Makassarese (mak), Palembang / Musi (mui), and Rejang (rej).

For the topic modeling task, we cover 8 topics, i.e., food \& beverages, sports, leisure, religion, culture \& heritage, a slice of life, technology, and business.

## Dataset Usage

Run `pip install nusacrowd` before loading the dataset through HuggingFace's `load_dataset`.

## Citation

```
@unpublished{anonymous2023nusawrites:,        
    title={NusaWrites: Constructing High-Quality Corpora for Underrepresented and Extremely Low-Resource Languages},        
    author={Anonymous},        
    journal={OpenReview Preprint},        
    year={2023},        
    note={anonymous preprint under review}    
}
```

## License

Creative Commons Attribution Share-Alike 4.0 International

## Homepage

[https://github.com/IndoNLP/nusa-writes](https://github.com/IndoNLP/nusa-writes)

### NusaCatalogue

For easy indexing and metadata: [https://indonlp.github.io/nusa-catalogue](https://indonlp.github.io/nusa-catalogue)