
# Dataset Card for "nepalitext-language-model-dataset"

### Dataset Summary

"NepaliText" language modeling dataset is a collection of over 13 million Nepali text sequences (phrases/sentences/paragraphs) extracted by combining the datasets: [OSCAR](https://huggingface.co/datasets/oscar) , [cc100](https://huggingface.co/datasets/cc100) and a set of scraped Nepali articles on Wikipedia. 

### Supported Tasks and Leaderboards

This dataset is intended to pre-train language models and word representations on Nepali Language.

### Languages

The data is focused on Nepali language, but may have instances of other languages as well.

## Dataset Structure

### Data Instances

An example:
```
{'text': 'घरेलु मैदानमा भएको च्याम्पियन्स लिगको दोस्रो लेगमा एथ्लेटिको मड्रिडले आर्सनललाई एक शून्यले हराउँदै समग्रमा दुई एकको अग्रताका साथ फाइनलमा प्रवेश गरेको हो ।\n'}
```

### Data Fields

The data fields are:
- `text`: a `string` feature.

### Data Splits

train|test|
----:|---:|
13141222|268189|


## Dataset Creation

### Curation Rationale

[More Information Needed]

### Source Data

#### Initial Data Collection and Normalization

[More Information Needed]

#### Who are the source language producers?

[More Information Needed]

### Annotations

The dataset does not contain any additional annotations.

#### Annotation process

[More Information Needed]

#### Who are the annotators?

[More Information Needed]

### Personal and Sensitive Information

Being extracted and scraped from variety of internet sources, Personal and sensitive information might be present. This must be considered before training deep learning models, specially in the case of text-generation models.

## Considerations for Using the Data

### Social Impact of Dataset

[More Information Needed]

### Discussion of Biases

[More Information Needed]

### Other Known Limitations

[More Information Needed]

## Additional Information

### Dataset Curators

[More Information Needed]

### Licensing Information

[More Information Needed]

### Citation Information

[More Information Needed]

### Contributions

Thanks to [@Sakonii](https://github.com/Sakonii) for adding this dataset.
task_categories:text-generation task_ids:language-modeling annotations_creators:no-annotation language_creators:found language_creators:other multilinguality:monolingual source_datasets:extended|oscar source_datasets:extended|cc100 language:ne license:cc0-1.0 region:us
{"annotations_creators": ["no-annotation"], "language_creators": ["found", "other"], "language": ["ne"], "license": ["cc0-1.0"], "multilinguality": ["monolingual"], "source_datasets": ["extended|oscar", "extended|cc100"], "task_categories": ["text-generation"], "task_ids": ["language-modeling"], "pretty_name": "nepalitext-language-model-dataset"}
TAGS
#task_categories-text-generation #task_ids-language-modeling #annotations_creators-no-annotation #language_creators-found #language_creators-other #multilinguality-monolingual #source_datasets-extended|oscar #source_datasets-extended|cc100 #language-Nepali (macrolanguage) #license-cc0-1.0 #region-us 

Dataset Card for "nepalitext-language-model-dataset"
====================================================


### Dataset Summary


"NepaliText" language modeling dataset is a collection of over 13 million Nepali text sequences (phrases/sentences/paragraphs) extracted by combining the datasets: OSCAR , cc100 and a set of scraped Nepali articles on Wikipedia.


### Supported Tasks and Leaderboards


This dataset is intended to pre-train language models and word representations on Nepali Language.


### Languages


The data is focused on Nepali language, but may have instances of other languages as well.


Dataset Structure
-----------------


### Data Instances


An example:


### Data Fields


The data fields are:


* 'text': a 'string' feature.


### Data Splits



Dataset Creation
----------------


### Curation Rationale


### Source Data


#### Initial Data Collection and Normalization


#### Who are the source language producers?


### Annotations


The dataset does not contain any additional annotations.


#### Annotation process


#### Who are the annotators?


### Personal and Sensitive Information


Being extracted and scraped from variety of internet sources, Personal and sensitive information might be present. This must be considered before training deep learning models, specially in the case of text-generation models.


Considerations for Using the Data
---------------------------------


### Social Impact of Dataset


### Discussion of Biases


### Other Known Limitations


Additional Information
----------------------


### Dataset Curators


### Licensing Information


### Contributions


Thanks to @Sakonii for adding this dataset.



### Dataset Summary


"NepaliText" language modeling dataset is a collection of over 13 million Nepali text sequences (phrases/sentences/paragraphs) extracted by combining the datasets: OSCAR , cc100 and a set of scraped Nepali articles on Wikipedia. ### Supported Tasks and Leaderboards


This dataset is intended to pre-train language models and word representations on Nepali Language. ### Languages


The data is focused on Nepali language, but may have instances of other languages as well.


Dataset Structure
----------------- ### Data Instances


An example: ### Data Fields


The data fields are:


* 'text': a 'string' feature. ### Data Splits



Dataset Creation
---------------- ### Curation Rationale ### Source Data #### Initial Data Collection and Normalization #### Who are the source language producers? ### Annotations


The dataset does not contain any additional annotations. #### Annotation process #### Who are the annotators? ### Personal and Sensitive Information


Being extracted and scraped from variety of internet sources, Personal and sensitive information might be present. This must be considered before training deep learning models, specially in the case of text-generation models.


Considerations for Using the Data
--------------------------------- ### Social Impact of Dataset ### Discussion of Biases ### Other Known Limitations


Additional Information
---------------------- ### Dataset Curators ### Licensing Information ### Contributions


Thanks to @Sakonii for adding this dataset.
TAGS
#task_categories-text-generation #task_ids-language-modeling #annotations_creators-no-annotation #language_creators-found #language_creators-other #multilinguality-monolingual #source_datasets-extended|oscar #source_datasets-extended|cc100 #language-Nepali (macrolanguage) #license-cc0-1.0 #region-us 
 ### Dataset Summary


"NepaliText" language modeling dataset is a collection of over 13 million Nepali text sequences (phrases/sentences/paragraphs) extracted by combining the datasets: OSCAR , cc100 and a set of scraped Nepali articles on Wikipedia. ### Supported Tasks and Leaderboards


This dataset is intended to pre-train language models and word representations on Nepali Language. ### Languages


The data is focused on Nepali language, but may have instances of other languages as well.


Dataset Structure
----------------- ### Data Instances


An example: ### Data Fields


The data fields are:


* 'text': a 'string' feature. ### Data Splits



Dataset Creation
---------------- ### Curation Rationale ### Source Data #### Initial Data Collection and Normalization #### Who are the source language producers? ### Annotations


The dataset does not contain any additional annotations. #### Annotation process #### Who are the annotators? ### Personal and Sensitive Information


Being extracted and scraped from variety of internet sources, Personal and sensitive information might be present. This must be considered before training deep learning models, specially in the case of text-generation models.


Considerations for Using the Data
--------------------------------- ### Social Impact of Dataset ### Discussion of Biases ### Other Known Limitations


Additional Information
---------------------- ### Dataset Curators ### Licensing Information ### Contributions


Thanks to @Sakonii for adding this dataset.
passage: TAGS
#task_categories-text-generation #task_ids-language-modeling #annotations_creators-no-annotation #language_creators-found #language_creators-other #multilinguality-monolingual #source_datasets-extended|oscar #source_datasets-extended|cc100 #language-Nepali (macrolanguage) #license-cc0-1.0 #region-us 
### Dataset Summary


"NepaliText" language modeling dataset is a collection of over 13 million Nepali text sequences (phrases/sentences/paragraphs) extracted by combining the datasets: OSCAR , cc100 and a set of scraped Nepali articles on Wikipedia.### Supported Tasks and Leaderboards


This dataset is intended to pre-train language models and word representations on Nepali Language.### Languages


The data is focused on Nepali language, but may have instances of other languages as well.


Dataset Structure
-----------------### Data Instances


An example:### Data Fields


The data fields are:


* 'text': a 'string' feature.### Data Splits



Dataset Creation
----------------### Curation Rationale### Source Data#### Initial Data Collection and Normalization#### Who are the source language producers?### Annotations


The dataset does not contain any additional annotations.#### Annotation process#### Who are the annotators?### Personal and Sensitive Information


Being extracted and scraped from variety of internet sources, Personal and sensitive information might be present. This must be considered before training deep learning models, specially in the case of text-generation models.


Considerations for Using the Data
---------------------------------### Social Impact of Dataset### Discussion of Biases### Other Known Limitations


Additional Information
----------------------### Dataset Curators### Licensing Information### Contributions


Thanks to @Sakonii for adding this dataset.