ROOTS Subset: roots_indic-ml_indic_nlp_corpus

# Indic NLP Corpus

- Dataset uid: `indic_nlp_corpus`

### Description

The IndicNLP corpus is a largescale, general-domain corpus containing 2.7 billion words for 10 Indian languages from two language families. s (IndoAryan branch and Dravidian). Each language has at least 100 million words (except Oriya).

### Homepage

https://github.com/AI4Bharat/indicnlp_corpus#publicly-available-classification-datasets

### Licensing

- non-commercial use
- cc-by-nc-sa-4.0: Creative Commons Attribution Non Commercial Share Alike 4.0 International


### Speaker Locations

- Southern Asia
- India


### Sizes

- 3.4019 % of total
- 44.4368 % of indic-hi
- 64.2943 % of indic-ta
- 70.5374 % of indic-ml
- 54.2394 % of indic-te
- 55.9105 % of indic-kn
- 61.6111 % of indic-mr
- 67.2242 % of indic-pa
- 68.1470 % of indic-or
- 64.3879 % of indic-gu
- 4.1495 % of indic-bn

### BigScience processing steps

#### Filters applied to: indic-hi

- dedup_document
- dedup_template_soft
- filter_remove_empty_docs
- filter_small_docs_bytes_300

#### Filters applied to: indic-ta

- dedup_document
- filter_remove_empty_docs
- filter_small_docs_bytes_300

#### Filters applied to: indic-ml

- dedup_document
- dedup_template_soft
- filter_remove_empty_docs
- filter_small_docs_bytes_300

#### Filters applied to: indic-te

- dedup_document
- dedup_template_soft
- filter_remove_empty_docs
- filter_small_docs_bytes_300

#### Filters applied to: indic-kn

- dedup_document
- dedup_template_soft
- filter_remove_empty_docs
- filter_small_docs_bytes_300

#### Filters applied to: indic-mr

- dedup_document
- dedup_template_soft
- filter_remove_empty_docs
- filter_small_docs_bytes_300

#### Filters applied to: indic-pa

- dedup_document
- dedup_template_soft
- filter_remove_empty_docs
- filter_small_docs_bytes_300

#### Filters applied to: indic-or

- dedup_document
- dedup_template_soft
- filter_remove_empty_docs

#### Filters applied to: indic-gu

- dedup_document
- dedup_template_soft
- filter_remove_empty_docs
- filter_small_docs_bytes_300

#### Filters applied to: indic-bn

- dedup_document
- dedup_template_soft
- filter_remove_empty_docs
- filter_small_docs_bytes_300


language:ml license:cc-by-nc-4.0 region:us
{"language": "ml", "license": "cc-by-nc-4.0", "extra_gated_prompt": "By accessing this dataset, you agree to abide by the BigScience Ethical Charter. The charter can be found at:\nhttps://hf.co/spaces/bigscience/ethical-charter", "extra_gated_fields": {"I have read and agree to abide by the BigScience Ethical Charter": "checkbox"}}
TAGS
#language-Malayalam #license-cc-by-nc-4.0 #region-us 

ROOTS Subset: roots_indic-ml_indic_nlp_corpus

# Indic NLP Corpus

- Dataset uid: 'indic_nlp_corpus'

### Description

The IndicNLP corpus is a largescale, general-domain corpus containing 2.7 billion words for 10 Indian languages from two language families. s (IndoAryan branch and Dravidian). Each language has at least 100 million words (except Oriya).

### Homepage

URL

### Licensing

- non-commercial use
- cc-by-nc-sa-4.0: Creative Commons Attribution Non Commercial Share Alike 4.0 International


### Speaker Locations

- Southern Asia
- India


### Sizes

- 3.4019 % of total
- 44.4368 % of indic-hi
- 64.2943 % of indic-ta
- 70.5374 % of indic-ml
- 54.2394 % of indic-te
- 55.9105 % of indic-kn
- 61.6111 % of indic-mr
- 67.2242 % of indic-pa
- 68.1470 % of indic-or
- 64.3879 % of indic-gu
- 4.1495 % of indic-bn

### BigScience processing steps

#### Filters applied to: indic-hi

- dedup_document
- dedup_template_soft
- filter_remove_empty_docs
- filter_small_docs_bytes_300

#### Filters applied to: indic-ta

- dedup_document
- filter_remove_empty_docs
- filter_small_docs_bytes_300

#### Filters applied to: indic-ml

- dedup_document
- dedup_template_soft
- filter_remove_empty_docs
- filter_small_docs_bytes_300

#### Filters applied to: indic-te

- dedup_document
- dedup_template_soft
- filter_remove_empty_docs
- filter_small_docs_bytes_300

#### Filters applied to: indic-kn

- dedup_document
- dedup_template_soft
- filter_remove_empty_docs
- filter_small_docs_bytes_300

#### Filters applied to: indic-mr

- dedup_document
- dedup_template_soft
- filter_remove_empty_docs
- filter_small_docs_bytes_300

#### Filters applied to: indic-pa

- dedup_document
- dedup_template_soft
- filter_remove_empty_docs
- filter_small_docs_bytes_300

#### Filters applied to: indic-or

- dedup_document
- dedup_template_soft
- filter_remove_empty_docs

#### Filters applied to: indic-gu

- dedup_document
- dedup_template_soft
- filter_remove_empty_docs
- filter_small_docs_bytes_300

#### Filters applied to: indic-bn

- dedup_document
- dedup_template_soft
- filter_remove_empty_docs
- filter_small_docs_bytes_300


# Indic NLP Corpus

- Dataset uid: 'indic_nlp_corpus' ### Description

The IndicNLP corpus is a largescale, general-domain corpus containing 2.7 billion words for 10 Indian languages from two language families. s (IndoAryan branch and Dravidian). Each language has at least 100 million words (except Oriya). ### Homepage

URL ### Licensing

- non-commercial use
- cc-by-nc-sa-4.0: Creative Commons Attribution Non Commercial Share Alike 4.0 International ### Speaker Locations

- Southern Asia
- India ### Sizes

- 3.4019 % of total
- 44.4368 % of indic-hi
- 64.2943 % of indic-ta
- 70.5374 % of indic-ml
- 54.2394 % of indic-te
- 55.9105 % of indic-kn
- 61.6111 % of indic-mr
- 67.2242 % of indic-pa
- 68.1470 % of indic-or
- 64.3879 % of indic-gu
- 4.1495 % of indic-bn ### BigScience processing steps #### Filters applied to: indic-hi

- dedup_document
- dedup_template_soft
- filter_remove_empty_docs
- filter_small_docs_bytes_300 #### Filters applied to: indic-ta

- dedup_document
- filter_remove_empty_docs
- filter_small_docs_bytes_300 #### Filters applied to: indic-ml

- dedup_document
- dedup_template_soft
- filter_remove_empty_docs
- filter_small_docs_bytes_300 #### Filters applied to: indic-te

- dedup_document
- dedup_template_soft
- filter_remove_empty_docs
- filter_small_docs_bytes_300 #### Filters applied to: indic-kn

- dedup_document
- dedup_template_soft
- filter_remove_empty_docs
- filter_small_docs_bytes_300 #### Filters applied to: indic-mr

- dedup_document
- dedup_template_soft
- filter_remove_empty_docs
- filter_small_docs_bytes_300 #### Filters applied to: indic-pa

- dedup_document
- dedup_template_soft
- filter_remove_empty_docs
- filter_small_docs_bytes_300 #### Filters applied to: indic-or

- dedup_document
- dedup_template_soft
- filter_remove_empty_docs #### Filters applied to: indic-gu

- dedup_document
- dedup_template_soft
- filter_remove_empty_docs
- filter_small_docs_bytes_300 #### Filters applied to: indic-bn

- dedup_document
- dedup_template_soft
- filter_remove_empty_docs
- filter_small_docs_bytes_300
TAGS
#language-Malayalam #license-cc-by-nc-4.0 #region-us 
 # Indic NLP Corpus

- Dataset uid: 'indic_nlp_corpus' ### Description

The IndicNLP corpus is a largescale, general-domain corpus containing 2.7 billion words for 10 Indian languages from two language families. s (IndoAryan branch and Dravidian). Each language has at least 100 million words (except Oriya). ### Homepage

URL ### Licensing

- non-commercial use
- cc-by-nc-sa-4.0: Creative Commons Attribution Non Commercial Share Alike 4.0 International ### Speaker Locations

- Southern Asia
- India ### Sizes

- 3.4019 % of total
- 44.4368 % of indic-hi
- 64.2943 % of indic-ta
- 70.5374 % of indic-ml
- 54.2394 % of indic-te
- 55.9105 % of indic-kn
- 61.6111 % of indic-mr
- 67.2242 % of indic-pa
- 68.1470 % of indic-or
- 64.3879 % of indic-gu
- 4.1495 % of indic-bn ### BigScience processing steps #### Filters applied to: indic-hi

- dedup_document
- dedup_template_soft
- filter_remove_empty_docs
- filter_small_docs_bytes_300 #### Filters applied to: indic-ta

- dedup_document
- filter_remove_empty_docs
- filter_small_docs_bytes_300 #### Filters applied to: indic-ml

- dedup_document
- dedup_template_soft
- filter_remove_empty_docs
- filter_small_docs_bytes_300 #### Filters applied to: indic-te

- dedup_document
- dedup_template_soft
- filter_remove_empty_docs
- filter_small_docs_bytes_300 #### Filters applied to: indic-kn

- dedup_document
- dedup_template_soft
- filter_remove_empty_docs
- filter_small_docs_bytes_300 #### Filters applied to: indic-mr

- dedup_document
- dedup_template_soft
- filter_remove_empty_docs
- filter_small_docs_bytes_300 #### Filters applied to: indic-pa

- dedup_document
- dedup_template_soft
- filter_remove_empty_docs
- filter_small_docs_bytes_300 #### Filters applied to: indic-or

- dedup_document
- dedup_template_soft
- filter_remove_empty_docs #### Filters applied to: indic-gu

- dedup_document
- dedup_template_soft
- filter_remove_empty_docs
- filter_small_docs_bytes_300 #### Filters applied to: indic-bn

- dedup_document
- dedup_template_soft
- filter_remove_empty_docs
- filter_small_docs_bytes_300
passage: TAGS
#language-Malayalam #license-cc-by-nc-4.0 #region-us 
# Indic NLP Corpus

- Dataset uid: 'indic_nlp_corpus'### Description

The IndicNLP corpus is a largescale, general-domain corpus containing 2.7 billion words for 10 Indian languages from two language families. s (IndoAryan branch and Dravidian). Each language has at least 100 million words (except Oriya).### Homepage

URL### Licensing

- non-commercial use
- cc-by-nc-sa-4.0: Creative Commons Attribution Non Commercial Share Alike 4.0 International### Speaker Locations

- Southern Asia
- India### Sizes

- 3.4019 % of total
- 44.4368 % of indic-hi
- 64.2943 % of indic-ta
- 70.5374 % of indic-ml
- 54.2394 % of indic-te
- 55.9105 % of indic-kn
- 61.6111 % of indic-mr
- 67.2242 % of indic-pa
- 68.1470 % of indic-or
- 64.3879 % of indic-gu
- 4.1495 % of indic-bn### BigScience processing steps#### Filters applied to: indic-hi

- dedup_document
- dedup_template_soft
- filter_remove_empty_docs
- filter_small_docs_bytes_300#### Filters applied to: indic-ta

- dedup_document
- filter_remove_empty_docs
- filter_small_docs_bytes_300#### Filters applied to: indic-ml

- dedup_document
- dedup_template_soft
- filter_remove_empty_docs
- filter_small_docs_bytes_300#### Filters applied to: indic-te

- dedup_document
- dedup_template_soft
- filter_remove_empty_docs
- filter_small_docs_bytes_300