# IndicCorpv2

[Github](https://github.com/AI4Bharat/IndicBERT) | [Paper](https://arxiv.org/abs/2212.05409)   
Languages: 24   
The largest collection of texts for Indic languages consisting of 20.9 billion tokens of which 14.4B tokens correspond to 23 Indic languages and 6.5B tokens of Indian English content curated from Indian websites.
<img src="https://huggingface.co/datasets/satpalsr/indicCorpv2/resolve/main/languages.png">

## Citation
```
@article{Doddapaneni2022towards,
  title={Towards Leaving No Indic Language Behind: Building Monolingual Corpora, Benchmark and Models for Indic Languages},
  author={Sumanth Doddapaneni and Rahul Aralikatte and Gowtham Ramesh and Shreyansh Goyal and Mitesh M. Khapra and Anoop Kunchukuttan and Pratyush Kumar},
  journal={ArXiv},
  year={2022},
  volume={abs/2212.05409}
}
```
task_categories:text-generation language:as language:brx language:bn language:doi language:en language:gom language:gu language:hi language:kha language:kn language:ks language:mai language:ml language:mni language:mr language:ne language:or language:pa language:sa language:sat language:sd language:ta language:te language:ur license:cc0-1.0 arxiv:2212.05409 region:us
{"language": ["as", "brx", "bn", "doi", "en", "gom", "gu", "hi", "kha", "kn", "ks", "mai", "ml", "mni", "mr", "ne", "or", "pa", "sa", "sat", "sd", "ta", "te", "ur"], "license": "cc0-1.0", "task_categories": ["text-generation"]}
TAGS
#task_categories-text-generation #language-Assamese #language-Bodo (India) #language-Bengali #language-Dogri (macrolanguage) #language-English #language-Goan Konkani #language-Gujarati #language-Hindi #language-Khasi #language-Kannada #language-Kashmiri #language-Maithili #language-Malayalam #language-Manipuri #language-Marathi #language-Nepali (macrolanguage) #language-Oriya (macrolanguage) #language-Panjabi #language-Sanskrit #language-Santali #language-Sindhi #language-Tamil #language-Telugu #language-Urdu #license-cc0-1.0 #arxiv-2212.05409 #region-us 

# IndicCorpv2

Github | Paper   
Languages: 24   
The largest collection of texts for Indic languages consisting of 20.9 billion tokens of which 14.4B tokens correspond to 23 Indic languages and 6.5B tokens of Indian English content curated from Indian websites.
<img src="URL


# IndicCorpv2

Github | Paper   
Languages: 24   
The largest collection of texts for Indic languages consisting of 20.9 billion tokens of which 14.4B tokens correspond to 23 Indic languages and 6.5B tokens of Indian English content curated from Indian websites.
<img src="URL
TAGS
#task_categories-text-generation #language-Assamese #language-Bodo (India) #language-Bengali #language-Dogri (macrolanguage) #language-English #language-Goan Konkani #language-Gujarati #language-Hindi #language-Khasi #language-Kannada #language-Kashmiri #language-Maithili #language-Malayalam #language-Manipuri #language-Marathi #language-Nepali (macrolanguage) #language-Oriya (macrolanguage) #language-Panjabi #language-Sanskrit #language-Santali #language-Sindhi #language-Tamil #language-Telugu #language-Urdu #license-cc0-1.0 #arxiv-2212.05409 #region-us 
 # IndicCorpv2

Github | Paper   
Languages: 24   
The largest collection of texts for Indic languages consisting of 20.9 billion tokens of which 14.4B tokens correspond to 23 Indic languages and 6.5B tokens of Indian English content curated from Indian websites.
<img src="URL
passage: TAGS
#task_categories-text-generation #language-Assamese #language-Bodo (India) #language-Bengali #language-Dogri (macrolanguage) #language-English #language-Goan Konkani #language-Gujarati #language-Hindi #language-Khasi #language-Kannada #language-Kashmiri #language-Maithili #language-Malayalam #language-Manipuri #language-Marathi #language-Nepali (macrolanguage) #language-Oriya (macrolanguage) #language-Panjabi #language-Sanskrit #language-Santali #language-Sindhi #language-Tamil #language-Telugu #language-Urdu #license-cc0-1.0 #arxiv-2212.05409 #region-us 
# IndicCorpv2

Github | Paper   
Languages: 24   
The largest collection of texts for Indic languages consisting of 20.9 billion tokens of which 14.4B tokens correspond to 23 Indic languages and 6.5B tokens of Indian English content curated from Indian websites.
<img src="URL