
## Dataset Description

- **Repository:** https://github.com/rahular/varta
- **Paper:** https://arxiv.org/abs/2305.05858

### Dataset Summary

Varta is a diverse, challenging, large-scale, multilingual, and high-quality headline-generation dataset containing 41.8 million news articles in 14 Indic languages and English. 
The data is crawled from DailyHunt, a popular news aggregator in India that pulls high-quality articles from multiple trusted and reputed news publishers. 

### Languages

Assamese, Bhojpuri, Bengali, English, Gujarati, Hindi, Kannada, Malayalam, Marathi, Nepali, Oriya, Punjabi, Tamil, Telugu, and Urdu.

## Dataset Structure

### Data Instances
```
{
"id":"n400000150",
"langCode":"as",
"source_url":"https://www.etvbharat.com/assamese/assam/bharat/militant-hideout-destroyed-on-srinagar-bandipora-highway/assam20220630074145729729173",
"dh_url":"https://m.dailyhunt.in/news/india/assamese/etvbharatassamese-epaper-dh6b381d65c3344bbcad9a06ee28b4ab2a/boma+nikshepeve+dhbans+kva+hl+santvasabadiv+aatmagopanasthali-newsid-n400000150"
}
```

### Data Fields

- id: unique identifier for the artilce on DailyHunt. This id will be used to recreate the dataset.
- langCode: ISO 639-1 language code
- source_url: the url that points to the article on the website of the original publisher
- dh_url: the url that points to the article on DailyHunt

### Data Splits

From every language, we randomly sample 10,000 articles each for validation and testing. We also ensure that at least 80% of a language’s data is available for training. 
Therefore, if a language has less than 100,000 articles, we restrict its validation and test splits to 10% of its size.

We also create a `small` training set by limiting the number of articles from each language to 100K.
This `small` training set with a size of 1.3M is used in all our fine-tuning experiments.
You can find the `small` training set [here](https://huggingface.co/datasets/rahular/varta/blob/main/varta/train/train_100k.json)

## Data Recreation
To recreate the dataset, follow this [README file](https://github.com/rahular/varta/tree/main/crawler#README.md).

## Misc
- Original source: https://m.dailyhunt.in/
- License: CC-BY 4.0

## Citation Information

```
@misc{aralikatte2023varta,
      title={V\=arta: A Large-Scale Headline-Generation Dataset for Indic Languages}, 
      author={Rahul Aralikatte and Ziling Cheng and Sumanth Doddapaneni and Jackie Chi Kit Cheung},
      year={2023},
      eprint={2305.05858},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}
```
task_categories:summarization task_categories:feature-extraction size_categories:1B<n<10B language:as language:bh language:bn language:en language:gu language:hi language:kn language:ml language:mr language:ne language:or language:pa language:ta language:te language:ur license:cc arxiv:2305.05858 region:us
{"language": ["as", "bh", "bn", "en", "gu", "hi", "kn", "ml", "mr", "ne", "or", "pa", "ta", "te", "ur"], "license": "cc", "size_categories": ["1B<n<10B"], "task_categories": ["summarization", "feature-extraction"], "pretty_name": "varta"}
TAGS
#task_categories-summarization #task_categories-feature-extraction #size_categories-1B<n<10B #language-Assamese #language-bh #language-Bengali #language-English #language-Gujarati #language-Hindi #language-Kannada #language-Malayalam #language-Marathi #language-Nepali (macrolanguage) #language-Oriya (macrolanguage) #language-Panjabi #language-Tamil #language-Telugu #language-Urdu #license-cc #arxiv-2305.05858 #region-us 


## Dataset Description

- Repository: URL
- Paper: URL

### Dataset Summary

Varta is a diverse, challenging, large-scale, multilingual, and high-quality headline-generation dataset containing 41.8 million news articles in 14 Indic languages and English. 
The data is crawled from DailyHunt, a popular news aggregator in India that pulls high-quality articles from multiple trusted and reputed news publishers. 

### Languages

Assamese, Bhojpuri, Bengali, English, Gujarati, Hindi, Kannada, Malayalam, Marathi, Nepali, Oriya, Punjabi, Tamil, Telugu, and Urdu.

## Dataset Structure

### Data Instances


### Data Fields

- id: unique identifier for the artilce on DailyHunt. This id will be used to recreate the dataset.
- langCode: ISO 639-1 language code
- source_url: the url that points to the article on the website of the original publisher
- dh_url: the url that points to the article on DailyHunt

### Data Splits

From every language, we randomly sample 10,000 articles each for validation and testing. We also ensure that at least 80% of a language’s data is available for training. 
Therefore, if a language has less than 100,000 articles, we restrict its validation and test splits to 10% of its size.

We also create a 'small' training set by limiting the number of articles from each language to 100K.
This 'small' training set with a size of 1.3M is used in all our fine-tuning experiments.
You can find the 'small' training set here

## Data Recreation
To recreate the dataset, follow this README file.

## Misc
- Original source: https://m.URL
- License: CC-BY 4.0




## Dataset Description

- Repository: URL
- Paper: URL ### Dataset Summary

Varta is a diverse, challenging, large-scale, multilingual, and high-quality headline-generation dataset containing 41.8 million news articles in 14 Indic languages and English. 
The data is crawled from DailyHunt, a popular news aggregator in India that pulls high-quality articles from multiple trusted and reputed news publishers. ### Languages

Assamese, Bhojpuri, Bengali, English, Gujarati, Hindi, Kannada, Malayalam, Marathi, Nepali, Oriya, Punjabi, Tamil, Telugu, and Urdu. ## Dataset Structure ### Data Instances ### Data Fields

- id: unique identifier for the artilce on DailyHunt. This id will be used to recreate the dataset.
- langCode: ISO 639-1 language code
- source_url: the url that points to the article on the website of the original publisher
- dh_url: the url that points to the article on DailyHunt ### Data Splits

From every language, we randomly sample 10,000 articles each for validation and testing. We also ensure that at least 80% of a language’s data is available for training. 
Therefore, if a language has less than 100,000 articles, we restrict its validation and test splits to 10% of its size.

We also create a 'small' training set by limiting the number of articles from each language to 100K.
This 'small' training set with a size of 1.3M is used in all our fine-tuning experiments.
You can find the 'small' training set here ## Data Recreation
To recreate the dataset, follow this README file. ## Misc
- Original source: https://m.URL
- License: CC-BY 4.0
TAGS
#task_categories-summarization #task_categories-feature-extraction #size_categories-1B<n<10B #language-Assamese #language-bh #language-Bengali #language-English #language-Gujarati #language-Hindi #language-Kannada #language-Malayalam #language-Marathi #language-Nepali (macrolanguage) #language-Oriya (macrolanguage) #language-Panjabi #language-Tamil #language-Telugu #language-Urdu #license-cc #arxiv-2305.05858 #region-us 
 ## Dataset Description

- Repository: URL
- Paper: URL ### Dataset Summary

Varta is a diverse, challenging, large-scale, multilingual, and high-quality headline-generation dataset containing 41.8 million news articles in 14 Indic languages and English. 
The data is crawled from DailyHunt, a popular news aggregator in India that pulls high-quality articles from multiple trusted and reputed news publishers. ### Languages

Assamese, Bhojpuri, Bengali, English, Gujarati, Hindi, Kannada, Malayalam, Marathi, Nepali, Oriya, Punjabi, Tamil, Telugu, and Urdu. ## Dataset Structure ### Data Instances ### Data Fields

- id: unique identifier for the artilce on DailyHunt. This id will be used to recreate the dataset.
- langCode: ISO 639-1 language code
- source_url: the url that points to the article on the website of the original publisher
- dh_url: the url that points to the article on DailyHunt ### Data Splits

From every language, we randomly sample 10,000 articles each for validation and testing. We also ensure that at least 80% of a language’s data is available for training. 
Therefore, if a language has less than 100,000 articles, we restrict its validation and test splits to 10% of its size.

We also create a 'small' training set by limiting the number of articles from each language to 100K.
This 'small' training set with a size of 1.3M is used in all our fine-tuning experiments.
You can find the 'small' training set here ## Data Recreation
To recreate the dataset, follow this README file. ## Misc
- Original source: https://m.URL
- License: CC-BY 4.0
passage: TAGS
#task_categories-summarization #task_categories-feature-extraction #size_categories-1B<n<10B #language-Assamese #language-bh #language-Bengali #language-English #language-Gujarati #language-Hindi #language-Kannada #language-Malayalam #language-Marathi #language-Nepali (macrolanguage) #language-Oriya (macrolanguage) #language-Panjabi #language-Tamil #language-Telugu #language-Urdu #license-cc #arxiv-2305.05858 #region-us 
## Dataset Description

- Repository: URL
- Paper: URL### Dataset Summary

Varta is a diverse, challenging, large-scale, multilingual, and high-quality headline-generation dataset containing 41.8 million news articles in 14 Indic languages and English. 
The data is crawled from DailyHunt, a popular news aggregator in India that pulls high-quality articles from multiple trusted and reputed news publishers.### Languages

Assamese, Bhojpuri, Bengali, English, Gujarati, Hindi, Kannada, Malayalam, Marathi, Nepali, Oriya, Punjabi, Tamil, Telugu, and Urdu.## Dataset Structure### Data Instances### Data Fields

- id: unique identifier for the artilce on DailyHunt. This id will be used to recreate the dataset.
- langCode: ISO 639-1 language code
- source_url: the url that points to the article on the website of the original publisher
- dh_url: the url that points to the article on DailyHunt### Data Splits

From every language, we randomly sample 10,000 articles each for validation and testing. We also ensure that at least 80% of a language’s data is available for training. 
Therefore, if a language has less than 100,000 articles, we restrict its validation and test splits to 10% of its size.

We also create a 'small' training set by limiting the number of articles from each language to 100K.
This 'small' training set with a size of 1.3M is used in all our fine-tuning experiments.
You can find the 'small' training set here