Update README.md

README.md

@@ -4,24 +4,16 @@
 {}
 ---
 
-# Varta T5 model
-
-<!-- Provide a quick summary of what the model is/does. -->
+# Varta-T5
 
 ## Model Description
-Varta T5 is a model pre-trained on full training set from Varta on English and 14 Indic languages (Assamese, Bhojpuri, Bengali, Gujarati, Hindi, Kannada, Malayalam, Marathi, Nepali, Oriya, Punjabi, Tamil, Telugu, and Urdu) using span corruption and gap-sentence generation as objectives from scratch. 
-Varta is a large-scale headline-generation dataset for Indic languages, including 41.8 million news articles in 14 different Indic languages (and English), which come from a variety of high-quality sources. 
-
-
-The dataset and the model were introduced in [this paper](https://arxiv.org/pdf/2305.05858.pdf). The code was released in [this repository](https://github.com/rahular/varta). The data was released in [this bucket](https://console.cloud.google.com/storage/browser/varta-eu/data-release).
+Varta-BERT is a model pre-trained on the `full` training set of Varta in 14 Indic languages (Assamese, Bhojpuri, Bengali, Gujarati, Hindi, Kannada, Malayalam, Marathi, Nepali, Oriya, Punjabi, Tamil, Telugu, and Urdu) and English, using a masked language modeling (MLM) objective. 
 
+Varta is a large-scale news corpus for Indic languages, including 41.8 million news articles in 14 different Indic languages (and English), which come from a variety of high-quality sources. 
+The dataset and the model are introduced in [this paper](https://arxiv.org/abs/2305.05858). The code is released in [this repository](https://github.com/rahular/varta). The data is released in [this bucket](https://console.cloud.google.com/storage/browser/varta-eu/data-release).
 
 ## Uses
-
-<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
-
-You can use the raw model for language modelling, but it's mostly intended to be fine-tuned on a downstream task. <br>
-
+You can use this model for causal language modeling, but it's mostly intended to be fine-tuned on a downstream task.
 
 Note that the text-to-text framework allows us to use the same model on any NLP task, including text generation tasks (e.g., machine translation, document summarization, question answering), and classification tasks (e.g., sentiment analysis). 
 
@@ -39,7 +31,7 @@ This work is mainly dedicated to the curation of a new multilingual dataset for
 
 ## How to Get Started with the Model
 
-You can use this model directly with a pipeline for language modeling.
+You can use this model directly for span in-filling. 
 
 ```python
 from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
@@ -54,37 +46,34 @@ model = AutoModelForSeq2SeqLM.from_pretrained("rahular/varta-t5")
 
 ### Training Data
 Varta contains 41.8 million high-quality news articles in 14 Indic languages and English. 
-With 34.5 million non-English article-headline pairs, it is the largest headline-generation dataset of its kind. 
+With 34.5 million non-English article-headline pairs, it is the largest document-level dataset of its kind. 
 
 ### Pretraining
-We use span corruption and gap-sentence generation as the pretraining objectives. 
-Both objectives are sampled uniformly during pretraining. 
-Span corruption is similar to masked language modeling except that instead of masking random tokens, we mask spans of tokens with an average length of 3. 
-In gap-sentence prediction, whole sentences are masked instead of spans. We follow the original work, and select sentences based on their `importance'. 
-Rouge-1 F1-score between the sentence and the document is used as a proxy for importance. 
-We use 0.15 and 0.2 as the masking ratios for span corruption and gap-sentence generation, respectively.
+- We use span corruption and gap-sentence generation as the pretraining objectives. 
+- Both objectives are sampled uniformly during pretraining. 
+- Span corruption is similar to masked language modeling except that instead of masking random tokens, we mask spans of tokens with an average length of 3. 
+- In gap-sentence prediction, whole sentences are masked instead of spans. We follow [the original work](https://arxiv.org/abs/1912.08777), and select sentences based on their `importance'. 
+- Rouge-1 F1-score between the sentence and the document is used as a proxy for importance. 
+- We use 0.15 and 0.2 as the masking ratios for span corruption and gap-sentence generation, respectively.
 
 Since data sizes across languages in Varta vary from 1.5K (Bhojpuri) to 14.4M articles (Hindi), we use standard temperature-based sampling to upsample data when necessary.
 
-We pretrain Varta-T5 using the T5 1.1 base architecture with 12 encoder and decoder layers. 
-We train with maximum sequence lengths of 512 and 256 for the encoder and decoder respectively. 
-We use 12 attention heads with an embedding dimension of 768 and a feed-forward width of 2048. 
-We use a 128K sentencepiece vocabulary. 
-In total, the model has 395M parameters. 
-The model is trained with Adafactor optimizer with a warm-up of 10K steps. 
-We use an initial learning rate of 1e-3 and use square root decay till we reach 2M steps. 
-We use an effective batch size of 256 and train the model on TPU v3-8 chips. 
-The model takes 11 days to train.
+- We pretrain Varta-T5 using the T5 1.1 base architecture with 12 encoder and decoder layers. 
+- We train with maximum sequence lengths of 512 and 256 for the encoder and decoder respectively. 
+- We use 12 attention heads with an embedding dimension of 768 and a feed-forward width of 2048. 
+- We use a 128K sentencepiece vocabulary. 
+- In total, the model has 395M parameters. 
+- The model is trained with Adafactor optimizer with a warm-up of 10K steps. 
+- We use an initial learning rate of 1e-3 and use square root decay till we reach 2M steps. 
+- We use an effective batch size of 256 and train the model on TPU v3-8 chips. 
+- The model takes 11 days to train.
 
 <!-- This should link to a Data Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
 
 ### Evaluation Results
-To come.
+Please see [the paper](https://arxiv.org/pdf/2305.05858.pdf).
 
 ## Citation
-
-<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
-
 ```
 @misc{aralikatte2023varta,
       title={V\=arta: A Large-Scale Headline-Generation Dataset for Indic Languages},