Release of KooBERT

.gitattributes

@@ -0,0 +1,34 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,186 @@
+---
+# For reference on model card metadata, see the spec: https://github.com/huggingface/hub-docs/blob/main/modelcard.md?plain=1
+# Doc / guide: https://huggingface.co/docs/hub/model-cards
+{}
+---
+
+# Model Card for KooBERT
+
+KooBERT is a masked language model trained on data from the multilingual micro-blogging social media platform [Koo India](https://www.kooapp.com/). <br>
+This model was built in collaboration with Koo India and AI4Bharat.
+
+## Model Details
+
+### Model Description
+
+On Koo platform, we have microblogs (Koos) which are limited to 400 characters and are available in multiple languages.
+The model was trained on a dataset that contains multilingual koos from Jan 2020 to Nov 2022 on masked language modeling task.
+
+
+- **Model type:** BERT based pretrained model
+- **Language(s) (NLP):** assamese, bengali, english, gujarati, hindi, kannada, malayalam, marathi, nigeran english, oriya, punjabi, tamil, telugu
+- **License:** KooBERT released under the MIT License.
+
+## Uses
+
+This model can be used to perform downstream tasks like content classification, toxicity detection, etc. for supported Indic languages <br>
+It can also be used with sentence-transformers library for the creation of multilingual vector embeddings for other uses.
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+## Bias, Risks, and Limitations
+As with any machine learning model, KooBERT may have limitations and biases. It is important to keep in mind that this model was trained on Koo Social Media data and may not generalize well to other domains. It is also possible that the model may have biases in the data it was trained on, which may affect its predictions. It is recommended to evaluate the model on your specific use case and data to ensure it is appropriate for your needs.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model for general finetuning tasks. Please note this is just a sample for finetuning.
+
+```
+import torch
+from datasets import load_dataset, load_metric
+from transformers import AutoTokenizer, AutoModel, TrainingArguments, Trainer
+import evaluate
+metric = evaluate.load("accuracy")
+def compute_metrics(eval_pred):
+    logits, labels = eval_pred
+    predictions = np.argmax(logits, axis=-1)
+    return metric.compute(predictions=predictions, references=labels)
+
+# Load the CoLA dataset
+cola_dataset = load_dataset("glue", "cola")
+
+cola_dataset = cola_dataset.rename_column('label', 'labels')
+cola_dataset = cola_dataset.rename_column('sentence', 'text')
+
+# Load the tokenizer and model
+tokenizer = AutoTokenizer.from_pretrained("Koodsml/KooBERT")
+model = AutoModel.from_pretrained("Koodsml/KooBERT", num_labels=2)
+
+def tokenize_function(examples):
+    return tokenizer(examples["text"], padding="max_length", truncation=True, max_length=128)
+
+
+cola_dataset = cola_dataset.map(tokenize_function, batched=True)
+
+# Set the device
+device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
+model.to(device)
+
+# Define the training arguments
+training_args = TrainingArguments(
+    output_dir='./results',
+    evaluation_strategy='epoch',
+#     eval_steps=100,
+    save_total_limit=1,
+    learning_rate=2e-5,
+    per_device_train_batch_size=8,
+    per_device_eval_batch_size=8,
+    num_train_epochs=3,
+    weight_decay=0.01,
+    push_to_hub=False,
+)
+
+# Define the trainer
+trainer = Trainer(
+    model=model,
+    args=training_args,
+    train_dataset=cola_dataset['train'],
+    eval_dataset=cola_dataset['validation'],
+#     tokenizer=tokenizer,
+    compute_metrics=compute_metrics
+)
+
+# Fine-tune on the CoLA dataset
+trainer.train()
+
+# Evaluate on the CoLA dataset
+eval_results = trainer.evaluate(eval_dataset=cola_dataset['validation'])
+print(eval_results)
+```
+
+We can also use KooBERT with the sentence-transformers library to create multilingual vector embeddings. Here is an example:
+```
+from sentence_transformers import SentenceTransformer
+
+# Load the KooBERT model
+koo_model = SentenceTransformer('Koodsml/KooBERT', device="cuda")
+
+# Define the text
+text = "यह हमेशा से हमारी सोच है"
+
+# Get the embedding
+embedding = koo_model.encode(text)
+print(embedding)
+```
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Data Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+Following is the distribution of tokens over languages:
+
+| Language         | Koos        | Avg Tokens per Koo  | Total Tokens |
+|------------------|-------------|---------------------|--------------|
+| assamese         | 562,050     | 16.4414198          | 9,240,900    |
+| bengali          | 2,110,380   | 12.08918773         | 25,512,780   |
+| english          | 17,889,600  | 10.93732057         | 195,664,290  |
+| gujarati         | 1,825,770   | 14.33965395         | 26,180,910   |
+| hindi            | 35,948,760  | 16.2337502          | 583,583,190  |
+| kannada          | 2,653,860   | 12.04577107         | 31,967,790   |
+| malayalam        | 71,370      | 10.32744851         | 737,070      |
+| marathi          | 1,894,080   | 14.81544602         | 28,061,640   |
+| nigeran english  | 255,330     | 17.11350018         | 4,369,590    |
+| oriya            | 87,930      | 14.1941317          | 1,248,090    |
+| punjabi          | 940,260     | 18.59961075         | 17,488,470   |
+| tamil            | 1,687,710   | 12.12147822         | 20,457,540   |
+| telugu           | 2,471,940   | 10.55735576         | 26,097,150   |
+
+
+Total Koos = 68,399,040<br>
+Total Tokens = 970,609,410 
+
+### Training Procedure 
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing
+Personal Identifiable Information (PII) was removed from data before training on microblogs.
+Temperature Sampling to upsample low resource languages. We used a temperature of value of 0.7 (Refer Sec 3.1 https://arxiv.org/pdf/1901.07291.pdf) 
+
+
+#### Training Hyperparameters
+
+- **Training regime** 
+Training steps - 1M steps
+Warm - 10k steps 
+Learning Rate - 5e-4
+Scheduler - Linear Decay
+Optimizer - Adam
+Batch Size of 4096 sequences
+Precision - fp32
+
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+The model has not been benchmarked yet. We shall be releasing the benchmark data in a future update.
+
+
+## Model Examination
+
+<!-- Relevant interpretability work for the model goes here -->
+
+### Model Architecture and Objective
+
+KooBERT is pretrained with BERT Architecture on Masked Language Modeling with a vocabulary size of 128k and max sequence length of 128 tokens.
+
+### Compute Infrastructure
+
+KooBERT was trained on TPU v3 with 128 cores which took over 5 days.
+
+
+## Contributors
+
+Mitesh Khapra ([[email protected]](mailto:[email protected]))- IITM AI4Bharat<br>
+Sumanth Doddapaneni ([[email protected]](mailto:[email protected]))- IITM AI4Bharat<br>
+Smiral Rashinkar ([[email protected]](mailto:[email protected]))- Koo India
+

config.json

@@ -0,0 +1,26 @@
+{
+  "_name_or_path": "/data/sumanth/conversion/koo/",
+  "architectures": [
+    "BertForMaskedLM"
+  ],
+  "attention_probs_dropout_prob": 0.1,
+  "classifier_dropout": null,
+  "embedding_size": 768,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 768,
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "layer_norm_eps": 1e-12,
+  "max_position_embeddings": 128,
+  "model_type": "bert",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 12,
+  "pad_token_id": 0,
+  "position_embedding_type": "absolute",
+  "torch_dtype": "float32",
+  "transformers_version": "4.21.1",
+  "type_vocab_size": 2,
+  "use_cache": true,
+  "vocab_size": 128000
+}

pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1e528f093db26d5c350a82ab3a6d2797f672ad7f2738d9de1268f45d47f09548
+size 736785579

special_tokens_map.json

@@ -0,0 +1,7 @@
+{
+  "cls_token": "[CLS]",
+  "mask_token": "[MASK]",
+  "pad_token": "[PAD]",
+  "sep_token": "[SEP]",
+  "unk_token": "[UNK]"
+}

tokenizer_config.json

@@ -0,0 +1,3 @@
+{
+  "tokenizer_class": "PreTrainedTokenizerFast"
+}