# Nep_Summ_BART:

<!-- Provide a quick summary of what the model is/does. -->

This model is pre-trained using BART on Nepali corpus and then fine-tuned on Nepali summary data.
<br>The model generates a summary for the text input.

The parameter size for the model is 101M.

## Model Details

### Model Description

<!-- Provide a longer summary of what this model is. -->

The model is trained using BART noising techniques like sentence permutation, token deletion, and random token masking.
<br>The noisy data is fed into the encoder of the transformer and the denoising task/ objective is fulfilled by the decoder of the transformer model.

Cross-entropy loss is used for both the pre-training and fine-tuning of the model.

The Loss for pre-training is as follows:

| Epoch   |      Training Loss      |  Val Loss |
|----------|:-------------:|------:|
| 1 |  0.8137 | 0.8010 |
| 2 |  0.7861 | 0.7524 |
| 3 |  0.7495 | 0.7290 |

The ROUGE Score after the fine-tuning, for the BBC XLSum Nepali Test Dataset is:

ROUGE1 : 0.177

ROUGE2 : 0.059

ROUGEL : 0.154

## Uses

<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
You can use this model for text summarization.

## How to Get Started with the Model

Use the code below to get started with the model.
```
import torch

# Load model directly
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

tokenizer = AutoTokenizer.from_pretrained("pascalrai/nep_summ_BART")
model = AutoModelForSeq2SeqLM.from_pretrained("pascalrai/nep_summ_BART")

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

sentence = """अत्यधिक माग भएका बेला दसैंमा चिनीको हाहाकार भएको थियो । उपत्यकाबाहिरका केही जिल्लामा चिनी पाइए पनि काठमाडौंमा भने अभाव नै कायम रहेको छ । प्रधानमन्त्री पुष्पकमल दाहालले बिहीबार बिहान उद्योग तथा वाणिज्य मन्त्री तथा मुख्यसचिवलाई चिनीको अभाव सिर्जना हुन नदिन सबै उपायको खोजी गर्न निर्देशन दिएका थिए ।

नेपाली चिनी उद्योगहरूले आम उपभोक्तालाई सहज हुने किसिमले बजारमा चिनी नपठाइ ठूला उद्योगलाई आपूर्ति गर्न गोदाममै राख्ने गरेको पनि भेटिएको छ । वाणिज्य विभागको तथ्यांक अनुसार, नेपालमा उत्पादन हुने चिनीको सत्तरी प्रतिशत चिनी बिभिन्न पेय पदार्थ, मिठाइ, चकलेट, विस्कुटलगायतका उद्योगहरुमा आपूर्ति हुने गर्दछ ।

नेपाल प्रहरीले नेपालमा रहेका सबै चिनी उद्योगको स्टक रेकर्ड चेक गर्ने तथा सो आधारमा बजारमा चिनी पठाउन उद्योगीहरूसँग छलफल गरिने विभागले जनाएको छ"""

inputs = tokenizer(sentence, max_length=1000, return_tensors="pt")
summary_ids = model.to(device).generate(inputs["input_ids"].to(device))

tokenizer.decode(summary_ids[0], skip_special_tokens=True, clean_up_tokenization_spaces=False)

'दशैंको मुखमा चिनीको चरम अभाव भएको भन्दै नेपाल प्रहरीले सबै चिनी उद्योगको स्टक रेकर्ड चेक गर्ने र बजारमा चिनी पठाउन उद्योगीहरूसँग छलफल गर्ने जनाएको छ।'

```
#### Hardware

The model was pre-trained continuously on a single A10G GPU in an AWS instance for 133 hours with each epoch taking 45 hours using bf16 quantization.

#### Possible Future Directions:

1. Use a decoder only model for pre-training and summarization.

#### Authors:

<a href="https://www.linkedin.com/in/bijaya-bhatta-69536018a/">Vijaya Bhatta</a>
<br><a href="https://www.linkedin.com/in/pascal-rai/">Pascal Rai</a>
<br><a href="https://www.linkedin.com/in/niranjan-shrestha-gem/">Niranjan Shrestha</a>
<br><a href="https://www.linkedin.com/in/dristi-sigdel-3120131b1/">Dristi Sigdel</a>
<br><a href="https://www.linkedin.com/in/sujan-neupane-596964211/">Sujan Neupane</a>
<br><a href="https://www.linkedin.com/in/sagar-kafle-a1b84b185/">Sagar Kafle</a>
{"language": ["ne"], "license": "mit", "tags": ["Nepali summary", "Nepali bart", "Nepali", "summary", "text", "nepali text summary"], "metrics": ["rouge"], "pipeline_tag": "text2text-generation", "widget": [{"text": "\u0905\u0924\u094d\u092f\u0927\u093f\u0915 \u092e\u093e\u0917 \u092d\u090f\u0915\u093e \u092c\u0947\u0932\u093e \u0926\u0938\u0948\u0902\u092e\u093e \u091a\u093f\u0928\u0940\u0915\u094b \u0939\u093e\u0939\u093e\u0915\u093e\u0930 \u092d\u090f\u0915\u094b \u0925\u093f\u092f\u094b \u0964 \u0909\u092a\u0924\u094d\u092f\u0915\u093e\u092c\u093e\u0939\u093f\u0930\u0915\u093e \u0915\u0947\u0939\u0940 \u091c\u093f\u0932\u094d\u0932\u093e\u092e\u093e \u091a\u093f\u0928\u0940 \u092a\u093e\u0907\u090f \u092a\u0928\u093f \u0915\u093e\u0920\u092e\u093e\u0921\u094c\u0902\u092e\u093e \u092d\u0928\u0947 \u0905\u092d\u093e\u0935 \u0928\u0948 \u0915\u093e\u092f\u092e \u0930\u0939\u0947\u0915\u094b \u091b \u0964 \u092a\u094d\u0930\u0927\u093e\u0928\u092e\u0928\u094d\u0924\u094d\u0930\u0940 \u092a\u0941\u0937\u094d\u092a\u0915\u092e\u0932 \u0926\u093e\u0939\u093e\u0932\u0932\u0947 \u092c\u093f\u0939\u0940\u092c\u093e\u0930 \u092c\u093f\u0939\u093e\u0928 \u0909\u0926\u094d\u092f\u094b\u0917 \u0924\u0925\u093e \u0935\u093e\u0923\u093f\u091c\u094d\u092f \u092e\u0928\u094d\u0924\u094d\u0930\u0940 \u0924\u0925\u093e \u092e\u0941\u0916\u094d\u092f\u0938\u091a\u093f\u0935\u0932\u093e\u0908 \u091a\u093f\u0928\u0940\u0915\u094b \u0905\u092d\u093e\u0935 \u0938\u093f\u0930\u094d\u091c\u0928\u093e \u0939\u0941\u0928 \u0928\u0926\u093f\u0928 \u0938\u092c\u0948 \u0909\u092a\u093e\u092f\u0915\u094b \u0916\u094b\u091c\u0940 \u0917\u0930\u094d\u0928 \u0928\u093f\u0930\u094d\u0926\u0947\u0936\u0928 \u0926\u093f\u090f\u0915\u093e \u0925\u093f\u090f \u0964 \u0928\u0947\u092a\u093e\u0932\u0940 \u091a\u093f\u0928\u0940 \u0909\u0926\u094d\u092f\u094b\u0917\u0939\u0930\u0942\u0932\u0947 \u0906\u092e \u0909\u092a\u092d\u094b\u0915\u094d\u0924\u093e\u0932\u093e\u0908 \u0938\u0939\u091c \u0939\u0941\u0928\u0947 \u0915\u093f\u0938\u093f\u092e\u0932\u0947 \u092c\u091c\u093e\u0930\u092e\u093e \u091a\u093f\u0928\u0940 \u0928\u092a\u0920\u093e\u0907 \u0920\u0942\u0932\u093e \u0909\u0926\u094d\u092f\u094b\u0917\u0932\u093e\u0908 \u0906\u092a\u0942\u0930\u094d\u0924\u093f \u0917\u0930\u094d\u0928 \u0917\u094b\u0926\u093e\u092e\u092e\u0948 \u0930\u093e\u0916\u094d\u0928\u0947 \u0917\u0930\u0947\u0915\u094b \u092a\u0928\u093f \u092d\u0947\u091f\u093f\u090f\u0915\u094b \u091b \u0964 \u0935\u093e\u0923\u093f\u091c\u094d\u092f \u0935\u093f\u092d\u093e\u0917\u0915\u094b \u0924\u0925\u094d\u092f\u093e\u0902\u0915 \u0905\u0928\u0941\u0938\u093e\u0930, \u0928\u0947\u092a\u093e\u0932\u092e\u093e \u0909\u0924\u094d\u092a\u093e\u0926\u0928 \u0939\u0941\u0928\u0947 \u091a\u093f\u0928\u0940\u0915\u094b \u0938\u0924\u094d\u0924\u0930\u0940 \u092a\u094d\u0930\u0924\u093f\u0936\u0924 \u091a\u093f\u0928\u0940 \u092c\u093f\u092d\u093f\u0928\u094d\u0928 \u092a\u0947\u092f \u092a\u0926\u093e\u0930\u094d\u0925, \u092e\u093f\u0920\u093e\u0907, \u091a\u0915\u0932\u0947\u091f, \u0935\u093f\u0938\u094d\u0915\u0941\u091f\u0932\u0917\u093e\u092f\u0924\u0915\u093e \u0909\u0926\u094d\u092f\u094b\u0917\u0939\u0930\u0941\u092e\u093e \u0906\u092a\u0942\u0930\u094d\u0924\u093f \u0939\u0941\u0928\u0947 \u0917\u0930\u094d\u0926\u091b \u0964 \u0928\u0947\u092a\u093e\u0932 \u092a\u094d\u0930\u0939\u0930\u0940\u0932\u0947 \u0928\u0947\u092a\u093e\u0932\u092e\u093e \u0930\u0939\u0947\u0915\u093e \u0938\u092c\u0948 \u091a\u093f\u0928\u0940 \u0909\u0926\u094d\u092f\u094b\u0917\u0915\u094b \u0938\u094d\u091f\u0915 \u0930\u0947\u0915\u0930\u094d\u0921 \u091a\u0947\u0915 \u0917\u0930\u094d\u0928\u0947 \u0924\u0925\u093e \u0938\u094b \u0906\u0927\u093e\u0930\u092e\u093e \u092c\u091c\u093e\u0930\u092e\u093e \u091a\u093f\u0928\u0940 \u092a\u0920\u093e\u0909\u0928 \u0909\u0926\u094d\u092f\u094b\u0917\u0940\u0939\u0930\u0942\u0938\u0901\u0917 \u091b\u0932\u092b\u0932 \u0917\u0930\u093f\u0928\u0947 \u0935\u093f\u092d\u093e\u0917\u0932\u0947 \u091c\u0928\u093e\u090f\u0915\u094b \u091b \u0964", "example_title": "Example 1"}]}
transformers safetensors bart text2text-generation Nepali summary Nepali bart Nepali summary text nepali text summary ne license:mit autotrain_compatible endpoints_compatible region:us