# Model Card for Backpack-GPT2-NLI
This is a fine-tuned version of [backpack-gpt2](https://huggingface.co/stanfordnlp/backpack-gpt2) with a NLI classification head on the [esnli](https://huggingface.co/datasets/esnli) dataset.
Results:
- On Validation Set:
  - CrossEntropyLoss: 0.3168
  - Accuracy: 0.9006
  - F1: 0.9004
- On Test Set:
  - CrossEntropyLoss: 0.3277
  - Accuracy: 0.8958
  - F1: 0.8955
### Model Description
- **Developed by:** [Erfan Moosavi Monazzah](https://huggingface.co/ErfanMoosaviMonazzah)
- **Model type:** Sequence Classifier
- **Language(s) (NLP):** English
- **License:** apache-2.0
- **Finetuned from model [optional]:** [Backpack-GPT2](https://huggingface.co/stanfordnlp/backpack-gpt2)


## How to Get Started with the Model
```python
from transformers import AutoTokenizer, AutoModelForSequenceClassification

tokenizer = AutoTokenizer.from_pretrained('gpt2')
tokenizer.pad_token = tokenizer.eos_token

def tokenize_function(examples):
    concatenated_sentences = [f'{premise.strip(".")}. ^ {hypothesis.strip(".")}.' for premise, hypothesis in zip(examples['premise'], examples['hypothesis'])]

    tokenized_inputs = tokenizer(
        concatenated_sentences,
        padding="max_length",
        truncation=True,
        max_length=512,
        return_tensors="pt",
    )
    return tokenized_inputs

model = AutoModelForSequenceClassification.from_pretrained('ErfanMoosaviMonazzah/backpack-gpt2-nli', trust_remote_code=True)
model.eval()

tokenized_sent = tokenize_function({
    'premise':['A boy is jumping on skateboard in the middle of a red bridge.',
               'Two women who just had lunch hugging and saying goodbye.',
               'Children smiling and waving at camera'],
    'hypothesis':['The boy does a skateboarding trick.',
                  'The friends have just met for the first time in 20 years, and have had a great time catching up.',
                  'The kids are frowning']
})
model.predict(input_ids=tokenized_sent['input_ids'], attention_mask=tokenized_sent['attention_mask'])
```
### Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 5e-5
- train_batch_size: 64
- eval_batch_size: 64
- seed: 2023
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_ratio: 0
- num_epochs: 3
### Training results
|Step        |Training Loss|Validation Loss|Precision|Recall  |F1      |Accuracy|
|------------|-------------|---------------|---------|--------|--------|--------|
|512         |0.614900     |0.463713       |0.826792 |0.824639|0.825133|0.824731|
|1024        |0.503300     |0.431796       |0.844831 |0.839414|0.839980|0.839565|
|1536        |0.475600     |0.400771       |0.848741 |0.847009|0.846287|0.847795|
|2048        |0.455900     |0.375981       |0.859064 |0.857357|0.857749|0.857448|
|2560        |0.440400     |0.365537       |0.862000 |0.862078|0.861917|0.862426|
|3072        |0.433100     |0.365180       |0.864717 |0.859693|0.860237|0.859785|
|3584        |0.425100     |0.346340       |0.872312 |0.870635|0.870865|0.870961|
|4096        |0.413300     |0.343761       |0.873606 |0.873046|0.873174|0.873298|
|4608        |0.412000     |0.344890       |0.882609 |0.882120|0.882255|0.882341|
|5120        |0.402600     |0.336744       |0.876463 |0.875629|0.875827|0.875737|
|5632        |0.390600     |0.323248       |0.882598 |0.880779|0.881129|0.880817|
|6144        |0.388300     |0.338029       |0.877255 |0.877041|0.877126|0.877261|
|6656        |0.390800     |0.333301       |0.876357 |0.876362|0.875965|0.876753|
|7168        |0.383800     |0.328297       |0.883593 |0.883675|0.883629|0.883967|
|7680        |0.380800     |0.331854       |0.882362 |0.880373|0.880764|0.880512|
|8192        |0.368400     |0.323076       |0.881730 |0.881378|0.881419|0.881528|
|8704        |0.367000     |0.313959       |0.889204 |0.889047|0.889053|0.889352|
|9216        |0.315600     |0.333637       |0.885518 |0.883965|0.884266|0.883967|
|9728        |0.303100     |0.319416       |0.888667 |0.888092|0.888256|0.888234|
|10240       |0.307200     |0.317827       |0.887575 |0.887647|0.887418|0.888031|
|10752       |0.300100     |0.311810       |0.890908 |0.890827|0.890747|0.891181|
|11264       |0.303400     |0.311010       |0.889871 |0.887939|0.888309|0.887929|
|11776       |0.300500     |0.309282       |0.891041 |0.889819|0.890077|0.889860|
|12288       |0.303600     |0.326918       |0.891272 |0.891250|0.890942|0.891689|
|12800       |0.300300     |0.301688       |0.894516 |0.894619|0.894481|0.894940|
|13312       |0.302200     |0.302173       |0.896441 |0.896527|0.896462|0.896769|
|13824       |0.299800     |0.293489       |0.895047 |0.895172|0.895084|0.895448|
|14336       |0.294600     |0.297645       |0.895865 |0.896012|0.895886|0.896261|
|14848       |0.296700     |0.300751       |0.895277 |0.895401|0.895304|0.895651|
|15360       |0.293100     |0.293049       |0.896855 |0.896705|0.896757|0.896871|
|15872       |0.293600     |0.294201       |0.895933 |0.895557|0.895624|0.895651|
|16384       |0.290100     |0.289367       |0.897847 |0.897889|0.897840|0.898090|
|16896       |0.293600     |0.283990       |0.898889 |0.898724|0.898789|0.898903|
|17408       |0.285800     |0.308257       |0.898250 |0.898102|0.898162|0.898293|
|17920       |0.252400     |0.327164       |0.898860 |0.898807|0.898831|0.899004|
|18432       |0.219500     |0.315286       |0.898877 |0.898835|0.898831|0.899004|
|18944       |0.217900     |0.312738       |0.898857 |0.898958|0.898886|0.899207|
|19456       |0.186400     |0.320669       |0.899252 |0.899166|0.899194|0.899411|
|19968       |0.199000     |0.316840       |0.900458 |0.900455|0.900426|0.900630|


## Model Card Authors

[Erfan Moosavi Monazzah](https://huggingface.co/ErfanMoosaviMonazzah)
{"language": ["en"], "license": "apache-2.0", "tags": ["Natural Language Inference", "Sequence Classification", "GPT2", "Backpack", "ESNLI"], "datasets": ["esnli"], "metrics": ["accuracy", "f1", "precision", "recall"], "pipeline_tag": "text-classification", "model-index": [{"name": "backpack-gpt2-nli", "results": [{"task": {"type": "text-classification", "name": "Natural Language Inference"}, "dataset": {"name": "e-SNLI", "type": "esnli", "split": "validation"}, "metrics": [{"type": "accuracy", "value": 0.9006299532615322, "name": "Accuracy"}, {"type": "f1", "value": 0.9004261302857443, "name": "F1"}, {"type": "precision", "value": 0.9004584180714215, "name": "Precision"}, {"type": "recall", "value": 0.9004554220756779, "name": "Recall"}]}, {"task": {"type": "text-classification", "name": "Natural Language Inference"}, "dataset": {"name": "e-SNLI", "type": "esnli", "split": "test"}, "metrics": [{"type": "accuracy", "value": 0.8957654723127035, "name": "Accuracy"}, {"type": "f1", "value": 0.8954702227331482, "name": "F1"}, {"type": "precision", "value": 0.8954036872157838, "name": "Precision"}, {"type": "recall", "value": 0.8955997285576146, "name": "Recall"}]}]}]}
transformers pytorch safetensors gpt2 text-classification Natural Language Inference Sequence Classification GPT2 Backpack ESNLI custom_code en dataset:esnli license:apache-2.0 model-index autotrain_compatible text-generation-inference region:us