peft safetensors generated_from_trainer base_model:mistralai/Mistral-7B-v0.1 license:apache-2.0 region:us
---
license: apache-2.0
library_name: peft
tags:
- generated_from_trainer
metrics:
- accuracy
- matthews_correlation
base_model: mistralai/Mistral-7B-v0.1
model-index:
- name: Mistral-7B-v0.1_colaMistral_scratch_cola
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# Mistral-7B-v0.1_colaMistral_scratch_cola

This model is a fine-tuned version of [mistralai/Mistral-7B-v0.1](https://huggingface.co/mistralai/Mistral-7B-v0.1) on an unknown dataset.
It achieves the following results on the evaluation set:
- Loss: 0.4281
- Accuracy: {'accuracy': 0.8387850467289719}
- Matthews Correlation: 0.6114

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 1e-05
- train_batch_size: 8
- eval_batch_size: 16
- seed: 2
- distributed_type: multi-GPU
- num_devices: 4
- gradient_accumulation_steps: 2
- total_train_batch_size: 64
- total_eval_batch_size: 64
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 20

### Training results

| Training Loss | Epoch | Step | Validation Loss | Accuracy                         | Matthews Correlation |
|:-------------:|:-----:|:----:|:---------------:|:--------------------------------:|:--------------------:|
| 1.9322        | 0.17  | 20   | 1.5215          | {'accuracy': 0.5743048897411314} | 0.0818               |
| 1.1953        | 0.33  | 40   | 0.9950          | {'accuracy': 0.660594439117929}  | 0.1870               |
| 0.6611        | 0.5   | 60   | 0.7549          | {'accuracy': 0.7353787152444871} | 0.3527               |
| 0.6165        | 0.66  | 80   | 0.6317          | {'accuracy': 0.7583892617449665} | 0.4081               |
| 0.5467        | 0.83  | 100  | 0.5667          | {'accuracy': 0.7842761265580057} | 0.5041               |
| 0.4864        | 1.0   | 120  | 0.5268          | {'accuracy': 0.7996164908916586} | 0.5385               |
| 0.478         | 1.16  | 140  | 0.4803          | {'accuracy': 0.8283796740172579} | 0.5859               |
| 0.439         | 1.33  | 160  | 0.4965          | {'accuracy': 0.8293384467881112} | 0.5818               |
| 0.4395        | 1.49  | 180  | 0.4669          | {'accuracy': 0.8283796740172579} | 0.5778               |
| 0.4202        | 1.66  | 200  | 0.5002          | {'accuracy': 0.825503355704698}  | 0.6192               |
| 0.3485        | 1.83  | 220  | 0.4360          | {'accuracy': 0.8389261744966443} | 0.6099               |
| 0.442         | 1.99  | 240  | 0.4391          | {'accuracy': 0.840843720038351}  | 0.6121               |
| 0.3752        | 2.16  | 260  | 0.4306          | {'accuracy': 0.8446788111217641} | 0.6474               |
| 0.3013        | 2.32  | 280  | 0.4163          | {'accuracy': 0.8427612655800575} | 0.6216               |
| 0.3395        | 2.49  | 300  | 0.4151          | {'accuracy': 0.8542665388302972} | 0.6592               |
| 0.3305        | 2.66  | 320  | 0.4096          | {'accuracy': 0.8475551294343241} | 0.6299               |
| 0.342         | 2.82  | 340  | 0.4101          | {'accuracy': 0.8465963566634708} | 0.6322               |
| 0.3183        | 2.99  | 360  | 0.4166          | {'accuracy': 0.8494726749760306} | 0.6364               |
| 0.2551        | 3.15  | 380  | 0.4321          | {'accuracy': 0.8542665388302972} | 0.6503               |


### Framework versions

- PEFT 0.7.1
- Transformers 4.36.2
- Pytorch 2.1.2+cu121
- Datasets 2.15.0
- Tokenizers 0.15.0