safetensors generated_from_trainer base_model:mistralai/Mistral-7B-v0.1 license:apache-2.0 region:us
---
license: apache-2.0
base_model: mistralai/Mistral-7B-v0.1
tags:
- generated_from_trainer
metrics:
- accuracy
- matthews_correlation
model-index:
- name: Mistral-7B-v0.1_cola_sparse_swiglu_scratch
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# Mistral-7B-v0.1_cola_sparse_swiglu_scratch

This model is a fine-tuned version of [mistralai/Mistral-7B-v0.1](https://huggingface.co/mistralai/Mistral-7B-v0.1) on an unknown dataset.
It achieves the following results on the evaluation set:
- Loss: 0.3823
- Accuracy: {'accuracy': 0.8621495327102804}
- Matthews Correlation: 0.6666

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 1e-05
- train_batch_size: 8
- eval_batch_size: 16
- seed: 2
- distributed_type: multi-GPU
- num_devices: 6
- gradient_accumulation_steps: 2
- total_train_batch_size: 96
- total_eval_batch_size: 96
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 40

### Training results

| Training Loss | Epoch | Step | Validation Loss | Accuracy                         | Matthews Correlation |
|:-------------:|:-----:|:----:|:---------------:|:--------------------------------:|:--------------------:|
| 1.0186        | 0.31  | 25   | 0.8870          | {'accuracy': 0.7152444870565676} | 0.2813               |
| 0.7573        | 0.62  | 50   | 0.6207          | {'accuracy': 0.788111217641419}  | 0.4870               |
| 0.5186        | 0.93  | 75   | 0.5819          | {'accuracy': 0.7986577181208053} | 0.4984               |
| 0.4259        | 1.24  | 100  | 0.5096          | {'accuracy': 0.8149568552253116} | 0.5459               |
| 0.5015        | 1.55  | 125  | 0.4887          | {'accuracy': 0.8302972195589645} | 0.6063               |
| 0.4622        | 1.86  | 150  | 0.4573          | {'accuracy': 0.8437200383509108} | 0.6273               |
| 0.3411        | 2.17  | 175  | 0.4755          | {'accuracy': 0.835091083413231}  | 0.5958               |
| 0.3637        | 2.48  | 200  | 0.4302          | {'accuracy': 0.8341323106423778} | 0.5941               |
| 0.3376        | 2.8   | 225  | 0.3974          | {'accuracy': 0.8446788111217641} | 0.6325               |
| 0.2879        | 3.11  | 250  | 0.5189          | {'accuracy': 0.8130393096836049} | 0.6287               |
| 0.2691        | 3.42  | 275  | 0.4033          | {'accuracy': 0.8331735378715245} | 0.6148               |


### Framework versions

- Transformers 4.35.2
- Pytorch 2.1.1+cu121
- Datasets 2.15.0
- Tokenizers 0.15.0
