safetensors generated_from_trainer base_model:mistralai/Mistral-7B-v0.1 license:apache-2.0 region:us
---
license: apache-2.0
base_model: mistralai/Mistral-7B-v0.1
tags:
- generated_from_trainer
metrics:
- accuracy
model-index:
- name: Mistral-7B-v0.1_cola_switrelu_log
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# Mistral-7B-v0.1_cola_switrelu_log

This model is a fine-tuned version of [mistralai/Mistral-7B-v0.1](https://huggingface.co/mistralai/Mistral-7B-v0.1) on an unknown dataset.
It achieves the following results on the evaluation set:
- Loss: 0.5679
- Accuracy: 0.7512

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 1e-05
- train_batch_size: 64
- eval_batch_size: 64
- seed: 2
- distributed_type: multi-GPU
- num_devices: 2
- gradient_accumulation_steps: 2
- total_train_batch_size: 256
- total_eval_batch_size: 128
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- training_steps: 750

### Training results

| Training Loss | Epoch | Step | Validation Loss | Accuracy |
|:-------------:|:-----:|:----:|:---------------:|:--------:|
| 5.6153        | 0.33  | 10   | 5.3822          | 0.3231   |
| 1.482         | 0.66  | 20   | 1.5553          | 0.6347   |
| 1.1295        | 0.98  | 30   | 1.0969          | 0.6251   |
| 0.9153        | 1.31  | 40   | 0.9182          | 0.6290   |
| 0.7577        | 1.64  | 50   | 0.8733          | 0.6500   |
| 0.8076        | 1.97  | 60   | 0.9483          | 0.6731   |
| 0.7947        | 2.3   | 70   | 0.9969          | 0.6424   |
| 0.9011        | 2.62  | 80   | 1.2108          | 0.6059   |
| 0.9677        | 2.95  | 90   | 1.4661          | 0.6261   |
| 0.8283        | 3.28  | 100  | 1.2659          | 0.5944   |
| 0.7966        | 3.61  | 110  | 0.7540          | 0.6673   |
| 0.7234        | 3.93  | 120  | 0.6601          | 0.6347   |
| 0.6215        | 4.26  | 130  | 0.6336          | 0.6635   |
| 0.6291        | 4.59  | 140  | 0.6215          | 0.6826   |
| 0.5683        | 4.92  | 150  | 0.6205          | 0.6663   |
| 0.636         | 5.25  | 160  | 0.6146          | 0.6913   |
| 0.6094        | 5.57  | 170  | 0.6047          | 0.6779   |
| 0.6069        | 5.9   | 180  | 0.6167          | 0.6481   |
| 0.5531        | 6.23  | 190  | 0.6094          | 0.6874   |
| 0.6018        | 6.56  | 200  | 0.6195          | 0.6711   |
| 0.5861        | 6.89  | 210  | 0.5953          | 0.6807   |
| 0.5658        | 7.21  | 220  | 0.6063          | 0.6865   |
| 0.6259        | 7.54  | 230  | 0.6302          | 0.6903   |
| 0.5858        | 7.87  | 240  | 0.6422          | 0.6894   |
| 0.5213        | 8.2   | 250  | 0.5937          | 0.6922   |
| 0.5589        | 8.52  | 260  | 0.5871          | 0.6961   |
| 0.5416        | 8.85  | 270  | 0.5912          | 0.6884   |
| 0.5449        | 9.18  | 280  | 0.5917          | 0.7028   |
| 0.6001        | 9.51  | 290  | 0.5889          | 0.6826   |
| 0.5101        | 9.84  | 300  | 0.5827          | 0.6913   |
| 0.5183        | 10.16 | 310  | 0.5768          | 0.7124   |
| 0.4761        | 10.49 | 320  | 0.5882          | 0.7143   |
| 0.4813        | 10.82 | 330  | 0.5971          | 0.6817   |
| 0.5052        | 11.15 | 340  | 0.5936          | 0.7028   |
| 0.5235        | 11.48 | 350  | 0.5949          | 0.6740   |
| 0.5136        | 11.8  | 360  | 0.5926          | 0.6894   |
| 0.5185        | 12.13 | 370  | 0.5784          | 0.7172   |
| 0.4815        | 12.46 | 380  | 0.5813          | 0.7085   |
| 0.5257        | 12.79 | 390  | 0.5880          | 0.7124   |
| 0.3868        | 13.11 | 400  | 0.6054          | 0.6942   |
| 0.468         | 13.44 | 410  | 0.5956          | 0.7028   |
| 0.4823        | 13.77 | 420  | 0.5949          | 0.7114   |
| 0.3725        | 14.1  | 430  | 0.5807          | 0.7095   |
| 0.3859        | 14.43 | 440  | 0.6012          | 0.7124   |
| 0.3893        | 14.75 | 450  | 0.5994          | 0.7239   |
| 0.4392        | 15.08 | 460  | 0.5814          | 0.7162   |
| 0.4174        | 15.41 | 470  | 0.6529          | 0.7181   |
| 0.3899        | 15.74 | 480  | 0.5867          | 0.7172   |
| 0.4336        | 16.07 | 490  | 0.6386          | 0.6884   |
| 0.3928        | 16.39 | 500  | 0.6296          | 0.7152   |
| 0.4384        | 16.72 | 510  | 0.6195          | 0.7028   |
| 0.3227        | 17.05 | 520  | 0.6315          | 0.7114   |
| 0.367         | 17.38 | 530  | 0.6490          | 0.7210   |
| 0.3692        | 17.7  | 540  | 0.6184          | 0.7152   |
| 0.3205        | 18.03 | 550  | 0.6415          | 0.7124   |
| 0.3596        | 18.36 | 560  | 0.6559          | 0.7191   |
| 0.37          | 18.69 | 570  | 0.6479          | 0.7133   |


### Framework versions

- Transformers 4.35.2
- Pytorch 2.1.1+cu121
- Datasets 2.15.0
- Tokenizers 0.15.0
