peft safetensors trl sft generated_from_trainer base_model:TheBloke/Mistral-7B-v0.1-GPTQ license:apache-2.0 region:us
---
license: apache-2.0
library_name: peft
tags:
- trl
- sft
- generated_from_trainer
base_model: TheBloke/Mistral-7B-v0.1-GPTQ
model-index:
- name: mistral-rand
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# mistral-rand

This model is a fine-tuned version of [TheBloke/Mistral-7B-v0.1-GPTQ](https://huggingface.co/TheBloke/Mistral-7B-v0.1-GPTQ) on the None dataset.
It achieves the following results on the evaluation set:
- Loss: 0.4471

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 2e-05
- train_batch_size: 4
- eval_batch_size: 8
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: cosine
- num_epochs: 15
- mixed_precision_training: Native AMP

### Training results

| Training Loss | Epoch | Step  | Validation Loss |
|:-------------:|:-----:|:-----:|:---------------:|
| 1.7543        | 0.03  | 50    | 0.9190          |
| 0.8445        | 0.05  | 100   | 0.7860          |
| 0.7819        | 0.07  | 150   | 0.7460          |
| 0.7231        | 0.1   | 200   | 0.7147          |
| 0.6985        | 0.12  | 250   | 0.6924          |
| 0.6887        | 0.15  | 300   | 0.6823          |
| 0.6836        | 0.17  | 350   | 0.6702          |
| 0.6624        | 0.2   | 400   | 0.6574          |
| 0.6712        | 0.23  | 450   | 0.6507          |
| 0.6354        | 0.25  | 500   | 0.6417          |
| 0.6089        | 0.28  | 550   | 0.6373          |
| 0.6236        | 0.3   | 600   | 0.6284          |
| 0.6161        | 0.33  | 650   | 0.6228          |
| 0.6367        | 0.35  | 700   | 0.6152          |
| 0.6329        | 0.38  | 750   | 0.6097          |
| 0.5944        | 0.4   | 800   | 0.6076          |
| 0.6036        | 0.42  | 850   | 0.6030          |
| 0.5767        | 0.45  | 900   | 0.5989          |
| 0.6079        | 0.47  | 950   | 0.5954          |
| 0.5915        | 0.5   | 1000  | 0.5916          |
| 0.5911        | 0.53  | 1050  | 0.5859          |
| 0.5752        | 0.55  | 1100  | 0.5847          |
| 0.5698        | 0.57  | 1150  | 0.5802          |
| 0.5813        | 0.6   | 1200  | 0.5754          |
| 0.5918        | 0.62  | 1250  | 0.5735          |
| 0.5587        | 0.65  | 1300  | 0.5677          |
| 0.5933        | 0.68  | 1350  | 0.5620          |
| 0.5262        | 0.7   | 1400  | 0.5522          |
| 0.5455        | 0.72  | 1450  | 0.5457          |
| 0.5472        | 0.75  | 1500  | 0.5416          |
| 0.536         | 0.78  | 1550  | 0.5400          |
| 0.527         | 0.8   | 1600  | 0.5393          |
| 0.5516        | 0.82  | 1650  | 0.5350          |
| 0.5578        | 0.85  | 1700  | 0.5356          |
| 0.5501        | 0.88  | 1750  | 0.5297          |
| 0.5316        | 0.9   | 1800  | 0.5288          |
| 0.5436        | 0.93  | 1850  | 0.5268          |
| 0.514         | 0.95  | 1900  | 0.5295          |
| 0.5249        | 0.97  | 1950  | 0.5246          |
| 0.538         | 1.0   | 2000  | 0.5226          |
| 0.4967        | 1.02  | 2050  | 0.5237          |
| 0.4991        | 1.05  | 2100  | 0.5261          |
| 0.5142        | 1.07  | 2150  | 0.5203          |
| 0.4891        | 1.1   | 2200  | 0.5174          |
| 0.5058        | 1.12  | 2250  | 0.5173          |
| 0.4895        | 1.15  | 2300  | 0.5182          |
| 0.4918        | 1.18  | 2350  | 0.5139          |
| 0.485         | 1.2   | 2400  | 0.5091          |
| 0.5173        | 1.23  | 2450  | 0.5121          |
| 0.5021        | 1.25  | 2500  | 0.5116          |
| 0.4834        | 1.27  | 2550  | 0.5097          |
| 0.4754        | 1.3   | 2600  | 0.5137          |
| 0.4907        | 1.32  | 2650  | 0.5059          |
| 0.5155        | 1.35  | 2700  | 0.5051          |
| 0.4965        | 1.38  | 2750  | 0.5050          |
| 0.5148        | 1.4   | 2800  | 0.5043          |
| 0.4709        | 1.43  | 2850  | 0.5032          |
| 0.4864        | 1.45  | 2900  | 0.5037          |
| 0.4794        | 1.48  | 2950  | 0.5029          |
| 0.4803        | 1.5   | 3000  | 0.5012          |
| 0.4843        | 1.52  | 3050  | 0.5017          |
| 0.4726        | 1.55  | 3100  | 0.4984          |
| 0.4773        | 1.57  | 3150  | 0.4968          |
| 0.4673        | 1.6   | 3200  | 0.4995          |
| 0.4803        | 1.62  | 3250  | 0.4990          |
| 0.4926        | 1.65  | 3300  | 0.4965          |
| 0.4814        | 1.68  | 3350  | 0.4973          |
| 0.4714        | 1.7   | 3400  | 0.4930          |
| 0.4797        | 1.73  | 3450  | 0.4903          |
| 0.4807        | 1.75  | 3500  | 0.4932          |
| 0.4815        | 1.77  | 3550  | 0.4888          |
| 0.4852        | 1.8   | 3600  | 0.4874          |
| 0.4802        | 1.82  | 3650  | 0.4887          |
| 0.4701        | 1.85  | 3700  | 0.4897          |
| 0.4572        | 1.88  | 3750  | 0.4873          |
| 0.4469        | 1.9   | 3800  | 0.4878          |
| 0.478         | 1.93  | 3850  | 0.4885          |
| 0.4449        | 1.95  | 3900  | 0.4866          |
| 0.4634        | 1.98  | 3950  | 0.4843          |
| 0.4718        | 2.0   | 4000  | 0.4838          |
| 0.4458        | 2.02  | 4050  | 0.4822          |
| 0.461         | 2.05  | 4100  | 0.4801          |
| 0.4247        | 2.08  | 4150  | 0.4856          |
| 0.4325        | 2.1   | 4200  | 0.4830          |
| 0.4354        | 2.12  | 4250  | 0.4827          |
| 0.4313        | 2.15  | 4300  | 0.4807          |
| 0.4753        | 2.17  | 4350  | 0.4812          |
| 0.4442        | 2.2   | 4400  | 0.4833          |
| 0.4431        | 2.23  | 4450  | 0.4851          |
| 0.4485        | 2.25  | 4500  | 0.4815          |
| 0.4416        | 2.27  | 4550  | 0.4813          |
| 0.4613        | 2.3   | 4600  | 0.4777          |
| 0.4121        | 2.33  | 4650  | 0.4775          |
| 0.4311        | 2.35  | 4700  | 0.4768          |
| 0.4532        | 2.38  | 4750  | 0.4765          |
| 0.4342        | 2.4   | 4800  | 0.4781          |
| 0.4189        | 2.42  | 4850  | 0.4743          |
| 0.443         | 2.45  | 4900  | 0.4742          |
| 0.4596        | 2.48  | 4950  | 0.4734          |
| 0.4193        | 2.5   | 5000  | 0.4719          |
| 0.4321        | 2.52  | 5050  | 0.4723          |
| 0.4456        | 2.55  | 5100  | 0.4713          |
| 0.4464        | 2.58  | 5150  | 0.4694          |
| 0.4273        | 2.6   | 5200  | 0.4700          |
| 0.4239        | 2.62  | 5250  | 0.4701          |
| 0.4282        | 2.65  | 5300  | 0.4687          |
| 0.4303        | 2.67  | 5350  | 0.4686          |
| 0.4559        | 2.7   | 5400  | 0.4695          |
| 0.4542        | 2.73  | 5450  | 0.4692          |
| 0.4532        | 2.75  | 5500  | 0.4685          |
| 0.4505        | 2.77  | 5550  | 0.4663          |
| 0.4533        | 2.8   | 5600  | 0.4660          |
| 0.4351        | 2.83  | 5650  | 0.4640          |
| 0.4354        | 2.85  | 5700  | 0.4651          |
| 0.4374        | 2.88  | 5750  | 0.4664          |
| 0.4571        | 2.9   | 5800  | 0.4662          |
| 0.4663        | 2.92  | 5850  | 0.4636          |
| 0.4211        | 2.95  | 5900  | 0.4645          |
| 0.4349        | 2.98  | 5950  | 0.4622          |
| 0.4167        | 3.0   | 6000  | 0.4634          |
| 0.4176        | 3.02  | 6050  | 0.4621          |
| 0.4387        | 3.05  | 6100  | 0.4607          |
| 0.395         | 3.08  | 6150  | 0.4638          |
| 0.4186        | 3.1   | 6200  | 0.4623          |
| 0.3993        | 3.12  | 6250  | 0.4622          |
| 0.4009        | 3.15  | 6300  | 0.4631          |
| 0.4033        | 3.17  | 6350  | 0.4640          |
| 0.389         | 3.2   | 6400  | 0.4662          |
| 0.4037        | 3.23  | 6450  | 0.4618          |
| 0.4287        | 3.25  | 6500  | 0.4617          |
| 0.3917        | 3.27  | 6550  | 0.4611          |
| 0.3944        | 3.3   | 6600  | 0.4626          |
| 0.4088        | 3.33  | 6650  | 0.4622          |
| 0.4205        | 3.35  | 6700  | 0.4604          |
| 0.4273        | 3.38  | 6750  | 0.4608          |
| 0.4139        | 3.4   | 6800  | 0.4607          |
| 0.3888        | 3.42  | 6850  | 0.4603          |
| 0.4353        | 3.45  | 6900  | 0.4573          |
| 0.4222        | 3.48  | 6950  | 0.4577          |
| 0.4083        | 3.5   | 7000  | 0.4571          |
| 0.4161        | 3.52  | 7050  | 0.4560          |
| 0.3879        | 3.55  | 7100  | 0.4540          |
| 0.3819        | 3.58  | 7150  | 0.4570          |
| 0.4345        | 3.6   | 7200  | 0.4551          |
| 0.4101        | 3.62  | 7250  | 0.4569          |
| 0.4194        | 3.65  | 7300  | 0.4543          |
| 0.4066        | 3.67  | 7350  | 0.4563          |
| 0.4144        | 3.7   | 7400  | 0.4553          |
| 0.4134        | 3.73  | 7450  | 0.4566          |
| 0.3906        | 3.75  | 7500  | 0.4550          |
| 0.4128        | 3.77  | 7550  | 0.4546          |
| 0.4227        | 3.8   | 7600  | 0.4535          |
| 0.4069        | 3.83  | 7650  | 0.4517          |
| 0.3927        | 3.85  | 7700  | 0.4548          |
| 0.3977        | 3.88  | 7750  | 0.4521          |
| 0.4184        | 3.9   | 7800  | 0.4516          |
| 0.3854        | 3.92  | 7850  | 0.4513          |
| 0.4129        | 3.95  | 7900  | 0.4524          |
| 0.3998        | 3.98  | 7950  | 0.4548          |
| 0.4227        | 4.0   | 8000  | 0.4534          |
| 0.3788        | 4.03  | 8050  | 0.4520          |
| 0.3732        | 4.05  | 8100  | 0.4501          |
| 0.375         | 4.08  | 8150  | 0.4565          |
| 0.3845        | 4.1   | 8200  | 0.4515          |
| 0.378         | 4.12  | 8250  | 0.4492          |
| 0.3874        | 4.15  | 8300  | 0.4508          |
| 0.3802        | 4.17  | 8350  | 0.4510          |
| 0.3596        | 4.2   | 8400  | 0.4524          |
| 0.4009        | 4.22  | 8450  | 0.4549          |
| 0.4105        | 4.25  | 8500  | 0.4515          |
| 0.3716        | 4.28  | 8550  | 0.4508          |
| 0.3673        | 4.3   | 8600  | 0.4497          |
| 0.3882        | 4.33  | 8650  | 0.4513          |
| 0.375         | 4.35  | 8700  | 0.4524          |
| 0.3654        | 4.38  | 8750  | 0.4503          |
| 0.3983        | 4.4   | 8800  | 0.4509          |
| 0.4067        | 4.42  | 8850  | 0.4487          |
| 0.3966        | 4.45  | 8900  | 0.4519          |
| 0.378         | 4.47  | 8950  | 0.4505          |
| 0.3755        | 4.5   | 9000  | 0.4508          |
| 0.3855        | 4.53  | 9050  | 0.4500          |
| 0.3938        | 4.55  | 9100  | 0.4527          |
| 0.3946        | 4.58  | 9150  | 0.4531          |
| 0.3752        | 4.6   | 9200  | 0.4506          |
| 0.3723        | 4.62  | 9250  | 0.4459          |
| 0.3704        | 4.65  | 9300  | 0.4467          |
| 0.3861        | 4.67  | 9350  | 0.4484          |
| 0.3965        | 4.7   | 9400  | 0.4481          |
| 0.3972        | 4.72  | 9450  | 0.4482          |
| 0.3917        | 4.75  | 9500  | 0.4447          |
| 0.3688        | 4.78  | 9550  | 0.4473          |
| 0.3861        | 4.8   | 9600  | 0.4491          |
| 0.3593        | 4.83  | 9650  | 0.4491          |
| 0.3916        | 4.85  | 9700  | 0.4432          |
| 0.3748        | 4.88  | 9750  | 0.4432          |
| 0.3921        | 4.9   | 9800  | 0.4459          |
| 0.3745        | 4.92  | 9850  | 0.4457          |
| 0.4002        | 4.95  | 9900  | 0.4443          |
| 0.3767        | 4.97  | 9950  | 0.4430          |
| 0.3537        | 5.0   | 10000 | 0.4470          |
| 0.3673        | 5.03  | 10050 | 0.4531          |
| 0.3506        | 5.05  | 10100 | 0.4474          |
| 0.3506        | 5.08  | 10150 | 0.4497          |
| 0.3622        | 5.1   | 10200 | 0.4471          |


### Framework versions

- PEFT 0.7.1
- Transformers 4.36.2
- Pytorch 2.1.2+cu121
- Datasets 2.16.1
- Tokenizers 0.15.0