transformers safetensors sparse_mistral text-generation generated_from_trainer custom_code base_model:mistralai/Mistral-7B-v0.1 license:apache-2.0 autotrain_compatible region:us
---
license: apache-2.0
base_model: mistralai/Mistral-7B-v0.1
tags:
- generated_from_trainer
model-index:
- name: Mistral_Sparse_refined_web_70p_2024-02-28
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# Mistral_Sparse_refined_web_70p_2024-02-28

This model is a fine-tuned version of [mistralai/Mistral-7B-v0.1](https://huggingface.co/mistralai/Mistral-7B-v0.1) on the None dataset.
It achieves the following results on the evaluation set:
- Loss: 2.1297

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 1e-05
- train_batch_size: 1
- eval_batch_size: 1
- seed: 0
- distributed_type: multi-GPU
- num_devices: 4
- gradient_accumulation_steps: 4
- total_train_batch_size: 16
- total_eval_batch_size: 4
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- training_steps: 3750

### Training results

| Training Loss | Epoch | Step | Validation Loss |
|:-------------:|:-----:|:----:|:---------------:|
| 2.7195        | 0.0   | 25   | 2.8171          |
| 2.4252        | 0.01  | 50   | 2.6962          |
| 2.4132        | 0.01  | 75   | 2.6180          |
| 2.3614        | 0.02  | 100  | 2.5748          |
| 2.3289        | 0.02  | 125  | 2.5433          |
| 2.4043        | 0.02  | 150  | 2.5219          |
| 2.3267        | 0.03  | 175  | 2.5078          |
| 2.3059        | 0.03  | 200  | 2.4980          |
| 2.3307        | 0.04  | 225  | 2.4895          |
| 2.3628        | 0.04  | 250  | 2.4807          |
| 2.2833        | 0.04  | 275  | 2.4741          |
| 2.294         | 0.05  | 300  | 2.4727          |
| 2.3001        | 0.05  | 325  | 2.4657          |
| 2.1377        | 0.06  | 350  | 2.4617          |
| 2.2947        | 0.06  | 375  | 2.4605          |
| 2.3341        | 0.06  | 400  | 2.4532          |
| 2.3318        | 0.07  | 425  | 2.4520          |
| 2.2793        | 0.07  | 450  | 2.4498          |
| 2.2009        | 0.08  | 475  | 2.4445          |
| 2.2665        | 0.08  | 500  | 2.4443          |
| 2.3313        | 0.08  | 525  | 2.4395          |
| 2.2272        | 0.09  | 550  | 2.4414          |
| 2.3625        | 0.09  | 575  | 2.4408          |
| 2.1607        | 0.1   | 600  | 2.4390          |
| 2.2494        | 0.1   | 625  | 2.4408          |
| 2.3305        | 0.1   | 650  | 2.4363          |
| 2.32          | 0.11  | 675  | 2.4367          |
| 2.2791        | 0.11  | 700  | 2.4322          |
| 2.2702        | 0.12  | 725  | 2.4273          |
| 2.1792        | 0.12  | 750  | 2.4286          |
| 2.2145        | 0.12  | 775  | 2.4304          |
| 2.3076        | 0.13  | 800  | 2.4273          |
| 2.3154        | 0.13  | 825  | 2.4287          |
| 2.2979        | 0.14  | 850  | 2.4295          |
| 2.1638        | 0.14  | 875  | 2.4261          |
| 2.2316        | 0.14  | 900  | 2.4287          |
| 2.1786        | 0.15  | 925  | 2.4270          |
| 2.3116        | 0.15  | 950  | 2.4246          |
| 2.2439        | 0.16  | 975  | 2.4232          |
| 2.2867        | 0.16  | 1000 | 2.4209          |
| 2.291         | 0.16  | 1025 | 2.4231          |
| 2.3092        | 0.17  | 1050 | 2.4261          |
| 2.3367        | 0.17  | 1075 | 2.4226          |
| 2.2603        | 0.18  | 1100 | 2.4185          |
| 2.2899        | 0.18  | 1125 | 2.4241          |
| 2.2575        | 0.18  | 1150 | 2.4241          |
| 2.3044        | 0.19  | 1175 | 2.4180          |
| 2.1659        | 0.19  | 1200 | 2.4153          |
| 2.2555        | 0.2   | 1225 | 2.4175          |
| 2.2407        | 0.2   | 1250 | 2.4174          |
| 2.2809        | 0.2   | 1275 | 2.4172          |
| 2.3096        | 0.21  | 1300 | 2.4159          |
| 2.2855        | 0.21  | 1325 | 2.4168          |
| 2.1743        | 0.22  | 1350 | 2.4171          |
| 2.2595        | 0.22  | 1375 | 2.4145          |
| 2.2657        | 0.22  | 1400 | 2.4205          |
| 2.2053        | 0.23  | 1425 | 2.4166          |
| 2.1895        | 0.23  | 1450 | 2.4209          |
| 2.1552        | 0.24  | 1475 | 2.4197          |
| 2.2134        | 0.24  | 1500 | 2.4188          |
| 2.1482        | 0.24  | 1525 | 2.4186          |
| 2.2142        | 0.25  | 1550 | 2.4222          |
| 2.3327        | 0.25  | 1575 | 2.4174          |
| 2.2899        | 0.26  | 1600 | 2.4167          |
| 2.2577        | 0.26  | 1625 | 2.4172          |
| 2.3402        | 0.26  | 1650 | 2.4211          |
| 2.2345        | 0.27  | 1675 | 2.4183          |
| 2.2808        | 0.27  | 1700 | 2.4164          |
| 2.2678        | 0.28  | 1725 | 2.4181          |
| 2.2512        | 0.28  | 1750 | 2.4161          |
| 2.231         | 0.28  | 1775 | 2.4152          |
| 2.1873        | 0.29  | 1800 | 2.4146          |
| 2.2662        | 0.29  | 1825 | 2.4135          |
| 2.262         | 0.3   | 1850 | 2.4188          |
| 2.2492        | 0.3   | 1875 | 2.4194          |
| 2.274         | 0.3   | 1900 | 2.4138          |
| 2.3257        | 0.31  | 1925 | 2.4152          |
| 2.2678        | 0.31  | 1950 | 2.4122          |
| 2.2877        | 0.32  | 1975 | 2.4153          |
| 2.2117        | 0.32  | 2000 | 2.4151          |
| 2.2312        | 0.32  | 2025 | 2.4163          |
| 2.2182        | 0.33  | 2050 | 2.4159          |
| 2.2126        | 0.33  | 2075 | 2.4121          |
| 2.3101        | 0.34  | 2100 | 2.4180          |
| 2.3467        | 0.34  | 2125 | 2.4173          |
| 2.2572        | 0.34  | 2150 | 2.4132          |
| 2.2068        | 0.35  | 2175 | 2.4127          |
| 2.2434        | 0.35  | 2200 | 2.4138          |
| 2.1937        | 0.36  | 2225 | 2.4165          |
| 2.1858        | 0.36  | 2250 | 2.4143          |
| 2.3188        | 0.36  | 2275 | 2.4138          |
| 2.2943        | 0.37  | 2300 | 2.4120          |
| 2.1959        | 0.37  | 2325 | 2.4095          |
| 2.2399        | 0.38  | 2350 | 2.4080          |
| 2.2694        | 0.38  | 2375 | 2.4090          |
| 2.2795        | 0.38  | 2400 | 2.4066          |
| 2.1325        | 0.39  | 2425 | 2.4071          |
| 2.1264        | 0.39  | 2450 | 2.4077          |
| 2.2166        | 0.4   | 2475 | 2.4082          |
| 2.2965        | 0.4   | 2500 | 2.4113          |
| 2.3058        | 0.4   | 2525 | 2.4066          |
| 2.1596        | 0.41  | 2550 | 2.4134          |
| 2.1838        | 0.41  | 2575 | 2.4110          |
| 2.2609        | 0.42  | 2600 | 2.4074          |
| 2.2689        | 0.42  | 2625 | 2.4092          |
| 2.3092        | 0.42  | 2650 | 2.4101          |
| 2.2789        | 0.43  | 2675 | 2.4096          |
| 2.3045        | 0.43  | 2700 | 2.4111          |
| 2.2974        | 0.44  | 2725 | 2.4134          |
| 2.2473        | 0.44  | 2750 | 2.4113          |
| 2.2701        | 0.44  | 2775 | 2.4087          |
| 2.3412        | 0.45  | 2800 | 2.4115          |
| 2.2427        | 0.45  | 2825 | 2.4100          |
| 2.2928        | 0.46  | 2850 | 2.4067          |
| 2.1879        | 0.46  | 2875 | 2.4104          |
| 2.2206        | 0.46  | 2900 | 2.4088          |
| 2.2282        | 0.47  | 2925 | 2.4065          |
| 2.3002        | 0.47  | 2950 | 2.4075          |
| 2.2273        | 0.48  | 2975 | 2.4045          |
| 2.2871        | 0.48  | 3000 | 2.4064          |
| 2.3817        | 0.48  | 3025 | 2.4054          |
| 2.2496        | 0.49  | 3050 | 2.4069          |
| 2.2985        | 0.49  | 3075 | 2.4083          |
| 2.3125        | 0.5   | 3100 | 2.4041          |
| 2.2139        | 0.5   | 3125 | 2.4064          |
| 2.2997        | 0.5   | 3150 | 2.4062          |
| 2.187         | 0.51  | 3175 | 2.4055          |
| 2.2584        | 0.51  | 3200 | 2.4062          |
| 2.1848        | 0.52  | 3225 | 2.4075          |
| 2.1779        | 0.52  | 3250 | 2.4066          |
| 2.2542        | 0.52  | 3275 | 2.4041          |
| 2.2406        | 0.53  | 3300 | 2.4066          |
| 2.1247        | 0.53  | 3325 | 2.4023          |
| 2.2576        | 0.54  | 3350 | 2.4041          |
| 2.1636        | 0.54  | 3375 | 2.4023          |
| 2.1781        | 0.54  | 3400 | 2.4056          |
| 2.1949        | 0.55  | 3425 | 2.4047          |
| 2.1119        | 0.55  | 3450 | 2.4070          |
| 2.2437        | 0.56  | 3475 | 2.4096          |
| 2.281         | 0.56  | 3500 | 2.4040          |
| 2.2499        | 0.56  | 3525 | 2.4063          |
| 2.2129        | 0.57  | 3550 | 2.4052          |
| 2.2115        | 0.57  | 3575 | 2.4050          |
| 2.375         | 0.58  | 3600 | 2.4050          |
| 2.1891        | 0.58  | 3625 | 2.4082          |
| 2.3929        | 0.58  | 3650 | 2.4038          |
| 2.1928        | 0.59  | 3675 | 2.4079          |
| 2.3194        | 0.59  | 3700 | 2.4067          |
| 2.2286        | 0.6   | 3725 | 2.4086          |
| 2.1629        | 0.6   | 3750 | 2.4058          |


### Framework versions

- Transformers 4.36.2
- Pytorch 2.1.2+cu121
- Datasets 2.15.0
- Tokenizers 0.15.0
