transformers safetensors mistral generated_from_trainer dataset:openwebtext base_model:mistralai/Mistral-7B-Instruct-v0.1 license:apache-2.0 endpoints_compatible text-generation-inference region:us
---
license: apache-2.0
base_model: mistralai/Mistral-7B-Instruct-v0.1
tags:
- generated_from_trainer
datasets:
- openwebtext
model-index:
- name: Mistral_Sparse_pretraining_80_percent_10000
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# Mistral_Sparse_pretraining_80_percent_10000

This model is a fine-tuned version of [mistralai/Mistral-7B-Instruct-v0.1](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.1) on the openwebtext dataset.
It achieves the following results on the evaluation set:
- Loss: 0.6872

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 1e-05
- train_batch_size: 8
- eval_batch_size: 32
- seed: 0
- distributed_type: multi-GPU
- num_devices: 6
- gradient_accumulation_steps: 2
- total_train_batch_size: 96
- total_eval_batch_size: 192
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- training_steps: 10000

### Training results

| Training Loss | Epoch | Step  | Validation Loss |
|:-------------:|:-----:|:-----:|:---------------:|
| 1.7461        | 0.05  | 50    | 1.7009          |
| 1.4034        | 0.1   | 100   | 1.3910          |
| 1.2302        | 0.15  | 150   | 1.2330          |
| 1.1363        | 0.19  | 200   | 1.1354          |
| 1.0699        | 0.24  | 250   | 1.0723          |
| 1.0316        | 0.29  | 300   | 1.0284          |
| 1.0044        | 0.34  | 350   | 0.9943          |
| 0.9719        | 0.39  | 400   | 0.9668          |
| 0.9391        | 0.44  | 450   | 0.9430          |
| 0.9194        | 0.48  | 500   | 0.9249          |
| 0.9131        | 0.53  | 550   | 0.9092          |
| 0.877         | 0.58  | 600   | 0.8953          |
| 0.8757        | 0.63  | 650   | 0.8852          |
| 0.8644        | 0.68  | 700   | 0.8749          |
| 0.8625        | 0.73  | 750   | 0.8679          |
| 0.867         | 0.78  | 800   | 0.8594          |
| 0.852         | 0.82  | 850   | 0.8529          |
| 0.8482        | 0.87  | 900   | 0.8473          |
| 0.8372        | 0.92  | 950   | 0.8421          |
| 0.8391        | 0.97  | 1000  | 0.8366          |
| 0.8209        | 1.02  | 1050  | 0.8327          |
| 0.8172        | 1.07  | 1100  | 0.8275          |
| 0.8094        | 1.11  | 1150  | 0.8247          |
| 0.8107        | 1.16  | 1200  | 0.8210          |
| 0.8137        | 1.21  | 1250  | 0.8168          |
| 0.8122        | 1.26  | 1300  | 0.8143          |
| 0.8047        | 1.31  | 1350  | 0.8115          |
| 0.804         | 1.36  | 1400  | 0.8083          |
| 0.7955        | 1.41  | 1450  | 0.8062          |
| 0.7939        | 1.45  | 1500  | 0.8040          |
| 0.7835        | 1.5   | 1550  | 0.8019          |
| 0.7983        | 1.55  | 1600  | 0.8001          |
| 0.7953        | 1.6   | 1650  | 0.7975          |
| 0.7903        | 1.65  | 1700  | 0.7945          |
| 0.7864        | 1.7   | 1750  | 0.7938          |
| 0.7972        | 1.75  | 1800  | 0.7914          |
| 0.7855        | 1.79  | 1850  | 0.7905          |
| 0.7834        | 1.84  | 1900  | 0.7878          |
| 0.7812        | 1.89  | 1950  | 0.7854          |
| 0.7865        | 1.94  | 2000  | 0.7847          |
| 0.7875        | 1.99  | 2050  | 0.7837          |
| 0.7764        | 2.04  | 2100  | 0.7815          |
| 0.7676        | 2.08  | 2150  | 0.7807          |
| 0.7716        | 2.13  | 2200  | 0.7796          |
| 0.777         | 2.18  | 2250  | 0.7781          |
| 0.7706        | 2.23  | 2300  | 0.7769          |
| 0.7669        | 2.28  | 2350  | 0.7748          |
| 0.771         | 2.33  | 2400  | 0.7742          |
| 0.7501        | 2.38  | 2450  | 0.7728          |
| 0.7653        | 2.42  | 2500  | 0.7713          |
| 0.7715        | 2.47  | 2550  | 0.7699          |
| 0.7588        | 2.52  | 2600  | 0.7694          |
| 0.7665        | 2.57  | 2650  | 0.7676          |
| 0.7616        | 2.62  | 2700  | 0.7658          |
| 0.7597        | 2.67  | 2750  | 0.7654          |
| 0.756         | 2.71  | 2800  | 0.7644          |
| 0.7517        | 2.76  | 2850  | 0.7628          |
| 0.7561        | 2.81  | 2900  | 0.7628          |
| 0.7413        | 2.86  | 2950  | 0.7620          |
| 0.7545        | 2.91  | 3000  | 0.7603          |
| 0.7442        | 2.96  | 3050  | 0.7592          |
| 0.7454        | 3.01  | 3100  | 0.7589          |
| 0.7575        | 3.05  | 3150  | 0.7583          |
| 0.739         | 3.1   | 3200  | 0.7571          |
| 0.7446        | 3.15  | 3250  | 0.7558          |
| 0.7428        | 3.2   | 3300  | 0.7557          |
| 0.737         | 3.25  | 3350  | 0.7553          |
| 0.7512        | 3.3   | 3400  | 0.7536          |
| 0.7447        | 3.34  | 3450  | 0.7525          |
| 0.7417        | 3.39  | 3500  | 0.7525          |
| 0.7403        | 3.44  | 3550  | 0.7512          |
| 0.761         | 3.49  | 3600  | 0.7502          |
| 0.7475        | 3.54  | 3650  | 0.7498          |
| 0.7535        | 3.59  | 3700  | 0.7486          |
| 0.733         | 3.64  | 3750  | 0.7483          |
| 0.7347        | 3.68  | 3800  | 0.7470          |
| 0.7439        | 3.73  | 3850  | 0.7470          |
| 0.7417        | 3.78  | 3900  | 0.7460          |
| 0.7383        | 3.83  | 3950  | 0.7460          |
| 0.7316        | 3.88  | 4000  | 0.7450          |
| 0.7273        | 3.93  | 4050  | 0.7442          |
| 0.7376        | 3.97  | 4100  | 0.7440          |
| 0.73          | 4.02  | 4150  | 0.7424          |
| 0.732         | 4.07  | 4200  | 0.7429          |
| 0.7278        | 4.12  | 4250  | 0.7419          |
| 0.721         | 4.17  | 4300  | 0.7416          |
| 0.7309        | 4.22  | 4350  | 0.7410          |
| 0.7273        | 4.27  | 4400  | 0.7400          |
| 0.7297        | 4.31  | 4450  | 0.7395          |
| 0.7321        | 4.36  | 4500  | 0.7385          |
| 0.7348        | 4.41  | 4550  | 0.7381          |
| 0.7251        | 4.46  | 4600  | 0.7371          |
| 0.7175        | 4.51  | 4650  | 0.7372          |
| 0.7356        | 4.56  | 4700  | 0.7368          |
| 0.7306        | 4.6   | 4750  | 0.7363          |
| 0.7248        | 4.65  | 4800  | 0.7359          |
| 0.7266        | 4.7   | 4850  | 0.7343          |
| 0.7243        | 4.75  | 4900  | 0.7349          |
| 0.7256        | 4.8   | 4950  | 0.7338          |
| 0.7301        | 4.85  | 5000  | 0.7335          |
| 0.7266        | 4.9   | 5050  | 0.7327          |
| 0.7229        | 4.94  | 5100  | 0.7321          |
| 0.7355        | 4.99  | 5150  | 0.7315          |
| 0.7207        | 5.04  | 5200  | 0.7317          |
| 0.7157        | 5.09  | 5250  | 0.7314          |
| 0.7214        | 5.14  | 5300  | 0.7299          |
| 0.7104        | 5.19  | 5350  | 0.7304          |
| 0.7059        | 5.24  | 5400  | 0.7296          |
| 0.7181        | 5.28  | 5450  | 0.7295          |
| 0.7226        | 5.33  | 5500  | 0.7286          |
| 0.7077        | 5.38  | 5550  | 0.7282          |
| 0.7239        | 5.43  | 5600  | 0.7276          |
| 0.7159        | 5.48  | 5650  | 0.7277          |
| 0.7169        | 5.53  | 5700  | 0.7271          |
| 0.7101        | 5.57  | 5750  | 0.7269          |
| 0.7146        | 5.62  | 5800  | 0.7262          |
| 0.7191        | 5.67  | 5850  | 0.7265          |
| 0.7124        | 5.72  | 5900  | 0.7248          |
| 0.7085        | 5.77  | 5950  | 0.7238          |
| 0.7052        | 5.82  | 6000  | 0.7235          |
| 0.7222        | 5.87  | 6050  | 0.7222          |
| 0.7089        | 5.91  | 6100  | 0.7221          |
| 0.7088        | 5.96  | 6150  | 0.7222          |
| 0.7017        | 6.01  | 6200  | 0.7218          |
| 0.7079        | 6.06  | 6250  | 0.7218          |
| 0.7209        | 6.11  | 6300  | 0.7211          |
| 0.691         | 6.16  | 6350  | 0.7210          |
| 0.7035        | 6.2   | 6400  | 0.7203          |
| 0.7075        | 6.25  | 6450  | 0.7207          |
| 0.7036        | 6.3   | 6500  | 0.7200          |
| 0.7023        | 6.35  | 6550  | 0.7189          |
| 0.7201        | 6.4   | 6600  | 0.7192          |
| 0.7021        | 6.45  | 6650  | 0.7188          |
| 0.6971        | 6.5   | 6700  | 0.7174          |
| 0.7087        | 6.54  | 6750  | 0.7184          |
| 0.7044        | 6.59  | 6800  | 0.7176          |
| 0.6921        | 6.64  | 6850  | 0.7179          |
| 0.7079        | 6.69  | 6900  | 0.7166          |
| 0.6908        | 6.74  | 6950  | 0.7158          |
| 0.687         | 6.79  | 7000  | 0.7158          |
| 0.696         | 6.83  | 7050  | 0.7148          |
| 0.6954        | 6.88  | 7100  | 0.7152          |
| 0.7103        | 6.93  | 7150  | 0.7143          |
| 0.6999        | 6.98  | 7200  | 0.7140          |
| 0.699         | 7.03  | 7250  | 0.7138          |
| 0.6959        | 7.08  | 7300  | 0.7138          |
| 0.6871        | 7.13  | 7350  | 0.7122          |
| 0.6941        | 7.17  | 7400  | 0.7131          |
| 0.6931        | 7.22  | 7450  | 0.7132          |
| 0.707         | 7.27  | 7500  | 0.7110          |
| 0.6911        | 7.32  | 7550  | 0.7122          |
| 0.7036        | 7.37  | 7600  | 0.7113          |
| 0.7105        | 7.42  | 7650  | 0.7107          |
| 0.7035        | 7.46  | 7700  | 0.7108          |
| 0.6901        | 7.51  | 7750  | 0.7113          |
| 0.6944        | 7.56  | 7800  | 0.7096          |
| 0.6927        | 7.61  | 7850  | 0.7093          |
| 0.7052        | 7.66  | 7900  | 0.7090          |
| 0.7046        | 7.71  | 7950  | 0.7082          |
| 0.6949        | 7.76  | 8000  | 0.7082          |
| 0.6888        | 7.8   | 8050  | 0.7071          |
| 0.6916        | 7.85  | 8100  | 0.7071          |
| 0.6937        | 7.9   | 8150  | 0.7067          |
| 0.7077        | 7.95  | 8200  | 0.7066          |
| 0.6847        | 8.0   | 8250  | 0.7057          |
| 0.6908        | 8.05  | 8300  | 0.7056          |
| 0.6813        | 8.1   | 8350  | 0.7060          |
| 0.6756        | 8.14  | 8400  | 0.7055          |
| 0.7006        | 8.19  | 8450  | 0.7052          |
| 0.6842        | 8.24  | 8500  | 0.7035          |
| 0.6851        | 8.29  | 8550  | 0.7044          |
| 0.6944        | 8.34  | 8600  | 0.7042          |
| 0.6929        | 8.39  | 8650  | 0.7040          |
| 0.6924        | 8.43  | 8700  | 0.7037          |
| 0.6843        | 8.48  | 8750  | 0.7037          |
| 0.7005        | 8.53  | 8800  | 0.7028          |
| 0.6795        | 8.58  | 8850  | 0.7022          |
| 0.6946        | 8.63  | 8900  | 0.7019          |
| 0.6761        | 8.68  | 8950  | 0.7016          |
| 0.6817        | 8.73  | 9000  | 0.7012          |
| 0.6838        | 8.77  | 9050  | 0.7012          |
| 0.6877        | 8.82  | 9100  | 0.7006          |
| 0.6812        | 8.87  | 9150  | 0.7004          |
| 0.6966        | 8.92  | 9200  | 0.7005          |
| 0.6778        | 8.97  | 9250  | 0.6993          |
| 0.6844        | 9.02  | 9300  | 0.6991          |
| 0.6853        | 9.06  | 9350  | 0.7000          |
| 0.6839        | 9.11  | 9400  | 0.6998          |
| 0.6813        | 9.16  | 9450  | 0.6984          |
| 0.6903        | 9.21  | 9500  | 0.6985          |
| 0.6819        | 9.26  | 9550  | 0.6987          |
| 0.6749        | 9.31  | 9600  | 0.6980          |
| 0.6782        | 9.36  | 9650  | 0.6979          |
| 0.6805        | 9.4   | 9700  | 0.6975          |
| 0.6907        | 9.45  | 9750  | 0.6974          |
| 0.6854        | 9.5   | 9800  | 0.6967          |
| 0.6803        | 9.55  | 9850  | 0.6969          |
| 0.6854        | 9.6   | 9900  | 0.6964          |
| 0.6761        | 9.65  | 9950  | 0.6966          |
| 0.6939        | 9.69  | 10000 | 0.6959          |


### Framework versions

- Transformers 4.35.2
- Pytorch 2.1.1+cu121
- Datasets 2.15.0
- Tokenizers 0.15.0
