transformers tensorboard safetensors mistral text-generation generated_from_trainer conversational autotrain_compatible endpoints_compatible text-generation-inference region:us
---
tags:
- generated_from_trainer
model-index:
- name: zephyr-7b-dpo-full-hydrox-safe
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# zephyr-7b-dpo-full-hydrox-safe

This model was trained from scratch on an unknown dataset.
It achieves the following results on the evaluation set:
- Loss: 0.0018
- Rewards/chosen: 1.5348
- Rewards/rejected: -41.0322
- Rewards/accuracies: 0.9992
- Rewards/margins: 42.5670
- Logps/rejected: -855.1025
- Logps/chosen: -270.7936
- Logits/rejected: -2.8721
- Logits/chosen: -3.0906

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 5e-07
- train_batch_size: 8
- eval_batch_size: 4
- seed: 42
- distributed_type: multi-GPU
- num_devices: 8
- total_train_batch_size: 64
- total_eval_batch_size: 32
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_ratio: 0.1
- num_epochs: 3

### Training results

| Training Loss | Epoch | Step | Validation Loss | Rewards/chosen | Rewards/rejected | Rewards/accuracies | Rewards/margins | Logps/rejected | Logps/chosen | Logits/rejected | Logits/chosen |
|:-------------:|:-----:|:----:|:---------------:|:--------------:|:----------------:|:------------------:|:---------------:|:--------------:|:------------:|:---------------:|:-------------:|
| 0.201         | 0.03  | 100  | 0.1790          | 0.8766         | -1.8347          | 0.9571             | 2.7113          | -463.1273      | -277.3755    | -2.9298         | -3.0062       |
| 0.1124        | 0.07  | 200  | 0.0931          | 1.4748         | -3.5536          | 0.9764             | 5.0284          | -480.3171      | -271.3938    | -2.9144         | -2.9984       |
| 0.0657        | 0.1   | 300  | 0.1207          | 1.7581         | -4.7603          | 0.9806             | 6.5185          | -492.3842      | -268.5603    | -2.9028         | -2.9908       |
| 0.0676        | 0.14  | 400  | 0.0446          | 1.3392         | -8.0193          | 0.9857             | 9.3585          | -524.9738      | -272.75      | -2.9171         | -3.0132       |
| 0.0477        | 0.17  | 500  | 0.0581          | 1.3777         | -9.9106          | 0.9882             | 11.2883         | -543.8868      | -272.3650    | -2.8810         | -2.9695       |
| 0.0404        | 0.2   | 600  | 0.0845          | 0.3594         | -12.1256         | 0.9840             | 12.4850         | -566.0372      | -282.5481    | -2.8164         | -2.9235       |
| 0.0122        | 0.24  | 700  | 0.0831          | 0.4374         | -19.3178         | 0.9848             | 19.7552         | -637.9587      | -281.7681    | -2.7024         | -2.8110       |
| 0.0183        | 0.27  | 800  | 0.0775          | -0.0400        | -18.4153         | 0.9891             | 18.3753         | -628.9333      | -286.5414    | -2.8078         | -2.9358       |
| 0.0555        | 0.31  | 900  | 0.0589          | 0.4664         | -23.4223         | 0.9924             | 23.8887         | -679.0039      | -281.4781    | -2.8207         | -2.9406       |
| 0.0376        | 0.34  | 1000 | 0.0552          | 0.6445         | -26.2370         | 0.9874             | 26.8814         | -707.1502      | -279.6971    | -2.7989         | -2.9241       |
| 0.0438        | 0.37  | 1100 | 0.0685          | 1.7090         | -23.4352         | 0.9874             | 25.1442         | -679.1324      | -269.0518    | -2.8438         | -3.0501       |
| 0.074         | 0.41  | 1200 | 0.0715          | 1.4961         | -24.8055         | 0.9882             | 26.3016         | -692.8359      | -271.1810    | -2.7269         | -2.8728       |
| 0.1029        | 0.44  | 1300 | 0.1098          | 0.9065         | -28.0158         | 0.9882             | 28.9223         | -724.9388      | -277.0765    | -2.7411         | -2.9059       |
| 0.0271        | 0.48  | 1400 | 0.0346          | 0.7923         | -26.9632         | 0.9907             | 27.7555         | -714.4124      | -278.2186    | -2.7555         | -2.9286       |
| 0.0219        | 0.51  | 1500 | 0.0405          | 0.5053         | -27.0039         | 0.9933             | 27.5092         | -714.8198      | -281.0890    | -2.7955         | -2.9553       |
| 0.0214        | 0.54  | 1600 | 0.0293          | 0.7598         | -22.6033         | 0.9941             | 23.3631         | -670.8132      | -278.5433    | -2.7650         | -2.9211       |
| 0.0221        | 0.58  | 1700 | 0.0206          | 0.6463         | -25.7837         | 0.9949             | 26.4300         | -702.6176      | -279.6782    | -2.7340         | -2.9148       |
| 0.0339        | 0.61  | 1800 | 0.0204          | 1.2579         | -22.2971         | 0.9966             | 23.5550         | -667.7516      | -273.5627    | -2.7337         | -2.8868       |
| 0.0662        | 0.65  | 1900 | 0.0290          | 0.9343         | -30.7737         | 0.9975             | 31.7081         | -752.5181      | -276.7985    | -2.7130         | -2.8608       |
| 0.1032        | 0.68  | 2000 | 0.0358          | 1.6616         | -22.3776         | 0.9975             | 24.0392         | -668.5566      | -269.5255    | -2.7454         | -2.8772       |
| 0.0059        | 0.71  | 2100 | 0.0148          | 0.9711         | -28.7176         | 0.9949             | 29.6887         | -731.9562      | -276.4307    | -2.6758         | -2.8253       |
| 0.0096        | 0.75  | 2200 | 0.0282          | 0.9406         | -25.7753         | 0.9949             | 26.7158         | -702.5333      | -276.7361    | -2.9417         | -3.1166       |
| 0.0117        | 0.78  | 2300 | 0.0489          | -0.3402        | -33.3639         | 0.9924             | 33.0238         | -778.4199      | -289.5432    | -3.0059         | -3.1712       |
| 0.0034        | 0.82  | 2400 | 0.0136          | 1.4644         | -27.4436         | 0.9966             | 28.9079         | -719.2164      | -271.4980    | -2.6572         | -2.7758       |
| 0.0448        | 0.85  | 2500 | 0.0122          | -0.7904        | -29.3707         | 0.9983             | 28.5803         | -738.4877      | -294.0459    | -2.7608         | -2.8870       |
| 0.0606        | 0.88  | 2600 | 0.0723          | 0.6824         | -40.6246         | 0.9949             | 41.3070         | -851.0272      | -279.3180    | -2.6804         | -2.8281       |
| 0.0172        | 0.92  | 2700 | 0.0151          | 1.7774         | -28.1840         | 0.9958             | 29.9614         | -726.6205      | -268.3677    | -2.7720         | -2.9085       |
| 0.0206        | 0.95  | 2800 | 0.0105          | 0.0966         | -39.5676         | 0.9975             | 39.6643         | -840.4568      | -285.1752    | -2.8397         | -2.9803       |
| 0.0093        | 0.99  | 2900 | 0.0165          | 0.6995         | -33.4341         | 0.9966             | 34.1336         | -779.1221      | -279.1465    | -2.8086         | -2.9511       |
| 0.0008        | 1.02  | 3000 | 0.0101          | -0.0075        | -36.2959         | 0.9983             | 36.2884         | -807.7395      | -286.2166    | -2.9213         | -3.0591       |
| 0.0015        | 1.05  | 3100 | 0.0091          | -0.2430        | -38.1496         | 0.9966             | 37.9066         | -826.2767      | -288.5718    | -2.8160         | -2.9881       |
| 0.001         | 1.09  | 3200 | 0.0069          | 0.7306         | -37.6657         | 0.9966             | 38.3964         | -821.4379      | -278.8353    | -2.8287         | -2.9686       |
| 0.0447        | 1.12  | 3300 | 0.0148          | 0.4641         | -39.1245         | 0.9975             | 39.5886         | -836.0254      | -281.5004    | -2.8502         | -2.9821       |
| 0.0015        | 1.16  | 3400 | 0.0145          | 1.2867         | -34.9331         | 0.9983             | 36.2198         | -794.1116      | -273.2742    | -2.8645         | -2.9793       |
| 0.0069        | 1.19  | 3500 | 0.0120          | 1.0980         | -33.3648         | 0.9983             | 34.4628         | -778.4290      | -275.1619    | -2.8298         | -2.9435       |
| 0.0006        | 1.22  | 3600 | 0.0144          | 0.4168         | -38.9632         | 0.9975             | 39.3800         | -834.4130      | -281.9736    | -2.8061         | -2.9050       |
| 0.0532        | 1.26  | 3700 | 0.0163          | 1.2493         | -35.1549         | 0.9983             | 36.4042         | -796.3295      | -273.6483    | -3.0117         | -3.1271       |
| 0.0041        | 1.29  | 3800 | 0.0138          | 0.6365         | -36.2498         | 0.9966             | 36.8863         | -807.2789      | -279.7767    | -2.9031         | -3.0283       |
| 0.0043        | 1.33  | 3900 | 0.0095          | 0.8546         | -36.5756         | 0.9992             | 37.4302         | -810.5366      | -277.5952    | -2.9713         | -3.0945       |
| 0.0016        | 1.36  | 4000 | 0.0098          | 0.4731         | -41.9454         | 0.9992             | 42.4186         | -864.2352      | -281.4105    | -3.0390         | -3.1458       |
| 0.0018        | 1.39  | 4100 | 0.0103          | 1.1650         | -31.5836         | 0.9992             | 32.7487         | -760.6172      | -274.4915    | -3.0057         | -3.1525       |
| 0.0015        | 1.43  | 4200 | 0.0097          | 0.7472         | -35.6321         | 0.9992             | 36.3793         | -801.1016      | -278.6696    | -2.9895         | -3.1466       |
| 0.0078        | 1.46  | 4300 | 0.0171          | 0.4428         | -47.4943         | 0.9983             | 47.9370         | -919.7234      | -281.7140    | -3.0189         | -3.1577       |
| 0.0006        | 1.5   | 4400 | 0.0117          | 2.3601         | -36.7779         | 0.9992             | 39.1379         | -812.5596      | -262.5412    | -2.9504         | -3.0998       |
| 0.0021        | 1.53  | 4500 | 0.0096          | 1.8317         | -33.0472         | 0.9983             | 34.8789         | -775.2529      | -267.8244    | -2.8650         | -3.0361       |
| 0.0022        | 1.56  | 4600 | 0.0124          | 1.8244         | -35.2193         | 0.9983             | 37.0437         | -796.9737      | -267.8973    | -2.8983         | -3.0580       |
| 0.0025        | 1.6   | 4700 | 0.0120          | 1.4261         | -35.4094         | 0.9983             | 36.8355         | -798.8742      | -271.8802    | -2.8938         | -3.0468       |
| 0.0154        | 1.63  | 4800 | 0.0070          | 0.7832         | -39.4533         | 0.9983             | 40.2365         | -839.3135      | -278.3098    | -2.9491         | -3.1257       |
| 0.0025        | 1.67  | 4900 | 0.0070          | 0.5157         | -40.3519         | 0.9983             | 40.8676         | -848.2994      | -280.9843    | -2.8537         | -3.0217       |
| 0.0022        | 1.7   | 5000 | 0.0060          | 1.8378         | -36.1887         | 0.9983             | 38.0265         | -806.6680      | -267.7638    | -2.8008         | -2.9944       |
| 0.0116        | 1.73  | 5100 | 0.0067          | 1.6571         | -30.2966         | 0.9983             | 31.9537         | -747.7468      | -269.5709    | -2.8245         | -3.0273       |
| 0.0027        | 1.77  | 5200 | 0.0056          | 1.8559         | -28.9170         | 0.9983             | 30.7729         | -733.9506      | -267.5823    | -2.8308         | -3.0333       |
| 0.0008        | 1.8   | 5300 | 0.0120          | 0.1806         | -39.8991         | 0.9983             | 40.0797         | -843.7715      | -284.3351    | -3.0642         | -3.2708       |
| 0.0008        | 1.84  | 5400 | 0.0042          | 1.8857         | -32.3213         | 0.9983             | 34.2070         | -767.9935      | -267.2849    | -2.9745         | -3.1835       |
| 0.0419        | 1.87  | 5500 | 0.0049          | 0.4195         | -42.5787         | 0.9983             | 42.9982         | -870.5677      | -281.9464    | -2.9390         | -3.1435       |
| 0.0055        | 1.9   | 5600 | 0.0039          | 0.8564         | -39.5863         | 0.9966             | 40.4428         | -840.6442      | -277.5772    | -2.9517         | -3.1465       |
| 0.0016        | 1.94  | 5700 | 0.0039          | 1.5280         | -38.4427         | 0.9983             | 39.9707         | -829.2079      | -270.8615    | -2.9334         | -3.1210       |
| 0.0015        | 1.97  | 5800 | 0.0031          | 2.5689         | -32.1821         | 0.9992             | 34.7510         | -766.6016      | -260.4530    | -2.9266         | -3.1170       |
| 0.0022        | 2.01  | 5900 | 0.0154          | 1.5272         | -38.5973         | 0.9983             | 40.1246         | -830.7538      | -270.8692    | -2.9905         | -3.1795       |
| 0.0001        | 2.04  | 6000 | 0.0153          | 1.4304         | -38.7250         | 0.9983             | 40.1554         | -832.0308      | -271.8378    | -3.0212         | -3.2078       |
| 0.0004        | 2.07  | 6100 | 0.0151          | 0.9974         | -40.5361         | 0.9983             | 41.5335         | -850.1418      | -276.1674    | -3.0092         | -3.1920       |
| 0.0032        | 2.11  | 6200 | 0.0143          | 2.2246         | -35.4872         | 0.9983             | 37.7117         | -799.6525      | -263.8961    | -3.0169         | -3.2017       |
| 0.0026        | 2.14  | 6300 | 0.0138          | 2.1149         | -36.0655         | 0.9983             | 38.1804         | -805.4358      | -264.9924    | -3.0121         | -3.1974       |
| 0.0052        | 2.18  | 6400 | 0.0093          | 2.0832         | -38.0872         | 0.9983             | 40.1704         | -825.6531      | -265.3097    | -2.9119         | -3.1190       |
| 0.0002        | 2.21  | 6500 | 0.0031          | 2.4808         | -33.4480         | 0.9992             | 35.9287         | -779.2603      | -261.3340    | -2.9056         | -3.1235       |
| 0.0014        | 2.24  | 6600 | 0.0029          | 2.1622         | -35.1487         | 0.9992             | 37.3110         | -796.2680      | -264.5192    | -2.9322         | -3.1505       |
| 0.0001        | 2.28  | 6700 | 0.0029          | 2.6987         | -34.3087         | 0.9992             | 37.0074         | -787.8676      | -259.1545    | -2.9528         | -3.1683       |
| 0.0004        | 2.31  | 6800 | 0.0028          | 2.2809         | -36.5033         | 0.9992             | 38.7842         | -809.8135      | -263.3324    | -2.9377         | -3.1456       |
| 0.0268        | 2.35  | 6900 | 0.0028          | 1.9206         | -37.6915         | 0.9983             | 39.6121         | -821.6957      | -266.9360    | -2.9110         | -3.1199       |
| 0.0012        | 2.38  | 7000 | 0.0026          | 2.0357         | -34.8535         | 0.9992             | 36.8892         | -793.3152      | -265.7842    | -2.9165         | -3.1204       |
| 0.0002        | 2.41  | 7100 | 0.0031          | 0.4887         | -42.0925         | 0.9983             | 42.5812         | -865.7061      | -281.2549    | -2.8696         | -3.0815       |
| 0.0006        | 2.45  | 7200 | 0.0027          | 1.6025         | -38.2160         | 0.9992             | 39.8185         | -826.9409      | -270.1166    | -2.8921         | -3.1009       |
| 0.0037        | 2.48  | 7300 | 0.0027          | 1.2715         | -42.6264         | 0.9992             | 43.8978         | -871.0444      | -273.4270    | -2.8870         | -3.0973       |
| 0.0023        | 2.52  | 7400 | 0.0026          | 1.3232         | -42.5318         | 0.9992             | 43.8549         | -870.0981      | -272.9100    | -2.8912         | -3.1000       |
| 0.0012        | 2.55  | 7500 | 0.0026          | 1.4663         | -41.9917         | 0.9992             | 43.4581         | -864.6981      | -271.4783    | -2.8990         | -3.1084       |
| 0.0012        | 2.58  | 7600 | 0.0025          | 1.2190         | -42.9826         | 0.9992             | 44.2016         | -874.6072      | -273.9521    | -2.8991         | -3.1070       |
| 0.0001        | 2.62  | 7700 | 0.0026          | 1.7469         | -40.1663         | 0.9992             | 41.9132         | -846.4437      | -268.6728    | -2.8615         | -3.0744       |
| 0.0001        | 2.65  | 7800 | 0.0026          | 1.7291         | -40.4524         | 0.9992             | 42.1815         | -849.3045      | -268.8504    | -2.8619         | -3.0751       |
| 0.003         | 2.69  | 7900 | 0.0018          | 1.8715         | -39.4482         | 0.9992             | 41.3197         | -839.2630      | -267.4267    | -2.8391         | -3.0504       |
| 0.0001        | 2.72  | 8000 | 0.0018          | 1.4707         | -42.0026         | 0.9992             | 43.4733         | -864.8069      | -271.4347    | -2.8587         | -3.0725       |
| 0.0001        | 2.75  | 8100 | 0.0018          | 1.7032         | -41.2255         | 0.9992             | 42.9287         | -857.0361      | -269.1100    | -2.8611         | -3.0771       |
| 0.0001        | 2.79  | 8200 | 0.0018          | 1.7053         | -41.5715         | 0.9992             | 43.2768         | -860.4957      | -269.0886    | -2.8608         | -3.0769       |
| 0.0           | 2.82  | 8300 | 0.0017          | 1.5198         | -41.7233         | 0.9992             | 43.2431         | -862.0136      | -270.9432    | -2.8563         | -3.0742       |
| 0.0           | 2.86  | 8400 | 0.0018          | 1.7544         | -40.1153         | 0.9992             | 41.8696         | -845.9333      | -268.5981    | -2.8694         | -3.0853       |
| 0.0003        | 2.89  | 8500 | 0.0018          | 1.6997         | -40.2116         | 0.9992             | 41.9113         | -846.8971      | -269.1451    | -2.8782         | -3.0951       |
| 0.0071        | 2.92  | 8600 | 0.0018          | 1.5913         | -40.7278         | 0.9992             | 42.3191         | -852.0585      | -270.2284    | -2.8728         | -3.0908       |
| 0.0013        | 2.96  | 8700 | 0.0018          | 1.5263         | -41.0232         | 0.9992             | 42.5495         | -855.0129      | -270.8784    | -2.8716         | -3.0900       |
| 0.0003        | 2.99  | 8800 | 0.0019          | 1.5305         | -41.0444         | 0.9992             | 42.5748         | -855.2242      | -270.8367    | -2.8724         | -3.0907       |


### Framework versions

- Transformers 4.35.0
- Pytorch 2.1.1+cu121
- Datasets 2.14.6
- Tokenizers 0.14.1
