
<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# zephyr-7b-dpo-full

This model is a fine-tuned version of [alignment-handbook/zephyr-7b-sft-full](https://huggingface.co/alignment-handbook/zephyr-7b-sft-full) on the None dataset.
It achieves the following results on the evaluation set:
- Loss: 0.6969
- Rewards/chosen: -2.1934
- Rewards/rejected: -5.6942
- Rewards/accuracies: 0.7738
- Rewards/margins: 3.5008
- Logps/rejected: -348.8867
- Logps/chosen: -389.1190
- Logits/rejected: -2.7684
- Logits/chosen: -2.7720

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 5e-07
- train_batch_size: 8
- eval_batch_size: 4
- seed: 42
- distributed_type: multi-GPU
- num_devices: 8
- total_train_batch_size: 64
- total_eval_batch_size: 32
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_ratio: 0.1
- num_epochs: 3

### Training results

| Training Loss | Epoch | Step | Validation Loss | Rewards/chosen | Rewards/rejected | Rewards/accuracies | Rewards/margins | Logps/rejected | Logps/chosen | Logits/rejected | Logits/chosen |
|:-------------:|:-----:|:----:|:---------------:|:--------------:|:----------------:|:------------------:|:---------------:|:--------------:|:------------:|:---------------:|:-------------:|
| 0.5488        | 0.1   | 100  | 0.5412          | 0.5276         | -0.1864          | 0.7579             | 0.7139          | -293.8083      | -361.9095    | -2.9356         | -2.9364       |
| 0.5514        | 0.21  | 200  | 0.5292          | 0.6050         | -0.5755          | 0.7659             | 1.1805          | -297.6997      | -361.1350    | -2.9822         | -2.9831       |
| 0.5912        | 0.31  | 300  | 0.5418          | 0.7641         | -0.4242          | 0.7381             | 1.1883          | -296.1867      | -359.5441    | -3.0357         | -3.0425       |
| 0.589         | 0.41  | 400  | 0.5529          | 0.5361         | -0.7827          | 0.7698             | 1.3188          | -299.7716      | -361.8236    | -2.9836         | -2.9787       |
| 0.5272        | 0.52  | 500  | 0.5443          | 0.5603         | -0.9137          | 0.7540             | 1.4740          | -301.0814      | -361.5821    | -3.0368         | -3.0271       |
| 0.6274        | 0.62  | 600  | 0.5343          | 0.4591         | -0.9165          | 0.7659             | 1.3755          | -301.1094      | -362.5942    | -2.9538         | -2.9489       |
| 0.6726        | 0.72  | 700  | 0.5271          | 0.3642         | -0.8879          | 0.7302             | 1.2521          | -300.8240      | -363.5435    | -2.9791         | -2.9672       |
| 0.525         | 0.83  | 800  | 0.5176          | 0.3174         | -1.1767          | 0.7460             | 1.4941          | -303.7114      | -364.0107    | -2.9522         | -2.9500       |
| 0.534         | 0.93  | 900  | 0.5167          | 0.2409         | -1.2183          | 0.7698             | 1.4592          | -304.1277      | -364.7764    | -2.9320         | -2.9388       |
| 0.1011        | 1.03  | 1000 | 0.5198          | 0.3164         | -1.4834          | 0.7659             | 1.7998          | -306.7792      | -364.0213    | -2.8653         | -2.8746       |
| 0.1087        | 1.14  | 1100 | 0.5456          | 0.2126         | -1.8894          | 0.7540             | 2.1020          | -310.8383      | -365.0587    | -2.9162         | -2.9230       |
| 0.1111        | 1.24  | 1200 | 0.5584          | -0.0121        | -2.0740          | 0.7698             | 2.0619          | -312.6852      | -367.3064    | -2.8811         | -2.8866       |
| 0.1032        | 1.34  | 1300 | 0.5655          | -0.1069        | -2.2999          | 0.7698             | 2.1930          | -314.9436      | -368.2544    | -2.8487         | -2.8573       |
| 0.1416        | 1.44  | 1400 | 0.5439          | -0.4170        | -2.3308          | 0.7738             | 1.9139          | -315.2530      | -371.3547    | -2.8707         | -2.8702       |
| 0.1101        | 1.55  | 1500 | 0.5555          | -0.4860        | -2.6838          | 0.7698             | 2.1978          | -318.7828      | -372.0454    | -2.8754         | -2.8769       |
| 0.1133        | 1.65  | 1600 | 0.5417          | -0.3562        | -2.3109          | 0.7698             | 1.9547          | -315.0535      | -370.7468    | -2.8890         | -2.8910       |
| 0.1016        | 1.75  | 1700 | 0.5871          | -0.4138        | -2.7895          | 0.7619             | 2.3756          | -319.8395      | -371.3235    | -2.8880         | -2.8884       |
| 0.1129        | 1.86  | 1800 | 0.5715          | -0.6833        | -2.9768          | 0.75               | 2.2935          | -321.7130      | -374.0183    | -2.8381         | -2.8352       |
| 0.1091        | 1.96  | 1900 | 0.5706          | -0.8101        | -2.8834          | 0.7619             | 2.0733          | -320.7788      | -375.2863    | -2.8804         | -2.8790       |
| 0.0182        | 2.06  | 2000 | 0.6236          | -1.1461        | -3.9707          | 0.7619             | 2.8246          | -331.6517      | -378.6460    | -2.8360         | -2.8396       |
| 0.0214        | 2.17  | 2100 | 0.6502          | -1.6566        | -4.6110          | 0.75               | 2.9545          | -338.0551      | -383.7507    | -2.8200         | -2.8229       |
| 0.0125        | 2.27  | 2200 | 0.6948          | -2.2004        | -5.4257          | 0.7619             | 3.2253          | -346.2019      | -389.1892    | -2.7760         | -2.7810       |
| 0.0198        | 2.37  | 2300 | 0.6849          | -2.0018        | -5.2050          | 0.7579             | 3.2032          | -343.9949      | -387.2029    | -2.7995         | -2.8036       |
| 0.0159        | 2.48  | 2400 | 0.6740          | -1.6117        | -4.8016          | 0.7738             | 3.1899          | -339.9604      | -383.3017    | -2.7894         | -2.7948       |
| 0.0153        | 2.58  | 2500 | 0.6984          | -1.9917        | -5.4895          | 0.7698             | 3.4978          | -346.8395      | -387.1017    | -2.7777         | -2.7830       |
| 0.0134        | 2.68  | 2600 | 0.7238          | -2.4703        | -6.1089          | 0.7738             | 3.6386          | -353.0335      | -391.8880    | -2.7779         | -2.7814       |
| 0.0103        | 2.79  | 2700 | 0.7101          | -2.4180        | -5.9765          | 0.7698             | 3.5585          | -351.7098      | -391.3647    | -2.7710         | -2.7748       |
| 0.0148        | 2.89  | 2800 | 0.7035          | -2.3061        | -5.8189          | 0.7738             | 3.5128          | -350.1333      | -390.2455    | -2.7674         | -2.7706       |
| 0.0087        | 2.99  | 2900 | 0.6965          | -2.1977        | -5.7044          | 0.7738             | 3.5067          | -348.9887      | -389.1620    | -2.7692         | -2.7724       |


### Framework versions

- Transformers 4.35.0
- Pytorch 2.1.0
- Datasets 2.14.6
- Tokenizers 0.14.1

{"license": "apache-2.0", "tags": ["generated_from_trainer"], "base_model": "alignment-handbook/zephyr-7b-sft-full", "model-index": [{"name": "zephyr-7b-dpo-full", "results": []}]}
transformers tensorboard safetensors mistral text-generation generated_from_trainer conversational base_model:alignment-handbook/zephyr-7b-sft-full license:apache-2.0 autotrain_compatible endpoints_compatible text-generation-inference region:us