
<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# zephyr-7b-dpo-qlora

This model is a fine-tuned version of [HuggingFaceH4/zephyr-7b-beta](https://huggingface.co/HuggingFaceH4/zephyr-7b-beta) on the HuggingFaceH4/ultrafeedback_binarized dataset.
It achieves the following results on the evaluation set:
- Loss: 0.3834
- Rewards/chosen: -3.1280
- Rewards/rejected: -4.8462
- Rewards/accuracies: 0.8290
- Rewards/margins: 1.7182
- Logps/rejected: -796.7263
- Logps/chosen: -599.3583
- Logits/rejected: -0.1921
- Logits/chosen: -0.6350

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 5e-06
- train_batch_size: 4
- eval_batch_size: 8
- seed: 42
- distributed_type: multi-GPU
- num_devices: 2
- gradient_accumulation_steps: 4
- total_train_batch_size: 32
- total_eval_batch_size: 16
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: cosine
- lr_scheduler_warmup_ratio: 0.1
- num_epochs: 1

### Training results

| Training Loss | Epoch | Step | Validation Loss | Rewards/chosen | Rewards/rejected | Rewards/accuracies | Rewards/margins | Logps/rejected | Logps/chosen | Logits/rejected | Logits/chosen |
|:-------------:|:-----:|:----:|:---------------:|:--------------:|:----------------:|:------------------:|:---------------:|:--------------:|:------------:|:---------------:|:-------------:|
| 0.6789        | 0.05  | 100  | 0.6775          | -0.0174        | -0.0516          | 0.8090             | 0.0342          | -317.2596      | -288.2941    | -2.3912         | -2.4395       |
| 0.5873        | 0.1   | 200  | 0.5557          | -0.8292        | -1.2862          | 0.7590             | 0.4570          | -440.7224      | -369.4810    | -2.1902         | -2.2432       |
| 0.533         | 0.16  | 300  | 0.5162          | -1.0093        | -1.6304          | 0.7800             | 0.6212          | -475.1447      | -387.4840    | -2.2229         | -2.2798       |
| 0.4781        | 0.21  | 400  | 0.4629          | -1.7713        | -2.7978          | 0.7960             | 1.0266          | -591.8876      | -463.6874    | -1.5199         | -1.6243       |
| 0.4156        | 0.26  | 500  | 0.4359          | -2.0095        | -3.2604          | 0.8140             | 1.2509          | -638.1469      | -487.5126    | -1.0398         | -1.2511       |
| 0.4055        | 0.31  | 600  | 0.4202          | -2.6156        | -4.1440          | 0.8150             | 1.5283          | -726.4991      | -548.1227    | -0.7806         | -1.0660       |
| 0.4267        | 0.37  | 700  | 0.4142          | -2.5405        | -4.0460          | 0.8280             | 1.5055          | -716.7000      | -540.6104    | -0.6594         | -1.0008       |
| 0.3549        | 0.42  | 800  | 0.4008          | -2.6201        | -4.2406          | 0.8240             | 1.6205          | -736.1599      | -548.5685    | -0.5974         | -0.9777       |
| 0.4165        | 0.47  | 900  | 0.3980          | -2.4267        | -3.8992          | 0.8240             | 1.4725          | -702.0190      | -529.2249    | -0.5918         | -0.9709       |
| 0.4031        | 0.52  | 1000 | 0.3935          | -2.9828        | -4.6816          | 0.8230             | 1.6987          | -780.2576      | -584.8433    | -0.2174         | -0.6399       |
| 0.3588        | 0.58  | 1100 | 0.3905          | -3.0421        | -4.7182          | 0.8270             | 1.6760          | -783.9185      | -590.7713    | -0.2252         | -0.6379       |
| 0.3597        | 0.63  | 1200 | 0.3881          | -3.4078        | -5.1661          | 0.8290             | 1.7583          | -828.7095      | -627.3367    | 0.0738          | -0.4042       |
| 0.3424        | 0.68  | 1300 | 0.3858          | -3.2762        | -5.0174          | 0.8270             | 1.7413          | -813.8470      | -614.1754    | -0.0505         | -0.5211       |
| 0.345         | 0.73  | 1400 | 0.3844          | -3.2367        | -4.9828          | 0.8290             | 1.7461          | -810.3834      | -610.2249    | -0.1123         | -0.5683       |
| 0.3982        | 0.79  | 1500 | 0.3843          | -3.1427        | -4.8630          | 0.8270             | 1.7203          | -798.4040      | -600.8328    | -0.1574         | -0.6033       |
| 0.3556        | 0.84  | 1600 | 0.3834          | -3.1877        | -4.9161          | 0.8260             | 1.7283          | -803.7087      | -605.3337    | -0.1366         | -0.5876       |
| 0.3685        | 0.89  | 1700 | 0.3833          | -3.1552        | -4.8869          | 0.8270             | 1.7317          | -800.7927      | -602.0740    | -0.1796         | -0.6240       |
| 0.4131        | 0.94  | 1800 | 0.3834          | -3.1247        | -4.8424          | 0.8250             | 1.7177          | -796.3429      | -599.0312    | -0.1954         | -0.6372       |
| 0.4336        | 0.99  | 1900 | 0.3834          | -3.1284        | -4.8462          | 0.8280             | 1.7178          | -796.7226      | -599.4030    | -0.1908         | -0.6336       |


### Framework versions

- PEFT 0.7.1
- Transformers 4.36.2
- Pytorch 2.1.2+cu121
- Datasets 2.14.6
- Tokenizers 0.15.1
{"license": "mit", "library_name": "peft", "tags": ["alignment-handbook", "generated_from_trainer", "trl", "dpo", "generated_from_trainer"], "datasets": ["HuggingFaceH4/ultrafeedback_binarized"], "base_model": "HuggingFaceH4/zephyr-7b-beta", "model-index": [{"name": "zephyr-7b-dpo-qlora", "results": []}]}
peft safetensors mistral alignment-handbook generated_from_trainer trl dpo dataset:HuggingFaceH4/ultrafeedback_binarized base_model:HuggingFaceH4/zephyr-7b-beta license:mit region:us