peft tensorboard safetensors mistral alignment-handbook generated_from_trainer trl dpo dataset:HuggingFaceH4/ultrafeedback_binarized base_model:mistralai/Mistral-7B-v0.1 license:apache-2.0 region:us
---
license: apache-2.0
library_name: peft
tags:
- alignment-handbook
- generated_from_trainer
- trl
- dpo
- generated_from_trainer
datasets:
- HuggingFaceH4/ultrafeedback_binarized
base_model: mistralai/Mistral-7B-v0.1
model-index:
- name: zephyr-7b-dpo-qlora
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# zephyr-7b-dpo-qlora

This model is a fine-tuned version of [alignment-handbook/zephyr-7b-sft-qlora](https://huggingface.co/alignment-handbook/zephyr-7b-sft-qlora) on the HuggingFaceH4/ultrafeedback_binarized dataset.
It achieves the following results on the evaluation set:
- Loss: 0.5036
- Rewards/chosen: -2.0892
- Rewards/rejected: -3.1197
- Rewards/accuracies: 0.7295
- Rewards/margins: 1.0304
- Logps/rejected: -560.7722
- Logps/chosen: -477.4810
- Logits/rejected: 2.3638
- Logits/chosen: 1.7891

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 5e-06
- train_batch_size: 4
- eval_batch_size: 8
- seed: 42
- distributed_type: multi-GPU
- gradient_accumulation_steps: 2
- total_train_batch_size: 8
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: cosine
- lr_scheduler_warmup_ratio: 0.1
- num_epochs: 1

### Training results

| Training Loss | Epoch | Step | Logits/chosen | Logits/rejected | Logps/chosen | Logps/rejected | Validation Loss | Rewards/accuracies | Rewards/chosen | Rewards/margins | Rewards/rejected |
|:-------------:|:-----:|:----:|:-------------:|:---------------:|:------------:|:--------------:|:---------------:|:------------------:|:--------------:|:---------------:|:----------------:|
| 0.6931        | 0.01  | 100  | -2.2163       | -2.1335         | -268.5095    | -248.7855      | 0.6930          | 0.5135             | 0.0005         | 0.0003          | 0.0002           |
| 0.6926        | 0.03  | 200  | -2.2157       | -2.1330         | -268.3331    | -248.7224      | 0.6924          | 0.5885             | 0.0023         | 0.0014          | 0.0008           |
| 0.6904        | 0.04  | 300  | -2.2194       | -2.1373         | -267.3080    | -248.1708      | 0.6901          | 0.6475             | 0.0125         | 0.0062          | 0.0064           |
| 0.6868        | 0.05  | 400  | -2.2179       | -2.1356         | -264.7627    | -247.1243      | 0.6830          | 0.6610             | 0.0380         | 0.0211          | 0.0168           |
| 0.6781        | 0.07  | 500  | -2.1590       | -2.0748         | -266.5388    | -252.3708      | 0.6679          | 0.6785             | 0.0202         | 0.0558          | -0.0356          |
| 0.6565        | 0.08  | 600  | -2.0685       | -1.9763         | -278.9226    | -272.4421      | 0.6403          | 0.6805             | -0.1036        | 0.1327          | -0.2364          |
| 0.6411        | 0.09  | 700  | -2.0181       | -1.9197         | -283.8720    | -282.3092      | 0.6254          | 0.6820             | -0.1531        | 0.1819          | -0.3350          |
| 0.6177        | 0.1   | 800  | -1.9304       | -1.8202         | -307.0186    | -313.1128      | 0.6134          | 0.6765             | -0.3846        | 0.2585          | -0.6431          |
| 0.6333        | 0.12  | 900  | -1.9660       | -1.8566         | -308.6199    | -317.1526      | 0.6082          | 0.6740             | -0.4006        | 0.2829          | -0.6835          |
| 0.5776        | 0.13  | 1000 | -2.0038       | -1.8956         | -335.0627    | -351.8794      | 0.6066          | 0.6735             | -0.6650        | 0.3657          | -1.0307          |
| 0.6093        | 0.14  | 1100 | -2.0022       | -1.9019         | -324.4846    | -341.5230      | 0.6075          | 0.6740             | -0.5592        | 0.3679          | -0.9272          |
| 0.5607        | 0.16  | 1200 | -1.9182       | -1.8081         | -352.8372    | -375.3466      | 0.5970          | 0.6800             | -0.8428        | 0.4226          | -1.2654          |
| 0.5627        | 0.17  | 1300 | -1.3203       | -1.1519         | -411.9446    | -433.7877      | 0.5935          | 0.6850             | -1.4339        | 0.4160          | -1.8498          |
| 0.5853        | 0.18  | 1400 | -1.0520       | -0.8708         | -389.5525    | -417.2325      | 0.5842          | 0.6950             | -1.2099        | 0.4743          | -1.6843          |
| 0.5622        | 0.2   | 1500 | -0.6561       | -0.4323         | -419.2693    | -453.9020      | 0.5712          | 0.6990             | -1.5071        | 0.5439          | -2.0510          |
| 0.4815        | 0.21  | 1600 | -0.5810       | -0.3415         | -421.0228    | -464.6043      | 0.5663          | 0.7035             | -1.5246        | 0.6333          | -2.1580          |
| 0.4698        | 0.22  | 1700 | 0.5697        | -1.8165         | -2.4986      | 0.6990         | 0.6821          | -498.6652          | -450.2103      | 0.5641          | 0.2594           |
| 0.5213        | 0.24  | 1800 | 0.5670        | -1.4236         | -2.1011      | 0.7055         | 0.6776          | -458.9214          | -410.9152      | 0.6173          | 0.2952           |
| 0.5295        | 0.25  | 1900 | 0.5606        | -1.9797         | -2.6952      | 0.6945         | 0.7155          | -518.3280          | -466.5294      | 0.8941          | 0.5819           |
| 0.6074        | 0.26  | 2000 | 0.5525        | -1.1848         | -1.7881      | 0.7165         | 0.6033          | -427.6170          | -387.0396      | 0.3449          | 0.0271           |
| 0.568         | 0.27  | 2100 | 0.5388        | -1.5667         | -2.2488      | 0.7220         | 0.6822          | -473.6912          | -425.2263      | 1.3497          | 0.9786           |
| 0.5643        | 0.29  | 2200 | 0.5539        | -1.8112         | -2.6184      | 0.7145         | 0.8072          | -510.6461          | -449.6774      | 1.9603          | 1.5565           |
| 0.5226        | 0.3   | 2300 | 0.5354        | -1.6020         | -2.3588      | 0.7245         | 0.7568          | -484.6839          | -428.7553      | 1.3673          | 0.9661           |
| 0.4144        | 0.31  | 2400 | 0.5338        | -2.0110         | -2.8276      | 0.7245         | 0.8167          | -531.5681          | -469.6557      | 1.6609          | 1.2542           |
| 0.5233        | 0.33  | 2500 | 0.5387        | -1.9001         | -2.7290      | 0.7245         | 0.8289          | -521.7109          | -458.5734      | 1.7390          | 1.3093           |
| 0.5425        | 0.34  | 2600 | 0.5430        | -2.4619         | -3.3366      | 0.7225         | 0.8747          | -582.4704          | -514.7514      | 2.4431          | 1.9262           |
| 0.4719        | 0.35  | 2700 | 0.5309        | -1.9122         | -2.7118      | 0.7285         | 0.7996          | -519.9872          | -459.7816      | 2.0586          | 1.6066           |
| 0.5543        | 0.37  | 2800 | 0.5268        | -1.7066         | -2.4623      | 0.7225         | 0.7557          | -495.0332          | -439.2162      | 1.5924          | 1.1721           |
| 0.5409        | 0.38  | 2900 | 0.5400        | -2.1879         | -3.1551      | 0.7175         | 0.9673          | -564.3220          | -487.3477      | 2.0890          | 1.6062           |
| 0.4956        | 0.39  | 3000 | 0.5285        | -1.8388         | -2.7165      | 0.7285         | 0.8777          | -520.4593          | -452.4431      | 1.6464          | 1.1679           |
| 0.4572        | 0.41  | 3100 | 0.5198        | -1.6639         | -2.4269      | 0.7265         | 0.7630          | -491.4958          | -434.9505      | 1.7627          | 1.2994           |
| 0.4962        | 0.42  | 3200 | 0.5181        | -1.6914         | -2.5214      | 0.7265         | 0.8300          | -500.9511          | -437.6994      | 1.6452          | 1.1780           |
| 0.6098        | 0.43  | 3300 | 0.5188        | -1.6044         | -2.4380      | 0.7310         | 0.8336          | -492.6022          | -428.9995      | 1.5141          | 1.0617           |
| 0.5349        | 0.44  | 3400 | 0.5210        | -1.4720         | -2.3090      | 0.7285         | 0.8370          | -479.7061          | -415.7578      | 1.4965          | 1.0371           |
| 0.4773        | 0.46  | 3500 | 0.5206        | -1.4425         | -2.2285      | 0.7280         | 0.7861          | -471.6597          | -412.8062      | 1.8090          | 1.3264           |
| 0.5312        | 0.47  | 3600 | 0.5196        | -1.8128         | -2.6719      | 0.7320         | 0.8591          | -515.9943          | -449.8387      | 2.5339          | 2.0191           |
| 0.5879        | 0.48  | 3700 | 0.5128        | -1.9225         | -2.7975      | 0.7355         | 0.8750          | -528.5556          | -460.8123      | 2.9390          | 2.3934           |
| 0.5202        | 0.5   | 3800 | 0.5155        | -1.8291         | -2.7153      | 0.7330         | 0.8863          | -520.3419          | -451.4667      | 2.2728          | 1.7445           |
| 0.5116        | 0.51  | 3900 | 0.5188        | -2.0732         | -3.0427      | 0.7285         | 0.9696          | -553.0799          | -475.8752      | 2.2721          | 1.7291           |
| 0.5521        | 0.52  | 4000 | 0.5161        | -2.3283         | -3.3054      | 0.7255         | 0.9771          | -579.3469          | -501.3872      | 2.2577          | 1.7449           |
| 0.5107        | 0.54  | 4100 | 0.5197        | -1.8192         | -2.7348      | 0.7215         | 0.9156          | -522.2897          | -450.4803      | 1.7678          | 1.2222           |
| 0.4773        | 0.55  | 4200 | 0.5163        | -2.1894         | -3.1554      | 0.7265         | 0.9660          | -564.3451          | -487.4992      | 1.8497          | 1.3121           |
| 0.4315        | 0.56  | 4300 | 0.5097        | -2.0873         | -3.0416      | 0.7340         | 0.9544          | -552.9705          | -477.2872      | 2.2039          | 1.6783           |
| 0.5176        | 0.58  | 4400 | 0.5097        | -2.2486         | -3.2409      | 0.7290         | 0.9924          | -572.8979          | -493.4146      | 2.0782          | 1.5387           |
| 0.4487        | 0.59  | 4500 | 0.5132        | -2.0257         | -3.0144      | 0.7245         | 0.9887          | -550.2475          | -471.1282      | 2.0676          | 1.4968           |
| 0.478         | 0.6   | 4600 | 0.5082        | -2.0565         | -3.0343      | 0.7270         | 0.9778          | -552.2376          | -474.2084      | 2.1065          | 1.5402           |
| 0.5351        | 0.62  | 4700 | 0.5038        | -1.9625         | -2.8993      | 0.7285         | 0.9368          | -538.7390          | -464.8120      | 2.0488          | 1.5017           |
| 0.4942        | 0.63  | 4800 | 0.5058        | -2.2570         | -3.2479      | 0.7305         | 0.9909          | -573.5954          | -494.2575      | 2.5210          | 1.9471           |
| 0.4918        | 0.64  | 4900 | 0.5129        | -2.4781         | -3.5322      | 0.7350         | 1.0541          | -602.0275          | -516.3653      | 2.8295          | 2.2468           |
| 0.4693        | 0.65  | 5000 | 0.5131        | -2.2974         | -3.3589      | 0.7315         | 1.0615          | -584.6987          | -498.2968      | 2.6931          | 2.1137           |
| 0.5796        | 0.67  | 5100 | 0.5084        | -2.1485         | -3.1709      | 0.7300         | 1.0224          | -565.8975          | -483.4113      | 2.4925          | 1.9365           |
| 0.5137        | 0.68  | 5200 | 0.5012        | -2.0083         | -2.9370      | 0.7365         | 0.9287          | -542.5073          | -469.3903      | 2.0969          | 1.5738           |
| 0.4484        | 0.69  | 5300 | 0.5022        | -2.1149         | -3.0765      | 0.7345         | 0.9616          | -556.4618          | -480.0531      | 2.2539          | 1.7154           |
| 0.4608        | 0.71  | 5400 | 0.5035        | -2.1639         | -3.1586      | 0.7380         | 0.9947          | -564.6663          | -484.9485      | 2.2224          | 1.6704           |
| 0.5746        | 0.72  | 5500 | 0.5045        | -2.3599         | -3.4023      | 0.7320         | 1.0424          | -589.0370          | -504.5520      | 2.2134          | 1.6562           |
| 0.5768        | 0.73  | 5600 | 0.5011        | -2.0662         | -3.0430      | 0.7375         | 0.9767          | -553.1031          | -475.1830      | 1.8199          | 1.2667           |
| 0.4359        | 0.75  | 5700 | 0.5032        | -2.0933         | -3.1100      | 0.7350         | 1.0166          | -559.8049          | -477.8932      | 1.9073          | 1.3503           |
| 0.4812        | 0.76  | 5800 | 0.5056        | -2.2931         | -3.3640      | 0.7320         | 1.0709          | -585.2068          | -497.8671      | 2.1234          | 1.5508           |
| 0.5048        | 0.77  | 5900 | 0.5036        | -1.9424         | -2.9286      | 0.7335         | 0.9862          | -541.6672          | -462.8024      | 1.7970          | 1.2367           |
| 0.4505        | 0.79  | 6000 | 0.5053        | -1.9881         | -2.9896      | 0.7330         | 1.0015          | -547.7703          | -467.3695      | 1.9582          | 1.3843           |
| 0.5197        | 0.8   | 6100 | 0.5071        | -2.0238         | -3.0391      | 0.7315         | 1.0152          | -552.7153          | -470.9445      | 2.0118          | 1.4341           |
| 0.6046        | 0.81  | 6200 | 0.5064        | -2.0803         | -3.1116      | 0.7310         | 1.0313          | -559.9708          | -476.5939      | 2.1151          | 1.5328           |
| 0.4669        | 0.82  | 6300 | 0.5072        | -2.1010         | -3.1541      | 0.7310         | 1.0531          | -564.2192          | -478.6570      | 2.2264          | 1.6394           |
| 0.5631        | 0.84  | 6400 | 0.5055        | -2.0938         | -3.1385      | 0.7305         | 1.0447          | -562.6528          | -477.9385      | 2.3072          | 1.7230           |
| 0.433         | 0.85  | 6500 | 0.5044        | -2.0630         | -3.0936      | 0.7290         | 1.0306          | -558.1638          | -474.8586      | 2.2760          | 1.6963           |
| 0.4908        | 0.86  | 6600 | 0.5043        | -2.0569         | -3.0863      | 0.7295         | 1.0294          | -557.4365          | -474.2540      | 2.3343          | 1.7557           |
| 0.522         | 0.88  | 6700 | 0.5039        | -2.0755         | -3.1060      | 0.7300         | 1.0304          | -559.4037          | -476.1125      | 2.3469          | 1.7706           |
| 0.4953        | 0.89  | 6800 | 0.5039        | -2.0918         | -3.1235      | 0.7290         | 1.0317          | -561.1605          | -477.7388      | 2.3881          | 1.8129           |
| 0.5683        | 0.9   | 6900 | 0.5036        | -2.0899         | -3.1203      | 0.7300         | 1.0304          | -560.8373          | -477.5472      | 2.3649          | 1.7897           |
| 0.5399        | 0.92  | 7000 | 0.5037        | -2.0831         | -3.1119      | 0.7295         | 1.0288          | -560.0004          | -476.8721      | 2.3590          | 1.7832           |
| 0.4628        | 0.93  | 7100 | 0.5035        | -2.0882         | -3.1188      | 0.7300         | 1.0307          | -560.6896          | -477.3761      | 2.3659          | 1.7910           |
| 0.5273        | 0.94  | 7200 | 0.5036        | -2.0897         | -3.1202      | 0.7295         | 1.0305          | -560.8275          | -477.5317      | 2.3594          | 1.7853           |
| 0.4445        | 0.96  | 7300 | 0.5035        | -2.0889         | -3.1197      | 0.7305         | 1.0308          | -560.7729          | -477.4447      | 2.3614          | 1.7871           |
| 0.4839        | 0.97  | 7400 | 0.5035        | -2.0894         | -3.1199      | 0.7310         | 1.0304          | -560.7961          | -477.5042      | 2.3646          | 1.7896           |
| 0.4425        | 0.98  | 7500 | 0.5036        | -2.0892         | -3.1197      | 0.7295         | 1.0304          | -560.7722          | -477.4810      | 2.3638          | 1.7891           |
| 0.5195        | 0.99  | 7600 | 0.5036        | -2.0892         | -3.1197      | 0.7295         | 1.0304          | -560.7722          | -477.4810      | 2.3638          | 1.7891           |


### Framework versions

- PEFT 0.7.1
- Transformers 4.36.2
- Pytorch 2.1.2+cu121
- Datasets 2.14.6
- Tokenizers 0.15.0