transformers tensorboard safetensors phi text-generation generated_from_trainer conversational custom_code base_model:microsoft/phi-2 license:mit autotrain_compatible endpoints_compatible text-generation-inference region:us
---
license: mit
base_model: microsoft/phi-2
tags:
- generated_from_trainer
model-index:
- name: phi-2-dpo-full-hydrox-safe
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# phi-2-dpo-full-hydrox-safe

This model is a fine-tuned version of [microsoft/phi-2](https://huggingface.co/microsoft/phi-2) on an unknown dataset.
It achieves the following results on the evaluation set:
- Loss: 0.0050
- Rewards/chosen: -0.0996
- Rewards/rejected: -19.3989
- Rewards/accuracies: 0.9992
- Rewards/margins: 19.2993
- Logps/rejected: -644.0859
- Logps/chosen: -275.4780
- Logits/rejected: 0.2021
- Logits/chosen: -0.3196

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 5e-07
- train_batch_size: 8
- eval_batch_size: 4
- seed: 42
- distributed_type: multi-GPU
- num_devices: 8
- total_train_batch_size: 64
- total_eval_batch_size: 32
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_ratio: 0.1
- num_epochs: 3

### Training results

| Training Loss | Epoch | Step | Validation Loss | Rewards/chosen | Rewards/rejected | Rewards/accuracies | Rewards/margins | Logps/rejected | Logps/chosen | Logits/rejected | Logits/chosen |
|:-------------:|:-----:|:----:|:---------------:|:--------------:|:----------------:|:------------------:|:---------------:|:--------------:|:------------:|:---------------:|:-------------:|
| 0.6535        | 0.03  | 100  | 0.6404          | 0.0912         | -0.0433          | 0.6936             | 0.1345          | -450.5297      | -273.5692    | 0.6266          | 0.1056        |
| 0.4293        | 0.07  | 200  | 0.4021          | 0.4957         | -0.4032          | 0.8998             | 0.8989          | -454.1287      | -269.5248    | 0.6425          | 0.1325        |
| 0.2685        | 0.1   | 300  | 0.2464          | 0.8729         | -1.0624          | 0.9386             | 1.9354          | -460.7211      | -265.7526    | 0.6700          | 0.1533        |
| 0.1918        | 0.14  | 400  | 0.1792          | 1.1981         | -1.7756          | 0.9453             | 2.9737          | -467.8528      | -262.5003    | 0.6918          | 0.1719        |
| 0.1409        | 0.17  | 500  | 0.1423          | 1.4079         | -2.4222          | 0.9503             | 3.8301          | -474.3184      | -260.4028    | 0.7139          | 0.1772        |
| 0.1437        | 0.2   | 600  | 0.1232          | 1.6054         | -3.1009          | 0.9537             | 4.7063          | -481.1060      | -258.4279    | 0.7179          | 0.1571        |
| 0.0747        | 0.24  | 700  | 0.1057          | 1.7190         | -3.9721          | 0.9613             | 5.6911          | -489.8179      | -257.2914    | 0.7179          | 0.1325        |
| 0.0848        | 0.27  | 800  | 0.0863          | 1.8540         | -4.3289          | 0.9689             | 6.1829          | -493.3858      | -255.9415    | 0.7047          | 0.1024        |
| 0.0853        | 0.31  | 900  | 0.0744          | 1.8328         | -5.0270          | 0.9722             | 6.8598          | -500.3666      | -256.1535    | 0.6746          | 0.0529        |
| 0.0561        | 0.34  | 1000 | 0.0639          | 1.7736         | -5.4555          | 0.9747             | 7.2291          | -504.6516      | -256.7455    | 0.6517          | 0.0185        |
| 0.0657        | 0.37  | 1100 | 0.0546          | 1.7966         | -5.6453          | 0.9840             | 7.4419          | -506.5500      | -256.5157    | 0.6738          | 0.0168        |
| 0.0445        | 0.41  | 1200 | 0.0499          | 1.7245         | -6.0645          | 0.9823             | 7.7889          | -510.7414      | -257.2369    | 0.6757          | 0.0176        |
| 0.0498        | 0.44  | 1300 | 0.0459          | 1.6804         | -6.6799          | 0.9848             | 8.3603          | -516.8956      | -257.6772    | 0.6719          | 0.0228        |
| 0.0554        | 0.48  | 1400 | 0.0549          | 1.5994         | -6.8731          | 0.9848             | 8.4726          | -518.8279      | -258.4872    | 0.6684          | 0.0151        |
| 0.0368        | 0.51  | 1500 | 0.0365          | 1.3551         | -7.8016          | 0.9865             | 9.1567          | -528.1130      | -260.9311    | 0.6464          | -0.0223       |
| 0.0514        | 0.54  | 1600 | 0.0333          | 1.3431         | -7.5785          | 0.9891             | 8.9215          | -525.8817      | -261.0511    | 0.6617          | -0.0159       |
| 0.0365        | 0.58  | 1700 | 0.0310          | 1.0743         | -8.8130          | 0.9933             | 9.8873          | -538.2264      | -263.7386    | 0.6354          | -0.0234       |
| 0.0214        | 0.61  | 1800 | 0.0300          | 1.1310         | -9.1633          | 0.9924             | 10.2942         | -541.7292      | -263.1721    | 0.6278          | -0.0199       |
| 0.0321        | 0.65  | 1900 | 0.0267          | 0.9514         | -9.9884          | 0.9924             | 10.9399         | -549.9811      | -264.9675    | 0.6044          | -0.0268       |
| 0.0239        | 0.68  | 2000 | 0.0249          | 0.9624         | -10.0608         | 0.9933             | 11.0232         | -550.7047      | -264.8574    | 0.5955          | -0.0628       |
| 0.0484        | 0.71  | 2100 | 0.0222          | 0.9176         | -11.0294         | 0.9941             | 11.9470         | -560.3909      | -265.3055    | 0.6043          | -0.0338       |
| 0.0239        | 0.75  | 2200 | 0.0209          | 0.8247         | -10.5070         | 0.9941             | 11.3317         | -555.1663      | -266.2345    | 0.5913          | -0.0648       |
| 0.0219        | 0.78  | 2300 | 0.0221          | 0.7334         | -10.5131         | 0.9924             | 11.2465         | -555.2272      | -267.1474    | 0.5942          | -0.0808       |
| 0.0168        | 0.82  | 2400 | 0.0200          | 0.8909         | -11.0818         | 0.9966             | 11.9727         | -560.9146      | -265.5724    | 0.6006          | -0.0770       |
| 0.0135        | 0.85  | 2500 | 0.0187          | 0.7273         | -11.3130         | 0.9958             | 12.0403         | -563.2269      | -267.2084    | 0.5948          | -0.0711       |
| 0.0211        | 0.88  | 2600 | 0.0190          | 0.5390         | -11.8492         | 0.9949             | 12.3883         | -568.5890      | -269.0913    | 0.5818          | -0.0719       |
| 0.0209        | 0.92  | 2700 | 0.0170          | 0.4957         | -12.3176         | 0.9949             | 12.8133         | -573.2725      | -269.5249    | 0.5537          | -0.0912       |
| 0.0168        | 0.95  | 2800 | 0.0175          | 0.5850         | -12.2737         | 0.9958             | 12.8587         | -572.8340      | -268.6321    | 0.5471          | -0.0870       |
| 0.0129        | 0.99  | 2900 | 0.0171          | 0.5691         | -12.7785         | 0.9949             | 13.3476         | -577.8817      | -268.7910    | 0.5441          | -0.0810       |
| 0.0134        | 1.02  | 3000 | 0.0188          | 0.3713         | -13.0254         | 0.9966             | 13.3966         | -580.3504      | -270.7691    | 0.5261          | -0.1163       |
| 0.0127        | 1.05  | 3100 | 0.0155          | 0.1769         | -14.3013         | 0.9958             | 14.4782         | -593.1094      | -272.7126    | 0.4928          | -0.1129       |
| 0.0072        | 1.09  | 3200 | 0.0147          | 0.2474         | -13.6499         | 0.9975             | 13.8973         | -586.5952      | -272.0074    | 0.4753          | -0.1479       |
| 0.0068        | 1.12  | 3300 | 0.0179          | 0.2168         | -14.6122         | 0.9958             | 14.8291         | -596.2192      | -272.3131    | 0.4722          | -0.1430       |
| 0.0323        | 1.16  | 3400 | 0.0218          | 0.2536         | -15.9479         | 0.9958             | 16.2015         | -609.5762      | -271.9459    | 0.4159          | -0.1430       |
| 0.0232        | 1.19  | 3500 | 0.0171          | 0.4273         | -14.2304         | 0.9958             | 14.6577         | -592.4003      | -270.2084    | 0.4598          | -0.1410       |
| 0.0112        | 1.22  | 3600 | 0.0165          | 0.3634         | -15.1069         | 0.9966             | 15.4704         | -601.1661      | -270.8472    | 0.4366          | -0.1521       |
| 0.0136        | 1.26  | 3700 | 0.0176          | 0.2922         | -14.4784         | 0.9966             | 14.7706         | -594.8809      | -271.5597    | 0.4553          | -0.1679       |
| 0.0089        | 1.29  | 3800 | 0.0155          | 0.2349         | -14.8970         | 0.9966             | 15.1319         | -599.0669      | -272.1331    | 0.4447          | -0.1721       |
| 0.01          | 1.33  | 3900 | 0.0123          | 0.1309         | -15.3620         | 0.9975             | 15.4930         | -603.7170      | -273.1724    | 0.4180          | -0.1834       |
| 0.0055        | 1.36  | 4000 | 0.0132          | 0.0227         | -16.5037         | 0.9975             | 16.5265         | -615.1339      | -274.2542    | 0.3836          | -0.1899       |
| 0.004         | 1.39  | 4100 | 0.0122          | 0.2637         | -15.2111         | 0.9975             | 15.4748         | -602.2075      | -271.8445    | 0.4087          | -0.1870       |
| 0.0057        | 1.43  | 4200 | 0.0117          | 0.1593         | -16.5411         | 0.9975             | 16.7003         | -615.5074      | -272.8891    | 0.4086          | -0.1518       |
| 0.0041        | 1.46  | 4300 | 0.0113          | -0.0815        | -17.4813         | 0.9975             | 17.3998         | -624.9092      | -275.2963    | 0.3722          | -0.1782       |
| 0.0039        | 1.5   | 4400 | 0.0115          | 0.2051         | -17.4309         | 0.9983             | 17.6359         | -624.4056      | -272.4310    | 0.3445          | -0.1975       |
| 0.0075        | 1.53  | 4500 | 0.0098          | 0.2810         | -15.5168         | 0.9992             | 15.7978         | -605.2650      | -271.6719    | 0.4021          | -0.1980       |
| 0.0034        | 1.56  | 4600 | 0.0123          | 0.2926         | -15.4764         | 0.9992             | 15.7690         | -604.8610      | -271.5560    | 0.4225          | -0.1766       |
| 0.0066        | 1.6   | 4700 | 0.0125          | 0.3192         | -14.8325         | 0.9992             | 15.1517         | -598.4222      | -271.2900    | 0.4068          | -0.2026       |
| 0.0079        | 1.63  | 4800 | 0.0124          | 0.1405         | -15.1594         | 0.9992             | 15.2999         | -601.6904      | -273.0765    | 0.3942          | -0.2089       |
| 0.0063        | 1.67  | 4900 | 0.0108          | -0.1407        | -17.4715         | 0.9975             | 17.3308         | -624.8116      | -275.8886    | 0.3294          | -0.2140       |
| 0.0026        | 1.7   | 5000 | 0.0116          | -0.0520        | -17.9554         | 0.9992             | 17.9034         | -629.6504      | -275.0013    | 0.3361          | -0.2062       |
| 0.0151        | 1.73  | 5100 | 0.0112          | 0.5727         | -14.3495         | 0.9975             | 14.9221         | -593.5914      | -268.7550    | 0.4234          | -0.1897       |
| 0.0049        | 1.77  | 5200 | 0.0083          | 0.2709         | -15.7593         | 0.9975             | 16.0301         | -607.6893      | -271.7730    | 0.3788          | -0.2054       |
| 0.0036        | 1.8   | 5300 | 0.0093          | 0.2336         | -16.3789         | 0.9992             | 16.6125         | -613.8856      | -272.1456    | 0.3798          | -0.1961       |
| 0.0076        | 1.84  | 5400 | 0.0086          | 0.2618         | -16.1654         | 0.9992             | 16.4272         | -611.7511      | -271.8639    | 0.3844          | -0.2090       |
| 0.0028        | 1.87  | 5500 | 0.0079          | 0.3444         | -15.7817         | 0.9983             | 16.1261         | -607.9135      | -271.0378    | 0.3874          | -0.2135       |
| 0.0102        | 1.9   | 5600 | 0.0081          | 0.3276         | -15.7265         | 0.9983             | 16.0541         | -607.3622      | -271.2061    | 0.3713          | -0.2243       |
| 0.0051        | 1.94  | 5700 | 0.0072          | 0.3979         | -15.8670         | 0.9983             | 16.2649         | -608.7670      | -270.5030    | 0.3562          | -0.2429       |
| 0.0068        | 1.97  | 5800 | 0.0069          | 0.2817         | -16.7894         | 0.9992             | 17.0711         | -617.9911      | -271.6647    | 0.3329          | -0.2424       |
| 0.0021        | 2.01  | 5900 | 0.0061          | 0.0725         | -17.3567         | 0.9992             | 17.4292         | -623.6634      | -273.7564    | 0.3142          | -0.2479       |
| 0.0031        | 2.04  | 6000 | 0.0066          | -0.0179        | -18.1905         | 0.9983             | 18.1726         | -632.0013      | -274.6606    | 0.2948          | -0.2445       |
| 0.0032        | 2.07  | 6100 | 0.0064          | 0.0140         | -17.8441         | 0.9992             | 17.8581         | -628.5374      | -274.3414    | 0.3043          | -0.2424       |
| 0.0043        | 2.11  | 6200 | 0.0066          | 0.0216         | -18.5824         | 0.9983             | 18.6041         | -635.9210      | -274.2653    | 0.2863          | -0.2416       |
| 0.0033        | 2.14  | 6300 | 0.0070          | 0.0355         | -18.3245         | 0.9983             | 18.3600         | -633.3419      | -274.1270    | 0.2783          | -0.2691       |
| 0.0036        | 2.18  | 6400 | 0.0065          | 0.1473         | -18.2292         | 0.9983             | 18.3765         | -632.3882      | -273.0083    | 0.2772          | -0.2744       |
| 0.0023        | 2.21  | 6500 | 0.0064          | 0.0450         | -17.9052         | 0.9992             | 17.9502         | -629.1483      | -274.0311    | 0.2750          | -0.2756       |
| 0.0025        | 2.24  | 6600 | 0.0061          | -0.0642        | -17.4265         | 0.9983             | 17.3623         | -624.3619      | -275.1237    | 0.2773          | -0.2881       |
| 0.0008        | 2.28  | 6700 | 0.0058          | 0.0951         | -17.3710         | 0.9992             | 17.4661         | -623.8071      | -273.5309    | 0.2703          | -0.2891       |
| 0.002         | 2.31  | 6800 | 0.0057          | -0.0533        | -18.2744         | 0.9992             | 18.2212         | -632.8410      | -275.0142    | 0.2544          | -0.2823       |
| 0.0052        | 2.35  | 6900 | 0.0073          | 0.0152         | -18.6276         | 0.9983             | 18.6427         | -636.3723      | -274.3301    | 0.2365          | -0.2928       |
| 0.0029        | 2.38  | 7000 | 0.0056          | 0.0321         | -18.4600         | 0.9992             | 18.4921         | -634.6968      | -274.1606    | 0.2366          | -0.2889       |
| 0.0013        | 2.41  | 7100 | 0.0058          | -0.1304        | -19.1740         | 0.9983             | 19.0436         | -641.8367      | -275.7859    | 0.2306          | -0.2898       |
| 0.0011        | 2.45  | 7200 | 0.0059          | -0.0977        | -19.2699         | 0.9992             | 19.1722         | -642.7958      | -275.4590    | 0.2227          | -0.2916       |
| 0.0058        | 2.48  | 7300 | 0.0059          | -0.0938        | -18.8625         | 0.9992             | 18.7687         | -638.7216      | -275.4195    | 0.2314          | -0.2945       |
| 0.004         | 2.52  | 7400 | 0.0067          | -0.1457        | -19.2264         | 0.9992             | 19.0807         | -642.3610      | -275.9389    | 0.2194          | -0.2959       |
| 0.0042        | 2.55  | 7500 | 0.0056          | -0.0100        | -18.6009         | 0.9992             | 18.5909         | -636.1058      | -274.5812    | 0.2398          | -0.3003       |
| 0.0033        | 2.58  | 7600 | 0.0057          | -0.1484        | -19.3114         | 0.9992             | 19.1630         | -643.2104      | -275.9655    | 0.2215          | -0.2965       |
| 0.0012        | 2.62  | 7700 | 0.0051          | -0.1574        | -19.2398         | 0.9992             | 19.0824         | -642.4946      | -276.0552    | 0.2210          | -0.3052       |
| 0.002         | 2.65  | 7800 | 0.0052          | -0.1220        | -19.5223         | 0.9992             | 19.4004         | -645.3201      | -275.7011    | 0.2102          | -0.3165       |
| 0.0053        | 2.69  | 7900 | 0.0051          | -0.1063        | -19.6889         | 0.9992             | 19.5826         | -646.9856      | -275.5444    | 0.2106          | -0.3074       |
| 0.0013        | 2.72  | 8000 | 0.0056          | -0.1357        | -19.8339         | 0.9992             | 19.6982         | -648.4356      | -275.8386    | 0.2029          | -0.3129       |
| 0.002         | 2.75  | 8100 | 0.0050          | -0.1024        | -19.6476         | 0.9992             | 19.5452         | -646.5726      | -275.5056    | 0.1993          | -0.3168       |
| 0.0016        | 2.79  | 8200 | 0.0052          | -0.0378        | -19.4058         | 0.9992             | 19.3680         | -644.1544      | -274.8596    | 0.2152          | -0.3120       |
| 0.0015        | 2.82  | 8300 | 0.0049          | -0.0549        | -19.2544         | 0.9992             | 19.1994         | -642.6404      | -275.0310    | 0.2093          | -0.3168       |
| 0.003         | 2.86  | 8400 | 0.0051          | -0.1218        | -19.2614         | 0.9983             | 19.1396         | -642.7106      | -275.7000    | 0.2057          | -0.3182       |
| 0.0019        | 2.89  | 8500 | 0.0050          | -0.1057        | -19.3041         | 0.9992             | 19.1983         | -643.1373      | -275.5387    | 0.2077          | -0.3151       |
| 0.0039        | 2.92  | 8600 | 0.0052          | -0.0863        | -19.2826         | 0.9992             | 19.1963         | -642.9229      | -275.3445    | 0.2057          | -0.3208       |
| 0.0042        | 2.96  | 8700 | 0.0048          | -0.1111        | -19.3837         | 0.9992             | 19.2726         | -643.9339      | -275.5926    | 0.2008          | -0.3215       |
| 0.0023        | 2.99  | 8800 | 0.0050          | -0.1085        | -19.4076         | 0.9992             | 19.2990         | -644.1723      | -275.5668    | 0.1985          | -0.3228       |


### Framework versions

- Transformers 4.38.0.dev0
- Pytorch 2.1.1+cu121
- Datasets 2.14.6
- Tokenizers 0.15.1
