transformers tensorboard safetensors mixtral text-generation trl dpo generated_from_trainer conversational base_model:ondevicellm/tinyllama_moe_sft_ultrachat_epochs3 license:apache-2.0 autotrain_compatible endpoints_compatible text-generation-inference region:us
---
license: apache-2.0
base_model: ondevicellm/tinyllama_moe_sft_ultrachat_epochs3
tags:
- trl
- dpo
- generated_from_trainer
model-index:
- name: tinyllama_moe_dpo_ultrafeedback_epochs5
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# tinyllama_moe_dpo_ultrafeedback_epochs5

This model is a fine-tuned version of [ondevicellm/tinyllama_moe_sft_ultrachat_epochs3](https://huggingface.co/ondevicellm/tinyllama_moe_sft_ultrachat_epochs3) on the None dataset.
It achieves the following results on the evaluation set:
- Loss: 0.5698
- Rewards/chosen: -1.5249
- Rewards/rejected: -2.1850
- Rewards/accuracies: 0.7460
- Rewards/margins: 0.6601
- Logps/rejected: -525.1185
- Logps/chosen: -501.3176
- Logits/rejected: -1.7144
- Logits/chosen: -1.8206

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 5e-07
- train_batch_size: 8
- eval_batch_size: 8
- seed: 42
- distributed_type: multi-GPU
- num_devices: 4
- gradient_accumulation_steps: 2
- total_train_batch_size: 64
- total_eval_batch_size: 32
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: cosine
- lr_scheduler_warmup_steps: 96
- num_epochs: 5

### Training results

| Training Loss | Epoch | Step | Validation Loss | Rewards/chosen | Rewards/rejected | Rewards/accuracies | Rewards/margins | Logps/rejected | Logps/chosen | Logits/rejected | Logits/chosen |
|:-------------:|:-----:|:----:|:---------------:|:--------------:|:----------------:|:------------------:|:---------------:|:--------------:|:------------:|:---------------:|:-------------:|
| 0.6916        | 0.1   | 100  | 0.6914          | 0.0044         | -0.0007          | 0.6290             | 0.0050          | -306.6867      | -348.3935    | -2.7893         | -2.8529       |
| 0.6838        | 0.21  | 200  | 0.6833          | 0.0222         | -0.0020          | 0.6548             | 0.0242          | -306.8183      | -346.6077    | -2.7745         | -2.8394       |
| 0.6683        | 0.31  | 300  | 0.6720          | 0.0087         | -0.0449          | 0.6647             | 0.0536          | -311.1083      | -347.9552    | -2.7457         | -2.8123       |
| 0.655         | 0.42  | 400  | 0.6583          | -0.0568        | -0.1516          | 0.6766             | 0.0948          | -321.7766      | -354.5066    | -2.6922         | -2.7610       |
| 0.6435        | 0.52  | 500  | 0.6453          | -0.1710        | -0.3165          | 0.6706             | 0.1455          | -338.2649      | -365.9251    | -2.6457         | -2.7160       |
| 0.641         | 0.63  | 600  | 0.6366          | -0.2120        | -0.3985          | 0.6825             | 0.1865          | -346.4684      | -370.0310    | -2.5893         | -2.6615       |
| 0.6207        | 0.73  | 700  | 0.6319          | -0.2615        | -0.4812          | 0.6706             | 0.2197          | -354.7403      | -374.9797    | -2.5381         | -2.6120       |
| 0.6025        | 0.84  | 800  | 0.6249          | -0.3306        | -0.5888          | 0.6964             | 0.2583          | -365.5025      | -381.8849    | -2.4644         | -2.5413       |
| 0.6317        | 0.94  | 900  | 0.6185          | -0.5597        | -0.8426          | 0.7063             | 0.2829          | -390.8784      | -404.7987    | -2.4027         | -2.4814       |
| 0.6087        | 1.05  | 1000 | 0.6137          | -0.5045        | -0.8126          | 0.7004             | 0.3081          | -387.8767      | -399.2817    | -2.3842         | -2.4637       |
| 0.5993        | 1.15  | 1100 | 0.6077          | -0.6040        | -0.9415          | 0.7044             | 0.3375          | -400.7663      | -409.2302    | -2.3436         | -2.4254       |
| 0.5628        | 1.26  | 1200 | 0.6026          | -0.8401        | -1.2238          | 0.7103             | 0.3837          | -429.0004      | -432.8431    | -2.2635         | -2.3475       |
| 0.5856        | 1.36  | 1300 | 0.5971          | -0.7421        | -1.1421          | 0.7242             | 0.3999          | -420.8279      | -423.0439    | -2.2233         | -2.3091       |
| 0.5672        | 1.47  | 1400 | 0.5930          | -0.7829        | -1.2202          | 0.7143             | 0.4373          | -428.6362      | -427.1146    | -2.1938         | -2.2804       |
| 0.5536        | 1.57  | 1500 | 0.5872          | -0.8347        | -1.2945          | 0.7202             | 0.4599          | -436.0717      | -432.2956    | -2.1433         | -2.2324       |
| 0.5669        | 1.67  | 1600 | 0.5858          | -0.7867        | -1.2636          | 0.7163             | 0.4769          | -432.9818      | -427.4996    | -2.1168         | -2.2065       |
| 0.5312        | 1.78  | 1700 | 0.5831          | -0.9925        | -1.4919          | 0.7262             | 0.4994          | -455.8103      | -448.0764    | -2.0492         | -2.1424       |
| 0.5596        | 1.88  | 1800 | 0.5798          | -1.0023        | -1.5297          | 0.7361             | 0.5274          | -459.5894      | -449.0625    | -2.0168         | -2.1124       |
| 0.5489        | 1.99  | 1900 | 0.5813          | -0.8832        | -1.3904          | 0.7202             | 0.5072          | -445.6621      | -437.1509    | -2.0039         | -2.0990       |
| 0.5327        | 2.09  | 2000 | 0.5795          | -0.9218        | -1.4418          | 0.7242             | 0.5200          | -450.7982      | -441.0125    | -1.9626         | -2.0594       |
| 0.5225        | 2.2   | 2100 | 0.5779          | -1.1696        | -1.7317          | 0.7401             | 0.5621          | -479.7868      | -465.7886    | -1.9207         | -2.0189       |
| 0.5085        | 2.3   | 2200 | 0.5769          | -1.1637        | -1.7425          | 0.7321             | 0.5789          | -480.8718      | -465.1949    | -1.8892         | -1.9891       |
| 0.5255        | 2.41  | 2300 | 0.5770          | -1.2191        | -1.7925          | 0.7341             | 0.5734          | -485.8650      | -470.7390    | -1.8632         | -1.9614       |
| 0.5116        | 2.51  | 2400 | 0.5742          | -1.1139        | -1.6936          | 0.7381             | 0.5798          | -475.9834      | -460.2151    | -1.8765         | -1.9749       |
| 0.5279        | 2.62  | 2500 | 0.5741          | -1.1556        | -1.7455          | 0.7361             | 0.5899          | -481.1734      | -464.3928    | -1.8664         | -1.9651       |
| 0.4795        | 2.72  | 2600 | 0.5745          | -1.1558        | -1.7459          | 0.7321             | 0.5900          | -481.2056      | -464.4143    | -1.8345         | -1.9355       |
| 0.5217        | 2.83  | 2700 | 0.5699          | -1.3475        | -1.9659          | 0.7440             | 0.6184          | -503.2092      | -483.5756    | -1.7956         | -1.8981       |
| 0.4945        | 2.93  | 2800 | 0.5699          | -1.3594        | -1.9727          | 0.7381             | 0.6132          | -503.8864      | -484.7731    | -1.8126         | -1.9141       |
| 0.477         | 3.04  | 2900 | 0.5721          | -1.3627        | -1.9877          | 0.7361             | 0.6250          | -505.3890      | -485.0972    | -1.7954         | -1.8980       |
| 0.4754        | 3.14  | 3000 | 0.5729          | -1.4117        | -2.0575          | 0.7321             | 0.6458          | -512.3726      | -490.0027    | -1.7473         | -1.8516       |
| 0.4696        | 3.24  | 3100 | 0.5708          | -1.5486        | -2.1921          | 0.7282             | 0.6435          | -525.8281      | -503.6902    | -1.7318         | -1.8363       |
| 0.4804        | 3.35  | 3200 | 0.5730          | -1.5037        | -2.1632          | 0.7321             | 0.6595          | -522.9344      | -499.1950    | -1.7097         | -1.8163       |
| 0.483         | 3.45  | 3300 | 0.5706          | -1.5793        | -2.2451          | 0.7302             | 0.6658          | -531.1252      | -506.7562    | -1.7082         | -1.8147       |
| 0.4791        | 3.56  | 3400 | 0.5723          | -1.4505        | -2.1095          | 0.7262             | 0.6590          | -517.5656      | -493.8777    | -1.7222         | -1.8274       |
| 0.4866        | 3.66  | 3500 | 0.5713          | -1.5091        | -2.1642          | 0.7381             | 0.6551          | -523.0358      | -499.7364    | -1.7191         | -1.8243       |
| 0.4651        | 3.77  | 3600 | 0.5731          | -1.4577        | -2.1177          | 0.7401             | 0.6600          | -518.3928      | -494.6030    | -1.7161         | -1.8217       |
| 0.483         | 3.87  | 3700 | 0.5708          | -1.4280        | -2.0759          | 0.7361             | 0.6479          | -514.2116      | -491.6330    | -1.7275         | -1.8325       |
| 0.4859        | 3.98  | 3800 | 0.5698          | -1.5249        | -2.1850          | 0.7460             | 0.6601          | -525.1185      | -501.3176    | -1.7144         | -1.8206       |
| 0.476         | 4.08  | 3900 | 0.5701          | -1.5060        | -2.1668          | 0.7440             | 0.6608          | -523.2975      | -499.4326    | -1.7157         | -1.8219       |
| 0.4553        | 4.19  | 4000 | 0.5705          | -1.5415        | -2.2042          | 0.7361             | 0.6626          | -527.0359      | -502.9834    | -1.7053         | -1.8120       |
| 0.4864        | 4.29  | 4100 | 0.5721          | -1.5310        | -2.1997          | 0.7381             | 0.6687          | -526.5859      | -501.9312    | -1.6982         | -1.8054       |
| 0.4402        | 4.4   | 4200 | 0.5720          | -1.5402        | -2.2110          | 0.7401             | 0.6708          | -527.7231      | -502.8538    | -1.6937         | -1.8008       |
| 0.4619        | 4.5   | 4300 | 0.5712          | -1.5462        | -2.2169          | 0.7361             | 0.6706          | -528.3046      | -503.4531    | -1.6931         | -1.8004       |
| 0.4421        | 4.6   | 4400 | 0.5710          | -1.5628        | -2.2323          | 0.7381             | 0.6695          | -529.8489      | -505.1078    | -1.6915         | -1.7989       |
| 0.4518        | 4.71  | 4500 | 0.5711          | -1.5704        | -2.2407          | 0.7361             | 0.6703          | -530.6893      | -505.8743    | -1.6913         | -1.7985       |
| 0.4508        | 4.81  | 4600 | 0.5715          | -1.5739        | -2.2436          | 0.7381             | 0.6697          | -530.9782      | -506.2146    | -1.6908         | -1.7981       |
| 0.484         | 4.92  | 4700 | 0.5716          | -1.5737        | -2.2419          | 0.7321             | 0.6682          | -530.8127      | -506.2016    | -1.6901         | -1.7976       |


### Framework versions

- Transformers 4.36.2
- Pytorch 2.1.2+cu118
- Datasets 2.14.6
- Tokenizers 0.15.0
