transformers safetensors llama text-generation trl dpo generated_from_trainer conversational base_model:meta-llama/Llama-2-7b-chat-hf autotrain_compatible endpoints_compatible text-generation-inference region:us
---
base_model: meta-llama/Llama-2-7b-chat-hf
tags:
- trl
- dpo
- generated_from_trainer
model-index:
- name: chat_700STEPS_1e4rate
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# chat_700STEPS_1e4rate

This model is a fine-tuned version of [meta-llama/Llama-2-7b-chat-hf](https://huggingface.co/meta-llama/Llama-2-7b-chat-hf) on an unknown dataset.
It achieves the following results on the evaluation set:
- Loss: 1.1848
- Rewards/chosen: -4.4236
- Rewards/rejected: -4.3538
- Rewards/accuracies: 0.4000
- Rewards/margins: -0.0698
- Logps/rejected: -62.3289
- Logps/chosen: -60.9807
- Logits/rejected: -4.5000
- Logits/chosen: -4.5000

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 0.0001
- train_batch_size: 4
- eval_batch_size: 1
- seed: 42
- gradient_accumulation_steps: 2
- total_train_batch_size: 8
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: cosine
- lr_scheduler_warmup_steps: 100
- training_steps: 700

### Training results

| Training Loss | Epoch | Step | Validation Loss | Rewards/chosen | Rewards/rejected | Rewards/accuracies | Rewards/margins | Logps/rejected | Logps/chosen | Logits/rejected | Logits/chosen |
|:-------------:|:-----:|:----:|:---------------:|:--------------:|:----------------:|:------------------:|:---------------:|:--------------:|:------------:|:---------------:|:-------------:|
| 1.2329        | 0.1   | 50   | 1.6243          | -8.0715        | -8.0351          | 0.4176             | -0.0364         | -99.1426       | -97.4598     | -0.6161         | -0.6164       |
| 1.3399        | 0.2   | 100  | 1.2323          | -4.6270        | -4.6104          | 0.3978             | -0.0167         | -64.8951       | -63.0151     | -2.3014         | -2.3014       |
| 1.316         | 0.29  | 150  | 1.2017          | -4.3807        | -4.3323          | 0.4000             | -0.0484         | -62.1140       | -60.5517     | -2.9891         | -2.9891       |
| 1.2778        | 0.39  | 200  | 1.1891          | -4.3216        | -4.2623          | 0.4044             | -0.0593         | -61.4138       | -59.9605     | -3.4116         | -3.4116       |
| 1.0721        | 0.49  | 250  | 1.1847          | -4.3234        | -4.2565          | 0.4110             | -0.0669         | -61.3561       | -59.9788     | -3.7977         | -3.7977       |
| 1.3775        | 0.59  | 300  | 1.1896          | -4.3481        | -4.2745          | 0.4176             | -0.0737         | -61.5360       | -60.2260     | -4.0911         | -4.0911       |
| 1.3232        | 0.68  | 350  | 1.1818          | -4.3486        | -4.2846          | 0.4088             | -0.0640         | -61.6375       | -60.2310     | -4.2490         | -4.2490       |
| 1.2476        | 0.78  | 400  | 1.1789          | -4.3705        | -4.3105          | 0.4044             | -0.0600         | -61.8961       | -60.4495     | -4.1920         | -4.1920       |
| 1.3082        | 0.88  | 450  | 1.1766          | -4.3831        | -4.3296          | 0.4000             | -0.0535         | -62.0870       | -60.5753     | -4.2348         | -4.2348       |
| 1.1007        | 0.98  | 500  | 1.1762          | -4.4022        | -4.3470          | 0.4000             | -0.0552         | -62.2610       | -60.7662     | -4.2010         | -4.2010       |
| 0.8786        | 1.07  | 550  | 1.1811          | -4.4150        | -4.3527          | 0.4000             | -0.0623         | -62.3185       | -60.8948     | -4.3351         | -4.3351       |
| 1.3113        | 1.17  | 600  | 1.1842          | -4.4174        | -4.3487          | 0.4000             | -0.0687         | -62.2785       | -60.9186     | -4.4858         | -4.4858       |
| 0.9783        | 1.27  | 650  | 1.1850          | -4.4234        | -4.3533          | 0.4000             | -0.0701         | -62.3242       | -60.9783     | -4.4997         | -4.4997       |
| 1.3696        | 1.37  | 700  | 1.1848          | -4.4236        | -4.3538          | 0.4000             | -0.0698         | -62.3289       | -60.9807     | -4.5000         | -4.5000       |


### Framework versions

- Transformers 4.37.2
- Pytorch 2.0.0+cu117
- Datasets 2.17.0
- Tokenizers 0.15.2
