transformers safetensors llama text-generation trl dpo generated_from_trainer conversational base_model:meta-llama/Llama-2-7b-chat-hf autotrain_compatible endpoints_compatible text-generation-inference region:us
---
base_model: meta-llama/Llama-2-7b-chat-hf
tags:
- trl
- dpo
- generated_from_trainer
model-index:
- name: chat_1000STEPS_1e7_05beta_DPO
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# chat_1000STEPS_1e7_05beta_DPO

This model is a fine-tuned version of [meta-llama/Llama-2-7b-chat-hf](https://huggingface.co/meta-llama/Llama-2-7b-chat-hf) on an unknown dataset.
It achieves the following results on the evaluation set:
- Loss: 0.6864
- Rewards/chosen: 0.0033
- Rewards/rejected: -0.0130
- Rewards/accuracies: 0.4571
- Rewards/margins: 0.0163
- Logps/rejected: -18.8173
- Logps/chosen: -16.7381
- Logits/rejected: -0.5974
- Logits/chosen: -0.5973

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 1e-07
- train_batch_size: 4
- eval_batch_size: 1
- seed: 42
- gradient_accumulation_steps: 2
- total_train_batch_size: 8
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: cosine
- lr_scheduler_warmup_steps: 100
- training_steps: 1000

### Training results

| Training Loss | Epoch | Step | Validation Loss | Rewards/chosen | Rewards/rejected | Rewards/accuracies | Rewards/margins | Logps/rejected | Logps/chosen | Logits/rejected | Logits/chosen |
|:-------------:|:-----:|:----:|:---------------:|:--------------:|:----------------:|:------------------:|:---------------:|:--------------:|:------------:|:---------------:|:-------------:|
| 0.6957        | 0.2   | 100  | 0.6926          | -0.0030        | -0.0058          | 0.4132             | 0.0028          | -18.8028       | -16.7506     | -0.5972         | -0.5971       |
| 0.6931        | 0.39  | 200  | 0.6899          | 0.0035         | -0.0050          | 0.4835             | 0.0085          | -18.8013       | -16.7376     | -0.5981         | -0.5980       |
| 0.6783        | 0.59  | 300  | 0.6915          | -0.0059        | -0.0111          | 0.4593             | 0.0052          | -18.8135       | -16.7564     | -0.5978         | -0.5977       |
| 0.6952        | 0.78  | 400  | 0.6904          | 0.0004         | -0.0075          | 0.4615             | 0.0079          | -18.8063       | -16.7439     | -0.5975         | -0.5973       |
| 0.6927        | 0.98  | 500  | 0.6904          | -0.0036        | -0.0115          | 0.4396             | 0.0080          | -18.8144       | -16.7518     | -0.5981         | -0.5980       |
| 0.6701        | 1.17  | 600  | 0.6878          | -0.0038        | -0.0170          | 0.4681             | 0.0132          | -18.8254       | -16.7522     | -0.5978         | -0.5977       |
| 0.6796        | 1.37  | 700  | 0.6886          | -0.0031        | -0.0150          | 0.4725             | 0.0119          | -18.8213       | -16.7508     | -0.5970         | -0.5969       |
| 0.6686        | 1.56  | 800  | 0.6881          | -0.0031        | -0.0158          | 0.4813             | 0.0127          | -18.8228       | -16.7508     | -0.5973         | -0.5972       |
| 0.6767        | 1.76  | 900  | 0.6901          | -0.0033        | -0.0123          | 0.4440             | 0.0091          | -18.8159       | -16.7511     | -0.5972         | -0.5971       |
| 0.6702        | 1.95  | 1000 | 0.6864          | 0.0033         | -0.0130          | 0.4571             | 0.0163          | -18.8173       | -16.7381     | -0.5974         | -0.5973       |


### Framework versions

- Transformers 4.37.2
- Pytorch 2.0.0+cu117
- Datasets 2.17.0
- Tokenizers 0.15.2
