safetensors generated_from_trainer base_model:dhmeltzer/llama-7b-SFT_eli5_wiki65k_1024_r_64_alpha_16_merged region:us
---
base_model: dhmeltzer/llama-7b-SFT_eli5_wiki65k_1024_r_64_alpha_16_merged
tags:
- generated_from_trainer
model-index:
- name: llama-7b-SFT-qlora-eli5-wiki_DPO_ds_RM_top_2_1024_r_64_alpha_16
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# llama-7b-SFT-qlora-eli5-wiki_DPO_ds_RM_top_2_1024_r_64_alpha_16

This model is a fine-tuned version of [dhmeltzer/llama-7b-SFT_eli5_wiki65k_1024_r_64_alpha_16_merged](https://huggingface.co/dhmeltzer/llama-7b-SFT_eli5_wiki65k_1024_r_64_alpha_16_merged) on an unknown dataset.
It achieves the following results on the evaluation set:
- Loss: 0.6538
- Rewards/chosen: 0.1408
- Rewards/rejected: -0.0291
- Rewards/accuracies: 0.6248
- Rewards/margins: 0.1699
- Logps/rejected: -199.6676
- Logps/chosen: -203.9681
- Logits/rejected: 0.8159
- Logits/chosen: 0.8393

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 0.0002
- train_batch_size: 32
- eval_batch_size: 32
- seed: 42
- gradient_accumulation_steps: 4
- total_train_batch_size: 128
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: cosine
- lr_scheduler_warmup_ratio: 0.03
- num_epochs: 2

### Training results

| Training Loss | Epoch | Step | Validation Loss | Rewards/chosen | Rewards/rejected | Rewards/accuracies | Rewards/margins | Logps/rejected | Logps/chosen | Logits/rejected | Logits/chosen |
|:-------------:|:-----:|:----:|:---------------:|:--------------:|:----------------:|:------------------:|:---------------:|:--------------:|:------------:|:---------------:|:-------------:|
| 0.6828        | 0.2   | 37   | 0.6867          | -0.3470        | -0.4719          | 0.5792             | 0.1249          | -201.8816      | -206.4072    | 0.7977          | 0.8213        |
| 0.6666        | 0.41  | 74   | 0.6731          | -0.1233        | -0.2593          | 0.5855             | 0.1361          | -200.8187      | -205.2885    | 0.8159          | 0.8381        |
| 0.6713        | 0.61  | 111  | 0.6645          | 0.0492         | -0.1110          | 0.6019             | 0.1602          | -200.0772      | -204.4260    | 0.8299          | 0.8526        |
| 0.6749        | 0.82  | 148  | 0.6593          | 0.2291         | 0.0917           | 0.5912             | 0.1374          | -199.0636      | -203.5266    | 0.8189          | 0.8414        |
| 0.6688        | 1.02  | 185  | 0.6538          | 0.1408         | -0.0291          | 0.6248             | 0.1699          | -199.6676      | -203.9681    | 0.8159          | 0.8393        |
| 0.3721        | 1.23  | 222  | 0.6911          | -0.3548        | -0.6171          | 0.6007             | 0.2623          | -202.6077      | -206.4462    | 0.8193          | 0.8406        |
| 0.2845        | 1.43  | 259  | 0.6989          | -0.3528        | -0.5968          | 0.5984             | 0.2441          | -202.5062      | -206.4359    | 0.7886          | 0.8059        |
| 0.2646        | 1.64  | 296  | 0.6991          | -0.4016        | -0.6359          | 0.5880             | 0.2343          | -202.7015      | -206.6800    | 0.7696          | 0.7875        |
| 0.2263        | 1.84  | 333  | 0.7063          | -0.4773        | -0.7137          | 0.5925             | 0.2365          | -203.0908      | -207.0584    | 0.7653          | 0.7833        |


### Framework versions

- Transformers 4.32.1
- Pytorch 2.0.1+cu118
- Datasets 2.14.4
- Tokenizers 0.13.3

# [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)
Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_dhmeltzer__llama-7b-SFT-qlora-eli5-wiki_DPO_ds_RM_top_2_1024_r_64_alpha_16)

| Metric                | Value                     |
|-----------------------|---------------------------|
| Avg.                  | 44.03   |
| ARC (25-shot)         | 54.1          |
| HellaSwag (10-shot)   | 78.74    |
| MMLU (5-shot)         | 45.44         |
| TruthfulQA (0-shot)   | 43.4   |
| Winogrande (5-shot)   | 73.64   |
| GSM8K (5-shot)        | 4.55        |
| DROP (3-shot)         | 8.35         |
