
<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# openhermes-mistral-dpo-gptq-rlhf

This model is a fine-tuned version of [TheBloke/OpenHermes-2-Mistral-7B-GPTQ](https://huggingface.co/TheBloke/OpenHermes-2-Mistral-7B-GPTQ) on the None dataset.
It achieves the following results on the evaluation set:
- Loss: 0.3258
- Rewards/chosen: -2.0828
- Rewards/rejected: -3.2293
- Rewards/accuracies: 0.9375
- Rewards/margins: 1.1465
- Logps/rejected: -172.3552
- Logps/chosen: -121.1924
- Logits/rejected: -2.1441
- Logits/chosen: -2.1987

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 0.0002
- train_batch_size: 1
- eval_batch_size: 8
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 2
- training_steps: 50
- mixed_precision_training: Native AMP

### Training results

| Training Loss | Epoch | Step | Validation Loss | Rewards/chosen | Rewards/rejected | Rewards/accuracies | Rewards/margins | Logps/rejected | Logps/chosen | Logits/rejected | Logits/chosen |
|:-------------:|:-----:|:----:|:---------------:|:--------------:|:----------------:|:------------------:|:---------------:|:--------------:|:------------:|:---------------:|:-------------:|
| 0.6895        | 0.0   | 10   | 2.0584          | 2.0411         | 2.9118           | 0.75               | -0.8707         | -110.9443      | -79.9538     | -2.3239         | -2.3541       |
| 0.6501        | 0.0   | 20   | 0.5405          | 0.4124         | 0.1311           | 0.75               | 0.2812          | -138.7508      | -96.2407     | -2.1975         | -2.2327       |
| 0.542         | 0.0   | 30   | 0.3799          | -0.7406        | -1.5830          | 0.75               | 0.8424          | -155.8916      | -107.7704    | -2.1702         | -2.2083       |
| 0.4462        | 0.01  | 40   | 0.3440          | -1.5490        | -2.6401          | 0.9375             | 1.0912          | -166.4635      | -115.8540    | -2.1463         | -2.1908       |
| 1.9657        | 0.01  | 50   | 0.3258          | -2.0828        | -3.2293          | 0.9375             | 1.1465          | -172.3552      | -121.1924    | -2.1441         | -2.1987       |


### Framework versions

- Transformers 4.35.2
- Pytorch 2.0.1+cu117
- Datasets 2.15.0
- Tokenizers 0.15.0

Fine Tuned OpenHermes-2-Mistral-7B-GPTQ on falcon_feedback_instruction rlhf dataset
Usage: 
```
! pip install torch==2.0.1 transformers datasets peft accelerate trl bitsandbytes optimum auto-gptq

from peft import AutoPeftModelForCausalLM
from transformers import GenerationConfig
from transformers import AutoTokenizer
import torch
tokenizer = AutoTokenizer.from_pretrained("umarigan/openhermes-mistral-dpo-gptq-rlhf")

inputs = tokenizer("""Do the new shift rules in Major League Baseball make the game more exciting?'""", return_tensors="pt").to("cuda")

model = AutoPeftModelForCausalLM.from_pretrained(
    "umarigan/openhermes-mistral-dpo-gptq-rlhf",
    low_cpu_mem_usage=True,
    return_dict=True,
    torch_dtype=torch.float16,
    device_map="cuda")

generation_config = GenerationConfig(
    do_sample=True,
    top_k=1,
    temperature=0.1,
    max_new_tokens=256,
    pad_token_id=tokenizer.eos_token_id
)

import time
st_time = time.time()
outputs = model.generate(**inputs, generation_config=generation_config)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
print(time.time()-st_time)
```
{"license": "apache-2.0", "tags": ["generated_from_trainer"], "base_model": "TheBloke/OpenHermes-2-Mistral-7B-GPTQ", "model-index": [{"name": "openhermes-mistral-dpo-gptq-rlhf", "results": []}]}
tensorboard safetensors generated_from_trainer base_model:TheBloke/OpenHermes-2-Mistral-7B-GPTQ license:apache-2.0 region:us