generated_from_trainer base_model:meta-llama/Llama-2-7b-hf region:us
---
base_model: meta-llama/Llama-2-7b-hf
tags:
- generated_from_trainer
model-index:
- name: llama2_lora_ice4
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# llama2_lora_ice4

This model is a fine-tuned version of [meta-llama/Llama-2-7b-hf](https://huggingface.co/meta-llama/Llama-2-7b-hf) on an unknown dataset.
It achieves the following results on the evaluation set:
- Loss: 1.8018

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 0.0001
- train_batch_size: 8
- eval_batch_size: 8
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- training_steps: 200000

### Training results

| Training Loss | Epoch | Step   | Validation Loss |
|:-------------:|:-----:|:------:|:---------------:|
| 2.1501        | 0.19  | 5000   | 2.1064          |
| 2.0868        | 0.38  | 10000  | 1.9864          |
| 1.9857        | 0.57  | 15000  | 1.9325          |
| 1.897         | 0.75  | 20000  | 1.8989          |
| 1.9629        | 0.94  | 25000  | 1.8773          |
| 1.8638        | 1.13  | 30000  | 1.8629          |
| 1.891         | 1.32  | 35000  | 1.8523          |
| 1.8903        | 1.51  | 40000  | 1.8422          |
| 1.8059        | 1.7   | 45000  | 1.8354          |
| 1.8479        | 1.88  | 50000  | 1.8291          |
| 1.8491        | 2.07  | 55000  | 1.8261          |
| 1.8497        | 2.26  | 60000  | 1.8228          |
| 1.8306        | 2.45  | 65000  | 1.8194          |
| 1.8482        | 2.64  | 70000  | 1.8162          |
| 1.8247        | 2.83  | 75000  | 1.8139          |
| 1.8428        | 3.02  | 80000  | 1.8122          |
| 1.8078        | 3.2   | 85000  | 1.8108          |
| 1.8398        | 3.39  | 90000  | 1.8090          |
| 1.7911        | 3.58  | 95000  | 1.8080          |
| 1.8222        | 3.77  | 100000 | 1.8067          |
| 1.8152        | 3.96  | 105000 | 1.8054          |
| 1.8237        | 4.15  | 110000 | 1.8053          |
| 1.7876        | 4.33  | 115000 | 1.8048          |
| 1.799         | 4.52  | 120000 | 1.8042          |
| 1.8215        | 4.71  | 125000 | 1.8037          |
| 1.8099        | 4.9   | 130000 | 1.8031          |
| 1.812         | 5.09  | 135000 | 1.8031          |
| 1.8308        | 5.28  | 140000 | 1.8028          |
| 1.8139        | 5.47  | 145000 | 1.8025          |
| 1.8233        | 5.65  | 150000 | 1.8025          |
| 1.8346        | 5.84  | 155000 | 1.8022          |
| 1.8173        | 6.03  | 160000 | 1.8020          |
| 1.8475        | 6.22  | 165000 | 1.8021          |
| 1.7865        | 6.41  | 170000 | 1.8019          |
| 1.7849        | 6.6   | 175000 | 1.8019          |
| 1.8436        | 6.78  | 180000 | 1.8019          |
| 1.7984        | 6.97  | 185000 | 1.8019          |
| 1.803         | 7.16  | 190000 | 1.8019          |
| 1.8692        | 7.35  | 195000 | 1.8018          |
| 1.7558        | 7.54  | 200000 | 1.8018          |


### Framework versions

- Transformers 4.34.1
- Pytorch 2.0.1+cu117
- Datasets 2.14.5
- Tokenizers 0.14.1
