generated_from_trainer base_model:DevaMalla/llama7b region:us
---
base_model: DevaMalla/llama7b
tags:
- generated_from_trainer
model-index:
- name: llama_7b_qlora_pds-eval
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# llama_7b_qlora_pds-eval

This model is a fine-tuned version of [DevaMalla/llama7b](https://huggingface.co/DevaMalla/llama7b) on an unknown dataset.
It achieves the following results on the evaluation set:
- Loss: 0.9304

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 0.0002
- train_batch_size: 10
- eval_batch_size: 2
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 1

### Training results

| Training Loss | Epoch | Step | Validation Loss |
|:-------------:|:-----:|:----:|:---------------:|
| 1.2419        | 0.03  | 50   | 1.0315          |
| 0.9965        | 0.05  | 100  | 0.9918          |
| 0.9747        | 0.08  | 150  | 0.9787          |
| 0.9771        | 0.11  | 200  | 0.9705          |
| 0.9653        | 0.14  | 250  | 0.9645          |
| 0.9507        | 0.16  | 300  | 0.9599          |
| 0.9596        | 0.19  | 350  | 0.9563          |
| 0.9497        | 0.22  | 400  | 0.9542          |
| 0.9481        | 0.24  | 450  | 0.9517          |
| 0.9475        | 0.27  | 500  | 0.9495          |
| 0.9376        | 0.3   | 550  | 0.9476          |
| 0.943         | 0.33  | 600  | 0.9463          |
| 0.952         | 0.35  | 650  | 0.9447          |
| 0.9489        | 0.38  | 700  | 0.9433          |
| 0.9347        | 0.41  | 750  | 0.9421          |
| 0.9307        | 0.44  | 800  | 0.9414          |
| 0.9366        | 0.46  | 850  | 0.9406          |
| 0.9246        | 0.49  | 900  | 0.9393          |
| 0.9267        | 0.52  | 950  | 0.9383          |
| 0.9358        | 0.54  | 1000 | 0.9373          |
| 0.9405        | 0.57  | 1050 | 0.9365          |
| 0.9276        | 0.6   | 1100 | 0.9359          |
| 0.9403        | 0.63  | 1150 | 0.9353          |
| 0.9218        | 0.65  | 1200 | 0.9347          |
| 0.9406        | 0.68  | 1250 | 0.9343          |
| 0.9257        | 0.71  | 1300 | 0.9337          |
| 0.92          | 0.73  | 1350 | 0.9334          |
| 0.9356        | 0.76  | 1400 | 0.9328          |
| 0.9279        | 0.79  | 1450 | 0.9322          |
| 0.9214        | 0.82  | 1500 | 0.9320          |
| 0.9214        | 0.84  | 1550 | 0.9316          |
| 0.9258        | 0.87  | 1600 | 0.9312          |
| 0.9308        | 0.9   | 1650 | 0.9310          |
| 0.9251        | 0.93  | 1700 | 0.9307          |
| 0.9207        | 0.95  | 1750 | 0.9306          |
| 0.926         | 0.98  | 1800 | 0.9304          |


### Framework versions

- Transformers 4.32.0.dev0
- Pytorch 2.0.0
- Datasets 2.12.0
- Tokenizers 0.13.3
