peft safetensors arxiv:1910.09700 base_model:meta-llama/Llama-2-7b-chat-hf region:us
---
library_name: peft
base_model: meta-llama/Llama-2-7b-chat-hf
---

# Model Card for Model ID

<!-- Provide a quick summary of what the model is/does. -->

```text
Start training at: 2024-01-05 08:36:20.951749
0%|              | 0/40220 [00:00<?, ?it/s]
{'loss': 0.8428, 'learning_rate': 4.75e-05, 'epoch': 1.0}
{'eval_loss': 0.7344576716423035, 'eval_runtime': 40.2922, 'eval_samples_per_second': 10.002, 'eval_steps_per_second': 1.266, 'epoch': 1.0}
{'loss': 0.6337, 'learning_rate': 4.5e-05, 'epoch': 2.0}
{'eval_loss': 0.635863184928894, 'eval_runtime': 40.4216, 'eval_samples_per_second': 9.97, 'eval_steps_per_second': 1.262, 'epoch': 2.0}
{'loss': 0.5509, 'learning_rate': 4.25e-05, 'epoch': 3.0}
{'eval_loss': 0.555050790309906, 'eval_runtime': 40.1906, 'eval_samples_per_second': 10.027, 'eval_steps_per_second': 1.269, 'epoch': 3.0}
{'loss': 0.4837, 'learning_rate': 4e-05, 'epoch': 4.0}
{'eval_loss': 0.5025414228439331, 'eval_runtime': 40.062, 'eval_samples_per_second': 10.059, 'eval_steps_per_second': 1.273, 'epoch': 4.0}
{'loss': 0.4289, 'learning_rate': 3.7500000000000003e-05, 'epoch': 5.0}
{'eval_loss': 0.4429796040058136, 'eval_runtime': 40.1244, 'eval_samples_per_second': 10.044, 'eval_steps_per_second': 1.271, 'epoch': 5.0}
{'loss': 0.3825, 'learning_rate': 3.5e-05, 'epoch': 6.0}
{'eval_loss': 0.40111732482910156, 'eval_runtime': 40.0455, 'eval_samples_per_second': 10.064, 'eval_steps_per_second': 1.274, 'epoch': 6.0}
{'loss': 0.3447, 'learning_rate': 3.2500000000000004e-05, 'epoch': 7.0}
{'eval_loss': 0.3600844144821167, 'eval_runtime': 40.0694, 'eval_samples_per_second': 10.058, 'eval_steps_per_second': 1.273, 'epoch': 7.0}
{'loss': 0.3121, 'learning_rate': 3e-05, 'epoch': 8.0}
{'eval_loss': 0.3315456211566925, 'eval_runtime': 39.9981, 'eval_samples_per_second': 10.075, 'eval_steps_per_second': 1.275, 'epoch': 8.0}
{'loss': 0.2843, 'learning_rate': 2.7500000000000004e-05, 'epoch': 9.0}
{'eval_loss': 0.3017001450061798, 'eval_runtime': 40.0311, 'eval_samples_per_second': 10.067, 'eval_steps_per_second': 1.274, 'epoch': 9.0}
{'loss': 0.2591, 'learning_rate': 2.5e-05, 'epoch': 10.0}
{'eval_loss': 0.27300581336021423, 'eval_runtime': 40.0794, 'eval_samples_per_second': 10.055, 'eval_steps_per_second': 1.272, 'epoch': 10.0}
{'loss': 0.2372, 'learning_rate': 2.25e-05, 'epoch': 11.0}
{'eval_loss': 0.2534460127353668, 'eval_runtime': 40.0164, 'eval_samples_per_second': 10.071, 'eval_steps_per_second': 1.274, 'epoch': 11.0}
{'loss': 0.2177, 'learning_rate': 2e-05, 'epoch': 12.0}
{'eval_loss': 0.23217065632343292, 'eval_runtime': 40.0506, 'eval_samples_per_second': 10.062, 'eval_steps_per_second': 1.273, 'epoch': 12.0}
{'loss': 0.1998, 'learning_rate': 1.75e-05, 'epoch': 13.0}
{'eval_loss': 0.215168759226799, 'eval_runtime': 40.0158, 'eval_samples_per_second': 10.071, 'eval_steps_per_second': 1.274, 'epoch': 13.0}
{'loss': 0.1841, 'learning_rate': 1.5e-05, 'epoch': 14.0}
{'eval_loss': 0.19979538023471832, 'eval_runtime': 40.0119, 'eval_samples_per_second': 10.072, 'eval_steps_per_second': 1.275, 'epoch': 14.0}
{'loss': 0.1704, 'learning_rate': 1.25e-05, 'epoch': 15.0}
{'eval_loss': 0.18658295273780823, 'eval_runtime': 40.0046, 'eval_samples_per_second': 10.074, 'eval_steps_per_second': 1.275, 'epoch': 15.0}
{'loss': 0.1588, 'learning_rate': 1e-05, 'epoch': 16.0}
{'eval_loss': 0.17462818324565887, 'eval_runtime': 40.0819, 'eval_samples_per_second': 10.054, 'eval_steps_per_second': 1.272, 'epoch': 16.0}
{'loss': 0.1481, 'learning_rate': 7.5e-06, 'epoch': 17.0}
{'eval_loss': 0.1658513993024826, 'eval_runtime': 40.0379, 'eval_samples_per_second': 10.065, 'eval_steps_per_second': 1.274, 'epoch': 17.0}
{'loss': 0.1393, 'learning_rate': 5e-06, 'epoch': 18.0}
{'eval_loss': 0.1572066694498062, 'eval_runtime': 40.0376, 'eval_samples_per_second': 10.066, 'eval_steps_per_second': 1.274, 'epoch': 18.0}
{'loss': 0.132, 'learning_rate': 2.5e-06, 'epoch': 19.0}
{'eval_loss': 0.15208856761455536, 'eval_runtime': 40.0185, 'eval_samples_per_second': 10.07, 'eval_steps_per_second': 1.274, 'epoch': 19.0}
{'loss': 0.1262, 'learning_rate': 0.0, 'epoch': 20.0}
{'eval_loss': 0.15022191405296326, 'eval_runtime': 40.0854, 'eval_samples_per_second': 10.054, 'eval_steps_per_second': 1.272, 'epoch': 20.0}
{'train_runtime': 11763.0766, 'train_samples_per_second': 3.419, 'train_steps_per_second': 3.419, 'train_loss': 0.3118171619811739, 'epoch': 20.0}
100%|███████████████████████| 40220/40220 [3:16:03<00:00,  3.42it/s]
Training done at: 2024-01-05 11:52:24.105524
```

```text
{
  "best_metric": 0.15022191405296326,
  "best_model_checkpoint": "./Lora-Meta-Llama2-7b-chat-hf-QandA_2g_v01-v04\\checkpoint-40220",
  "epoch": 20.0,
  "eval_steps": 500,
  "global_step": 40220,
  "is_hyper_param_search": false,
  "is_local_process_zero": true,
  "is_world_process_zero": true,
  "log_history": [
    {
      "epoch": 1.0,
      "learning_rate": 4.75e-05,
      "loss": 0.8428,
      "step": 2011
    },
    {
      "epoch": 1.0,
      "eval_loss": 0.7344576716423035,
      "eval_runtime": 40.2922,
      "eval_samples_per_second": 10.002,
      "eval_steps_per_second": 1.266,
      "step": 2011
    },
    {
      "epoch": 2.0,
      "learning_rate": 4.5e-05,
      "loss": 0.6337,
      "step": 4022
    },
    {
      "epoch": 2.0,
      "eval_loss": 0.635863184928894,
      "eval_runtime": 40.4216,
      "eval_samples_per_second": 9.97,
      "eval_steps_per_second": 1.262,
      "step": 4022
    },
    {
      "epoch": 3.0,
      "learning_rate": 4.25e-05,
      "loss": 0.5509,
      "step": 6033
    },
    {
      "epoch": 3.0,
      "eval_loss": 0.555050790309906,
      "eval_runtime": 40.1906,
      "eval_samples_per_second": 10.027,
      "eval_steps_per_second": 1.269,
      "step": 6033
    },
    {
      "epoch": 4.0,
      "learning_rate": 4e-05,
      "loss": 0.4837,
      "step": 8044
    },
    {
      "epoch": 4.0,
      "eval_loss": 0.5025414228439331,
      "eval_runtime": 40.062,
      "eval_samples_per_second": 10.059,
      "eval_steps_per_second": 1.273,
      "step": 8044
    },
    {
      "epoch": 5.0,
      "learning_rate": 3.7500000000000003e-05,
      "loss": 0.4289,
      "step": 10055
    },
    {
      "epoch": 5.0,
      "eval_loss": 0.4429796040058136,
      "eval_runtime": 40.1244,
      "eval_samples_per_second": 10.044,
      "eval_steps_per_second": 1.271,
      "step": 10055
    },
    {
      "epoch": 6.0,
      "learning_rate": 3.5e-05,
      "loss": 0.3825,
      "step": 12066
    },
    {
      "epoch": 6.0,
      "eval_loss": 0.40111732482910156,
      "eval_runtime": 40.0455,
      "eval_samples_per_second": 10.064,
      "eval_steps_per_second": 1.274,
      "step": 12066
    },
    {
      "epoch": 7.0,
      "learning_rate": 3.2500000000000004e-05,
      "loss": 0.3447,
      "step": 14077
    },
    {
      "epoch": 7.0,
      "eval_loss": 0.3600844144821167,
      "eval_runtime": 40.0694,
      "eval_samples_per_second": 10.058,
      "eval_steps_per_second": 1.273,
      "step": 14077
    },
    {
      "epoch": 8.0,
      "learning_rate": 3e-05,
      "loss": 0.3121,
      "step": 16088
    },
    {
      "epoch": 8.0,
      "eval_loss": 0.3315456211566925,
      "eval_runtime": 39.9981,
      "eval_samples_per_second": 10.075,
      "eval_steps_per_second": 1.275,
      "step": 16088
    },
    {
      "epoch": 9.0,
      "learning_rate": 2.7500000000000004e-05,
      "loss": 0.2843,
      "step": 18099
    },
    {
      "epoch": 9.0,
      "eval_loss": 0.3017001450061798,
      "eval_runtime": 40.0311,
      "eval_samples_per_second": 10.067,
      "eval_steps_per_second": 1.274,
      "step": 18099
    },
    {
      "epoch": 10.0,
      "learning_rate": 2.5e-05,
      "loss": 0.2591,
      "step": 20110
    },
    {
      "epoch": 10.0,
      "eval_loss": 0.27300581336021423,
      "eval_runtime": 40.0794,
      "eval_samples_per_second": 10.055,
      "eval_steps_per_second": 1.272,
      "step": 20110
    },
    {
      "epoch": 11.0,
      "learning_rate": 2.25e-05,
      "loss": 0.2372,
      "step": 22121
    },
    {
      "epoch": 11.0,
      "eval_loss": 0.2534460127353668,
      "eval_runtime": 40.0164,
      "eval_samples_per_second": 10.071,
      "eval_steps_per_second": 1.274,
      "step": 22121
    },
    {
      "epoch": 12.0,
      "learning_rate": 2e-05,
      "loss": 0.2177,
      "step": 24132
    },
    {
      "epoch": 12.0,
      "eval_loss": 0.23217065632343292,
      "eval_runtime": 40.0506,
      "eval_samples_per_second": 10.062,
      "eval_steps_per_second": 1.273,
      "step": 24132
    },
    {
      "epoch": 13.0,
      "learning_rate": 1.75e-05,
      "loss": 0.1998,
      "step": 26143
    },
    {
      "epoch": 13.0,
      "eval_loss": 0.215168759226799,
      "eval_runtime": 40.0158,
      "eval_samples_per_second": 10.071,
      "eval_steps_per_second": 1.274,
      "step": 26143
    },
    {
      "epoch": 14.0,
      "learning_rate": 1.5e-05,
      "loss": 0.1841,
      "step": 28154
    },
    {
      "epoch": 14.0,
      "eval_loss": 0.19979538023471832,
      "eval_runtime": 40.0119,
      "eval_samples_per_second": 10.072,
      "eval_steps_per_second": 1.275,
      "step": 28154
    },
    {
      "epoch": 15.0,
      "learning_rate": 1.25e-05,
      "loss": 0.1704,
      "step": 30165
    },
    {
      "epoch": 15.0,
      "eval_loss": 0.18658295273780823,
      "eval_runtime": 40.0046,
      "eval_samples_per_second": 10.074,
      "eval_steps_per_second": 1.275,
      "step": 30165
    },
    {
      "epoch": 16.0,
      "learning_rate": 1e-05,
      "loss": 0.1588,
      "step": 32176
    },
    {
      "epoch": 16.0,
      "eval_loss": 0.17462818324565887,
      "eval_runtime": 40.0819,
      "eval_samples_per_second": 10.054,
      "eval_steps_per_second": 1.272,
      "step": 32176
    },
    {
      "epoch": 17.0,
      "learning_rate": 7.5e-06,
      "loss": 0.1481,
      "step": 34187
    },
    {
      "epoch": 17.0,
      "eval_loss": 0.1658513993024826,
      "eval_runtime": 40.0379,
      "eval_samples_per_second": 10.065,
      "eval_steps_per_second": 1.274,
      "step": 34187
    },
    {
      "epoch": 18.0,
      "learning_rate": 5e-06,
      "loss": 0.1393,
      "step": 36198
    },
    {
      "epoch": 18.0,
      "eval_loss": 0.1572066694498062,
      "eval_runtime": 40.0376,
      "eval_samples_per_second": 10.066,
      "eval_steps_per_second": 1.274,
      "step": 36198
    },
    {
      "epoch": 19.0,
      "learning_rate": 2.5e-06,
      "loss": 0.132,
      "step": 38209
    },
    {
      "epoch": 19.0,
      "eval_loss": 0.15208856761455536,
      "eval_runtime": 40.0185,
      "eval_samples_per_second": 10.07,
      "eval_steps_per_second": 1.274,
      "step": 38209
    },
    {
      "epoch": 20.0,
      "learning_rate": 0.0,
      "loss": 0.1262,
      "step": 40220
    },
    {
      "epoch": 20.0,
      "eval_loss": 0.15022191405296326,
      "eval_runtime": 40.0854,
      "eval_samples_per_second": 10.054,
      "eval_steps_per_second": 1.272,
      "step": 40220
    }
  ],
  "logging_steps": 500,
  "max_steps": 40220,
  "num_train_epochs": 20,
  "save_steps": 500,
  "total_flos": 2.344189529530368e+17,
  "trial_name": null,
  "trial_params": null
}
```

## Model Details

### Model Description

<!-- Provide a longer summary of what this model is. -->



- **Developed by:** [More Information Needed]
- **Funded by [optional]:** [More Information Needed]
- **Shared by [optional]:** [More Information Needed]
- **Model type:** [More Information Needed]
- **Language(s) (NLP):** [More Information Needed]
- **License:** [More Information Needed]
- **Finetuned from model [optional]:** [More Information Needed]

### Model Sources [optional]

<!-- Provide the basic links for the model. -->

- **Repository:** [More Information Needed]
- **Paper [optional]:** [More Information Needed]
- **Demo [optional]:** [More Information Needed]

## Uses

<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->

### Direct Use

<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->

[More Information Needed]

### Downstream Use [optional]

<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->

[More Information Needed]

### Out-of-Scope Use

<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->

[More Information Needed]

## Bias, Risks, and Limitations

<!-- This section is meant to convey both technical and sociotechnical limitations. -->

[More Information Needed]

### Recommendations

<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->

Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.

## How to Get Started with the Model

Use the code below to get started with the model.

[More Information Needed]

## Training Details

### Training Data

<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->

[More Information Needed]

### Training Procedure 

<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->

#### Preprocessing [optional]

[More Information Needed]


#### Training Hyperparameters

- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->

#### Speeds, Sizes, Times [optional]

<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->

[More Information Needed]

## Evaluation

<!-- This section describes the evaluation protocols and provides the results. -->

### Testing Data, Factors & Metrics

#### Testing Data

<!-- This should link to a Dataset Card if possible. -->

[More Information Needed]

#### Factors

<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->

[More Information Needed]

#### Metrics

<!-- These are the evaluation metrics being used, ideally with a description of why. -->

[More Information Needed]

### Results

[More Information Needed]

#### Summary



## Model Examination [optional]

<!-- Relevant interpretability work for the model goes here -->

[More Information Needed]

## Environmental Impact

<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->

Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).

- **Hardware Type:** [More Information Needed]
- **Hours used:** [More Information Needed]
- **Cloud Provider:** [More Information Needed]
- **Compute Region:** [More Information Needed]
- **Carbon Emitted:** [More Information Needed]

## Technical Specifications [optional]

### Model Architecture and Objective

[More Information Needed]

### Compute Infrastructure

[More Information Needed]

#### Hardware

[More Information Needed]

#### Software

[More Information Needed]

## Citation [optional]

<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->

**BibTeX:**

[More Information Needed]

**APA:**

[More Information Needed]

## Glossary [optional]

<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->

[More Information Needed]

## More Information [optional]

[More Information Needed]

## Model Card Authors [optional]

[More Information Needed]

## Model Card Contact

[More Information Needed]


## Training procedure


The following `bitsandbytes` quantization config was used during training:
- quant_method: bitsandbytes
- load_in_8bit: False
- load_in_4bit: True
- llm_int8_threshold: 6.0
- llm_int8_skip_modules: None
- llm_int8_enable_fp32_cpu_offload: False
- llm_int8_has_fp16_weight: False
- bnb_4bit_quant_type: nf4
- bnb_4bit_use_double_quant: True
- bnb_4bit_compute_dtype: float16

### Framework versions


- PEFT 0.6.2
