peft safetensors arxiv:1910.09700 base_model:meta-llama/Llama-2-7b-hf region:us
---
library_name: peft
base_model: meta-llama/Llama-2-7b-hf
---

# Model Card for Model ID

<!-- Provide a quick summary of what the model is/does. -->

```text
DATASET_FILE_NAME = 'QandA_2g_v01.json'
adapter_name = "Lora-Meta-Llama2-7b-hf-QandA_2g_v01-v03"

Full dataset size: 2011
Eval dataset size: 403
Epochs: 20
Steps: 40220 
Total training time: 3.26 horas

Start training at: 2024-01-04 14:34:08.062743
0%|                                                                                        | 0/40220 [00:00<?, ?it/s]
{'loss': 0.7906, 'learning_rate': 4.75e-05, 'epoch': 1.0}
{'eval_loss': 0.7224752902984619, 'eval_runtime': 41.5025, 'eval_samples_per_second': 9.71, 'eval_steps_per_second': 1.229, 'epoch': 1.0}
{'loss': 0.6145, 'learning_rate': 4.5e-05, 'epoch': 2.0}
{'eval_loss': 0.6293410062789917, 'eval_runtime': 41.4281, 'eval_samples_per_second': 9.728, 'eval_steps_per_second': 1.231, 'epoch': 2.0}
{'loss': 0.5393, 'learning_rate': 4.25e-05, 'epoch': 3.0}
{'eval_loss': 0.5556005239486694, 'eval_runtime': 41.4426, 'eval_samples_per_second': 9.724, 'eval_steps_per_second': 1.231, 'epoch': 3.0}
{'loss': 0.4774, 'learning_rate': 4e-05, 'epoch': 4.0}
{'eval_loss': 0.5019810199737549, 'eval_runtime': 41.4936, 'eval_samples_per_second': 9.712, 'eval_steps_per_second': 1.229, 'epoch': 4.0}
{'loss': 0.4244, 'learning_rate': 3.7500000000000003e-05, 'epoch': 5.0}
{'eval_loss': 0.4469846785068512, 'eval_runtime': 41.5516, 'eval_samples_per_second': 9.699, 'eval_steps_per_second': 1.227, 'epoch': 5.0}
{'loss': 0.3797, 'learning_rate': 3.5e-05, 'epoch': 6.0}
{'eval_loss': 0.4054427742958069, 'eval_runtime': 41.445, 'eval_samples_per_second': 9.724, 'eval_steps_per_second': 1.231, 'epoch': 6.0}
{'loss': 0.3424, 'learning_rate': 3.2500000000000004e-05, 'epoch': 7.0}
{'eval_loss': 0.3668522238731384, 'eval_runtime': 41.438, 'eval_samples_per_second': 9.725, 'eval_steps_per_second': 1.231, 'epoch': 7.0}
{'loss': 0.3111, 'learning_rate': 3e-05, 'epoch': 8.0}
{'eval_loss': 0.34070536494255066, 'eval_runtime': 41.4578, 'eval_samples_per_second': 9.721, 'eval_steps_per_second': 1.23, 'epoch': 8.0}
{'loss': 0.2837, 'learning_rate': 2.7500000000000004e-05, 'epoch': 9.0}
{'eval_loss': 0.3126145303249359, 'eval_runtime': 41.4312, 'eval_samples_per_second': 9.727, 'eval_steps_per_second': 1.231, 'epoch': 9.0}
{'loss': 0.2599, 'learning_rate': 2.5e-05, 'epoch': 10.0}
{'eval_loss': 0.28896018862724304, 'eval_runtime': 41.4486, 'eval_samples_per_second': 9.723, 'eval_steps_per_second': 1.23, 'epoch': 10.0}
{'loss': 0.2393, 'learning_rate': 2.25e-05, 'epoch': 11.0}
{'eval_loss': 0.26615414023399353, 'eval_runtime': 41.4467, 'eval_samples_per_second': 9.723, 'eval_steps_per_second': 1.23, 'epoch': 11.0}
{'loss': 0.2203, 'learning_rate': 2e-05, 'epoch': 12.0}
{'eval_loss': 0.24929480254650116, 'eval_runtime': 41.4549, 'eval_samples_per_second': 9.721, 'eval_steps_per_second': 1.23, 'epoch': 12.0}
{'loss': 0.2031, 'learning_rate': 1.75e-05, 'epoch': 13.0}
{'eval_loss': 0.23146237432956696, 'eval_runtime': 41.4637, 'eval_samples_per_second': 9.719, 'eval_steps_per_second': 1.23, 'epoch': 13.0}
{'loss': 0.1879, 'learning_rate': 1.5e-05, 'epoch': 14.0}
{'eval_loss': 0.216956228017807, 'eval_runtime': 41.4526, 'eval_samples_per_second': 9.722, 'eval_steps_per_second': 1.23, 'epoch': 14.0}
{'loss': 0.1742, 'learning_rate': 1.25e-05, 'epoch': 15.0}
{'eval_loss': 0.20527032017707825, 'eval_runtime': 41.4392, 'eval_samples_per_second': 9.725, 'eval_steps_per_second': 1.231, 'epoch': 15.0}
{'loss': 0.1624, 'learning_rate': 1e-05, 'epoch': 16.0}
{'eval_loss': 0.19351345300674438, 'eval_runtime': 41.402, 'eval_samples_per_second': 9.734, 'eval_steps_per_second': 1.232, 'epoch': 16.0}
{'loss': 0.1517, 'learning_rate': 7.5e-06, 'epoch': 17.0}
{'eval_loss': 0.1841140240430832, 'eval_runtime': 41.3718, 'eval_samples_per_second': 9.741, 'eval_steps_per_second': 1.233, 'epoch': 17.0}
{'loss': 0.1428, 'learning_rate': 5e-06, 'epoch': 18.0}
{'eval_loss': 0.17686299979686737, 'eval_runtime': 41.4581, 'eval_samples_per_second': 9.721, 'eval_steps_per_second': 1.23, 'epoch': 18.0}
{'loss': 0.1356, 'learning_rate': 2.5e-06, 'epoch': 19.0}
{'eval_loss': 0.17180992662906647, 'eval_runtime': 41.4578, 'eval_samples_per_second': 9.721, 'eval_steps_per_second': 1.23, 'epoch': 19.0}
{'loss': 0.1301, 'learning_rate': 0.0, 'epoch': 20.0}
{'eval_loss': 0.16986818611621857, 'eval_runtime': 41.4657, 'eval_samples_per_second': 9.719, 'eval_steps_per_second': 1.23, 'epoch': 20.0}
{'train_runtime': 11767.2453, 'train_samples_per_second': 3.418, 'train_steps_per_second': 3.418, 'train_loss': 0.30853292274569943, 'epoch': 20.0}
100%|██████████████████████████████████████████████████████████████████████████| 40220/40220 [3:16:07<00:00,  3.42it/s]
Training done at: 2024-01-04 17:50:15.380360


{
  "best_metric": 0.16986818611621857,
  "best_model_checkpoint": "./Lora-Meta-Llama2-7b-hf-QandA_2g_v01-v03\\checkpoint-40220",
  "epoch": 20.0,
  "eval_steps": 500,
  "global_step": 40220,
  "is_hyper_param_search": false,
  "is_local_process_zero": true,
  "is_world_process_zero": true,
  "log_history": [
    {
      "epoch": 1.0,
      "learning_rate": 4.75e-05,
      "loss": 0.7906,
      "step": 2011
    },
    {
      "epoch": 1.0,
      "eval_loss": 0.7224752902984619,
      "eval_runtime": 41.5025,
      "eval_samples_per_second": 9.71,
      "eval_steps_per_second": 1.229,
      "step": 2011
    },
    {
      "epoch": 2.0,
      "learning_rate": 4.5e-05,
      "loss": 0.6145,
      "step": 4022
    },
    {
      "epoch": 2.0,
      "eval_loss": 0.6293410062789917,
      "eval_runtime": 41.4281,
      "eval_samples_per_second": 9.728,
      "eval_steps_per_second": 1.231,
      "step": 4022
    },
    {
      "epoch": 3.0,
      "learning_rate": 4.25e-05,
      "loss": 0.5393,
      "step": 6033
    },
    {
      "epoch": 3.0,
      "eval_loss": 0.5556005239486694,
      "eval_runtime": 41.4426,
      "eval_samples_per_second": 9.724,
      "eval_steps_per_second": 1.231,
      "step": 6033
    },
    {
      "epoch": 4.0,
      "learning_rate": 4e-05,
      "loss": 0.4774,
      "step": 8044
    },
    {
      "epoch": 4.0,
      "eval_loss": 0.5019810199737549,
      "eval_runtime": 41.4936,
      "eval_samples_per_second": 9.712,
      "eval_steps_per_second": 1.229,
      "step": 8044
    },
    {
      "epoch": 5.0,
      "learning_rate": 3.7500000000000003e-05,
      "loss": 0.4244,
      "step": 10055
    },
    {
      "epoch": 5.0,
      "eval_loss": 0.4469846785068512,
      "eval_runtime": 41.5516,
      "eval_samples_per_second": 9.699,
      "eval_steps_per_second": 1.227,
      "step": 10055
    },
    {
      "epoch": 6.0,
      "learning_rate": 3.5e-05,
      "loss": 0.3797,
      "step": 12066
    },
    {
      "epoch": 6.0,
      "eval_loss": 0.4054427742958069,
      "eval_runtime": 41.445,
      "eval_samples_per_second": 9.724,
      "eval_steps_per_second": 1.231,
      "step": 12066
    },
    {
      "epoch": 7.0,
      "learning_rate": 3.2500000000000004e-05,
      "loss": 0.3424,
      "step": 14077
    },
    {
      "epoch": 7.0,
      "eval_loss": 0.3668522238731384,
      "eval_runtime": 41.438,
      "eval_samples_per_second": 9.725,
      "eval_steps_per_second": 1.231,
      "step": 14077
    },
    {
      "epoch": 8.0,
      "learning_rate": 3e-05,
      "loss": 0.3111,
      "step": 16088
    },
    {
      "epoch": 8.0,
      "eval_loss": 0.34070536494255066,
      "eval_runtime": 41.4578,
      "eval_samples_per_second": 9.721,
      "eval_steps_per_second": 1.23,
      "step": 16088
    },
    {
      "epoch": 9.0,
      "learning_rate": 2.7500000000000004e-05,
      "loss": 0.2837,
      "step": 18099
    },
    {
      "epoch": 9.0,
      "eval_loss": 0.3126145303249359,
      "eval_runtime": 41.4312,
      "eval_samples_per_second": 9.727,
      "eval_steps_per_second": 1.231,
      "step": 18099
    },
    {
      "epoch": 10.0,
      "learning_rate": 2.5e-05,
      "loss": 0.2599,
      "step": 20110
    },
    {
      "epoch": 10.0,
      "eval_loss": 0.28896018862724304,
      "eval_runtime": 41.4486,
      "eval_samples_per_second": 9.723,
      "eval_steps_per_second": 1.23,
      "step": 20110
    },
    {
      "epoch": 11.0,
      "learning_rate": 2.25e-05,
      "loss": 0.2393,
      "step": 22121
    },
    {
      "epoch": 11.0,
      "eval_loss": 0.26615414023399353,
      "eval_runtime": 41.4467,
      "eval_samples_per_second": 9.723,
      "eval_steps_per_second": 1.23,
      "step": 22121
    },
    {
      "epoch": 12.0,
      "learning_rate": 2e-05,
      "loss": 0.2203,
      "step": 24132
    },
    {
      "epoch": 12.0,
      "eval_loss": 0.24929480254650116,
      "eval_runtime": 41.4549,
      "eval_samples_per_second": 9.721,
      "eval_steps_per_second": 1.23,
      "step": 24132
    },
    {
      "epoch": 13.0,
      "learning_rate": 1.75e-05,
      "loss": 0.2031,
      "step": 26143
    },
    {
      "epoch": 13.0,
      "eval_loss": 0.23146237432956696,
      "eval_runtime": 41.4637,
      "eval_samples_per_second": 9.719,
      "eval_steps_per_second": 1.23,
      "step": 26143
    },
    {
      "epoch": 14.0,
      "learning_rate": 1.5e-05,
      "loss": 0.1879,
      "step": 28154
    },
    {
      "epoch": 14.0,
      "eval_loss": 0.216956228017807,
      "eval_runtime": 41.4526,
      "eval_samples_per_second": 9.722,
      "eval_steps_per_second": 1.23,
      "step": 28154
    },
    {
      "epoch": 15.0,
      "learning_rate": 1.25e-05,
      "loss": 0.1742,
      "step": 30165
    },
    {
      "epoch": 15.0,
      "eval_loss": 0.20527032017707825,
      "eval_runtime": 41.4392,
      "eval_samples_per_second": 9.725,
      "eval_steps_per_second": 1.231,
      "step": 30165
    },
    {
      "epoch": 16.0,
      "learning_rate": 1e-05,
      "loss": 0.1624,
      "step": 32176
    },
    {
      "epoch": 16.0,
      "eval_loss": 0.19351345300674438,
      "eval_runtime": 41.402,
      "eval_samples_per_second": 9.734,
      "eval_steps_per_second": 1.232,
      "step": 32176
    },
    {
      "epoch": 17.0,
      "learning_rate": 7.5e-06,
      "loss": 0.1517,
      "step": 34187
    },
    {
      "epoch": 17.0,
      "eval_loss": 0.1841140240430832,
      "eval_runtime": 41.3718,
      "eval_samples_per_second": 9.741,
      "eval_steps_per_second": 1.233,
      "step": 34187
    },
    {
      "epoch": 18.0,
      "learning_rate": 5e-06,
      "loss": 0.1428,
      "step": 36198
    },
    {
      "epoch": 18.0,
      "eval_loss": 0.17686299979686737,
      "eval_runtime": 41.4581,
      "eval_samples_per_second": 9.721,
      "eval_steps_per_second": 1.23,
      "step": 36198
    },
    {
      "epoch": 19.0,
      "learning_rate": 2.5e-06,
      "loss": 0.1356,
      "step": 38209
    },
    {
      "epoch": 19.0,
      "eval_loss": 0.17180992662906647,
      "eval_runtime": 41.4578,
      "eval_samples_per_second": 9.721,
      "eval_steps_per_second": 1.23,
      "step": 38209
    },
    {
      "epoch": 20.0,
      "learning_rate": 0.0,
      "loss": 0.1301,
      "step": 40220
    },
    {
      "epoch": 20.0,
      "eval_loss": 0.16986818611621857,
      "eval_runtime": 41.4657,
      "eval_samples_per_second": 9.719,
      "eval_steps_per_second": 1.23,
      "step": 40220
    }
  ],
  "logging_steps": 500,
  "max_steps": 40220,
  "num_train_epochs": 20,
  "save_steps": 500,
  "total_flos": 2.344189529530368e+17,
  "trial_name": null,
  "trial_params": null
}
```

## Model Details

### Model Description

<!-- Provide a longer summary of what this model is. -->



- **Developed by:** [More Information Needed]
- **Funded by [optional]:** [More Information Needed]
- **Shared by [optional]:** [More Information Needed]
- **Model type:** [More Information Needed]
- **Language(s) (NLP):** [More Information Needed]
- **License:** [More Information Needed]
- **Finetuned from model [optional]:** [More Information Needed]

### Model Sources [optional]

<!-- Provide the basic links for the model. -->

- **Repository:** [More Information Needed]
- **Paper [optional]:** [More Information Needed]
- **Demo [optional]:** [More Information Needed]

## Uses

<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->

### Direct Use

<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->

[More Information Needed]

### Downstream Use [optional]

<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->

[More Information Needed]

### Out-of-Scope Use

<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->

[More Information Needed]

## Bias, Risks, and Limitations

<!-- This section is meant to convey both technical and sociotechnical limitations. -->

[More Information Needed]

### Recommendations

<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->

Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.

## How to Get Started with the Model

Use the code below to get started with the model.

[More Information Needed]

## Training Details

### Training Data

<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->

[More Information Needed]

### Training Procedure 

<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->

#### Preprocessing [optional]

[More Information Needed]


#### Training Hyperparameters

- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->

#### Speeds, Sizes, Times [optional]

<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->

[More Information Needed]

## Evaluation

<!-- This section describes the evaluation protocols and provides the results. -->

### Testing Data, Factors & Metrics

#### Testing Data

<!-- This should link to a Dataset Card if possible. -->

[More Information Needed]

#### Factors

<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->

[More Information Needed]

#### Metrics

<!-- These are the evaluation metrics being used, ideally with a description of why. -->

[More Information Needed]

### Results

[More Information Needed]

#### Summary



## Model Examination [optional]

<!-- Relevant interpretability work for the model goes here -->

[More Information Needed]

## Environmental Impact

<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->

Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).

- **Hardware Type:** [More Information Needed]
- **Hours used:** [More Information Needed]
- **Cloud Provider:** [More Information Needed]
- **Compute Region:** [More Information Needed]
- **Carbon Emitted:** [More Information Needed]

## Technical Specifications [optional]

### Model Architecture and Objective

[More Information Needed]

### Compute Infrastructure

[More Information Needed]

#### Hardware

[More Information Needed]

#### Software

[More Information Needed]

## Citation [optional]

<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->

**BibTeX:**

[More Information Needed]

**APA:**

[More Information Needed]

## Glossary [optional]

<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->

[More Information Needed]

## More Information [optional]

[More Information Needed]

## Model Card Authors [optional]

[More Information Needed]

## Model Card Contact

[More Information Needed]


## Training procedure


The following `bitsandbytes` quantization config was used during training:
- quant_method: bitsandbytes
- load_in_8bit: False
- load_in_4bit: True
- llm_int8_threshold: 6.0
- llm_int8_skip_modules: None
- llm_int8_enable_fp32_cpu_offload: False
- llm_int8_has_fp16_weight: False
- bnb_4bit_quant_type: nf4
- bnb_4bit_use_double_quant: True
- bnb_4bit_compute_dtype: float16

### Framework versions


- PEFT 0.6.2
## Training procedure


The following `bitsandbytes` quantization config was used during training:
- quant_method: bitsandbytes
- load_in_8bit: False
- load_in_4bit: True
- llm_int8_threshold: 6.0
- llm_int8_skip_modules: None
- llm_int8_enable_fp32_cpu_offload: False
- llm_int8_has_fp16_weight: False
- bnb_4bit_quant_type: nf4
- bnb_4bit_use_double_quant: True
- bnb_4bit_compute_dtype: float16

### Framework versions


- PEFT 0.6.2
## Training procedure


The following `bitsandbytes` quantization config was used during training:
- quant_method: bitsandbytes
- load_in_8bit: False
- load_in_4bit: True
- llm_int8_threshold: 6.0
- llm_int8_skip_modules: None
- llm_int8_enable_fp32_cpu_offload: False
- llm_int8_has_fp16_weight: False
- bnb_4bit_quant_type: nf4
- bnb_4bit_use_double_quant: True
- bnb_4bit_compute_dtype: float16

### Framework versions


- PEFT 0.6.2
