peft safetensors arxiv:1910.09700 base_model:meta-llama/Llama-2-7b-hf region:us
---
library_name: peft
base_model: meta-llama/Llama-2-7b-hf
---

# Model Card for Model ID

<!-- Provide a quick summary of what the model is/does. -->

```text
Start training at: 2024-01-08 08:10:15.228841
0%|                | 0/80440 [00:00<?, ?it/s]You're using a LlamaTokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.
{'loss': 0.7902, 'learning_rate': 4.875e-05, 'epoch': 1.0}
{'eval_loss': 0.7449761629104614, 'eval_runtime': 41.5791, 'eval_samples_per_second': 9.692, 'eval_steps_per_second': 1.227, 'epoch': 1.0}
{'loss': 0.6126, 'learning_rate': 4.75e-05, 'epoch': 2.0}
{'eval_loss': 0.6517715454101562, 'eval_runtime': 41.9682, 'eval_samples_per_second': 9.603, 'eval_steps_per_second': 1.215, 'epoch': 2.0}
{'loss': 0.5359, 'learning_rate': 4.6250000000000006e-05, 'epoch': 3.0}
{'eval_loss': 0.5711696147918701, 'eval_runtime': 41.7642, 'eval_samples_per_second': 9.649, 'eval_steps_per_second': 1.221, 'epoch': 3.0}
{'loss': 0.4728, 'learning_rate': 4.5e-05, 'epoch': 4.0}
{'eval_loss': 0.5220186114311218, 'eval_runtime': 41.7716, 'eval_samples_per_second': 9.648, 'eval_steps_per_second': 1.221, 'epoch': 4.0}
{'loss': 0.4196, 'learning_rate': 4.375e-05, 'epoch': 5.0}
{'eval_loss': 0.4640021324157715, 'eval_runtime': 41.7921, 'eval_samples_per_second': 9.643, 'eval_steps_per_second': 1.22, 'epoch': 5.0}
{'loss': 0.3746, 'learning_rate': 4.25e-05, 'epoch': 6.0}
{'eval_loss': 0.41543692350387573, 'eval_runtime': 41.9208, 'eval_samples_per_second': 9.613, 'eval_steps_per_second': 1.217, 'epoch': 6.0}
{'loss': 0.3376, 'learning_rate': 4.125e-05, 'epoch': 7.0}
{'eval_loss': 0.3779241740703583, 'eval_runtime': 41.7454, 'eval_samples_per_second': 9.654, 'eval_steps_per_second': 1.222, 'epoch': 7.0}
{'loss': 0.3064, 'learning_rate': 4e-05, 'epoch': 8.0}
{'eval_loss': 0.34859710931777954, 'eval_runtime': 42.0013, 'eval_samples_per_second': 9.595, 'eval_steps_per_second': 1.214, 'epoch': 8.0}
{'loss': 0.2792, 'learning_rate': 3.875e-05, 'epoch': 9.0}
{'eval_loss': 0.3179122507572174, 'eval_runtime': 41.657, 'eval_samples_per_second': 9.674, 'eval_steps_per_second': 1.224, 'epoch': 9.0}
{'loss': 0.2544, 'learning_rate': 3.7500000000000003e-05, 'epoch': 10.0}
{'eval_loss': 0.29244551062583923, 'eval_runtime': 41.628, 'eval_samples_per_second': 9.681, 'eval_steps_per_second': 1.225, 'epoch': 10.0}
{'loss': 0.2334, 'learning_rate': 3.625e-05, 'epoch': 11.0}
{'eval_loss': 0.27090445160865784, 'eval_runtime': 41.6294, 'eval_samples_per_second': 9.681, 'eval_steps_per_second': 1.225, 'epoch': 11.0}
{'loss': 0.2139, 'learning_rate': 3.5e-05, 'epoch': 12.0}
{'eval_loss': 0.2475050389766693, 'eval_runtime': 41.722, 'eval_samples_per_second': 9.659, 'eval_steps_per_second': 1.222, 'epoch': 12.0}
{'loss': 0.1964, 'learning_rate': 3.375000000000001e-05, 'epoch': 13.0}
{'eval_loss': 0.22930796444416046, 'eval_runtime': 41.7015, 'eval_samples_per_second': 9.664, 'eval_steps_per_second': 1.223, 'epoch': 13.0}
{'loss': 0.1807, 'learning_rate': 3.2500000000000004e-05, 'epoch': 14.0}
{'eval_loss': 0.21222631633281708, 'eval_runtime': 41.7061, 'eval_samples_per_second': 9.663, 'eval_steps_per_second': 1.223, 'epoch': 14.0}
{'loss': 0.1671, 'learning_rate': 3.125e-05, 'epoch': 15.0}
{'eval_loss': 0.19732101261615753, 'eval_runtime': 41.7192, 'eval_samples_per_second': 9.66, 'eval_steps_per_second': 1.222, 'epoch': 15.0}
{'loss': 0.155, 'learning_rate': 3e-05, 'epoch': 16.0}
{'eval_loss': 0.18540407717227936, 'eval_runtime': 41.6288, 'eval_samples_per_second': 9.681, 'eval_steps_per_second': 1.225, 'epoch': 16.0}
{'loss': 0.144, 'learning_rate': 2.8749999999999997e-05, 'epoch': 17.0}
{'eval_loss': 0.1721782088279724, 'eval_runtime': 41.7432, 'eval_samples_per_second': 9.654, 'eval_steps_per_second': 1.222, 'epoch': 17.0}
{'loss': 0.1348, 'learning_rate': 2.7500000000000004e-05, 'epoch': 18.0}
{'eval_loss': 0.15782776474952698, 'eval_runtime': 41.6345, 'eval_samples_per_second': 9.679, 'eval_steps_per_second': 1.225, 'epoch': 18.0}
{'loss': 0.1258, 'learning_rate': 2.625e-05, 'epoch': 19.0}
{'eval_loss': 0.1482909917831421, 'eval_runtime': 41.6668, 'eval_samples_per_second': 9.672, 'eval_steps_per_second': 1.224, 'epoch': 19.0}
{'loss': 0.1178, 'learning_rate': 2.5e-05, 'epoch': 20.0}
{'eval_loss': 0.13850367069244385, 'eval_runtime': 41.7142, 'eval_samples_per_second': 9.661, 'eval_steps_per_second': 1.223, 'epoch': 20.0}
{'loss': 0.1105, 'learning_rate': 2.375e-05, 'epoch': 21.0}
{'eval_loss': 0.12867872416973114, 'eval_runtime': 41.6334, 'eval_samples_per_second': 9.68, 'eval_steps_per_second': 1.225, 'epoch': 21.0}
{'loss': 0.1049, 'learning_rate': 2.25e-05, 'epoch': 22.0}
{'eval_loss': 0.12203275412321091, 'eval_runtime': 41.7291, 'eval_samples_per_second': 9.658, 'eval_steps_per_second': 1.222, 'epoch': 22.0}
{'loss': 0.0998, 'learning_rate': 2.125e-05, 'epoch': 23.0}
{'eval_loss': 0.11384650319814682, 'eval_runtime': 41.7428, 'eval_samples_per_second': 9.654, 'eval_steps_per_second': 1.222, 'epoch': 23.0}
{'loss': 0.0946, 'learning_rate': 2e-05, 'epoch': 24.0}
{'eval_loss': 0.10876920819282532, 'eval_runtime': 41.7306, 'eval_samples_per_second': 9.657, 'eval_steps_per_second': 1.222, 'epoch': 24.0}
{'loss': 0.0906, 'learning_rate': 1.8750000000000002e-05, 'epoch': 25.0}
{'eval_loss': 0.1036558523774147, 'eval_runtime': 41.6568, 'eval_samples_per_second': 9.674, 'eval_steps_per_second': 1.224, 'epoch': 25.0}
{'loss': 0.0866, 'learning_rate': 1.75e-05, 'epoch': 26.0}
{'eval_loss': 0.09655934572219849, 'eval_runtime': 41.6692, 'eval_samples_per_second': 9.671, 'eval_steps_per_second': 1.224, 'epoch': 26.0}
{'loss': 0.0835, 'learning_rate': 1.6250000000000002e-05, 'epoch': 27.0}
{'eval_loss': 0.09233643114566803, 'eval_runtime': 41.7504, 'eval_samples_per_second': 9.653, 'eval_steps_per_second': 1.222, 'epoch': 27.0}
{'loss': 0.0805, 'learning_rate': 1.5e-05, 'epoch': 28.0}
{'eval_loss': 0.08887409418821335, 'eval_runtime': 41.73, 'eval_samples_per_second': 9.657, 'eval_steps_per_second': 1.222, 'epoch': 28.0}
{'loss': 0.0779, 'learning_rate': 1.3750000000000002e-05, 'epoch': 29.0}
{'eval_loss': 0.08545327931642532, 'eval_runtime': 41.7924, 'eval_samples_per_second': 9.643, 'eval_steps_per_second': 1.22, 'epoch': 29.0}
{'loss': 0.0758, 'learning_rate': 1.25e-05, 'epoch': 30.0}
{'eval_loss': 0.08104873448610306, 'eval_runtime': 41.7832, 'eval_samples_per_second': 9.645, 'eval_steps_per_second': 1.221, 'epoch': 30.0}
{'loss': 0.0738, 'learning_rate': 1.125e-05, 'epoch': 31.0}
{'eval_loss': 0.07830855250358582, 'eval_runtime': 41.745, 'eval_samples_per_second': 9.654, 'eval_steps_per_second': 1.222, 'epoch': 31.0}
{'loss': 0.072, 'learning_rate': 1e-05, 'epoch': 32.0}
{'eval_loss': 0.07554436475038528, 'eval_runtime': 41.6983, 'eval_samples_per_second': 9.665, 'eval_steps_per_second': 1.223, 'epoch': 32.0}
{'loss': 0.0705, 'learning_rate': 8.75e-06, 'epoch': 33.0}
{'eval_loss': 0.07316704839468002, 'eval_runtime': 41.7684, 'eval_samples_per_second': 9.648, 'eval_steps_per_second': 1.221, 'epoch': 33.0}
{'loss': 0.0692, 'learning_rate': 7.5e-06, 'epoch': 34.0}
{'eval_loss': 0.07176368683576584, 'eval_runtime': 41.6547, 'eval_samples_per_second': 9.675, 'eval_steps_per_second': 1.224, 'epoch': 34.0}
{'loss': 0.068, 'learning_rate': 6.25e-06, 'epoch': 35.0}
{'eval_loss': 0.06996949017047882, 'eval_runtime': 41.7824, 'eval_samples_per_second': 9.645, 'eval_steps_per_second': 1.221, 'epoch': 35.0}
{'loss': 0.067, 'learning_rate': 5e-06, 'epoch': 36.0}
{'eval_loss': 0.06851284950971603, 'eval_runtime': 41.7116, 'eval_samples_per_second': 9.662, 'eval_steps_per_second': 1.223, 'epoch': 36.0}
{'loss': 0.066, 'learning_rate': 3.75e-06, 'epoch': 37.0}
{'eval_loss': 0.06739722192287445, 'eval_runtime': 41.7517, 'eval_samples_per_second': 9.652, 'eval_steps_per_second': 1.222, 'epoch': 37.0}
{'loss': 0.0651, 'learning_rate': 2.5e-06, 'epoch': 38.0}
{'eval_loss': 0.0664171576499939, 'eval_runtime': 41.7491, 'eval_samples_per_second': 9.653, 'eval_steps_per_second': 1.222, 'epoch': 38.0}
{'loss': 0.0644, 'learning_rate': 1.25e-06, 'epoch': 39.0}
{'eval_loss': 0.06586799770593643, 'eval_runtime': 41.7935, 'eval_samples_per_second': 9.643, 'eval_steps_per_second': 1.22, 'epoch': 39.0}
{'loss': 0.0637, 'learning_rate': 0.0, 'epoch': 40.0}
{'eval_loss': 0.06568873673677444, 'eval_runtime': 41.8305, 'eval_samples_per_second': 9.634, 'eval_steps_per_second': 1.219, 'epoch': 40.0}
{'train_runtime': 24034.2593, 'train_samples_per_second': 3.347, 'train_steps_per_second': 3.347, 'train_loss': 0.19091044956626493, 'epoch': 40.0}
100%|█████████████| 80440/80440 [6:40:34<00:00,  3.35it/s]
Training done at: 2024-01-08 14:50:49.566594
For Model: meta-llama/Llama-2-7b-hf
In Adapter: Lora-Meta-Llama2-7b-chat-hf-QandA_2g_v01-v05
```

```text
{
  "best_metric": 0.06568873673677444,
  "best_model_checkpoint": "./Lora-Meta-Llama2-7b-chat-hf-QandA_2g_v01-v05\\checkpoint-80440",
  "epoch": 40.0,
  "eval_steps": 500,
  "global_step": 80440,
  "is_hyper_param_search": false,
  "is_local_process_zero": true,
  "is_world_process_zero": true,
  "log_history": [
    {
      "epoch": 1.0,
      "learning_rate": 4.875e-05,
      "loss": 0.7902,
      "step": 2011
    },
    {
      "epoch": 1.0,
      "eval_loss": 0.7449761629104614,
      "eval_runtime": 41.5791,
      "eval_samples_per_second": 9.692,
      "eval_steps_per_second": 1.227,
      "step": 2011
    },
    {
      "epoch": 2.0,
      "learning_rate": 4.75e-05,
      "loss": 0.6126,
      "step": 4022
    },
    {
      "epoch": 2.0,
      "eval_loss": 0.6517715454101562,
      "eval_runtime": 41.9682,
      "eval_samples_per_second": 9.603,
      "eval_steps_per_second": 1.215,
      "step": 4022
    },
    {
      "epoch": 3.0,
      "learning_rate": 4.6250000000000006e-05,
      "loss": 0.5359,
      "step": 6033
    },
    {
      "epoch": 3.0,
      "eval_loss": 0.5711696147918701,
      "eval_runtime": 41.7642,
      "eval_samples_per_second": 9.649,
      "eval_steps_per_second": 1.221,
      "step": 6033
    },
    {
      "epoch": 4.0,
      "learning_rate": 4.5e-05,
      "loss": 0.4728,
      "step": 8044
    },
    {
      "epoch": 4.0,
      "eval_loss": 0.5220186114311218,
      "eval_runtime": 41.7716,
      "eval_samples_per_second": 9.648,
      "eval_steps_per_second": 1.221,
      "step": 8044
    },
    {
      "epoch": 5.0,
      "learning_rate": 4.375e-05,
      "loss": 0.4196,
      "step": 10055
    },
    {
      "epoch": 5.0,
      "eval_loss": 0.4640021324157715,
      "eval_runtime": 41.7921,
      "eval_samples_per_second": 9.643,
      "eval_steps_per_second": 1.22,
      "step": 10055
    },
    {
      "epoch": 6.0,
      "learning_rate": 4.25e-05,
      "loss": 0.3746,
      "step": 12066
    },
    {
      "epoch": 6.0,
      "eval_loss": 0.41543692350387573,
      "eval_runtime": 41.9208,
      "eval_samples_per_second": 9.613,
      "eval_steps_per_second": 1.217,
      "step": 12066
    },
    {
      "epoch": 7.0,
      "learning_rate": 4.125e-05,
      "loss": 0.3376,
      "step": 14077
    },
    {
      "epoch": 7.0,
      "eval_loss": 0.3779241740703583,
      "eval_runtime": 41.7454,
      "eval_samples_per_second": 9.654,
      "eval_steps_per_second": 1.222,
      "step": 14077
    },
    {
      "epoch": 8.0,
      "learning_rate": 4e-05,
      "loss": 0.3064,
      "step": 16088
    },
    {
      "epoch": 8.0,
      "eval_loss": 0.34859710931777954,
      "eval_runtime": 42.0013,
      "eval_samples_per_second": 9.595,
      "eval_steps_per_second": 1.214,
      "step": 16088
    },
    {
      "epoch": 9.0,
      "learning_rate": 3.875e-05,
      "loss": 0.2792,
      "step": 18099
    },
    {
      "epoch": 9.0,
      "eval_loss": 0.3179122507572174,
      "eval_runtime": 41.657,
      "eval_samples_per_second": 9.674,
      "eval_steps_per_second": 1.224,
      "step": 18099
    },
    {
      "epoch": 10.0,
      "learning_rate": 3.7500000000000003e-05,
      "loss": 0.2544,
      "step": 20110
    },
    {
      "epoch": 10.0,
      "eval_loss": 0.29244551062583923,
      "eval_runtime": 41.628,
      "eval_samples_per_second": 9.681,
      "eval_steps_per_second": 1.225,
      "step": 20110
    },
    {
      "epoch": 11.0,
      "learning_rate": 3.625e-05,
      "loss": 0.2334,
      "step": 22121
    },
    {
      "epoch": 11.0,
      "eval_loss": 0.27090445160865784,
      "eval_runtime": 41.6294,
      "eval_samples_per_second": 9.681,
      "eval_steps_per_second": 1.225,
      "step": 22121
    },
    {
      "epoch": 12.0,
      "learning_rate": 3.5e-05,
      "loss": 0.2139,
      "step": 24132
    },
    {
      "epoch": 12.0,
      "eval_loss": 0.2475050389766693,
      "eval_runtime": 41.722,
      "eval_samples_per_second": 9.659,
      "eval_steps_per_second": 1.222,
      "step": 24132
    },
    {
      "epoch": 13.0,
      "learning_rate": 3.375000000000001e-05,
      "loss": 0.1964,
      "step": 26143
    },
    {
      "epoch": 13.0,
      "eval_loss": 0.22930796444416046,
      "eval_runtime": 41.7015,
      "eval_samples_per_second": 9.664,
      "eval_steps_per_second": 1.223,
      "step": 26143
    },
    {
      "epoch": 14.0,
      "learning_rate": 3.2500000000000004e-05,
      "loss": 0.1807,
      "step": 28154
    },
    {
      "epoch": 14.0,
      "eval_loss": 0.21222631633281708,
      "eval_runtime": 41.7061,
      "eval_samples_per_second": 9.663,
      "eval_steps_per_second": 1.223,
      "step": 28154
    },
    {
      "epoch": 15.0,
      "learning_rate": 3.125e-05,
      "loss": 0.1671,
      "step": 30165
    },
    {
      "epoch": 15.0,
      "eval_loss": 0.19732101261615753,
      "eval_runtime": 41.7192,
      "eval_samples_per_second": 9.66,
      "eval_steps_per_second": 1.222,
      "step": 30165
    },
    {
      "epoch": 16.0,
      "learning_rate": 3e-05,
      "loss": 0.155,
      "step": 32176
    },
    {
      "epoch": 16.0,
      "eval_loss": 0.18540407717227936,
      "eval_runtime": 41.6288,
      "eval_samples_per_second": 9.681,
      "eval_steps_per_second": 1.225,
      "step": 32176
    },
    {
      "epoch": 17.0,
      "learning_rate": 2.8749999999999997e-05,
      "loss": 0.144,
      "step": 34187
    },
    {
      "epoch": 17.0,
      "eval_loss": 0.1721782088279724,
      "eval_runtime": 41.7432,
      "eval_samples_per_second": 9.654,
      "eval_steps_per_second": 1.222,
      "step": 34187
    },
    {
      "epoch": 18.0,
      "learning_rate": 2.7500000000000004e-05,
      "loss": 0.1348,
      "step": 36198
    },
    {
      "epoch": 18.0,
      "eval_loss": 0.15782776474952698,
      "eval_runtime": 41.6345,
      "eval_samples_per_second": 9.679,
      "eval_steps_per_second": 1.225,
      "step": 36198
    },
    {
      "epoch": 19.0,
      "learning_rate": 2.625e-05,
      "loss": 0.1258,
      "step": 38209
    },
    {
      "epoch": 19.0,
      "eval_loss": 0.1482909917831421,
      "eval_runtime": 41.6668,
      "eval_samples_per_second": 9.672,
      "eval_steps_per_second": 1.224,
      "step": 38209
    },
    {
      "epoch": 20.0,
      "learning_rate": 2.5e-05,
      "loss": 0.1178,
      "step": 40220
    },
    {
      "epoch": 20.0,
      "eval_loss": 0.13850367069244385,
      "eval_runtime": 41.7142,
      "eval_samples_per_second": 9.661,
      "eval_steps_per_second": 1.223,
      "step": 40220
    },
    {
      "epoch": 21.0,
      "learning_rate": 2.375e-05,
      "loss": 0.1105,
      "step": 42231
    },
    {
      "epoch": 21.0,
      "eval_loss": 0.12867872416973114,
      "eval_runtime": 41.6334,
      "eval_samples_per_second": 9.68,
      "eval_steps_per_second": 1.225,
      "step": 42231
    },
    {
      "epoch": 22.0,
      "learning_rate": 2.25e-05,
      "loss": 0.1049,
      "step": 44242
    },
    {
      "epoch": 22.0,
      "eval_loss": 0.12203275412321091,
      "eval_runtime": 41.7291,
      "eval_samples_per_second": 9.658,
      "eval_steps_per_second": 1.222,
      "step": 44242
    },
    {
      "epoch": 23.0,
      "learning_rate": 2.125e-05,
      "loss": 0.0998,
      "step": 46253
    },
    {
      "epoch": 23.0,
      "eval_loss": 0.11384650319814682,
      "eval_runtime": 41.7428,
      "eval_samples_per_second": 9.654,
      "eval_steps_per_second": 1.222,
      "step": 46253
    },
    {
      "epoch": 24.0,
      "learning_rate": 2e-05,
      "loss": 0.0946,
      "step": 48264
    },
    {
      "epoch": 24.0,
      "eval_loss": 0.10876920819282532,
      "eval_runtime": 41.7306,
      "eval_samples_per_second": 9.657,
      "eval_steps_per_second": 1.222,
      "step": 48264
    },
    {
      "epoch": 25.0,
      "learning_rate": 1.8750000000000002e-05,
      "loss": 0.0906,
      "step": 50275
    },
    {
      "epoch": 25.0,
      "eval_loss": 0.1036558523774147,
      "eval_runtime": 41.6568,
      "eval_samples_per_second": 9.674,
      "eval_steps_per_second": 1.224,
      "step": 50275
    },
    {
      "epoch": 26.0,
      "learning_rate": 1.75e-05,
      "loss": 0.0866,
      "step": 52286
    },
    {
      "epoch": 26.0,
      "eval_loss": 0.09655934572219849,
      "eval_runtime": 41.6692,
      "eval_samples_per_second": 9.671,
      "eval_steps_per_second": 1.224,
      "step": 52286
    },
    {
      "epoch": 27.0,
      "learning_rate": 1.6250000000000002e-05,
      "loss": 0.0835,
      "step": 54297
    },
    {
      "epoch": 27.0,
      "eval_loss": 0.09233643114566803,
      "eval_runtime": 41.7504,
      "eval_samples_per_second": 9.653,
      "eval_steps_per_second": 1.222,
      "step": 54297
    },
    {
      "epoch": 28.0,
      "learning_rate": 1.5e-05,
      "loss": 0.0805,
      "step": 56308
    },
    {
      "epoch": 28.0,
      "eval_loss": 0.08887409418821335,
      "eval_runtime": 41.73,
      "eval_samples_per_second": 9.657,
      "eval_steps_per_second": 1.222,
      "step": 56308
    },
    {
      "epoch": 29.0,
      "learning_rate": 1.3750000000000002e-05,
      "loss": 0.0779,
      "step": 58319
    },
    {
      "epoch": 29.0,
      "eval_loss": 0.08545327931642532,
      "eval_runtime": 41.7924,
      "eval_samples_per_second": 9.643,
      "eval_steps_per_second": 1.22,
      "step": 58319
    },
    {
      "epoch": 30.0,
      "learning_rate": 1.25e-05,
      "loss": 0.0758,
      "step": 60330
    },
    {
      "epoch": 30.0,
      "eval_loss": 0.08104873448610306,
      "eval_runtime": 41.7832,
      "eval_samples_per_second": 9.645,
      "eval_steps_per_second": 1.221,
      "step": 60330
    },
    {
      "epoch": 31.0,
      "learning_rate": 1.125e-05,
      "loss": 0.0738,
      "step": 62341
    },
    {
      "epoch": 31.0,
      "eval_loss": 0.07830855250358582,
      "eval_runtime": 41.745,
      "eval_samples_per_second": 9.654,
      "eval_steps_per_second": 1.222,
      "step": 62341
    },
    {
      "epoch": 32.0,
      "learning_rate": 1e-05,
      "loss": 0.072,
      "step": 64352
    },
    {
      "epoch": 32.0,
      "eval_loss": 0.07554436475038528,
      "eval_runtime": 41.6983,
      "eval_samples_per_second": 9.665,
      "eval_steps_per_second": 1.223,
      "step": 64352
    },
    {
      "epoch": 33.0,
      "learning_rate": 8.75e-06,
      "loss": 0.0705,
      "step": 66363
    },
    {
      "epoch": 33.0,
      "eval_loss": 0.07316704839468002,
      "eval_runtime": 41.7684,
      "eval_samples_per_second": 9.648,
      "eval_steps_per_second": 1.221,
      "step": 66363
    },
    {
      "epoch": 34.0,
      "learning_rate": 7.5e-06,
      "loss": 0.0692,
      "step": 68374
    },
    {
      "epoch": 34.0,
      "eval_loss": 0.07176368683576584,
      "eval_runtime": 41.6547,
      "eval_samples_per_second": 9.675,
      "eval_steps_per_second": 1.224,
      "step": 68374
    },
    {
      "epoch": 35.0,
      "learning_rate": 6.25e-06,
      "loss": 0.068,
      "step": 70385
    },
    {
      "epoch": 35.0,
      "eval_loss": 0.06996949017047882,
      "eval_runtime": 41.7824,
      "eval_samples_per_second": 9.645,
      "eval_steps_per_second": 1.221,
      "step": 70385
    },
    {
      "epoch": 36.0,
      "learning_rate": 5e-06,
      "loss": 0.067,
      "step": 72396
    },
    {
      "epoch": 36.0,
      "eval_loss": 0.06851284950971603,
      "eval_runtime": 41.7116,
      "eval_samples_per_second": 9.662,
      "eval_steps_per_second": 1.223,
      "step": 72396
    },
    {
      "epoch": 37.0,
      "learning_rate": 3.75e-06,
      "loss": 0.066,
      "step": 74407
    },
    {
      "epoch": 37.0,
      "eval_loss": 0.06739722192287445,
      "eval_runtime": 41.7517,
      "eval_samples_per_second": 9.652,
      "eval_steps_per_second": 1.222,
      "step": 74407
    },
    {
      "epoch": 38.0,
      "learning_rate": 2.5e-06,
      "loss": 0.0651,
      "step": 76418
    },
    {
      "epoch": 38.0,
      "eval_loss": 0.0664171576499939,
      "eval_runtime": 41.7491,
      "eval_samples_per_second": 9.653,
      "eval_steps_per_second": 1.222,
      "step": 76418
    },
    {
      "epoch": 39.0,
      "learning_rate": 1.25e-06,
      "loss": 0.0644,
      "step": 78429
    },
    {
      "epoch": 39.0,
      "eval_loss": 0.06586799770593643,
      "eval_runtime": 41.7935,
      "eval_samples_per_second": 9.643,
      "eval_steps_per_second": 1.22,
      "step": 78429
    },
    {
      "epoch": 40.0,
      "learning_rate": 0.0,
      "loss": 0.0637,
      "step": 80440
    },
    {
      "epoch": 40.0,
      "eval_loss": 0.06568873673677444,
      "eval_runtime": 41.8305,
      "eval_samples_per_second": 9.634,
      "eval_steps_per_second": 1.219,
      "step": 80440
    }
  ],
  "logging_steps": 500,
  "max_steps": 80440,
  "num_train_epochs": 40,
  "save_steps": 500,
  "total_flos": 4.688379059060736e+17,
  "trial_name": null,
  "trial_params": null
}
```

## Model Details

### Model Description

<!-- Provide a longer summary of what this model is. -->



- **Developed by:** [More Information Needed]
- **Funded by [optional]:** [More Information Needed]
- **Shared by [optional]:** [More Information Needed]
- **Model type:** [More Information Needed]
- **Language(s) (NLP):** [More Information Needed]
- **License:** [More Information Needed]
- **Finetuned from model [optional]:** [More Information Needed]

### Model Sources [optional]

<!-- Provide the basic links for the model. -->

- **Repository:** [More Information Needed]
- **Paper [optional]:** [More Information Needed]
- **Demo [optional]:** [More Information Needed]

## Uses

<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->

### Direct Use

<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->

[More Information Needed]

### Downstream Use [optional]

<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->

[More Information Needed]

### Out-of-Scope Use

<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->

[More Information Needed]

## Bias, Risks, and Limitations

<!-- This section is meant to convey both technical and sociotechnical limitations. -->

[More Information Needed]

### Recommendations

<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->

Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.

## How to Get Started with the Model

Use the code below to get started with the model.

[More Information Needed]

## Training Details

### Training Data

<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->

[More Information Needed]

### Training Procedure 

<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->

#### Preprocessing [optional]

[More Information Needed]


#### Training Hyperparameters

- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->

#### Speeds, Sizes, Times [optional]

<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->

[More Information Needed]

## Evaluation

<!-- This section describes the evaluation protocols and provides the results. -->

### Testing Data, Factors & Metrics

#### Testing Data

<!-- This should link to a Dataset Card if possible. -->

[More Information Needed]

#### Factors

<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->

[More Information Needed]

#### Metrics

<!-- These are the evaluation metrics being used, ideally with a description of why. -->

[More Information Needed]

### Results

[More Information Needed]

#### Summary



## Model Examination [optional]

<!-- Relevant interpretability work for the model goes here -->

[More Information Needed]

## Environmental Impact

<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->

Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).

- **Hardware Type:** [More Information Needed]
- **Hours used:** [More Information Needed]
- **Cloud Provider:** [More Information Needed]
- **Compute Region:** [More Information Needed]
- **Carbon Emitted:** [More Information Needed]

## Technical Specifications [optional]

### Model Architecture and Objective

[More Information Needed]

### Compute Infrastructure

[More Information Needed]

#### Hardware

[More Information Needed]

#### Software

[More Information Needed]

## Citation [optional]

<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->

**BibTeX:**

[More Information Needed]

**APA:**

[More Information Needed]

## Glossary [optional]

<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->

[More Information Needed]

## More Information [optional]

[More Information Needed]

## Model Card Authors [optional]

[More Information Needed]

## Model Card Contact

[More Information Needed]


## Training procedure


The following `bitsandbytes` quantization config was used during training:
- quant_method: bitsandbytes
- load_in_8bit: False
- load_in_4bit: True
- llm_int8_threshold: 6.0
- llm_int8_skip_modules: None
- llm_int8_enable_fp32_cpu_offload: False
- llm_int8_has_fp16_weight: False
- bnb_4bit_quant_type: nf4
- bnb_4bit_use_double_quant: True
- bnb_4bit_compute_dtype: float16

### Framework versions


- PEFT 0.6.2
