peft tensorboard safetensors generated_from_trainer base_model:meta-llama/Llama-2-7b-hf region:us
---
library_name: peft
tags:
- generated_from_trainer
base_model: meta-llama/Llama-2-7b-hf
model-index:
- name: ice_llama2
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# ice_llama2

This model is a fine-tuned version of [meta-llama/Llama-2-7b-hf](https://huggingface.co/meta-llama/Llama-2-7b-hf) on an unknown dataset.
It achieves the following results on the evaluation set:
- Loss: 2.3080

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 2e-05
- train_batch_size: 4
- eval_batch_size: 4
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: cosine
- num_epochs: 3

### Training results

| Training Loss | Epoch | Step    | Validation Loss |
|:-------------:|:-----:|:-------:|:---------------:|
| 2.6464        | 0.02  | 10000   | 2.5455          |
| 2.4927        | 0.03  | 20000   | 2.4736          |
| 2.5238        | 0.05  | 30000   | 2.4381          |
| 2.4711        | 0.07  | 40000   | 2.4124          |
| 2.4345        | 0.08  | 50000   | 2.3989          |
| 2.3715        | 0.1   | 60000   | 2.3878          |
| 2.4212        | 0.12  | 70000   | 2.3786          |
| 2.434         | 0.14  | 80000   | 2.3724          |
| 2.4798        | 0.15  | 90000   | 2.3670          |
| 2.4343        | 0.17  | 100000  | 2.3615          |
| 2.4365        | 0.19  | 110000  | 2.3571          |
| 2.398         | 0.2   | 120000  | 2.3538          |
| 2.4734        | 0.22  | 130000  | 2.3510          |
| 2.4356        | 0.24  | 140000  | 2.3499          |
| 2.4512        | 0.25  | 150000  | 2.3458          |
| 2.4331        | 0.27  | 160000  | 2.3438          |
| 2.3749        | 0.29  | 170000  | 2.3409          |
| 2.4375        | 0.3   | 180000  | 2.3397          |
| 2.3301        | 0.32  | 190000  | 2.3378          |
| 2.4341        | 0.34  | 200000  | 2.3364          |
| 2.3286        | 0.36  | 210000  | 2.3356          |
| 2.3173        | 0.37  | 220000  | 2.3332          |
| 2.4105        | 0.39  | 230000  | 2.3333          |
| 2.3553        | 0.41  | 240000  | 2.3311          |
| 2.386         | 0.42  | 250000  | 2.3307          |
| 2.3761        | 0.44  | 260000  | 2.3294          |
| 2.42          | 0.46  | 270000  | 2.3277          |
| 2.44          | 0.47  | 280000  | 2.3267          |
| 2.4169        | 0.49  | 290000  | 2.3276          |
| 2.3254        | 0.51  | 300000  | 2.3265          |
| 2.4326        | 0.52  | 310000  | 2.3252          |
| 2.4203        | 0.54  | 320000  | 2.3244          |
| 2.4401        | 0.56  | 330000  | 2.3242          |
| 2.4005        | 0.57  | 340000  | 2.3221          |
| 2.3734        | 0.59  | 350000  | 2.3223          |
| 2.331         | 0.61  | 360000  | 2.3223          |
| 2.4652        | 0.63  | 370000  | 2.3220          |
| 2.3741        | 0.64  | 380000  | 2.3211          |
| 2.3634        | 0.66  | 390000  | 2.3193          |
| 2.3688        | 0.68  | 400000  | 2.3201          |
| 2.3284        | 0.69  | 410000  | 2.3195          |
| 2.4486        | 0.71  | 420000  | 2.3184          |
| 2.4113        | 0.73  | 430000  | 2.3183          |
| 2.3442        | 0.74  | 440000  | 2.3175          |
| 2.3414        | 0.76  | 450000  | 2.3176          |
| 2.3534        | 0.78  | 460000  | 2.3166          |
| 2.3384        | 0.79  | 470000  | 2.3170          |
| 2.4021        | 0.81  | 480000  | 2.3163          |
| 2.2645        | 0.83  | 490000  | 2.3161          |
| 2.3392        | 0.85  | 500000  | 2.3158          |
| 2.3499        | 0.86  | 510000  | 2.3155          |
| 2.3111        | 0.88  | 520000  | 2.3149          |
| 2.3604        | 0.9   | 530000  | 2.3139          |
| 2.4264        | 0.91  | 540000  | 2.3144          |
| 2.4964        | 0.93  | 550000  | 2.3138          |
| 2.5121        | 0.95  | 560000  | 2.3129          |
| 2.3389        | 0.96  | 570000  | 2.3129          |
| 2.3755        | 0.98  | 580000  | 2.3131          |
| 2.414         | 1.0   | 590000  | 2.3129          |
| 2.2656        | 1.01  | 600000  | 2.3127          |
| 2.3372        | 1.03  | 610000  | 2.3120          |
| 2.3479        | 1.05  | 620000  | 2.3126          |
| 2.3752        | 1.07  | 630000  | 2.3121          |
| 2.3685        | 1.08  | 640000  | 2.3118          |
| 2.4521        | 1.1   | 650000  | 2.3117          |
| 2.426         | 1.12  | 660000  | 2.3113          |
| 2.4583        | 1.13  | 670000  | 2.3114          |
| 2.414         | 1.15  | 680000  | 2.3107          |
| 2.3833        | 1.17  | 690000  | 2.3107          |
| 2.3053        | 1.18  | 700000  | 2.3107          |
| 2.2819        | 1.2   | 710000  | 2.3108          |
| 2.4079        | 1.22  | 720000  | 2.3106          |
| 2.4529        | 1.23  | 730000  | 2.3101          |
| 2.3139        | 1.25  | 740000  | 2.3102          |
| 2.3249        | 1.27  | 750000  | 2.3104          |
| 2.3575        | 1.28  | 760000  | 2.3102          |
| 2.3568        | 1.3   | 770000  | 2.3102          |
| 2.3686        | 1.32  | 780000  | 2.3096          |
| 2.34          | 1.34  | 790000  | 2.3098          |
| 2.4017        | 1.35  | 800000  | 2.3096          |
| 2.4174        | 1.37  | 810000  | 2.3096          |
| 2.3793        | 1.39  | 820000  | 2.3092          |
| 2.4362        | 1.4   | 830000  | 2.3094          |
| 2.4015        | 1.42  | 840000  | 2.3091          |
| 2.4451        | 1.44  | 850000  | 2.3089          |
| 2.369         | 1.45  | 860000  | 2.3091          |
| 2.408         | 1.47  | 870000  | 2.3091          |
| 2.4213        | 1.49  | 880000  | 2.3088          |
| 2.4088        | 1.5   | 890000  | 2.3090          |
| 2.3602        | 1.52  | 900000  | 2.3090          |
| 2.3525        | 1.54  | 910000  | 2.3087          |
| 2.421         | 1.56  | 920000  | 2.3086          |
| 2.4586        | 1.57  | 930000  | 2.3083          |
| 2.4173        | 1.59  | 940000  | 2.3084          |
| 2.3366        | 1.61  | 950000  | 2.3084          |
| 2.3604        | 1.62  | 960000  | 2.3084          |
| 2.3662        | 1.64  | 970000  | 2.3082          |
| 2.3754        | 1.66  | 980000  | 2.3085          |
| 2.3966        | 1.67  | 990000  | 2.3083          |
| 2.3537        | 1.69  | 1000000 | 2.3083          |
| 2.4711        | 1.71  | 1010000 | 2.3084          |
| 2.335         | 1.72  | 1020000 | 2.3082          |
| 2.3608        | 1.74  | 1030000 | 2.3082          |
| 2.3669        | 1.76  | 1040000 | 2.3080          |
| 2.463         | 1.78  | 1050000 | 2.3081          |
| 2.3369        | 1.79  | 1060000 | 2.3082          |
| 2.3314        | 1.81  | 1070000 | 2.3082          |
| 2.3681        | 1.83  | 1080000 | 2.3081          |
| 2.3867        | 1.84  | 1090000 | 2.3081          |
| 2.4012        | 1.86  | 1100000 | 2.3081          |
| 2.367         | 1.88  | 1110000 | 2.3080          |
| 2.415         | 1.89  | 1120000 | 2.3080          |
| 2.3738        | 1.91  | 1130000 | 2.3080          |
| 2.4199        | 1.93  | 1140000 | 2.3080          |
| 2.3884        | 1.94  | 1150000 | 2.3080          |
| 2.3609        | 1.96  | 1160000 | 2.3080          |
| 2.3343        | 1.98  | 1170000 | 2.3080          |
| 2.3575        | 1.99  | 1180000 | 2.3079          |
| 2.4116        | 2.01  | 1190000 | 2.3079          |
| 2.3074        | 2.03  | 1200000 | 2.3080          |
| 2.4261        | 2.05  | 1210000 | 2.3080          |
| 2.3793        | 2.06  | 1220000 | 2.3079          |
| 2.3543        | 2.08  | 1230000 | 2.3080          |
| 2.404         | 2.1   | 1240000 | 2.3081          |
| 2.4234        | 2.11  | 1250000 | 2.3079          |
| 2.3582        | 2.13  | 1260000 | 2.3080          |
| 2.3922        | 2.15  | 1270000 | 2.3079          |
| 2.3461        | 2.16  | 1280000 | 2.3080          |
| 2.3395        | 2.18  | 1290000 | 2.3080          |
| 2.3755        | 2.2   | 1300000 | 2.3080          |
| 2.3718        | 2.21  | 1310000 | 2.3079          |
| 2.3894        | 2.23  | 1320000 | 2.3079          |
| 2.3955        | 2.25  | 1330000 | 2.3079          |
| 2.3445        | 2.27  | 1340000 | 2.3079          |
| 2.3894        | 2.28  | 1350000 | 2.3080          |
| 2.3648        | 2.3   | 1360000 | 2.3080          |
| 2.3999        | 2.32  | 1370000 | 2.3080          |
| 2.3102        | 2.33  | 1380000 | 2.3079          |
| 2.3311        | 2.35  | 1390000 | 2.3078          |
| 2.3408        | 2.37  | 1400000 | 2.3079          |
| 2.3844        | 2.38  | 1410000 | 2.3080          |
| 2.3825        | 2.4   | 1420000 | 2.3080          |
| 2.398         | 2.42  | 1430000 | 2.3080          |
| 2.3305        | 2.43  | 1440000 | 2.3080          |
| 2.4646        | 2.45  | 1450000 | 2.3078          |
| 2.4326        | 2.47  | 1460000 | 2.3080          |
| 2.351         | 2.49  | 1470000 | 2.3080          |
| 2.4074        | 2.5   | 1480000 | 2.3080          |
| 2.3806        | 2.52  | 1490000 | 2.3080          |
| 2.4329        | 2.54  | 1500000 | 2.3080          |
| 2.421         | 2.55  | 1510000 | 2.3079          |
| 2.3322        | 2.57  | 1520000 | 2.3079          |
| 2.393         | 2.59  | 1530000 | 2.3079          |
| 2.3183        | 2.6   | 1540000 | 2.3080          |
| 2.3257        | 2.62  | 1550000 | 2.3080          |
| 2.3745        | 2.64  | 1560000 | 2.3080          |
| 2.409         | 2.65  | 1570000 | 2.3080          |
| 2.4447        | 2.67  | 1580000 | 2.3078          |
| 2.3805        | 2.69  | 1590000 | 2.3079          |
| 2.4323        | 2.7   | 1600000 | 2.3079          |
| 2.3654        | 2.72  | 1610000 | 2.3079          |
| 2.4017        | 2.74  | 1620000 | 2.3079          |
| 2.3168        | 2.76  | 1630000 | 2.3079          |
| 2.3304        | 2.77  | 1640000 | 2.3079          |
| 2.4072        | 2.79  | 1650000 | 2.3080          |
| 2.3232        | 2.81  | 1660000 | 2.3080          |
| 2.3563        | 2.82  | 1670000 | 2.3079          |
| 2.4327        | 2.84  | 1680000 | 2.3080          |
| 2.3932        | 2.86  | 1690000 | 2.3080          |
| 2.3134        | 2.87  | 1700000 | 2.3079          |
| 2.3421        | 2.89  | 1710000 | 2.3080          |
| 2.4805        | 2.91  | 1720000 | 2.3080          |
| 2.4306        | 2.92  | 1730000 | 2.3079          |
| 2.3511        | 2.94  | 1740000 | 2.3081          |
| 2.3509        | 2.96  | 1750000 | 2.3080          |
| 2.3598        | 2.98  | 1760000 | 2.3080          |
| 2.4583        | 2.99  | 1770000 | 2.3080          |


### Framework versions

- PEFT 0.8.2
- Transformers 4.38.1
- Pytorch 2.2.0+cu118
- Datasets 2.17.1
- Tokenizers 0.15.2