peft tensorboard safetensors trl sft generated_from_trainer base_model:google/gemma-2b license:other region:us
---
license: other
library_name: peft
tags:
- trl
- sft
- generated_from_trainer
base_model: google/gemma-2b
model-index:
- name: Gemma-2b-chat
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# Gemma-2b-chat

This model is a fine-tuned version of [google/gemma-2b](https://huggingface.co/google/gemma-2b) on the None dataset.
It achieves the following results on the evaluation set:
- Loss: 2.3272

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 1e-05
- train_batch_size: 1
- eval_batch_size: 1
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: cosine
- num_epochs: 3
- mixed_precision_training: Native AMP

### Training results

| Training Loss | Epoch | Step  | Validation Loss |
|:-------------:|:-----:|:-----:|:---------------:|
| 2.974         | 0.02  | 100   | 2.9419          |
| 3.1998        | 0.04  | 200   | 2.9117          |
| 2.9839        | 0.06  | 300   | 2.8745          |
| 2.9379        | 0.08  | 400   | 2.8328          |
| 2.9392        | 0.1   | 500   | 2.7909          |
| 2.7735        | 0.12  | 600   | 2.7461          |
| 2.7835        | 0.14  | 700   | 2.7075          |
| 2.7794        | 0.16  | 800   | 2.6695          |
| 2.7355        | 0.18  | 900   | 2.6363          |
| 2.7805        | 0.2   | 1000  | 2.6104          |
| 2.556         | 0.22  | 1100  | 2.5871          |
| 2.596         | 0.24  | 1200  | 2.5652          |
| 2.5838        | 0.26  | 1300  | 2.5432          |
| 2.6598        | 0.28  | 1400  | 2.5247          |
| 2.5475        | 0.3   | 1500  | 2.5100          |
| 2.4459        | 0.31  | 1600  | 2.4954          |
| 2.502         | 0.33  | 1700  | 2.4829          |
| 2.557         | 0.35  | 1800  | 2.4702          |
| 2.4944        | 0.37  | 1900  | 2.4604          |
| 2.4774        | 0.39  | 2000  | 2.4528          |
| 2.4287        | 0.41  | 2100  | 2.4453          |
| 2.5386        | 0.43  | 2200  | 2.4381          |
| 2.363         | 0.45  | 2300  | 2.4322          |
| 2.514         | 0.47  | 2400  | 2.4272          |
| 2.413         | 0.49  | 2500  | 2.4225          |
| 2.4667        | 0.51  | 2600  | 2.4176          |
| 2.4724        | 0.53  | 2700  | 2.4128          |
| 2.3949        | 0.55  | 2800  | 2.4084          |
| 2.4822        | 0.57  | 2900  | 2.4044          |
| 2.4556        | 0.59  | 3000  | 2.4009          |
| 2.4067        | 0.61  | 3100  | 2.3977          |
| 2.3911        | 0.63  | 3200  | 2.3947          |
| 2.3446        | 0.65  | 3300  | 2.3923          |
| 2.3358        | 0.67  | 3400  | 2.3891          |
| 2.3213        | 0.69  | 3500  | 2.3867          |
| 2.4041        | 0.71  | 3600  | 2.3840          |
| 2.4759        | 0.73  | 3700  | 2.3818          |
| 2.4622        | 0.75  | 3800  | 2.3801          |
| 2.3512        | 0.77  | 3900  | 2.3778          |
| 2.3653        | 0.79  | 4000  | 2.3760          |
| 2.3455        | 0.81  | 4100  | 2.3744          |
| 2.4364        | 0.83  | 4200  | 2.3724          |
| 2.2805        | 0.85  | 4300  | 2.3706          |
| 2.5448        | 0.87  | 4400  | 2.3681          |
| 2.3061        | 0.89  | 4500  | 2.3674          |
| 2.2572        | 0.9   | 4600  | 2.3657          |
| 2.3259        | 0.92  | 4700  | 2.3645          |
| 2.4078        | 0.94  | 4800  | 2.3633          |
| 2.3841        | 0.96  | 4900  | 2.3618          |
| 2.5439        | 0.98  | 5000  | 2.3604          |
| 2.4556        | 1.0   | 5100  | 2.3593          |
| 2.3752        | 1.02  | 5200  | 2.3582          |
| 2.3415        | 1.04  | 5300  | 2.3567          |
| 2.2824        | 1.06  | 5400  | 2.3555          |
| 2.3748        | 1.08  | 5500  | 2.3541          |
| 2.2535        | 1.1   | 5600  | 2.3534          |
| 2.3277        | 1.12  | 5700  | 2.3530          |
| 2.394         | 1.14  | 5800  | 2.3518          |
| 2.4876        | 1.16  | 5900  | 2.3511          |
| 2.4705        | 1.18  | 6000  | 2.3503          |
| 2.4394        | 1.2   | 6100  | 2.3499          |
| 2.3898        | 1.22  | 6200  | 2.3488          |
| 2.3789        | 1.24  | 6300  | 2.3483          |
| 2.4315        | 1.26  | 6400  | 2.3472          |
| 2.4065        | 1.28  | 6500  | 2.3463          |
| 2.3331        | 1.3   | 6600  | 2.3456          |
| 2.3415        | 1.32  | 6700  | 2.3452          |
| 2.3433        | 1.34  | 6800  | 2.3448          |
| 2.337         | 1.36  | 6900  | 2.3434          |
| 2.4492        | 1.38  | 7000  | 2.3425          |
| 2.3757        | 1.4   | 7100  | 2.3419          |
| 2.4124        | 1.42  | 7200  | 2.3412          |
| 2.2778        | 1.44  | 7300  | 2.3408          |
| 2.3127        | 1.46  | 7400  | 2.3401          |
| 2.2558        | 1.48  | 7500  | 2.3398          |
| 2.4419        | 1.49  | 7600  | 2.3394          |
| 2.3052        | 1.51  | 7700  | 2.3388          |
| 2.3212        | 1.53  | 7800  | 2.3387          |
| 2.3989        | 1.55  | 7900  | 2.3376          |
| 2.3201        | 1.57  | 8000  | 2.3372          |
| 2.4111        | 1.59  | 8100  | 2.3364          |
| 2.3243        | 1.61  | 8200  | 2.3361          |
| 2.3158        | 1.63  | 8300  | 2.3360          |
| 2.3065        | 1.65  | 8400  | 2.3357          |
| 2.3627        | 1.67  | 8500  | 2.3353          |
| 2.4604        | 1.69  | 8600  | 2.3348          |
| 2.2451        | 1.71  | 8700  | 2.3346          |
| 2.3559        | 1.73  | 8800  | 2.3342          |
| 2.4832        | 1.75  | 8900  | 2.3338          |
| 2.5064        | 1.77  | 9000  | 2.3335          |
| 2.2961        | 1.79  | 9100  | 2.3336          |
| 2.4394        | 1.81  | 9200  | 2.3334          |
| 2.4337        | 1.83  | 9300  | 2.3332          |
| 2.2984        | 1.85  | 9400  | 2.3328          |
| 2.2544        | 1.87  | 9500  | 2.3325          |
| 2.4421        | 1.89  | 9600  | 2.3321          |
| 2.2737        | 1.91  | 9700  | 2.3322          |
| 2.4483        | 1.93  | 9800  | 2.3319          |
| 2.4371        | 1.95  | 9900  | 2.3314          |
| 2.3184        | 1.97  | 10000 | 2.3312          |
| 2.2936        | 1.99  | 10100 | 2.3308          |
| 2.432         | 2.01  | 10200 | 2.3304          |
| 2.3306        | 2.03  | 10300 | 2.3301          |
| 2.3926        | 2.05  | 10400 | 2.3301          |
| 2.358         | 2.07  | 10500 | 2.3300          |
| 2.341         | 2.08  | 10600 | 2.3298          |
| 2.3886        | 2.1   | 10700 | 2.3297          |
| 2.2559        | 2.12  | 10800 | 2.3296          |
| 2.4121        | 2.14  | 10900 | 2.3294          |
| 2.3301        | 2.16  | 11000 | 2.3292          |
| 2.2807        | 2.18  | 11100 | 2.3290          |
| 2.3028        | 2.2   | 11200 | 2.3288          |
| 2.2957        | 2.22  | 11300 | 2.3289          |
| 2.296         | 2.24  | 11400 | 2.3289          |
| 2.248         | 2.26  | 11500 | 2.3288          |
| 2.3639        | 2.28  | 11600 | 2.3286          |
| 2.4383        | 2.3   | 11700 | 2.3284          |
| 2.2921        | 2.32  | 11800 | 2.3282          |
| 2.4594        | 2.34  | 11900 | 2.3282          |
| 2.4243        | 2.36  | 12000 | 2.3280          |
| 2.344         | 2.38  | 12100 | 2.3280          |
| 2.3063        | 2.4   | 12200 | 2.3279          |
| 2.3875        | 2.42  | 12300 | 2.3280          |
| 2.3502        | 2.44  | 12400 | 2.3278          |
| 2.3034        | 2.46  | 12500 | 2.3278          |
| 2.4234        | 2.48  | 12600 | 2.3277          |
| 2.2829        | 2.5   | 12700 | 2.3277          |
| 2.3965        | 2.52  | 12800 | 2.3277          |
| 2.4046        | 2.54  | 12900 | 2.3274          |
| 2.3374        | 2.56  | 13000 | 2.3274          |
| 2.1988        | 2.58  | 13100 | 2.3274          |
| 2.3893        | 2.6   | 13200 | 2.3274          |
| 2.3621        | 2.62  | 13300 | 2.3273          |
| 2.2888        | 2.64  | 13400 | 2.3273          |
| 2.3928        | 2.66  | 13500 | 2.3273          |
| 2.3523        | 2.68  | 13600 | 2.3272          |
| 2.3158        | 2.69  | 13700 | 2.3273          |
| 2.3453        | 2.71  | 13800 | 2.3273          |
| 2.3113        | 2.73  | 13900 | 2.3272          |
| 2.3878        | 2.75  | 14000 | 2.3272          |
| 2.3361        | 2.77  | 14100 | 2.3273          |
| 2.2343        | 2.79  | 14200 | 2.3273          |
| 2.2963        | 2.81  | 14300 | 2.3271          |
| 2.252         | 2.83  | 14400 | 2.3272          |
| 2.4307        | 2.85  | 14500 | 2.3272          |
| 2.2778        | 2.87  | 14600 | 2.3272          |
| 2.3832        | 2.89  | 14700 | 2.3272          |
| 2.3611        | 2.91  | 14800 | 2.3272          |
| 2.3556        | 2.93  | 14900 | 2.3271          |
| 2.3712        | 2.95  | 15000 | 2.3272          |
| 2.3667        | 2.97  | 15100 | 2.3272          |
| 2.3816        | 2.99  | 15200 | 2.3272          |


### Framework versions

- PEFT 0.9.0
- Transformers 4.38.2
- Pytorch 2.1.2
- Datasets 2.17.1
- Tokenizers 0.15.2