zeeshan73/mistral_7b_cosine_w_restart_lr

mistral_7b_cosine_lr

This model is a fine-tuned version of mistralai/Mistral-7B-Instruct-v0.3 on the generator dataset. It achieves the following results on the evaluation set:

• Loss: 0.3803

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0002
• train_batch_size: 2
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 8
• total_train_batch_size: 16
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine_with_restarts
• lr_scheduler_warmup_ratio: 0.03
• lr_scheduler_warmup_steps: 15
• num_epochs: 3

Training results

Training Loss	Epoch	Step	Validation Loss
1.0242	0.0366	10	0.6227
0.5884	0.0732	20	0.5310
0.519	0.1098	30	0.4930
0.4818	0.1465	40	0.4653
0.4722	0.1831	50	0.4537
0.4513	0.2197	60	0.4440
0.4481	0.2563	70	0.4377
0.4455	0.2929	80	0.4321
0.4344	0.3295	90	0.4271
0.4345	0.3661	100	0.4233
0.4296	0.4027	110	0.4186
0.4255	0.4394	120	0.4166
0.4173	0.4760	130	0.4131
0.4195	0.5126	140	0.4098
0.4143	0.5492	150	0.4067
0.4103	0.5858	160	0.4043
0.4124	0.6224	170	0.4021
0.4069	0.6590	180	0.3988
0.4041	0.6957	190	0.3981
0.4044	0.7323	200	0.3951
0.3989	0.7689	210	0.3912
0.3947	0.8055	220	0.3895
0.3945	0.8421	230	0.3868
0.3876	0.8787	240	0.3849
0.3877	0.9153	250	0.3839
0.3922	0.9519	260	0.3817
0.3844	0.9886	270	0.3796
0.3491	1.0252	280	0.3832
0.3291	1.0618	290	0.3821
0.3267	1.0984	300	0.3803

Framework versions

• PEFT 0.13.2
• Transformers 4.45.2
• Pytorch 2.4.1+cu121
• Datasets 3.0.1
• Tokenizers 0.20.0