Progen2_Kinase_PhosphositeGen_dkz_traindata

This model is a fine-tuned version of hugohrban/progen2-base on the None dataset. It achieves the following results on the evaluation set:

• Loss: 2.0955
• Perplexity: 8.1296

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0005
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 2
• total_train_batch_size: 16
• optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• training_steps: 5000

Training results

Training Loss	Epoch	Step	Validation Loss	Perplexity
4.7229	0.1455	100	2.1862	8.9015
4.2858	0.2909	200	2.1091	8.2405
4.2112	0.4364	300	2.0519	7.7824
4.1146	0.5818	400	2.0049	7.4252
4.0772	0.7273	500	1.9859	7.2855
3.9871	0.8727	600	1.9478	7.0130
3.891	1.0175	700	1.9204	6.8236
3.4841	1.1629	800	1.8889	6.6122
3.4596	1.3084	900	1.8696	6.4854
3.4659	1.4538	1000	1.8430	6.3152
3.433	1.5993	1100	1.8105	6.1137
3.3728	1.7447	1200	1.7991	6.0441
3.3853	1.8902	1300	1.7924	6.0040
3.1832	2.0349	1400	1.7975	6.0348
2.8198	2.1804	1500	1.7924	6.0041
2.7867	2.3258	1600	1.7604	5.8149
2.8669	2.4713	1700	1.7437	5.7183
2.795	2.6167	1800	1.7307	5.6445
2.8152	2.7622	1900	1.7188	5.5779
2.7734	2.9076	2000	1.6911	5.4256
2.5299	3.0524	2100	1.7682	5.8605
2.2126	3.1978	2200	1.7346	5.6669
2.2435	3.3433	2300	1.7104	5.5310
2.2663	3.4887	2400	1.7144	5.5536
2.2463	3.6342	2500	1.7338	5.6620
2.3117	3.7796	2600	1.6879	5.4079
2.2655	3.9251	2700	1.6946	5.4447
1.9936	4.0698	2800	1.8444	6.3244
1.7929	4.2153	2900	1.8653	6.4577
1.8214	4.3607	3000	1.7600	5.8123
1.8505	4.5062	3100	1.7855	5.9628
1.8382	4.6516	3200	1.7955	6.0225
1.7945	4.7971	3300	1.7754	5.9028
1.8238	4.9425	3400	1.7820	5.9418
1.573	5.0873	3500	1.8691	6.4823
1.4562	5.2327	3600	1.8905	6.6225
1.47	5.3782	3700	2.0037	7.4163
1.4649	5.5236	3800	1.8911	6.6268
1.4778	5.6691	3900	1.9307	6.8940
1.4985	5.8145	4000	1.9265	6.8655
1.4587	5.96	4100	1.9128	6.7720
1.258	6.1047	4200	2.0383	7.6773
1.2239	6.2502	4300	2.0444	7.7244
1.2186	6.3956	4400	2.0497	7.7658
1.2174	6.5411	4500	2.0454	7.7323
1.2051	6.6865	4600	2.0195	7.5345
1.2189	6.832	4700	2.0461	7.7376
1.2061	6.9775	4800	2.0435	7.7176
1.0575	7.1222	4900	2.0885	8.0727
1.048	7.2676	5000	2.0955	8.1296

Framework versions

• PEFT 0.13.2
• Transformers 4.47.1
• Pytorch 2.1.0.post301
• Datasets 3.0.2
• Tokenizers 0.21.0