Progen2_Kinase_PhosphositeGen

This model is a fine-tuned version of hugohrban/progen2-base on the None dataset. It achieves the following results on the evaluation set:

• Loss: 2.0025
• Perplexity: 7.4078

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0005
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 2
• total_train_batch_size: 16
• optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• training_steps: 5000

Training results

Training Loss	Epoch	Step	Validation Loss	Perplexity
4.9303	0.1415	100	2.1811	8.8564
4.2874	0.2831	200	2.1198	8.3296
4.1941	0.4246	300	2.0694	7.9202
4.0803	0.5662	400	2.0362	7.6616
4.0613	0.7077	500	2.0053	7.4284
3.9511	0.8493	600	1.9922	7.3315
3.9216	0.9908	700	1.9477	7.0124
3.5053	1.1316	800	1.9200	6.8208
3.4311	1.2732	900	1.9035	6.7094
3.4238	1.4147	1000	1.8714	6.4974
3.392	1.5563	1100	1.8527	6.3772
3.3621	1.6978	1200	1.8317	6.2444
3.3577	1.8393	1300	1.8237	6.1945
3.3419	1.9809	1400	1.7889	5.9826
2.8256	2.1217	1500	1.7977	6.0356
2.8061	2.2633	1600	1.7860	5.9653
2.7837	2.4048	1700	1.7666	5.8507
2.7504	2.5464	1800	1.7428	5.7133
2.829	2.6879	1900	1.7288	5.6337
2.7567	2.8294	2000	1.7088	5.5225
2.7443	2.9710	2100	1.6986	5.4664
2.3409	3.1118	2200	1.7382	5.6869
2.2568	3.2534	2300	1.7487	5.7471
2.2481	3.3949	2400	1.7181	5.5740
2.2323	3.5364	2500	1.7058	5.5059
2.2654	3.6780	2600	1.7031	5.4912
2.2611	3.8195	2700	1.6707	5.3157
2.256	3.9611	2800	1.6719	5.3222
1.8849	4.1019	2900	1.7899	5.9886
1.771	4.2435	3000	1.7697	5.8694
1.7992	4.3850	3100	1.7880	5.9775
1.838	4.5265	3200	1.7871	5.9722
1.8285	4.6681	3300	1.7342	5.6644
1.8127	4.8096	3400	1.7196	5.5825
1.8353	4.9512	3500	1.7471	5.7377
1.5511	5.0920	3600	1.8285	6.2248
1.4449	5.2335	3700	1.8683	6.4770
1.4631	5.3751	3800	1.8880	6.6063
1.4525	5.5166	3900	1.8807	6.5581
1.4516	5.6582	4000	1.8723	6.5031
1.4423	5.7997	4100	1.8828	6.5716
1.4626	5.9413	4200	1.8535	6.3824
1.3065	6.0821	4300	1.9369	6.9369
1.1889	6.2236	4400	1.9767	7.2191
1.1865	6.3652	4500	1.9845	7.2752
1.1927	6.5067	4600	2.0029	7.4103
1.1937	6.6483	4700	1.9931	7.3380
1.1893	6.7898	4800	1.9814	7.2532
1.1654	6.9314	4900	1.9931	7.3383
1.1036	7.0722	5000	2.0025	7.4078

Framework versions

• PEFT 0.13.2
• Transformers 4.47.1
• Pytorch 2.1.0.post301
• Datasets 3.0.2
• Tokenizers 0.21.0