defect-classification-scibert-baseline-25-epochs

This model is a fine-tuned version of allenai/scibert_scivocab_uncased on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 0.2317
• Accuracy: 0.9075

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-05
• train_batch_size: 256
• eval_batch_size: 256
• seed: 42
• optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• num_epochs: 25

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy
1.3655	1.0	2124	1.1784	0.8208
0.9439	2.0	4248	0.6611	0.8474
0.7364	3.0	6372	0.5215	0.8561
0.6355	4.0	8496	0.4250	0.8669
0.5592	5.0	10620	0.3694	0.8740
0.5411	6.0	12744	0.3482	0.8754
0.5056	7.0	14868	0.3418	0.8757
0.4838	8.0	16992	0.2492	0.9132
0.469	9.0	19116	0.3220	0.8785
0.4468	10.0	21240	0.2693	0.8964
0.432	11.0	23364	0.2756	0.8913
0.4256	12.0	25488	0.2545	0.9019
0.4158	13.0	27612	0.2431	0.9061
0.3961	14.0	29736	0.2532	0.9001
0.3972	15.0	31860	0.2441	0.9025
0.3931	16.0	33984	0.2456	0.9008
0.3882	17.0	36108	0.2536	0.8971
0.3846	18.0	38232	0.2421	0.9029
0.3806	19.0	40356	0.2547	0.8960
0.3761	20.0	42480	0.2527	0.8956
0.3648	21.0	44604	0.2475	0.8979
0.3684	22.0	46728	0.2333	0.9079
0.3679	23.0	48852	0.2300	0.9095
0.3683	24.0	50976	0.2347	0.9060
0.3632	25.0	53100	0.2317	0.9075

Framework versions

• Transformers 4.47.0
• Pytorch 2.5.1+cu124
• Datasets 3.2.0
• Tokenizers 0.21.0