w2v-bert-2.0-lg-CV-Fleurs-20hrs-v10

This model is a fine-tuned version of facebook/w2v-bert-2.0 on the None dataset. It achieves the following results on the evaluation set:

• Loss: 0.5879
• Wer: 0.3384
• Cer: 0.0706

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 3e-05
• train_batch_size: 8
• eval_batch_size: 4
• seed: 42
• gradient_accumulation_steps: 2
• total_train_batch_size: 16
• optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• num_epochs: 100
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Wer	Cer
0.6539	0.9992	664	0.3779	0.4182	0.0890
0.2976	2.0	1329	0.3493	0.4013	0.0823
0.2421	2.9992	1993	0.3409	0.3655	0.0774
0.2038	4.0	2658	0.3221	0.3604	0.0739
0.1789	4.9992	3322	0.3093	0.3711	0.0767
0.158	6.0	3987	0.3472	0.3584	0.0740
0.1388	6.9992	4651	0.3467	0.3763	0.0790
0.1245	8.0	5316	0.3358	0.3644	0.0753
0.1087	8.9992	5980	0.3607	0.3468	0.0724
0.0933	10.0	6645	0.3673	0.3454	0.0719
0.0815	10.9992	7309	0.3767	0.3507	0.0728
0.0684	12.0	7974	0.4179	0.3452	0.0728
0.058	12.9992	8638	0.4341	0.3524	0.0743
0.0516	14.0	9303	0.4370	0.3647	0.0744
0.0423	14.9992	9967	0.4761	0.3587	0.0752
0.0385	16.0	10632	0.4817	0.3432	0.0729
0.033	16.9992	11296	0.4761	0.3575	0.0760
0.0316	18.0	11961	0.5045	0.3485	0.0733
0.026	18.9992	12625	0.5423	0.3482	0.0731
0.0235	20.0	13290	0.5298	0.3442	0.0716
0.0213	20.9992	13954	0.5624	0.3332	0.0715
0.0206	22.0	14619	0.5381	0.3428	0.0711
0.0186	22.9992	15283	0.5958	0.3461	0.0734
0.0176	24.0	15948	0.5234	0.3472	0.0727
0.016	24.9992	16612	0.5626	0.3438	0.0726
0.0155	26.0	17277	0.5592	0.3448	0.0737
0.0137	26.9992	17941	0.5726	0.3357	0.0708
0.0127	28.0	18606	0.5851	0.3371	0.0713
0.0116	28.9992	19270	0.6023	0.3422	0.0720
0.0118	30.0	19935	0.5801	0.3402	0.0732
0.011	30.9992	20599	0.5811	0.3430	0.0721
0.01	32.0	21264	0.5863	0.3412	0.0722
0.0101	32.9992	21928	0.5879	0.3384	0.0706

Framework versions

• Transformers 4.46.3
• Pytorch 2.1.0+cu118
• Datasets 3.1.0
• Tokenizers 0.20.3