w2v-bert-2.0-lg-CV-Fleurs-50hrs-v10

This model is a fine-tuned version of facebook/w2v-bert-2.0 on the None dataset. It achieves the following results on the evaluation set:

• Loss: 0.4777
• Wer: 0.3015
• Cer: 0.0640

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 3e-05
• train_batch_size: 8
• eval_batch_size: 4
• seed: 42
• gradient_accumulation_steps: 2
• total_train_batch_size: 16
• optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• num_epochs: 100
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Wer	Cer
0.404	0.9998	2145	0.3121	0.3712	0.0779
0.2269	2.0	4291	0.2865	0.3381	0.0707
0.1917	2.9998	6436	0.2710	0.3346	0.0696
0.1663	4.0	8582	0.2615	0.3191	0.0659
0.148	4.9998	10727	0.2710	0.3149	0.0659
0.1328	6.0	12873	0.2755	0.3053	0.0647
0.1199	6.9998	15018	0.2748	0.3076	0.0652
0.1081	8.0	17164	0.2753	0.3038	0.0640
0.0984	8.9998	19309	0.2787	0.3076	0.0637
0.0875	10.0	21455	0.2958	0.3050	0.0641
0.0776	10.9998	23600	0.2975	0.3096	0.0648
0.0679	12.0	25746	0.3140	0.3108	0.0654
0.0598	12.9998	27891	0.3183	0.3017	0.0630
0.0523	14.0	30037	0.3284	0.3027	0.0633
0.0443	14.9998	32182	0.3883	0.3067	0.0634
0.0387	16.0	34328	0.3608	0.3104	0.0650
0.0347	16.9998	36473	0.3989	0.3051	0.0642
0.0299	18.0	38619	0.3820	0.3168	0.0653
0.0271	18.9998	40764	0.4055	0.3048	0.0638
0.0245	20.0	42910	0.4294	0.3140	0.0655
0.0217	20.9998	45055	0.3997	0.3176	0.0642
0.0199	22.0	47201	0.4121	0.3104	0.0649
0.0182	22.9998	49346	0.4364	0.3112	0.0647
0.0165	24.0	51492	0.4817	0.3093	0.0665
0.0153	24.9998	53637	0.4777	0.3015	0.0640

Framework versions

• Transformers 4.46.3
• Pytorch 2.1.0+cu118
• Datasets 3.1.0
• Tokenizers 0.20.3