W2V2_Bert_BIG-C_BEMBA_5hr_v1

This model is a fine-tuned version of facebook/w2v-bert-2.0 on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: inf
• Wer: 0.4852
• Cer: 0.1215

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 16
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 2
• total_train_batch_size: 32
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_ratio: 0.01
• num_epochs: 100
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Wer	Cer
2.7125	1.0	80	inf	0.8011	0.2453
0.9662	2.0	160	inf	0.6760	0.1957
0.8283	3.0	240	inf	0.5605	0.1560
0.747	4.0	320	inf	0.6229	0.2060
0.6936	5.0	400	inf	0.6425	0.1831
0.6788	6.0	480	inf	0.5411	0.1585
0.6271	7.0	560	inf	0.5229	0.1509
0.7234	8.0	640	inf	0.6888	0.2353
1.1405	9.0	720	inf	0.9791	0.5775
2.4003	10.0	800	inf	0.9988	0.9226
2.6328	11.0	880	inf	0.9986	0.9117
2.9233	12.0	960	inf	1.0	0.9986
3.6687	13.0	1040	inf	1.0	0.9970
3.6827	14.0	1120	inf	1.0	0.9970
3.6799	15.0	1200	inf	1.0	0.9970
3.65	16.0	1280	inf	1.0	0.9970
3.6764	17.0	1360	inf	1.0	0.9970

Framework versions

• Transformers 4.44.2
• Pytorch 2.2.0+cu121
• Datasets 2.21.0
• Tokenizers 0.19.1