w2v-bert-2.0-Fleurs_AMMI_AFRIVOICE_LRSC-ln-109hrs-v4

This model is a fine-tuned version of facebook/w2v-bert-2.0 on the None dataset. It achieves the following results on the evaluation set:

• Loss: 0.7091
• Wer: 0.1781
• Cer: 0.0585

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 4
• eval_batch_size: 2
• seed: 42
• gradient_accumulation_steps: 2
• total_train_batch_size: 8
• optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 500
• num_epochs: 80
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Wer	Cer
0.9031	1.0	2880	0.3538	0.2711	0.0833
0.5044	2.0	5760	0.3253	0.2325	0.0732
0.4598	3.0	8640	0.3055	0.2258	0.0700
0.4309	4.0	11520	0.2902	0.2164	0.0666
0.4074	5.0	14400	0.2737	0.2065	0.0669
0.3799	6.0	17280	0.2915	0.2066	0.0692
0.3535	7.0	20160	0.2991	0.1983	0.0649
0.3266	8.0	23040	0.3086	0.1991	0.0638
0.3023	9.0	25920	0.3162	0.2263	0.0956
0.2805	10.0	28800	0.3065	0.1972	0.0706
0.262	11.0	31680	0.3115	0.1853	0.0597
0.2449	12.0	34560	0.3411	0.1949	0.0639
0.2312	13.0	37440	0.3472	0.1868	0.0609
0.218	14.0	40320	0.3413	0.1866	0.0619
0.2044	15.0	43200	0.3433	0.1854	0.0591
0.1911	16.0	46080	0.3707	0.1832	0.0606
0.178	17.0	48960	0.3853	0.1894	0.0645
0.1655	18.0	51840	0.4232	0.1864	0.0634
0.154	19.0	54720	0.4142	0.1907	0.0624
0.1419	20.0	57600	0.4450	0.1967	0.0645
0.1286	21.0	60480	0.4470	0.1887	0.0623
0.1175	22.0	63360	0.4616	0.1827	0.0595
0.1077	23.0	66240	0.4958	0.1827	0.0598
0.0961	24.0	69120	0.4994	0.1933	0.0625
0.0871	25.0	72000	0.5498	0.1921	0.0627
0.0769	26.0	74880	0.5651	0.1864	0.0603
0.0697	27.0	77760	0.5475	0.1805	0.0593
0.061	28.0	80640	0.5627	0.1898	0.0612
0.0539	29.0	83520	0.5609	0.1869	0.0610
0.0471	30.0	86400	0.5873	0.1886	0.0599
0.0411	31.0	89280	0.5947	0.1833	0.0594
0.0353	32.0	92160	0.6101	0.1831	0.0587
0.0312	33.0	95040	0.6111	0.1842	0.0591
0.0271	34.0	97920	0.6419	0.1773	0.0576
0.024	35.0	100800	0.6625	0.1894	0.0603
0.0212	36.0	103680	0.6262	0.1807	0.0581
0.0188	37.0	106560	0.6520	0.1855	0.0592
0.0166	38.0	109440	0.6937	0.1854	0.0592
0.0145	39.0	112320	0.7056	0.1811	0.0596
0.0135	40.0	115200	0.6605	0.1784	0.0589
0.012	41.0	118080	0.6902	0.1888	0.0600
0.0105	42.0	120960	0.6909	0.1803	0.0595
0.0099	43.0	123840	0.6989	0.1829	0.0601
0.0089	44.0	126720	0.7174	0.1790	0.0582
0.0082	45.0	129600	0.7089	0.1810	0.0598
0.0073	46.0	132480	0.6911	0.1808	0.0590
0.0069	47.0	135360	0.7039	0.1765	0.0581
0.0062	48.0	138240	0.7128	0.1775	0.0580
0.006	49.0	141120	0.7091	0.1781	0.0585

Framework versions

• Transformers 4.46.3
• Pytorch 2.1.0+cu118
• Datasets 3.1.0
• Tokenizers 0.20.3