--- library_name: transformers license: mit base_model: facebook/w2v-bert-2.0 tags: - generated_from_trainer metrics: - wer model-index: - name: w2v-bert-2.0-lg-CV-Fleurs-313hrs-v11 results: [] --- # w2v-bert-2.0-lg-CV-Fleurs-313hrs-v11 This model is a fine-tuned version of [facebook/w2v-bert-2.0](https://huggingface.co/facebook/w2v-bert-2.0) on the None dataset. It achieves the following results on the evaluation set: - Loss: 0.4627 - Wer: 0.2300 - Cer: 0.0518 ## Model description More information needed ## Intended uses & limitations More information needed ## Training and evaluation data More information needed ## Training procedure ### Training hyperparameters The following hyperparameters were used during training: - learning_rate: 3e-05 - train_batch_size: 8 - eval_batch_size: 4 - seed: 42 - gradient_accumulation_steps: 2 - total_train_batch_size: 16 - optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments - lr_scheduler_type: linear - lr_scheduler_warmup_ratio: 0.1 - num_epochs: 100 - mixed_precision_training: Native AMP ### Training results | Training Loss | Epoch | Step | Validation Loss | Wer | Cer | |:-------------:|:-------:|:-------:|:---------------:|:------:|:------:| | 0.7159 | 1.0000 | 15243 | 0.2954 | 0.3514 | 0.0735 | | 0.1577 | 2.0 | 30487 | 0.2537 | 0.3192 | 0.0654 | | 0.1273 | 3.0000 | 45730 | 0.2309 | 0.2936 | 0.0607 | | 0.1125 | 4.0 | 60974 | 0.2272 | 0.2882 | 0.0600 | | 0.1027 | 5.0000 | 76217 | 0.2281 | 0.2882 | 0.0592 | | 0.096 | 6.0 | 91461 | 0.2365 | 0.2879 | 0.0589 | | 0.0913 | 7.0000 | 106704 | 0.2247 | 0.3017 | 0.0614 | | 0.0873 | 8.0 | 121948 | 0.2261 | 0.2811 | 0.0588 | | 0.0838 | 9.0000 | 137191 | 0.2151 | 0.2681 | 0.0580 | | 0.0804 | 10.0 | 152435 | 0.2227 | 0.2731 | 0.0583 | | 0.0758 | 11.0000 | 167678 | 0.2159 | 0.2738 | 0.0586 | | 0.068 | 12.0 | 182922 | 0.2150 | 0.2757 | 0.0581 | | 0.0601 | 13.0000 | 198165 | 0.2212 | 0.2704 | 0.0576 | | 0.053 | 14.0 | 213409 | 0.2236 | 0.2642 | 0.0558 | | 0.0456 | 15.0000 | 228652 | 0.2222 | 0.2679 | 0.0575 | | 0.0395 | 16.0 | 243896 | 0.2276 | 0.2821 | 0.0587 | | 0.0337 | 17.0000 | 259139 | 0.2236 | 0.2609 | 0.0565 | | 0.0289 | 18.0 | 274383 | 0.2348 | 0.2719 | 0.0577 | | 0.0254 | 19.0000 | 289626 | 0.2564 | 0.2654 | 0.0571 | | 0.0222 | 20.0 | 304870 | 0.2417 | 0.2538 | 0.0554 | | 0.0196 | 21.0000 | 320113 | 0.2754 | 0.2550 | 0.0561 | | 0.0178 | 22.0 | 335357 | 0.2715 | 0.2576 | 0.0563 | | 0.0163 | 23.0000 | 350600 | 0.2567 | 0.2528 | 0.0562 | | 0.0149 | 24.0 | 365844 | 0.2585 | 0.2654 | 0.0573 | | 0.0137 | 25.0000 | 381087 | 0.2742 | 0.2593 | 0.0567 | | 0.013 | 26.0 | 396331 | 0.2678 | 0.2463 | 0.0540 | | 0.0119 | 27.0000 | 411574 | 0.2808 | 0.2503 | 0.0553 | | 0.0108 | 28.0 | 426818 | 0.2941 | 0.2532 | 0.0543 | | 0.0102 | 29.0000 | 442061 | 0.3024 | 0.2601 | 0.0563 | | 0.0096 | 30.0 | 457305 | 0.3066 | 0.2571 | 0.0568 | | 0.009 | 31.0000 | 472548 | 0.3088 | 0.2532 | 0.0552 | | 0.0085 | 32.0 | 487792 | 0.2955 | 0.2518 | 0.0552 | | 0.0079 | 33.0000 | 503035 | 0.3238 | 0.2532 | 0.0563 | | 0.0074 | 34.0 | 518279 | 0.3071 | 0.2523 | 0.0546 | | 0.0068 | 35.0000 | 533522 | 0.3127 | 0.2509 | 0.0553 | | 0.0065 | 36.0 | 548766 | 0.3080 | 0.2561 | 0.0567 | | 0.0062 | 37.0000 | 564009 | 0.3228 | 0.2635 | 0.0571 | | 0.0057 | 38.0 | 579253 | 0.3168 | 0.2503 | 0.0551 | | 0.0054 | 39.0000 | 594496 | 0.3293 | 0.2441 | 0.0549 | | 0.0052 | 40.0 | 609740 | 0.3417 | 0.2494 | 0.0562 | | 0.0048 | 41.0000 | 624983 | 0.3337 | 0.2474 | 0.0549 | | 0.0045 | 42.0 | 640227 | 0.3583 | 0.2441 | 0.0547 | | 0.0043 | 43.0000 | 655470 | 0.3429 | 0.2538 | 0.0562 | | 0.0042 | 44.0 | 670714 | 0.3344 | 0.2430 | 0.0536 | | 0.0039 | 45.0000 | 685957 | 0.3506 | 0.2509 | 0.0557 | | 0.0037 | 46.0 | 701201 | 0.3511 | 0.2496 | 0.0555 | | 0.0035 | 47.0000 | 716444 | 0.3570 | 0.2469 | 0.0548 | | 0.0034 | 48.0 | 731688 | 0.3608 | 0.2472 | 0.0546 | | 0.0031 | 49.0000 | 746931 | 0.3475 | 0.2360 | 0.0532 | | 0.0031 | 50.0 | 762175 | 0.3516 | 0.2413 | 0.0544 | | 0.0029 | 51.0000 | 777418 | 0.3539 | 0.2477 | 0.0552 | | 0.0027 | 52.0 | 792662 | 0.3546 | 0.2390 | 0.0531 | | 0.0025 | 53.0000 | 807905 | 0.3552 | 0.2399 | 0.0534 | | 0.0025 | 54.0 | 823149 | 0.3693 | 0.2437 | 0.0547 | | 0.0023 | 55.0000 | 838392 | 0.3656 | 0.2362 | 0.0531 | | 0.0022 | 56.0 | 853636 | 0.3766 | 0.2392 | 0.0546 | | 0.0021 | 57.0000 | 868879 | 0.3910 | 0.2362 | 0.0535 | | 0.002 | 58.0 | 884123 | 0.3720 | 0.2397 | 0.0532 | | 0.0019 | 59.0000 | 899366 | 0.3770 | 0.2416 | 0.0543 | | 0.0018 | 60.0 | 914610 | 0.3983 | 0.2463 | 0.0545 | | 0.0018 | 61.0000 | 929853 | 0.3796 | 0.2348 | 0.0523 | | 0.0017 | 62.0 | 945097 | 0.3935 | 0.2403 | 0.0541 | | 0.0015 | 63.0000 | 960340 | 0.3673 | 0.2368 | 0.0531 | | 0.0015 | 64.0 | 975584 | 0.3581 | 0.2358 | 0.0526 | | 0.0014 | 65.0000 | 990827 | 0.4068 | 0.2406 | 0.0541 | | 0.0013 | 66.0 | 1006071 | 0.3957 | 0.2406 | 0.0537 | | 0.0012 | 67.0000 | 1021314 | 0.4079 | 0.2421 | 0.0536 | | 0.0012 | 68.0 | 1036558 | 0.3942 | 0.2393 | 0.0538 | | 0.0011 | 69.0000 | 1051801 | 0.4336 | 0.2392 | 0.0540 | | 0.001 | 70.0 | 1067045 | 0.3805 | 0.2363 | 0.0533 | | 0.0009 | 71.0000 | 1082288 | 0.4245 | 0.2269 | 0.0512 | | 0.001 | 72.0 | 1097532 | 0.3981 | 0.2330 | 0.0516 | | 0.0008 | 73.0000 | 1112775 | 0.4157 | 0.2335 | 0.0524 | | 0.0009 | 74.0 | 1128019 | 0.4189 | 0.2376 | 0.0530 | | 0.0009 | 75.0000 | 1143262 | 0.4098 | 0.2372 | 0.0534 | | 0.0008 | 76.0 | 1158506 | 0.4215 | 0.2361 | 0.0532 | | 0.0007 | 77.0000 | 1173749 | 0.4226 | 0.2368 | 0.0538 | | 0.0006 | 78.0 | 1188993 | 0.4699 | 0.2394 | 0.0533 | | 0.0006 | 79.0000 | 1204236 | 0.4195 | 0.2330 | 0.0524 | | 0.0006 | 80.0 | 1219480 | 0.4138 | 0.2327 | 0.0515 | | 0.0005 | 81.0000 | 1234723 | 0.4582 | 0.2347 | 0.0527 | | 0.0005 | 82.0 | 1249967 | 0.4278 | 0.2375 | 0.0530 | | 0.0004 | 83.0000 | 1265210 | 0.4469 | 0.2297 | 0.0513 | | 0.0004 | 84.0 | 1280454 | 0.4607 | 0.2320 | 0.0518 | | 0.0004 | 85.0000 | 1295697 | 0.4226 | 0.2290 | 0.0522 | | 0.0004 | 86.0 | 1310941 | 0.4627 | 0.2300 | 0.0518 | ### Framework versions - Transformers 4.46.3 - Pytorch 2.1.0+cu118 - Datasets 3.1.0 - Tokenizers 0.20.3