transformers tensorboard safetensors wav2vec2-bert automatic-speech-recognition mozilla-foundation/common_voice_16_0 generated_from_trainer en base_model:facebook/w2v-bert-2.0 license:mit endpoints_compatible region:us
---
language:
- en
license: mit
base_model: facebook/w2v-bert-2.0
tags:
- automatic-speech-recognition
- mozilla-foundation/common_voice_16_0
- generated_from_trainer
metrics:
- wer
model-index:
- name: wav2vec2-bert-CV16-en-cv-2
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# wav2vec2-bert-CV16-en-cv-2

This model is a fine-tuned version of [facebook/w2v-bert-2.0](https://huggingface.co/facebook/w2v-bert-2.0) on the MOZILLA-FOUNDATION/COMMON_VOICE_16_0 - EN dataset.
It achieves the following results on the evaluation set:
- Loss: 0.4363
- Wer: 0.1304
- Cer: 0.0527

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 3e-05
- train_batch_size: 12
- eval_batch_size: 12
- seed: 42
- distributed_type: multi-GPU
- num_devices: 3
- gradient_accumulation_steps: 3
- total_train_batch_size: 108
- total_eval_batch_size: 36
- optimizer: Adam with betas=(0.9,0.96) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 10000
- num_epochs: 50.0
- mixed_precision_training: Native AMP

### Training results

| Training Loss | Epoch | Step   | Validation Loss | Wer    | Cer    |
|:-------------:|:-----:|:------:|:---------------:|:------:|:------:|
| 0.1884        | 0.49  | 5000   | 0.3214          | 0.1872 | 0.0699 |
| 0.1862        | 0.98  | 10000  | 0.2829          | 0.1759 | 0.0674 |
| 0.1493        | 1.47  | 15000  | 0.2955          | 0.1680 | 0.0658 |
| 0.1346        | 1.96  | 20000  | 0.2727          | 0.1649 | 0.0647 |
| 0.1273        | 2.45  | 25000  | 0.2677          | 0.1608 | 0.0631 |
| 0.1241        | 2.94  | 30000  | 0.2613          | 0.1594 | 0.0631 |
| 0.1059        | 3.43  | 35000  | 0.2737          | 0.1600 | 0.0636 |
| 0.1201        | 3.92  | 40000  | 0.2633          | 0.1553 | 0.0618 |
| 0.0961        | 4.4   | 45000  | 0.2666          | 0.1549 | 0.0617 |
| 0.0971        | 4.89  | 50000  | 0.2647          | 0.1536 | 0.0616 |
| 0.088         | 5.38  | 55000  | 0.2504          | 0.1500 | 0.0597 |
| 0.0931        | 5.87  | 60000  | 0.2494          | 0.1500 | 0.0599 |
| 0.0906        | 6.36  | 65000  | 0.2604          | 0.1507 | 0.0602 |
| 0.0788        | 6.85  | 70000  | 0.2629          | 0.1506 | 0.0603 |
| 0.0692        | 7.34  | 75000  | 0.2500          | 0.1484 | 0.0590 |
| 0.0896        | 7.83  | 80000  | 0.2525          | 0.1468 | 0.0586 |
| 0.0794        | 8.32  | 85000  | 0.2494          | 0.1458 | 0.0583 |
| 0.0908        | 8.81  | 90000  | 0.2539          | 0.1475 | 0.0587 |
| 0.0646        | 9.3   | 95000  | 0.2539          | 0.1457 | 0.0578 |
| 0.0877        | 9.79  | 100000 | 0.2503          | 0.1453 | 0.0583 |
| 0.0657        | 10.28 | 105000 | 0.2589          | 0.1457 | 0.0586 |
| 0.0715        | 10.77 | 110000 | 0.2638          | 0.1462 | 0.0586 |
| 0.0704        | 11.26 | 115000 | 0.2562          | 0.1419 | 0.0571 |
| 0.0582        | 11.75 | 120000 | 0.2663          | 0.1425 | 0.0577 |
| 0.0583        | 12.23 | 125000 | 0.2615          | 0.1408 | 0.0573 |
| 0.0776        | 12.72 | 130000 | 0.2593          | 0.1434 | 0.0575 |
| 0.0624        | 13.21 | 135000 | 0.2603          | 0.1438 | 0.0577 |
| 0.0619        | 13.7  | 140000 | 0.2512          | 0.1391 | 0.0564 |
| 0.0642        | 14.19 | 145000 | 0.2539          | 0.1386 | 0.0558 |
| 0.0593        | 14.68 | 150000 | 0.2609          | 0.1406 | 0.0567 |
| 0.0596        | 15.17 | 155000 | 0.2567          | 0.1390 | 0.0562 |
| 0.0671        | 15.66 | 160000 | 0.2618          | 0.1404 | 0.0574 |
| 0.0537        | 16.15 | 165000 | 0.2668          | 0.1391 | 0.0565 |
| 0.0543        | 16.64 | 170000 | 0.2583          | 0.1379 | 0.0560 |
| 0.056         | 17.13 | 175000 | 0.2612          | 0.1395 | 0.0564 |
| 0.0605        | 17.62 | 180000 | 0.2654          | 0.1375 | 0.0557 |
| 0.0536        | 18.11 | 185000 | 0.2703          | 0.1356 | 0.0549 |
| 0.0469        | 18.6  | 190000 | 0.2571          | 0.1348 | 0.0548 |
| 0.0548        | 19.09 | 195000 | 0.2621          | 0.1359 | 0.0551 |
| 0.0507        | 19.58 | 200000 | 0.2628          | 0.1348 | 0.0549 |
| 0.0513        | 20.06 | 205000 | 0.2722          | 0.1367 | 0.0558 |
| 0.0423        | 20.55 | 210000 | 0.2752          | 0.1349 | 0.0546 |
| 0.049         | 21.04 | 215000 | 0.2777          | 0.1358 | 0.0552 |
| 0.0434        | 21.53 | 220000 | 0.2671          | 0.1336 | 0.0549 |
| 0.0443        | 22.02 | 225000 | 0.2815          | 0.1333 | 0.0544 |
| 0.0533        | 22.51 | 230000 | 0.2674          | 0.1334 | 0.0542 |
| 0.0458        | 23.0  | 235000 | 0.2746          | 0.1320 | 0.0541 |
| 0.0527        | 23.49 | 240000 | 0.2750          | 0.1351 | 0.0546 |
| 0.0458        | 23.98 | 245000 | 0.2748          | 0.1322 | 0.0539 |
| 0.0434        | 24.47 | 250000 | 0.2774          | 0.1317 | 0.0538 |
| 0.0434        | 24.96 | 255000 | 0.2756          | 0.1322 | 0.0534 |
| 0.041         | 25.45 | 260000 | 0.2786          | 0.1337 | 0.0542 |
| 0.0408        | 25.94 | 265000 | 0.2785          | 0.1320 | 0.0534 |
| 0.0486        | 26.43 | 270000 | 0.2882          | 0.1325 | 0.0536 |
| 0.0469        | 26.92 | 275000 | 0.2796          | 0.1315 | 0.0532 |
| 0.041         | 27.41 | 280000 | 0.2786          | 0.1319 | 0.0536 |
| 0.0333        | 27.89 | 285000 | 0.2893          | 0.1316 | 0.0532 |
| 0.0391        | 28.38 | 290000 | 0.2893          | 0.1318 | 0.0537 |
| 0.0427        | 28.87 | 295000 | 0.3006          | 0.1326 | 0.0535 |
| 0.0469        | 29.36 | 300000 | 0.2846          | 0.1308 | 0.0530 |
| 0.0317        | 29.85 | 305000 | 0.3140          | 0.1311 | 0.0534 |
| 0.0373        | 30.34 | 310000 | 0.2951          | 0.1314 | 0.0533 |
| 0.0367        | 30.83 | 315000 | 0.2976          | 0.1309 | 0.0532 |
| 0.0385        | 31.32 | 320000 | 0.3068          | 0.1311 | 0.0534 |
| 0.0372        | 31.81 | 325000 | 0.3085          | 0.1295 | 0.0527 |
| 0.0342        | 32.3  | 330000 | 0.3150          | 0.1290 | 0.0527 |
| 0.035         | 32.79 | 335000 | 0.3133          | 0.1299 | 0.0530 |
| 0.0331        | 33.28 | 340000 | 0.3201          | 0.1303 | 0.0530 |
| 0.0334        | 33.77 | 345000 | 0.3310          | 0.1294 | 0.0527 |
| 0.0353        | 34.26 | 350000 | 0.3105          | 0.1287 | 0.0523 |
| 0.0367        | 34.75 | 355000 | 0.3180          | 0.1302 | 0.0530 |
| 0.0397        | 35.24 | 360000 | 0.3322          | 0.1297 | 0.0528 |
| 0.0384        | 35.72 | 365000 | 0.3290          | 0.1303 | 0.0530 |
| 0.0349        | 36.21 | 370000 | 0.3358          | 0.1298 | 0.0529 |
| 0.0352        | 36.7  | 375000 | 0.3351          | 0.1347 | 0.0546 |
| 0.0333        | 37.19 | 380000 | 0.3420          | 0.1296 | 0.0528 |
| 0.0282        | 37.68 | 385000 | 0.3426          | 0.1317 | 0.0534 |
| 0.0247        | 38.17 | 390000 | 0.3606          | 0.1318 | 0.0531 |
| 0.0312        | 38.66 | 395000 | 0.3509          | 0.1494 | 0.0601 |
| 0.0288        | 39.15 | 400000 | 0.3516          | 0.1325 | 0.0536 |
| 0.0281        | 39.64 | 405000 | 0.3489          | 0.1303 | 0.0528 |
| 0.0208        | 40.13 | 410000 | 0.3661          | 0.1430 | 0.0576 |
| 0.0276        | 40.62 | 415000 | 0.3620          | 0.1300 | 0.0524 |
| 0.0253        | 41.11 | 420000 | 0.3786          | 0.1328 | 0.0538 |
| 0.025         | 41.6  | 425000 | 0.3782          | 0.1321 | 0.0535 |
| 0.02          | 42.09 | 430000 | 0.3721          | 0.1297 | 0.0523 |
| 0.0192        | 42.58 | 435000 | 0.4099          | 0.1308 | 0.0528 |
| 0.0197        | 43.07 | 440000 | 0.3970          | 0.1292 | 0.0525 |
| 0.0177        | 43.55 | 445000 | 0.3946          | 0.1306 | 0.0531 |
| 0.0185        | 44.04 | 450000 | 0.4060          | 0.1293 | 0.0526 |
| 0.0176        | 44.53 | 455000 | 0.3968          | 0.1303 | 0.0529 |
| 0.0126        | 45.02 | 460000 | 0.3994          | 0.1304 | 0.0529 |
| 0.0142        | 45.51 | 465000 | 0.3975          | 0.1300 | 0.0527 |
| 0.0122        | 46.0  | 470000 | 0.4055          | 0.1287 | 0.0523 |
| 0.0115        | 46.49 | 475000 | 0.4211          | 0.1303 | 0.0526 |
| 0.0102        | 46.98 | 480000 | 0.4148          | 0.1298 | 0.0525 |
| 0.0119        | 47.47 | 485000 | 0.4238          | 0.1301 | 0.0527 |
| 0.0098        | 47.96 | 490000 | 0.4293          | 0.1299 | 0.0526 |
| 0.0125        | 48.45 | 495000 | 0.4375          | 0.1300 | 0.0526 |
| 0.0095        | 48.94 | 500000 | 0.4268          | 0.1303 | 0.0527 |
| 0.0055        | 49.43 | 505000 | 0.4286          | 0.1305 | 0.0527 |
| 0.0089        | 49.92 | 510000 | 0.4371          | 0.1304 | 0.0528 |


### Framework versions

- Transformers 4.37.0.dev0
- Pytorch 2.1.0+cu121
- Datasets 2.16.1
- Tokenizers 0.15.0
