
# Whisper Small - Italian

This model is a fine-tuned version of [openai/whisper-small](https://huggingface.co/openai/whisper-small) 
on the [Common-voice-11.0 dataset](https://huggingface.co/datasets/mozilla-foundation/common_voice_11_0).
It achieves the following results on the evaluation set:
- Loss: 0.4549
- Wer: 200.40

## Model description

Whisper is a pre-trained model for automatic speech recognition (ASR)
published in [September 2022](https://openai.com/blog/whisper/) by the authors
Alec Radford et al. from OpenAI. Unlike many of its predecessors, such as
[Wav2Vec 2.0](https://arxiv.org/abs/2006.11477), which are pre-trained
on un-labelled audio data, Whisper is pre-trained on a vast quantity of
**labelled** audio-transcription data, 680,000 hours to be precise.
This is an order of magnitude more data than the un-labelled audio data used
to train Wav2Vec 2.0 (60,000 hours). What is more, 117,000 hours of this
pre-training data is multilingual ASR data. This results in checkpoints
that can be applied to over 96 languages, many of which are considered
_low-resource_.

When scaled to 680,000 hours of labelled pre-training data, Whisper models
demonstrate a strong ability to generalise to many datasets and domains.
The pre-trained checkpoints achieve competitive results to state-of-the-art
ASR systems, with near 3% word error rate (WER) on the test-clean subset of
LibriSpeech ASR and a new state-of-the-art on TED-LIUM with 4.7% WER (_c.f._
Table 8 of the [Whisper paper](https://cdn.openai.com/papers/whisper.pdf)).
The extensive multilingual ASR knowledge acquired by Whisper during pre-training
can be leveraged for other low-resource languages; through fine-tuning, the
pre-trained checkpoints can be adapted for specific datasets and languages
to further improve upon these results.

## Intended uses & limitations

This fine-tuned model goals are to experiment and to allow the authors to
gain skills and knowledge on how this process is carried out. The model
serves as basis for the development of a small [gradio-hosted](here) application
that transcribes recordings and audio files in italian. This application also
allows to insert a YouTube link of an Italian video ad gain a transciption.

The limitations of this project mainly regard the limited resources available 
to fine-tune the model, namely Google Colab free-version and a Google Drive
used as feature storage, that had a limited space. The time dedicated to this 
project was also limited, as it had to fit within academic deadlines.

## Training and evaluation data

The training was carried out on Google Colab platform, and the evalutation data
(as the whole dataset) was taken from the [Common-voice-11.0 dataset](https://huggingface.co/datasets/mozilla-foundation/common_voice_11_0)
reducing the dataset to only 10% of the original dataset, to avoid the training the model for too much time. 

## Training procedure

The training was conducted on Google Colab, using Jupyter Notebook to write code and document the training. Google Drive was used as Feature store.
Due to the limited resources of the free version of Google Colab, checkpointing was used to save partial results and then resume in a 
following run. The notebook was run 15 times, with approximately 40 min for each 100 steps of training for a total of 26.5h of training.
Keep in mind that Google Colab was available to us for no more than 4 h a day, so around 7 days were necessary for training alone.

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 1e-05
- train_batch_size: 16
- eval_batch_size: 8
- training_steps: 4000
- gradient_accumulation_steps: 2
- save_steps: 100
- eval_steps: 100
  
### Training results

| Run Number    | Step         | Training Loss     | Validation Loss                | Wer                        |
|:-------------:|:------------:|:-----------------:|:------------------------------:|:--------------------------:|
| 1             | 100          | 1.2396            | 1.2330                         | 176.40                     |
| 2             | 200          | 0.7389            | 0.8331                         |  80.49                     |
| 2             | 300          | 0.2951            | 0.4261                         |  70.20                     |
| 2             | 400          | 0.2703            | 0.4051                         | 101.60                     |
| 3             | 500          | 0.2491            | 0.3923                         | 112.20                     |
| 3             | 600          | 0.1700            | 0.3860                         | 107.10                     |
| 3             | 700          | 0.1603            | 0.3836                         |  90.36                     |
| 4             | 800          | 0.1607            | 0.3786                         | 135.00                     |
| 4             | 900          | 0.1540            | 0.3783                         |  99.05                     |
| 4             | 1000         | 0.1562            | 0.3667                         |  98.32                     |
| 4             | 1100         | 0.0723            | 0.3757                         | 158.90                     |
| 5             | 1200         | 0.0769            | 0.3789                         | 215.20                     |
| 5             | 1300         | 0.0814            | 0.3779                         | 170.50                     |
| 5             | 1400         | 0.0786            | 0.3770                         | 140.60                     |
| 5             | 1500         | 0.0673            | 0.3777                         | 137.10                     |
| 6             | 1600         | 0.0339            | 0.3892                         | 166.50                     |
| 7             | 1700         | 0.0324            | 0.3963                         | 170.90                     |
| 7             | 1800         | 0.0348            | 0.4004                         | 163.40                     |
| 8             | 1900         | 0.0345            | 0.4016                         | 158.60                     |
| 8             | 2000         | 0.0346            | 0.4020                         | 176.10                     |
| 8             | 2100         | 0.0317            | 0.4001                         | 134.70                     |
| 9             | 2200         | 0.0173            | 0.4141                         | 189.30                     |
| 9             | 2300         | 0.0174            | 0.4106                         | 175.00                     |
| 9             | 2400         | 0.0165            | 0.4204                         | 179.60                     |
| 10            | 2500         | 0.0172            | 0.4185                         | 186.10                     |
| 10            | 2600         | 0.0142            | 0.4175                         | 181.10                     |
| 11            | 2700         | 0.0090            | 0.4325                         | 161.70                     |
| 11            | 2800         | 0.0069            | 0.4362                         | 161.20                     |
| 11            | 2900         | 0.0093            | 0.4342                         | 157.50                     |
| 12            | 3000         | 0.0076            | 0.4352                         | 154.50                     |
| 12            | 3100         | 0.0089            | 0.4394                         | 184.30                     |
| 13            | 3200         | 0.0063            | 0.4454                         | 166.00                     |
| 13            | 3300         | 0.0059            | 0.4476                         | 179.20                     |
| 13            | 3400         | 0.0058            | 0.4490                         | 189.60                     |
| 14            | 3500         | 0.0051            | 0.4502                         | 194.20                     |
| 14            | 3600         | 0.0064            | 0.4512                         | 187.40                     |
| 14            | 3700         | 0.0053            | 0.4520                         | 190.20                     |
| 14            | 3800         | 0.0049            | 0.4545                         | 194.90                     |
| 15            | 3900         | 0.0052            | 0.4546                         | 199.60                     |
| 15            | 4000         | 0.0054            | 0.4549                         | 200.40                     |

### Framework versions
- Transformers 4.36.0.dev0
- Pytorch 2.1.0+cu118
- Datasets 2.15.0
- Tokenizers 0.15.0
{"language": ["it"], "license": "apache-2.0", "tags": ["hf-asr-leaderboard", "generated_from_trainer"], "datasets": ["mozilla-foundation/common_voice_11_0"], "metrics": ["wer"], "base_model": "openai/whisper-small", "model-index[0]": [{"name": "Whisper Small IT"}, {"name": "results", "results": [{"task": {"name": "Text Classification", "type": "text-classification"}, "dataset": {"name": "emotion", "type": "emotion", "args": "default"}, "metrics": [{"name": "Accuracy", "type": "accuracy", "value": 0.925}, {"name": "F1", "type": "f1", "value": 0.9251012149383893}]}]}]}
transformers tensorboard safetensors whisper automatic-speech-recognition hf-asr-leaderboard generated_from_trainer it dataset:mozilla-foundation/common_voice_11_0 arxiv:2006.11477 base_model:openai/whisper-small license:apache-2.0 endpoints_compatible has_space region:us