How to apply SpecAugment to a Whisper?
I’m trying to learn how I can apply SpecAugment when fine tuning a Whisper model. First of all, I’m not sure if applying such augmentation is common practice for fine tuning or not (maybe it’s only used for training?). And second, I don’t even know how to implement it.I see that PyTorch has alibraryimplementing the technique. But I guess I need to take into account the mask for each sample before using that library. And I don’t know how to do that.I appreciate any guidance I can get. Thanks.
2023-07-25T05:31:21Z
HI, I share the same question as well. Have you managed to sovle this issue?
2023-08-13T20:37:28Z
I’m not if this is the right way to do it but it seem to work:from transformers import WhisperForConditionalGeneration

model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-small")

model.config.forced_decoder_ids = None
model.config.suppress_tokens = []

model.config.apply_spec_augment = True
model.config.mask_time_prob = 0.05
model.config.mask_feature_prob = 0.05The last three lines enable the SpecAug for the Whisper model. The right values to set need to be researched.
2024-02-09T07:43:41Z
Hey@mehranhave you observed any noticeable improvements in performance metrics after applying the SpecAugment?