
Speech2Text
Overview
The Speech2Text model was proposed in fairseq S2T: Fast Speech-to-Text Modeling with fairseq by Changhan Wang, Yun Tang, Xutai Ma, Anne Wu, Dmytro Okhonko, Juan Pino. It's a
transformer-based seq2seq (encoder-decoder) model designed for end-to-end Automatic Speech Recognition (ASR) and Speech
Translation (ST). It uses a convolutional downsampler to reduce the length of speech inputs by 3/4th before they are
fed into the encoder. The model is trained with standard autoregressive cross-entropy loss and generates the
transcripts/translations autoregressively. Speech2Text has been fine-tuned on several datasets for ASR and ST:
LibriSpeech, CoVoST 2, MuST-C.
This model was contributed by valhalla. The original code can be found here.
Inference
Speech2Text is a speech model that accepts a float tensor of log-mel filter-bank features extracted from the speech
signal. It's a transformer-based seq2seq model, so the transcripts/translations are generated autoregressively. The
generate() method can be used for inference.
The [Speech2TextFeatureExtractor] class is responsible for extracting the log-mel filter-bank
features. The [Speech2TextProcessor] wraps [Speech2TextFeatureExtractor] and
[Speech2TextTokenizer] into a single instance to both extract the input features and decode the
predicted token ids.
The feature extractor depends on torchaudio and the tokenizer depends on sentencepiece so be sure to
install those packages before running the examples. You could either install those as extra speech dependencies with
pip install transformers"[speech, sentencepiece]" or install the packages separately with pip install torchaudio sentencepiece. Also torchaudio requires the development version of the libsndfile package which can be installed via a system package manager. On Ubuntu it can
be installed as follows: apt install libsndfile1-dev