What are the existing approaches for the lip-reading task?
The lip-reading task has attracted many researchers
(Cooke et al., 2006; Afouras et al., 2018a), aim-
ing to recognize spoken sentences according to
the given video of lip movements without relying
on the audio stream. With the support of a large
amount of visual speech utterances (Afouras et al.,
2018a,b), Assael et al. (2016) first proposes to use
neural networks for lip reading. (Ma et al., 2021)
and (Shi et al., 2022) adopt different pre-training
strategies, attempting to obtain fine-grained lip rep-
resentation by pre-training on a large amount of ad-
ditional unlabeled audio-visual utterances (Chung
et al., 2018). Some works (Makino et al., 2019;
Serdyuk et al., 2021) adopt massive labeled visual
utterances (even more than 90,000h) for training
and promote the generalization of the model. How-
ever, none of these methods can train lip-reading
models without large amounts of labeled visual ut-
terances, making them unusable in low-resource
domains where labeled visual utterances are scarce
or unavailable. This is also the original intention of
our work, for which we propose a training system
that can employ only labeled audio utterances to
train the lip-reading model.
VoxCeleb2: Deep Speaker Recognition
The objective of this paper is speaker recognition under noisy and unconstrained conditions. We make two key contributions. First, we introduce a very large-scale audio-visual speaker recognition dataset collected from open-source media. Using a fully automated pipeline, we curate VoxCeleb2 which contains over a million utterances from over 6,000 speakers. This is several times larger than any publicly available speaker recognition dataset. Second, we develop and compare Convolutional Neural Network (CNN) models and training strategies that can effectively recognise identities from voice under various conditions. The models trained on the VoxCeleb2 dataset surpass the performance of previous works on a benchmark dataset by a significant margin.
[{"first":"Joon","last":"Chung","middle":["Son"]},{"first":"Arsha","last":"Nagrani","middle":[]},{"first":"Andrew","last":"Zisserman","middle":[]}]
Recurrent Neural Network Transducer for Audio-Visual Speech Recognition
This work presents a large-scale audio-visual speech recognition system based on a recurrent neural network transducer (RNN-T) architecture. To support the development of such a system, we built a large audio-visual (A/V) dataset of segmented utterances extracted from YouTube public videos, leading to 31k hours of audio-visual training content. The performance of an audio-only, visual-only, and audio-visual system are compared on two large-vocabulary test sets: a set of utterance segments from public YouTube videos called YTDEV18 and the publicly available LRS3-TED set. To highlight the contribution of the visual modality, we also evaluated the performance of our system on the YTDEV18 set artificially corrupted with background noise and overlapping speech. To the best of our knowledge, our system significantly improves the state-of-the-art on the LRS3-TED set.
[{"first":"Takaki","last":"Makino","middle":[]},{"first":"Hank","last":"Liao","middle":[]},{"first":"Yannis","last":"Assael","middle":[]},{"first":"Brendan","last":"Shillingford","middle":[]},{"first":"Basilio","last":"Garcia","middle":[]},{"first":"Otavio","last":"Braga","middle":[]},{"first":"Olivier","last":"Siohan","middle":[]}]
Deep Audio-Visual Speech Recognition
The goal of this work is to recognise phrases and sentences being spoken by a talking face, with or without the audio. Unlike previous works that have focussed on recognising a limited number of words or phrases, we tackle lip reading as an open-world problem - unconstrained natural language sentences, and in the wild videos. Our key contributions are: (1) we compare two models for lip reading, one using a CTC loss, and the other using a sequence-to-sequence loss. Both models are built on top of the transformer self-attention architecture; (2) we investigate to what extent lip reading is complementary to audio speech recognition, especially when the audio signal is noisy; (3) we introduce and publicly release a new dataset for audio-visual speech recognition, LRS2-BBC, consisting of thousands of natural sentences from British television. The models that we train surpass the performance of all previous work on a lip reading benchmark dataset by a significant margin.
[{"first":"Triantafyllos","last":"Afouras","middle":[]},{"first":"Joon","last":"Chung","middle":["Son"]},{"first":"Andrew","last":"Senior","middle":[]},{"first":"Oriol","last":"Vinyals","middle":[]},{"first":"Andrew","last":"Zisserman","middle":[]}]
LipNet: End-to-End Sentence-level Lipreading
Lipreading is the task of decoding text from the movement of a speaker's mouth. Traditional approaches separated the problem into two stages: designing or learning visual features, and prediction. More recent deep lipreading approaches are end-to-end trainable (Wand et al., 2016; Chung&Zisserman, 2016a). However, existing work on models trained end-to-end perform only word classification, rather than sentence-level sequence prediction. Studies have shown that human lipreading performance increases for longer words (Easton&Basala, 1982), indicating the importance of features capturing temporal context in an ambiguous communication channel. Motivated by this observation, we present LipNet, a model that maps a variable-length sequence of video frames to text, making use of spatiotemporal convolutions, a recurrent network, and the connectionist temporal classification loss, trained entirely end-to-end. To the best of our knowledge, LipNet is the first end-to-end sentence-level lipreading model that simultaneously learns spatiotemporal visual features and a sequence model. On the GRID corpus, LipNet achieves 95.2% accuracy in sentence-level, overlapped speaker split task, outperforming experienced human lipreaders and the previous 86.4% word-level state-of-the-art accuracy (Gergen et al., 2016).
[{"first":"Yannis","last":"Assael","middle":["M."]},{"first":"Brendan","last":"Shillingford","middle":[]},{"first":"Shimon","last":"Whiteson","middle":[]},{"first":"Nando","last":"Freitas","middle":["de"]}]
LiRA: Learning Visual Speech Representations from Audio through Self-supervision
The large amount of audiovisual content being shared online today has drawn substantial attention to the prospect of audiovisual self-supervised learning. Recent works have focused on each of these modalities separately, while others have attempted to model both simultaneously in a cross-modal fashion. However, comparatively little attention has been given to leveraging one modality as a training objective to learn from the other. In this work, we propose Learning visual speech Representations from Audio via self-supervision (LiRA). Specifically, we train a ResNet+Conformer model to predict acoustic features from unlabelled visual speech. We find that this pre-trained model can be leveraged towards word-level and sentence-level lip-reading through feature extraction and fine-tuning experiments. We show that our approach significantly outperforms other self-supervised methods on the Lip Reading in the Wild (LRW) dataset and achieves state-of-the-art performance on Lip Reading Sentences 2 (LRS2) using only a fraction of the total labelled data.
[{"first":"Pingchuan","last":"Ma","middle":[]},{"first":"Rodrigo","last":"Mira","middle":[]},{"first":"Stavros","last":"Petridis","middle":[]},{"first":"Bjorn","last":"Schuller","middle":["W."]},{"first":"Maja","last":"Pantic","middle":[]}]
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
Video recordings of speech contain correlated audio and visual information, providing a strong signal for speech representation learning from the speaker's lip movements and the produced sound. We introduce Audio-Visual Hidden Unit BERT (AV-HuBERT), a self-supervised representation learning framework for audio-visual speech, which masks multi-stream video input and predicts automatically discovered and iteratively refined multimodal hidden units. AV-HuBERT learns powerful audio-visual speech representation benefiting both lip-reading and automatic speech recognition. On the largest public lip-reading benchmark LRS3 (433 hours), AV-HuBERT achieves 32.5% WER with only 30 hours of labeled data, outperforming the former state-of-the-art approach (33.6%) trained with a thousand times more transcribed video data (31K hours). The lip-reading WER is further reduced to 26.9% when using all 433 hours of labeled data from LRS3 and combined with self-training. Using our audio-visual representation on the same benchmark for audio-only speech recognition leads to a 40% relative WER reduction over the state-of-the-art performance (1.3% vs 2.3%). Our code and models are available at https://github.com/facebookresearch/av_hubert
[{"first":"Bowen","last":"Shi","middle":[]},{"first":"Wei-Ning","last":"Hsu","middle":[]},{"first":"Kushal","last":"Lakhotia","middle":[]},{"first":"Abdelrahman","last":"Mohamed","middle":[]}]
VoxCeleb2: Deep Speaker Recognition
Despite recent advances in the field of speaker recognition, producing single, compact representations for speaker segments that can be used efficiently under noisy and unconstrained conditions is still a significant challenge. In this paper, we present a deep CNN based neural speaker embedding system, named VGGVox, trained to map voice spectrograms to a compact Euclidean space where distances directly correspond to a measure of speaker similarity. Once such a space has been produced, other tasks such as speaker verification, clustering and diarisation can be straightforwardly implemented using standard techniques, with our embeddings as features.
VoxCeleb2: Deep Speaker Recognition
Such a mapping has been learnt effectively for face images, through the use of deep CNN architectures [1,2,3] trained on large-scale face datasets [4,5,6]. Unfortunately, speaker recognition still faces a dearth of large-scale freely available datasets in the wild. VoxCeleb1 [7] and SITW [8] are valuable contributions, however they are still an order of magnitude smaller than popular face datasets, which contain millions of images. To address this issue we curate VoxCeleb2, a large scale speaker recognition dataset obtained automatically from open-source media. VoxCeleb2 consists of over a million utterances from over 6k speakers. Since the dataset is collected 'in the wild', the speech segments are corrupted with real world noise including laughter, cross-talk, channel effects, music and other sounds. The dataset is also multilingual, with speech from speakers of 145 different nationalities, covering a wide range of accents, ages, ethnicities and languages. The dataset is audio-visual, so is also useful for a number of other applications, for example -visual speech synthesis [9,10], speech separation [11,12], † These authors contributed equally to this work. cross-modal transfer from face to voice or vice versa [13,14] and training face recognition from video to complement existing face recognition datasets [4,5,6]. Both audio and video for the dataset will be released.
VoxCeleb2: Deep Speaker Recognition
We train VGGVox on this dataset in order to learn speaker discriminative embeddings. Our system consists of three main variable parts: an underlying deep CNN trunk architecture, which is used to extract the features, a pooling method which is used to aggregate features to provide a single embedding for a given utterance, and a pairwise loss trained on the features to directly optimise the mapping itself. We experiment with both VGG-M [15] and ResNet [16] based trunk CNN architectures.
VoxCeleb2: Deep Speaker Recognition
We make the following four contributions: (i) we curate and release a large-scale dataset which is significantly larger than any other speaker verification dataset. It also addresses a lack of ethnic diversity in the VoxCeleb1 dataset (section 3); (ii) we propose deep ResNet-based architectures for speaker embedding suitable for spectrogram inputs (section 4); (iii) we beat the current state of the art for speaker verification on the VoxCeleb1 test set using our embeddings (section 5); and (iv) we propose and evaluate on a new verification benchmark test set which involves the entire VoxCeleb1 dataset.
VoxCeleb2: Deep Speaker Recognition
The VoxCeleb2 dataset can be downloaded from http: //www.robots.ox.ac.uk/˜vgg/data/voxceleb2.
VoxCeleb2: Deep Speaker Recognition
Traditional methods. Traditionally, the field of speaker recognition has been dominated by i-vectors [17], classified using techniques such as heavy-tailed PLDA [18] and Gauss-PLDA [19]. While defining the state-of-the-art for a long time, such methods are disadvantaged by their reliance on handcrafted feature engineering. An in-depth review of these traditional methods is given in [20]. Deep learning methods. The success of deep learning in computer vision and speech recognition has motivated the use of deep neural networks (DNN) as feature extractors combined with classifiers, though not trained end-to-end [21,22,23,24,25]. While such fusion methods are highly effective, they still require hand-crafted engineering. In contrast, CNN architectures can be applied directly to raw spectrograms and trained in an end-to-end manner. For example, [26] uses a Siamese feedforward DNN to discriminatively compare two voices, however this relies on pre-computed MFCC features, whilst [27] also learns the features instead of using MFCCs. The most relevant to our work is [28], who train a neural embedding system using the triplet loss. However, they use private internal datasets for both training and evaluation, and hence a direct comparison with their work is not possible. Datasets. Existing speaker recognition datasets usually suffer from one or more of the following limitations: (i) they are ei-  ther obtained under controlled conditions (e.g., from telephone calls [29,30] or acoustic laboratories [31,32,33]), (ii) they are manually annotated and hence limited in size [8], or (iii) not freely available to the speaker community [34,33] (see [7] for a full review of existing datasets). In contrast, the VoxCeleb2 dataset does not suffer from any of these limitations.
VoxCeleb2: Deep Speaker Recognition
VoxCeleb2 contains over 1 million utterances for over 6,000 celebrities, extracted from videos uploaded to YouTube. The dataset is fairly gender balanced, with 61% of the speakers male. The speakers span a wide range of different ethnicities, accents, professions and ages. Videos included in the dataset are shot in a large number of challenging visual and auditory environments. These include interviews from red carpets, outdoor stadiums and quiet indoor studios, speeches given to large audiences, excerpts from professionally shot multimedia, and even crude videos shot on hand-held devices. Audio segments present in the dataset are degraded with background chatter, laughter, overlapping speech and varying room acoustics. We also provide face detections and face-tracks for the speakers in the dataset, and the face images are similarly 'in the wild', with variations in pose (including profiles), lighting, image quality and motion blur. Table 1  The dataset contains both development (train/val) and test sets. However, since we use the VoxCeleb1 dataset for testing, only the development set will be used for the speaker recognition task (Sections 4 and 5). The VoxCeleb2 test set should prove useful for other applications of audio-visual learning for which the dataset might be used. The split is given in Table 2. The development set of VoxCeleb2 has no overlap with the identities in the VoxCeleb1 or SITW datasets.
VoxCeleb2: Deep Speaker Recognition
We use an automatic computer vision pipeline to curate VoxCeleb2. While the pipeline is similar to that used to compile VoxCeleb1 [7], the details have been modified to increase efficiency and allow talking faces to be recognised from multiple poses, not only near-frontal. In fact, we change the implementation of every key component of the pipeline: the face detector, the face tracker, the SyncNet model used to perform active speaker verification, and the final face recognition model at the end. We also add an additional step for automatic dupli-cate removal. This pipeline allows us to obtain a dataset that is five times the size of [7]. We also note that the list of celebrity names spans a wider range of nationalities, and hence unlike [7], the dataset obtained is multi-lingual. For the sake of clarity, the key stages are discussed in the following paragraphs: Stage 1. Candidate list of Persons of Interest (POIs). The first stage is to obtain a list of POIs. We start from the list of people that appear in the VGGFace2 dataset [4], which has considerable ethnic diversity and diversity in profession. This list contains over 9,000 identities, ranging from actors and sportspeople to politicians. Identities that overlap with those of VoxCeleb1 and SITW are removed from the development set. Stage 2. Downloading videos. The top 100 videos for each of the POIs are automatically downloaded using YouTube search. The word 'interview' is appended to the name of the POI in search queries to increase the likelihood that the videos contain an instance of the POI speaking, as opposed to sports or music videos. Stage 3. Face tracking. The CNN face detector based on the Single Shot MultiBox Detector (SSD) [35] is used to detect face appearances on every frame of the video. This detector is a distinct improvement from that used in [7], allowing the detection of faces in profile and extreme poses. We used the same tracker as [7] based on ROI overlap. Stage 4. Face verification. A face recognition CNN is used to classify the face tracks into whether they are of the POI or not. The classification network used here is based on the ResNet-50 [16] trained on the VGGFace2 dataset. Verification is done by directly using this classification score. Stage 5. Active speaker verification. The goal of this stage is to determine if the visible face is the speaker. This is done by using a multi-view adaptation [36] of 'SyncNet' [37,38], a twostream CNN which determines the active speaker by estimating the correlation between the audio track and the mouth motion of the video. The method can reject clips that contain dubbing or voice-over. Stage 6. Duplicate removal. A caveat of using YouTube as a source for videos is that often the same video (or a section of a video) can be uploaded twice, albeit with different URLs. Duplicates are identified and removed as follows: each speech segment is represented by a 1024D vector using the model in [7] as a feature extractor. The Euclidean distance is computed between all pairs of features from the same speaker. If any two speech segments have a distance smaller than a very conservative threshold (of 0.1), then the the speech segments are deemed to be identical, and one is removed. This method will certainly identify all exact duplicates, and in practice we find that it also succeeds in identifying near-duplicates, e.g. speech segments of the same source that are differently trimmed. Stage 7. Obtaining nationality labels. Nationality labels are crawled from Wikipedia for all the celebrities in the dataset. We crawl for country of citizenship, and not ethnicity, as this is often more indicative of accent. In total, nationality labels are obtained for all but 428 speakers, who were labelled as unknown. Speakers in the dataset were found to hail from 145 nationalities (compared to 36 for VoxCeleb1), yielding a far more ethnically diverse dataset (See Figure 1 (bottom, right) for the distribution of nationalities). Note also the percentage of U.S. speakers is smaller in VoxCeleb2 (29%) compared to VoxCeleb1 (64%) where it dominates. Discussion. In order to ensure that our system is extremely confident that a person has been correctly identified (Stage 4), and that are speaking (Stage 5) without any manual interference, we set conservative thresholds in order to minimise the number of false positives. Since VoxCeleb2 is designed primarily as a training-only dataset, the thresholds are less strict compared to those used to compile VoxCeleb1, so that fewer videos are discarded. Despite this, we have only found very few label errors after manual inspection of a significant subset of the dataset.
VoxCeleb2: Deep Speaker Recognition
In this section we describe our neural embedding system, called VGGVox. The system is trained on short-term magnitude spectrograms extracted directly from raw audio segments, with no other pre-processing. A deep neural network trunk architecture is used to extract frame level features, which are pooled to obtain utterance-level speaker embeddings. The entire model is then trained using contrastive loss. Pre-training using a softmax layer and cross-entropy over a fixed list of speakers improves model performance; hence we pre-train the trunk architecture model for the task of identification first.
VoxCeleb2: Deep Speaker Recognition
The model is trained on the VoxCeleb2 dataset. At train time, pairs are sampled on-line using the method described in Section 4.3. The testing is done on the VoxCeleb1 dataset, with the test pairs provided in that dataset. We report two performance metrics: (i) the Equal Error Rate (EER) which is the rate at which both acceptance and rejection errors are equal; and (ii) the cost function
VoxCeleb2: Deep Speaker Recognition
where we assume a prior target probability Ptar of 0.01 and equal weights of 1.0 between misses Cmiss and false alarms C f a . Both metrics are commonly used for evaluating identity verification systems.
VoxCeleb2: Deep Speaker Recognition
The baseline trunk architecture is the CNN introduced in [7]. This architecture is a modification of the VGG-M [15] CNN, known for high efficiency and good classification performance on image data. In particular, the fully connected fc6 layer from the original VGG-M is replaced by two layersa fully connected layer of 9 × 1 (support in the frequency domain), and an average pool layer with support 1 × n, where n depends on the length of the input speech segment (for example for a 3 second segment, n = 8). The benefit of this modification is that the network becomes invariant to temporal position but not frequency, which is desirable for speech, but not for images. It also helps to keep the output dimensions the same as those of the original fully connected layer, and reduces the number of network parameters by fivefold. ResNets: The residual-network (ResNet) architecture [16] is similar to a standard multi-layer CNN, but with added skip connections such that the layers add residuals to an identity mapping on the channel outputs. We experiment with both ResNet-34 and ResNet-50 architectures, and modify the layers to adapt to the spectrogram input. We apply batch normalisation before computing rectified linear unit (ReLU) activations. The architectures are specified in Table 3.  
VoxCeleb2: Deep Speaker Recognition
We employ a contrastive loss [39,40] on paired embeddings, which seeks to minimise the distance between the embeddings of positive pairs and penalises the negative pair distances for being smaller than a margin parameter α. Pair-wise losses such as the contrastive loss are notoriously difficult to train [41], and hence to avoid suboptimal local minima early on in training, we proceed in two stages: first, pre-training for identifcation using a softmax loss, then, second, fine-tuning with the contrastive loss.
VoxCeleb2: Deep Speaker Recognition
Pre-training for identification: Our first strategy is to use softmax pre-training to initialise the weights of the network. The cross entropy loss produces more stable convergence than the contrastive loss, possibly because softmax training is not impacted by the difficulty of pairs when using the contrastive loss.
VoxCeleb2: Deep Speaker Recognition
To evaluate the identification performance, we create a held-out validation test which consists of all the speech segments from a single video for each identity.
VoxCeleb2: Deep Speaker Recognition
Learning an embedding with contrastive loss -hard negative mining: We take the model pre-trained on the identification task, and replace the 5994-way classification layer with a fully connected layer of output dimension 512. This network is trained with contrastive loss.
VoxCeleb2: Deep Speaker Recognition
A key challenge associated with learning embeddings via the contrastive loss is that as the dataset gets larger, the number of possible pairs grows quadratically. In such a scenario, the network rapidly learns to correctly map the easy examples, and hard negative mining is often required to improve performance to provide the network with a more useful learning signal. We use an offline hard negative mining strategy, which allows us to select harder negatives (e.g. top 1-percent of randomly generated pairs) than is possible with online (in-batch) hard negative mining methods [42,41,43] limited by the batch size. We do not mine hard positives, since false positive pairs are much more likely to occur than false negative pairs in a random sample (due to possible label noise on the face verification), and these label errors will lead to poor learning dynamics. [7] use average pooling at test time by evaluating the entire test utterance at once by changing the size of the apool6 layer. Here, we experiment with different augmentation protocols for evaluating the performance at test time. We propose three methods: (1) Baseline: variable average pooling as described in [7];
VoxCeleb2: Deep Speaker Recognition
(2) Sample ten 3-second temporal crops from each test segment, and use the mean of the features;
VoxCeleb2: Deep Speaker Recognition
(3) Sample ten 3-second temporal crops from each test segment, compute the distances between the every possible pair of crops (10 × 10 = 100) from the two speech segments, and use the mean of the 100 distances. The method results in a marginal improvement in performance, as shown in Table 4.
VoxCeleb2: Deep Speaker Recognition
Input features. Spectrograms are computed from raw audio in a sliding window fashion using a hamming window of width 25ms and step 10ms, in exactly the same manner as [7]. This gives spectrograms of size 512 x 300 for 3 seconds of speech. Mean and variance normalisation is performed on every frequency bin of the spectrum. Training. During training, we randomly sample 3-second segments from each utterance. Our implementation is based on the deep learning toolbox MatConvNet [44]. Each network is trained on three Titan X GPUs for 30 epochs or until the validation error stops decreasing, whichever is sooner, using a batch-size of 64. We use SGD with momentum (0.9), weight decay (5E − 4) and a logarithmically decaying learning rate (initialised to 10 −2 and decaying to 10 −8 ).
VoxCeleb2: Deep Speaker Recognition
Original VoxCeleb1 test set. Table 4 provides the performance of our models on the original VoxCeleb1 test set.
VoxCeleb2: Deep Speaker Recognition
As might be expected, performance improves with greater network depth, and also with more training data (VoxCeleb2 vs VoxCeleb1). This also demonstrates that VoxCeleb2 provides a suitable training regime for use on other datasets.
VoxCeleb2: Deep Speaker Recognition
Trained on C min det EER (%) I-vectors + PLDA (1)   New VoxCeleb1-E test set -using the entire dataset. Popular speaker verification test sets in the wild [7,8] are limited in the number of speakers. This yields the possible danger of optimising performance to overfit the small number of speakers in the test set, and results are not always indicative of good generalised performance. Hence we propose a new evaluation protocol consisting of 581,480 random pairs sampled from the entire VoxCeleb1 dataset, covering 1,251 speakers, and set benchmark performance for this test set. The result is given in Table 5.
VoxCeleb2: Deep Speaker Recognition
New VoxCeleb1-H test set -within the same nationality and gender. By using the whole of VoxCeleb1 as a test set, we are able to test only on the pairs with same nationality and gender. We propose a new evaluation list consisting of 552,536 pairs sampled from the VoxCeleb1 dataset, all of which are from the same nationality and gender. 18 nationality-gender combinations each with at least 5 individuals are used to generate this list, of which 'USA-Male' is the most common. The result is given in Table 5.
VoxCeleb2: Deep Speaker Recognition
Tested on C min det EER (%) ResNet-50 (3) VoxCeleb1-E 0.524 4.42
VoxCeleb2: Deep Speaker Recognition
VoxCeleb1-H 0.673 7.33 Table 5: Results for verification on the extended VoxCeleb1 test sets.
VoxCeleb2: Deep Speaker Recognition
In this paper, we have introduced new architectures and training strategies for the task of speaker verification, and demonstrated state-of-the-art performance on the VoxCeleb1 dataset. Our learnt identity embeddings are compact (512D) and hence easy to store and useful for other tasks such as diarisation and retrieval. We have also introduced the VoxCeleb2 dataset, which is several times larger than any speaker recognition dataset, and have re-purposed the VoxCeleb1 dataset, so that the entire dataset of 1,251 speakers can be used as a test set for speaker verification. Choosing pairs from all speakers allows a better assessment of performance than from the 40 speakers of the original test set. We hope that this new test set will be adopted, alongside SITW, as a standard for the speech community to evaluate on.
Recurrent Neural Network Transducer for Audio-Visual Speech Recognition
While the performance of automatic speech recognition (ASR) systems has significantly improved over the past several years, outstanding challenges remain for ubiquitous ASR. In particular, state-of-the-art recognizers can fail in noisy environments or in presence of overlapping speech. In some applications such as auto-captioning of videos, multiple modalities are available when transcribing speech. The visual signal can supply the spelling of obscure names and terms when shown on-screen, or provide contextual information related to the visual scene [1]. Further, the motion of the lips constrains the possible phonemes and hence words that can be spoken. With the availability of sophisticated neural network architectures and large amounts of multimedia data, it is intriguing to explore how audio and visual modalities can be combined to yield improvements not otherwise available in a unimodal setting.
Recurrent Neural Network Transducer for Audio-Visual Speech Recognition
Our work is motivated by the recent success in automatic lip reading by [2] and audio-visual speech recognition in [3,4]. In the former, a state-of-the-art Vision-to-Phoneme model (V2P) is developed using a carefully and unprecedentedlysized visual corpus of speaking faces with their corresponding text transcripts extracted from YouTube public videos. A large VGG-inspired neural network was used for extracting visual features, with a connectionist temporal classification (CTC) based system [5] to predict phonemes. In the latter, using a large audio-visual data set with captions derived mainly from British television, the authors were able to use transformer-based self-attention network blocks [6] to model the audio and visual modalities with a sequence-to-sequence loss [7]. With their approach called TM-seq2seq, they demonstrate that combining audio and visual features yields better performance than audio-only models on a large vocabulary recognition task of transcribing TED talk videos, released as a data set called LRS3-TED [8].
Recurrent Neural Network Transducer for Audio-Visual Speech Recognition
As for most machine learning tasks, the performance of an AV-ASR system is driven by the availability of high-quality training datasets. In this work, we first use the approach described in [9] to mine a 150k hours, audio-only speech recognition corpus from YouTube with labels derived from user-uploaded captions. Next, we filter this dataset using face tracking technology similar to the work in [8,10] to select segmented utterances with a matching on-screen speaking face, leading to a 31k hours audio-visual corpus. Similar to [10], we apply face landmark smoothing to stabilize the face thumbnails. But in contrast to previous AV-ASR work, we synchronize audio to video frames by adjusting the audio feature frame rate while maintaining a fixed short-term Fourier transform analysis window.
Recurrent Neural Network Transducer for Audio-Visual Speech Recognition
In terms of modeling, as observed in [3,11], a weakness of CTC modeling as used in V2P is that the network outputs are conditionally independent of each other and require an external language model, such as a N-gram Finite State Transducer to perform well. In contrast, using a sequence-tosequence loss allows the language model to be learned implicitly as part of the decoder. We advocate using the recurrent neural network transducer (RNN-T), first described in [12], as a more natural model for speech recognition. The RNN-T consists of two components, a transcription or encoder network and a recurrent prediction or decoder network. The encoder can be viewed as the acoustic or visual model, with in our case, inputs being mel-filterbank coefficients [13] or visual embeddings derived using a VGG inspired network [14] as 3-D convolutions in V2P. The decoder network models the output of the current symbol given the previous output symbol, in our case character-level outputs. The encoder and decoder outputs are fed into a joint network where the output is normalized to yield a softmax over characters. We believe that the RNN-T is a better model for speech since the output dependencies tend to be local and the use of LSTM layers for the encoder and decoder better models this locality. Compared to sequence-to-sequence models, the use of attention may be less desirable since the model spends capacity learning that the most important features for the output are the nearest input frames. In other words, for speech recognition the alignment of input features to output symbols is monotonic. The use of character level outputs, rather than phonemes, provides a simple way of achieving an open-vocabulary system.
Recurrent Neural Network Transducer for Audio-Visual Speech Recognition
We evaluate our approach on a general YouTube transcription task for audio, video, and audio-visual speech recognition. To illustrate the impact of the visual modality on ASR performance, we artificially corrupted our evaluation dataset by adding either babble noise at various signal-to-noise ratios (SNR), or a few seconds of overlapping speech at the beginning or end of each evaluation utterance. We also provide experimental results on the publicly available large-vocabulary benchmark LRS3-TED [8] and report, to the best of our knowledge, state-of-the-art results on that set.
Recurrent Neural Network Transducer for Audio-Visual Speech Recognition
The paper is organized as follows. In Section 2 we describe the way we construct our training set and present our test sets. Section 3 focuses on the architecture of our system, starting with our process to generate synchronized audio and visual features, followed by our RNN-T approach that operates on either audio, visual, or AV features. Experiments are discussed in Section 4, Section 5 highlights the AI Principles followed in this work, and Section 6 concludes the paper.
Recurrent Neural Network Transducer for Audio-Visual Speech Recognition
The impact of the visual modality on human speech perception has been documented as early as 1954 [15]. In particular, the place of articulation can provide valuable information to help differentiate acoustically confusable sounds [16]. As a result, and as highlighted in several reviews of AV-ASR technology [17,18], attempts were made in the early 90's to combine audio and visual modalities for ASR. Until a few years ago, the large majority of the publicly available AV datasets suitable for AV-ASR were limited in size and scope, consisting mostly of tens of hours of spoken digits or command-andcontrol word sequences with a limited number of speakers recorded in a laboratory environment under controlled lighting conditions and face orientation [18].
Recurrent Neural Network Transducer for Audio-Visual Speech Recognition
In 2016, researchers proposed a novel end-to-end trained model for visual speech recognition [19] and approach to construct large AV datasets suitable for AV-ASR [20]. Starting from TV shows and their corresponding closed-captions provided by the BBC, they applied computer vision techniques to select short audio segments with an on-screen speaking face matching the audio. In subsequent studies [21,22], they extended their approach to a larger amount of data, leading to the LRS2 & LRS3 datasets consisting of videos from the BBC and TED-talks, respectively, for a total of 800 hours of transcribed AV content. More recently, applying this method to public videos from YouTube lead to an AV-ASR dataset consisting of 3.9k hours of videos [2].
Recurrent Neural Network Transducer for Audio-Visual Speech Recognition
In this paper, we expand the approach of [2] to construct a significantly larger dataset. As in [2], we build on the semisupervised approach described in [9] to mine a large audio dataset from YouTube videos. When YouTube users and content creators upload a public video on YouTube, they have the option of uploading subtitles alongside their video. A forced alignment procedure is then used to synchronize the useruploaded transcripts with the audio to generate time-aligned captions. We take advantage of user-uploaded captions to automatically mine our AV dataset. First, we run ASR on the uploaded video and string-align the ASR transcripts with the user-uploaded transcripts. Audio segments where the ASR and user-uploaded transcripts agree are selected as training utterances, treating the user-uploaded transcripts as groundtruth reference. Because of the scale of YouTube, this leads to the construction of a large audio training set, in the order of 150k hours of data for American English. Next, we extract the video snippets corresponding to the selected utterances and run face tracking to locate all on-screen faces [23,24]. A visual speech classifier is used to identify the speaking face, if any, that spans each audio segment, followed by an audiovisual synchrony detector to reject dubbed videos.
Recurrent Neural Network Transducer for Audio-Visual Speech Recognition
The result of that process is a collection of short utterances (from a few seconds to tens of seconds long) totaling 31k hours of data, where with high confidence, the audio matches the user-uploaded transcript, and the selected face video track matches the audio. This exceeds the amount of training data used for training the V2P model of [2] by a factor of 5, and the TM-seq2seq model of [7] by a factor of 10. The size of our training set is mitigated by the fact that we do not augment the training data by perturbing the audio or image data. We believe the main reason we obtain much more data than in the V2P work is due to accepting a greater range of face pan and tilt angles, i.e. more than +/-30 degrees. Unlike the LRW/LRS2/LRS3 datasets that are restricted to professionally generated video content, our dataset spans a much greater variety of content of speaking faces in the wild. We use this dataset to train our unimodal audio and video systems, as well as our AV ASR system, and extract 70 hours of data as development set to tune our models.
Recurrent Neural Network Transducer for Audio-Visual Speech Recognition
Unlike our training set that is automatically constructed from videos with user-uploaded captions, we built our evaluation set from a collection of manually transcribed YouTube videos. Starting from a set of 1000 hours of transcribed videos, we applied the same face tracking process as in training to select a collection of utterances with a matching onscreen speaking face. Out of that process, we retained a set of 20k segmented utterances, with their corresponding manual reference transcripts and face tracks, called YTDEV18 and totaling 25 hours of data. We checked the video IDs to confirm that the videos used in the YTDEV18 and LRS3-TED evaluation sets are not included in the training data.
Recurrent Neural Network Transducer for Audio-Visual Speech Recognition
To study the impact of the visual modality on recognition performance, we artificially corrupt the YTDEV18 utterances in two ways. First, we add babble noise randomly selected from the NoiseX [25] dataset at 0dB, 10dB, and 20dB SNR to each utterance. Second, we add overlapping speech from a competing speaker at the beginning/end of each utterance at an equal energy level.
Recurrent Neural Network Transducer for Audio-Visual Speech Recognition
A challenge in audio-visual speech recognition is dealing with the difference in data rates between audio and visual features. In this work we start with audio sampled and mixed down to a 16kHz, single-channel signal using ffmpeg. A 25ms Hanning window with 10 ms shift is used to compute a short-time Fourier transform for a spectral representation of the audio. A mel-spaced bank of 80 triangular filters is applied followed by a log function to yield mel filterbank coefficients [13]. These are stacked 2 frames to the left and 2 to the right to yield a 400 dimensional feature vector, while only keeping one in three frames similar to [26], for a final frame rate of 33 1 /3 fps.
Recurrent Neural Network Transducer for Audio-Visual Speech Recognition
Because our data is heterogeneous in terms of video standards and frame rates, we follow an approach similar to [2] and downsample high frame rates down to a maximum of 30 fps, retaining only videos between 23 fps to 30 fps, so that 23.98/24 fps (cinemas), 25 fps (PAL/SECAM), 29.97 fps (NTSC) and 30 fps videos are kept. FaceNet [24] is applied to detect and track faces. Face landmarks are then extracted from these tracks and smoothed using a temporal Gaussian kernel. A difference from the V2P data processing pipeline is that we do not filter out segments with extreme face pan or tilt angles to train a more robust model, however we do enforce a minimum eye distance of 80 pixels. Crops around the mouth are then extracted as 128x128 RGB pixel thumbnails.
Recurrent Neural Network Transducer for Audio-Visual Speech Recognition
Variable video frame rates typically do not match the audio frame rate leading to unsynchronized audio and visual features as illustrated in Figure 1. In [3], the video is downsampled to 25 fps and the audio feature stacked and sampled to match this frame rate to yield synchronized frame rates. We believe that the down-sampling of the video stream may degrade the visual features and instead propose to operate on the variable video frame rate and extract audio features at a matching rate. Since the original audio has a high sampling rate, the 25 ms STFT window can be advanced at a rate proportional the video frame rate, here 1 /3 of the video frame length, as shown in Figure 2. This provides an alternate means of getting synchronized audio-visual features without changing the visual features. In Section 4.3, we illustrate the difference in performance between extracting features at the proposed variable frame rate vs. fixed frame rate.
Recurrent Neural Network Transducer for Audio-Visual Speech Recognition
Audio features are derived as discussed in the previous sec- volutional filters as the features used in V2P [10]. The number of filters increases from 64 in the first block, to 128, 256, 512 and 512 in the final layer. After each convolutional layer, before pooling, group norm [27] is applied with 32 groups, over the spatial dimensions, but not the temporal, which gives a small improvement. The output of the video model yields an embedding vector of 512 coefficients for every input frame. The video model has 11.7M parameters. A detailed description of the video model can be found in [2]. To create a visual transcription model, the 512 dimensional output is fed into the encoder by turning on the 'video switch' in Figure 3.
Recurrent Neural Network Transducer for Audio-Visual Speech Recognition
For an A/V model, the audio and visual features are enabled and concatenated to form a 912-dimensional feature vector that is fed into the encoder as shown in Figure 3. A summary of the parameters, allocated to the various parts of the network for the full A/V model, is shown in Table 1 minus any temporary variables used only in training, e.g. moment estimates. For simplicity, the shape information listed is for the main kernel of the layer only and for example does not list the gamma and beta terms from layer norm, however the parameter count includes them. Layer normalization [28] is used in all the LSTM layers to make training more stable.
Recurrent Neural Network Transducer for Audio-Visual Speech Recognition
To train the models in this work, we use TensorFlow to implement our neural network models. An Adam optimizer [29] is used for training. Unless otherwise noted, we used a learning rate schedule warming up linearly to 2 × 10 −3 in 20k steps, staying constant for 50k steps, before exponentially decaying. We train our models on tensor processing units (TPUs) with 128 cores and a batch size of 16 sent to each core for an effective batch size per step of 2048 utterances. For all results, we select the model checkpoint that produces the best result on a held-out validation shard of the training data. We present results first on a selection of YouTube utterances where we  Table 1. Audio-visual RNN-T model architecture.
Recurrent Neural Network Transducer for Audio-Visual Speech Recognition
found they had visual tracks that covered most of the speech in the segment and were well synchronized. This set of utterances is collected from over 1000 videos totaling over 20k utterances and called YTDEV18. We then present results on the publicly available LRS3-TED test set which contains about 10k words and 1436 utterances. The results are measured in word error rates (WER) of the output of frame-synchronized beam decoder [30], where at most 4 highest scoring candidates are retained at every step during decoding. CI denotes the half width of the 95% confidence intervals based on [31].
Recurrent Neural Network Transducer for Audio-Visual Speech Recognition
Training an audio-visual model can be difficult since the speech signal is far more informative than the visual. However, our results in Table 2 show that an A/V RNN-T model can be trained from scratch on A/V data without pre-training any components. On the 31k training set, an audio-only sytem can be trained that yields a 21.5% word error rate (WER) on YTDEV18. In comparison, the video-only system gives a 48.5% error rate. In combination, an A/V system trained on both modalities, leads to a better system at 20.5%.
Recurrent Neural Network Transducer for Audio-Visual Speech Recognition
Train & Eval Mode WER (%) ± 95% CI Audio 21.5 ± 0.5 YTDEV18 Visual 48.5 ± 0.6 Audio+Visual 20.5 ± 0.5 Table 2. Results on YTDEV18 where the training modalities match the test.
Recurrent Neural Network Transducer for Audio-Visual Speech Recognition
Our best results so far have been with an A/V model trained on both audio and visual data. However it is interesting to see how such system behaves when one of these modes are missing. We evaluated our ASR system that trained from both audio and visual modalities with unimodal test data. As shown  Table 3. Results with A+V models trained with different drop-out rates and tested on audio-only (A), visual-only (V) and audio-visual (A+V) modes. All values are WER (%) ± 95% CI.
Recurrent Neural Network Transducer for Audio-Visual Speech Recognition
in the first row of results in Table 3, when the A/V trained model is evaluated on a audio-only version of the test set, we found that the performance degrades from 20.5% to 24.0% demonstrating that the model has learned to depend on both modes for recognition. More surprising, is that when evaluated on a visual-only version of the test set, the results become extremely poor at 98.8%. In real-world data, both modalities may not always be present so it may be prudent to improve results on unimodal data for robustness.
Recurrent Neural Network Transducer for Audio-Visual Speech Recognition
To boost results with missing modalities, we simulate the effect during training by randomly dropping out a modality at the sentence level as in [32]. As shown in Table 3, the first row of results indicates when no drop-out is applied. In the second row, there is a 30% chance the audio is dropped during training. This makes the model learn to use the video signal almost as well as the visual-only system: 50.3% compared to 48.5% and the overall A/V system is better at 19.8%. The cost is the results on only audio-only data are much worse, now at 46%. To improve the audio-only results, we also tried training with a 30% chance of audio being dropped, 10% chance of video being dropped, or neither. For this model with audio and video drop-out, we use a different learning rate schedule (4 × 10 −3 and kept constant until 100k steps) to compensate for slower learning. This did not have an appreciable affect on the audio-only and A/V results but surprisingly did harm the visual-only results by 10% relative. Thus, for the rest of paper we report results with 30% audio dropout, but no visual drop-out. Overall, these results suggest that it was easier for the network to learn from audio modality than from visual modality, unless it is forced by drop-out during training. They also indicate that the current strategy of modality fusion through the concatenation of features is not robust to these different input conditions.
Recurrent Neural Network Transducer for Audio-Visual Speech Recognition
We compared the performance with variable frame rate to those of the fixed frame rates. For fixed frame rates, we chose 33 1 /3 frames-per-second, to match a conventional 10ms×3 audio frame system; 30 fps since about half of the videos are 30 or 29.97 fps; and 24 fps to avoid frame duplication. Table 4 shows the results with different frame rate conditions. The difference between these frame rate conditions are not significant, as with a different learning rate schedule we  Table 4. Results on YTDEV18 with different frame rate conditions. All values are WER (%) ± 95% CI.
Recurrent Neural Network Transducer for Audio-Visual Speech Recognition
observed that fixed 30fps model achieved slightly better accuracy than variable frame rate model. We can conclude, using a variable frame rate is as effective as using fixed frame rates with an RNN-T model. Table 5 presents results on noisy datasets. In all cases, the models are tested with the same set of modalities as used in training. We find that there is about a 5% relative improvement from having an audio-visual model over an audio-only on the YTDEV18 test set without added noise. We can see that the advantage of the multimodal model is greater on the noisier versions of the test set. The visual-only model performs the same regardless of added noise, which in the case of the severe 0dB babble noise conditions, is better than either using audio-only or audio-visual systems. This is not entirely unexpected because there is no added noise in our audio training data. We can artificially add overlapping speech to the audio training data by randomly selecting an utterance from the training set as a form of multi-style training [33]. We do   Table 6. Results on noisy versions of YTDEV18 for audio (A), visual (V) and audio-visual (A+V) trained models where overlapping speech is added to the training audio 10% of the time. All values are WER(%) ± 95% CI.. so with a 10% chance at a level between 0-20dB randomly selected over a uniform distribution. In Table 6, we report results with models trained in such a manner. The audio-only model shows improvement from the overlap training across all the noise condidtions. The audio-visual model shows even greater relative improvements, especially on the 0db babble and overlapping speech tests. We believe that providing the visual signal allows for the A/V model to better ignore competing speech, similar to speaker seperation/speech enhancement with visual features [34,35].
Recurrent Neural Network Transducer for Audio-Visual Speech Recognition
To provide a reference on how our models work on a publicly available data set, we report results on the LRS3-TED dataset, using our models trained only on the YouTube 31khr data set, in Table 7 along with some previously published results. The CTC-V2P system in [2] was shown to yield better transcription results than professional human lipreaders on YouTube video and despite not training on the available training data had a word error rate of 55.1% on this test set. Another comparison is the TM-seq2seq model from [3]. TM-seq2seq was shown to better than CTC and demonstrated how combining audio and video could yield improved results on a large-vocabulary task. Our system, using far more data and a RNN-T model makes significant improvement over both of these systems, achieving a 33.6% word error rate training for video-only training and testing, and 4.5% for a combined A/V system. The improvement in using both audio and visual modalities over audio holds for this test set as well. 
Recurrent Neural Network Transducer for Audio-Visual Speech Recognition
The development of Audio-Visual ASR technology, and more specifically lip-reading, raises issues related to privacy, especially in light of the performance of our system on the LRS3-TED dataset when using the visual modality only. It should be noted that this level of performance relates to the nature of the LRS3-TED task which involves professionally produced content under good lighting conditions (no shadows), high video frame-rate (≥ 24 fps), and a cooperative speaker facing the camera. This is illustrated by the significantly higher visual-only WER on the YTDEV18 set (48.5%), which reflects "in-the-wild" conditions, compared to the LRS3-TED set (33.6%), which corresponds to studio-quality recordings. We also evaluated the performance of our video-only system as a function of the image quality by labeling utterances from the YTDEV18 set as "high quality" when the face had a minimum eye distance of 80 pixels, a bounding box diagonal of 300 pixels, a maximum absolute pan angle of 30 degrees and a tilt angle less than 10 degrees. Everything else was labeled as "low quality", with 70% of the utterances falling in that category. The visual-only WER on the low quality utterances was 57.0%, to be compared with 37.1% on the high quality utterances. We would expect an even greater degradation in performance on video streams originating from low resolution and low frame rate devices such as CCTV cameras.
Recurrent Neural Network Transducer for Audio-Visual Speech Recognition
We are aware of the risk-benefit trade-off for use of this technology and our work abides by Google AI Principles [36]. We are hoping that this work, by improving the robustness of speech recognition systems, will increase the reach of ASR technology to a larger population of users, as well as the development of assistive technology. One area of interest is enabling people with impaired speech, such as for example people suffering from Lou Gehrig's disease, to continue operating speech-enabled devices and equipment by relying more on the visual modality.
Recurrent Neural Network Transducer for Audio-Visual Speech Recognition
Last, it should be noted that the data and models developed in this work are restricted to a small group of researchers working on this project and are handled in compliance with the European Union General Data Protection Regulation [37].
Recurrent Neural Network Transducer for Audio-Visual Speech Recognition
In this paper, we presented an RNN-T based speech recognition system operating on audio-visual content. We adopted a fully automated approach to mine a large audio-visual training set out of YouTube public videos, relying on advances in ASR and computer vision to select utterances with an onscreen speaking face, its corresponding audio signal and its matching user-uploaded captions. We used a V2P frontend to extract visual features from thumbnails located on the speaker mouth region and enforced extracting audio features at the same frame rate as the visual features. This enabled concatenating the audio and visual features at the input of the RNN-T encoder, leading to a similar modeling architecture for our audio, visual, and AV system. We illustrated that our AV system slightly improves over an audio-only system when trained on the same amount of training data, but leads to significant performance improvement in presence of babble noise or overlapping speech. We also described how the use of such a large training set leads to state-of-the-art performance on the publicly available LRS3-TED set. Future work will focus on comparing the performance of our system with other alternative approaches such as attention-based and transformed-based models and in exploring the use of modality-imbalanced training sets.
Deep Audio-Visual Speech Recognition
L IP READING, the ability to recognize what is being said from visual information alone, is an impressive skill, and very challenging for a novice. It is inherently ambiguous at the word level due to homophones -different characters that produce exactly the same lip sequence (e.g. 'p' and 'b'). However, such ambiguities can be resolved to an extent using the context of neighboring words in a sentence, and/or a language model. A machine that can lip read opens up a host of applications: 'dictating' instructions or messages to a phone in a noisy environment; transcribing and re-dubbing archival silent films; resolving multi-talker simultaneous speech; and, improving the performance of automated speech recognition in general.
Deep Audio-Visual Speech Recognition
That such automation is now possible is due to two developments that are well known across computer vision tasks: the use of deep neural network models [30,44,47]; and, the availability of a large scale dataset for training [41]. In this case, the lip reading models are based on recent encoder-decoder architectures that have been developed for speech recognition and machine translation [5,7,22,23,46].
Deep Audio-Visual Speech Recognition
The objective of this paper is to develop neural transcription architectures for lip reading sentences. We compare two models: one using a Connectionist Temporal Classification (CTC) loss [22], and the other using a sequence-to-sequence (seq2seq) loss [9,46]. Both models are based on the transformer self-attention architecture [49], so that the advantages and disadvantages of the two losses can be compared head-to-head, with as much of the rest of the architecture in common as possible. The dataset developed in this paper to train and evaluate the models, are based on thousands of hours of videos that have talking faces together with subtitles of what is being said.
Deep Audio-Visual Speech Recognition
We also investigate how lip reading can contribute to audio based speech recognition. There is a large literature on this contribution, particularly in noisy environments, as well as the • T. Afouras  converse where some derived measure of audio can contribute to lip reading for the deaf or hard of hearing. To investigate this aspect we train a model to recognize characters from both audio and visual input, and then systematically disturb the audio channel.
Deep Audio-Visual Speech Recognition
Our models output at the character level. In the case of the CTC, these outputs are independent of each other. In the case of the sequence-to-sequence loss a language model is learnt implicitly, and the architecture incorporates a novel dual attention mechanism that can operate over visual input only, audio input only, or both. The architectures are described in Section 3. Both models are decoded with a beam search, in which we can optionally incorporate an external language model. Section 4, we describe the generation and statistics of a new large scale dataset, LRS2-BBC, that is used to train and evaluate the models. The dataset contains talking faces together with subtitles of what is said. The videos contain faces 'in the wild' with a significant variety of pose, expressions, lighting, backgrounds and ethnic origin. Section 5 describes the network training, where we report a form of curriculum learning that is used to accelerate training. Finally, Section 6 evaluates the performance of the models, including for visual (lips) input only, for audio and visual inputs, and for synchronization errors between the audio and visual streams. On the content: This submission is based on the conference paper [12]. We replace the WLAS model in the original paper with two variants of a Transformer-based model [49]. One variant was published in [2], and the second variant (using the CTC loss) is an original contribution in this paper. We also update the visual frontend with a ResNet-based one proposed by [45]. The new frontend and back-end architectures contribute to over 22% absolute improvement in Word Error Rate (WER) over the model proposed in [12]. Finally, we publicly release a new dataset, LRS2-BBC, that supersedes the original LRS dataset in [12] which could not be made public due to license restrictions.
Deep Audio-Visual Speech Recognition
For the most part, end-to-end deep learning approaches for sequence prediction can be divided into two types.  Fig. 1: Outline of the audio-visual speech recognition pipeline.
Deep Audio-Visual Speech Recognition
The first type uses a neural network as an emission model which outputs the likelihood of each output symbol (e.g. phonemes) given the input sequence (e.g. audio). These methods generally employ a second phase of decoding using a Hidden Markov Model [25]. One such version of this variant is the Connectionist Temporal Classification (CTC) [22], where the model predicts frame-wise labels and then looks for the optimal alignment between the frame-wise predictions and the output sequence. The main weakness of CTC is that the output labels are not conditioned on each other (it assumes each unit is independent), and hence a language model is employed as a post-processing step. Note that some alternatives to jointly train the two step process has been proposed [21]. Another limitation of this approach is that it assumes a monotonic ordering between input and output sequences. This assumption is suitable for ASR and transcription for example, but not for machine translation.
Deep Audio-Visual Speech Recognition
The second type is sequence-to-sequence models [9,46] (seq2seq) that first read all of the input sequence before predicting the output sentence. A number of papers have adopted this approach for speech recognition [10,11]: for example, Chan et al. [7] proposes an elegant sequence-to-sequence method to transcribe audio signal to characters. Sequence-to-sequence decodes an output symbol at time t (e.g. character or word) conditioned on previous outputs 1, . . . , t − 1. Thus, unlike CTC-based models, the model implicitly learns a language model over output symbols, and no further processing is required. However, it has been shown [7,26] that it is beneficial to incorporate an external language model in the decoding of sequence-to-sequence models as well. This way it is possible to leverage larger text-only corpora that contain much richer natural language information than the limited aligned data used for training the acoustic model.
Deep Audio-Visual Speech Recognition
Regarding architectures, while CTC-based or seq2seq approaches traditionally relied on recurrent networks, recently there has been a shift towards purely convolutional models [6]. For example, fully convolutional networks have been used for ASR with CTC [51,55] or a simplified variant [16,32,54].
Deep Audio-Visual Speech Recognition
Lip reading. There is a large body of work on lip reading using non deep learning methods. These methods are thoroughly reviewed in [56], and we will not repeat this here. A number of papers have used Convolutional Neural Networks (CNNs) to predict phonemes [37] or visemes [29] from still images, as opposed to recognising to full words or sentences. A phoneme is the smallest distinguishable unit of sound that collectively make up a spoken word; a viseme is its visual equivalent.
Deep Audio-Visual Speech Recognition
For recognising full words, Petridis et al. [39] train an LSTM classifier on a discrete cosine transform (DCT) and deep bottle-neck features (DBF). Similarly, Wand et al. [50] use an LSTM with HOG input features to recognise short phrases. The shortage of training data in lip reading presumably contributes to the continued use of hand crafted features. Existing datasets consist of videos with only a small number of subjects, and also a limited vocabulary (<60 words), which is also an obstacle to progress. Chung and Zisserman [13] tackles the small-lexicon problem by using faces in television broadcasts to assemble the LRW dataset with a vocabulary size of 500 words. However, as with any wordlevel classification task, the setting is still distant from the realworld, given that the word boundaries must be known beforehand. Assael et al. [4] uses a CNN and LSTM-based network and (CTC) [22] to compute the labelling. This reports strong speakerindependent performance on the constrained grammar and 51 word vocabulary of the GRID dataset [17].
Deep Audio-Visual Speech Recognition
A deeper architecture than LipNet [4] is used by [45], who propose a residual network with 3D convolutions to extract more powerful representations. The network is trained with a crossentropy loss to recognise words from the LRW dataset. Here, the standard ResNet architecture [24] is modified to process 3D image sequences by changing the first convolutional and pooling blocks from 2D to 3D.
Deep Audio-Visual Speech Recognition
In our earlier work [12], we proposed a WLAS sequenceto-sequence model based on the LAS ASR model of [7] (the acronym WLAS are for Watch, Listen, Attend and Spell, and LAS for Listen, Attend and Spell). The WLAS model had a dual attention mechanism -one for the visual (lip) stream, and the other for the audio (speech) stream. It transcribed spoken sentences to characters, and could handle an input of vision only, audio only, or both.
Deep Audio-Visual Speech Recognition
In independent and concurrent work, Shillingford et al. [43], design a lip reading pipeline that uses a network which outputs phoneme probabilities and is trained with CTC loss. At inference time, they use a decoder based on finite state transducers to convert the phoneme distributions into word sequences. The network is trained on a very large scale lip reading dataset constructed from YouTube videos and achieves a remarkable 40.9% word error rate. Audio-visual speech recognition. The problems of audio-visual speech recognition (AVSR) and lip reading are closely linked. Mroueh et al. [36] employs feed-forward Deep Neural Networks (DNNs) to perform phoneme classification using a large nonpublic audio-visual dataset. The use of HMMs together with handcrafted or pre-trained visual features have proved popular - [48] encodes input images using DBF; [20] used DCT; and [38] uses a CNN pre-trained to classify phonemes; all three combine these features with HMMs to classify spoken digits or isolated words. As with lip reading, there has been little attempt to develop AVSR systems that generalise to real-world settings.
Deep Audio-Visual Speech Recognition
Petridis et al. [40] use an extended version of the architecture of [45] to learn representations from raw pixels and waveforms which they then concatenate and feed to a bidirectional recurrent network that jointly models the audio and video sequences and outputs word labels.
Deep Audio-Visual Speech Recognition
In this section, we describe model architectures for audio-visual speech recognition, for which we explore two variants, based on the recently proposed Transformer model [49]: i) an encoderdecoder attention structure for training in a seq2seq manner and ii) a stack of self-attention blocks for training with CTC loss. The  architecture is outlined in Figure 2. The general model receives two input streams, one for video (V) and one for audio (A).
Deep Audio-Visual Speech Recognition
For the acoustic representation we use 321-dimensional spectral magnitudes, computed with a 40ms window and 10ms hop-length, at a 16 kHz sample rate. Since the video is sampled at 25 fps (40 ms per frame), every video input frame corresponds to 4 acoustic feature frames. We concatenate the audio features in groups of 4, in order to reduce the input sequence length as is common for stable CTC training [8,42], while at the same time achieving a common temporal-scale for both modalities.
Deep Audio-Visual Speech Recognition
The input images are 224×224 pixels, sampled at 25 fps and contain the speaker's face. We crop a 112×112 patch covering the region around the mouth, as shown in Figure 3. To extract visual features representing the lip movement, we use a spatio-temporal visual front-end that is based on [45]. The network applies 3D convolutions on the input image sequence, with a filter width of 5 frames, followed by a 2D ResNet that gradually decreases the spatial dimensions with depth. The layers are listed in full detail in Appendix A. For an input sequence of T × H × W frames, the output is a T × H 32 × W 32 ×512 tensor (i.e. the temporal resolution is preserved) that is then average-pooled over the spatial dimensions, yielding a 512-dimensional feature vector for every input video frame.
Deep Audio-Visual Speech Recognition
Both variants that we consider use the same self-attention-based encoder architecture. The encoder is a stack of multi-head selfattention layers, where the input tensor serves as the query, key and value for the attention at the same time. A separate encoder is used for each modality as shown in Figure 2 (a). The information about the sequence order of the inputs is fed to the model via fixed positional embeddings in the form of sinusoid functions.
Deep Audio-Visual Speech Recognition
In this variant, separate attention heads are used for attending on the video and audio embeddings. In every decoder layer, the resulting video and audio contexts are concatenated over the channel dimension and propagated to the feedforward block. The attention mechanisms for both modalities receive as queries the output of the previous decoding layer (or the decoder input in the case of the first layer). The decoder produces character probabilities which are directly matched to the ground truth labels and trained with a cross-entropy loss. More details about the multi-head attention and feed-forward building blocks are given in Appendix B.
Deep Audio-Visual Speech Recognition
The TM-CTC model concatenates the video and audio encodings and propagates the result through a stack of self-attention / feedforward blocks, same as the one used in the encoders. The outputs of the network are the CTC posterior probabilities for every input frame and the whole stack is trained with CTC loss.
Deep Audio-Visual Speech Recognition
For decoding both variants, during inference, we use a characterlevel language model. It is a recurrent network with 4 unidirectional layers of 1024 LSTM cells each. The language model is trained to predict one character at a time, receiving only the previous character as input. Decoding for both models is performed with a left-to-right beam search where the LM logprobabilities are combined with the model's outputs via shallow fusion [26]. More details on decoding are given in Appendices C and D.
Deep Audio-Visual Speech Recognition
The audio-visual models described in this section can be used when only one of the two modalities is present. Instead of concatenating the attention vectors for TM-seq2seq or the encodings for TM-CTC, only the vector from the available modality is used.
Deep Audio-Visual Speech Recognition
In this section, we describe the multi-stage pipeline for automatically generating a large-scale dataset, LRS2-BBC, for audiovisual speech recognition. Using this pipeline, we have been able to collect thousands of hours of spoken sentences and phrases along with the corresponding facetrack. We use a variety of BBC programs from Dragon's Den to Top Gear and Countryfile. The processing pipeline is summarised in Figure 4. Most of the steps are based on the methods described in [13] and [14], but we give a brief sketch of the method here.
Deep Audio-Visual Speech Recognition
Video preparation. A CNN face detector based on the Single Shot MultiBox Detector (SSD) [33] is used to detect face appearances in the individual frames. Unlike the HOG-based detector [27] used by previous works, the SSD detects faces from all angles, and shows a more robust performance whilst being faster to run.
Deep Audio-Visual Speech Recognition
The shot boundaries are determined by comparing color histograms across consecutive frames [31]. Within each shot, face tracks are generated from face detections based on their positions, as feature-based trackers such as KLT [34] often fail when there are extreme changes in viewpoints.
Deep Audio-Visual Speech Recognition
Audio and text preparation. The subtitles in television are not broadcast in sync with the audio. The Penn Phonetics Lab Forced Aligner [53] is used to force-align the subtitle to the audio signal. Errors exist in the alignment as the transcript is not verbatimtherefore the aligned labels are filtered by checking against the commercial IBM Watson Speech to Text service.
Deep Audio-Visual Speech Recognition
AV sync and speaker detection. In broadcast videos, the audio and the video streams can be out of sync by up to around one second, which can cause problems when the facetrack corresponding to a sentence is being extracted. A multi-view adaptation [15] of the two-stream network described in [14] is used to synchronise the two streams. The same network is also used to determine which face's lip movements match the audio, and if none matches, the clip is rejected as being a voice-over.
Deep Audio-Visual Speech Recognition
Sentence extraction. The videos are divided into individual sentences/ phrases using the punctuations in the transcript. The sentences are separated by full stops, commas and question marks; and are clipped to 100 characters or 10 seconds, due to GPU memory constraints. We do not impose any restrictions on the vocabulary size.
Deep Audio-Visual Speech Recognition
The LRS2-BBC dataset is divided into development (train/val) and test sets according to broadcast date. The dataset also has a "pre-train" set that contains sentence excerpts which may be shorter or longer than the full sentences included in the development set, and are annotated with the alignment boundaries of every word. The statistics of these sets are given in Table 1. The table also compares the 'Lip Reading Sentences' (LRS) series of datasets to the largest existing public datasets. In addition to LRS2-BBC, we use MV-LRS and LRS3-TED for training and evaluation.
Deep Audio-Visual Speech Recognition
Datasets for training external language models. To train the language models used for evaluation on each audio-visual dataset, we use a text corpus containing the full subtitles of the videos from which the dataset's training set was generated. The text-only corpus contains 26M words.
Deep Audio-Visual Speech Recognition
In this section, we describe the strategy used to effectively train the models, making best use of the limited amount of data available. The training proceeds in four stages: i) the visual front-end module is trained; ii) visual features are generated for all the training data using the vision module; iii) the sequence processing module is trained on the frozen visual features; iv) the whole network is trained end-to-end.
Deep Audio-Visual Speech Recognition
We pre-train the visual front-end on word excerpts from the MV-LRS [15] dataset, using a 2-layer temporal convolution back-end to classify every clip with a word label similarly to [45]. We perform data augmentation in the form of horizontal flipping, removal of random frames [4,45], and random shifts of up to ±5 pixels in the spatial dimensions and of ±2 frames in the temporal dimension.
Deep Audio-Visual Speech Recognition
Sequence to sequence learning has been reported to converge very slowly when the number of timesteps is large, because the decoder initially has a hard time extracting the relevant information from all the input steps [7]. Even though our models do not contain any recurrent modules, we found it beneficial to follow a curriculum instead of immediately training on full sentences.
Deep Audio-Visual Speech Recognition
We introduce a new strategy where we start training only on single word examples, and then let the sequence length grow as the network trains. These short sequences are parts of the longer sentences in the dataset. We observe that the rate of convergence on the training set is several times faster, while the curriculum also significantly reduces overfitting, presumably because it works as a natural way of augmenting the data.
Deep Audio-Visual Speech Recognition
The networks are first trained on the frozen features of the pre-train sets from MV-LRS, LRS2-BBC and LRS3-TED. We deal with the difference in utterance lengths by zero-padding the sequences to a maximum length, which we gradually increase. We then separately fine-tune end-to-end on the train-val set of LRS2-BBC or LRS3-TED, according to which set we are evaluating on.    Fig. 4: Pipeline to generate the dataset.
Deep Audio-Visual Speech Recognition
The audio-only models are initially trained with clean input audio. Networks with multi-modal inputs can often be dominated by one of the modes [19]. In our case we observe that for the audio-visual models the audio signal dominates, because speech recognition is a significantly easier problem than lip reading. To help prevent this from happening, we add babble noise with 0dB SNR to the audio stream with probability p n = 0.25 during training.
Deep Audio-Visual Speech Recognition
To assess and improve tolerance to audio noise, we then finetune the audio-only and audio-visual models in a setting where babble noise with 0dB SNR is always added to the original audio. We synthesize the babble noise samples by mixing the signals of 20 different audio samples from the LRS2-BBC dataset.
Deep Audio-Visual Speech Recognition
The output size of the network is 40, accounting for the 26 characters in the alphabet, the 10   the decoder, while during inference we feed back the decoder prediction.
Deep Audio-Visual Speech Recognition
Our implementation is based on the TensorFlow library [1] and trained on a single GeForce GTX 1080 Ti GPU with 11GB memory. The network is trained using the ADAM optimiser [28] with the default parameters and an initial learning rate of 10 −4 , which is reduced by a factor of 2 every time the validation error plateaus, down to a final learning rate of 10 −6 . For all the models we use dropout with p = 0.1 and label smoothing.
Deep Audio-Visual Speech Recognition
In this section we evaluate and compare the proposed architectures and training strategies. We also compare our methods to the previous state of the art.
Deep Audio-Visual Speech Recognition
We train as described in section 5.2 and evaluate the fine-tuned models for LRS2-BBC and LRS3-TED on the independent test set of the respective dataset. The inference and evaluation procedures are described below.
Deep Audio-Visual Speech Recognition
Test time augmentation. During inference we perform 9 random transforms (horizontal flipping of the video frames and spatial shifts up to ±5 pixels) on every video sample, and pass the perturbed sequences through the network, in addition to the original. For TM-seq2seq we average the resulting logits whereas for TM-CTC we average the visual features. Experimental setup. The rest of this section is structured as follows: First we present results on lip reading, where only the video is used as input. We then use the full models for audio-visual speech recognition, where the video and audio are assumed to be properly synchronised. To assess the robustness of our models in noisy environments we also train and test in a setting where babble noise is artificially added to the utterances. Finally we present some experiments on non-synchronised video and audio. The results for all experiments are summarized in Table 2, where we report word error rates depending on whether a language model is used during decoding or not.
Deep Audio-Visual Speech Recognition
Results. The best performing network is TM-seq2seq, which achieves a WER of 48.3% on LRS2-BBC when decoded with a language model, an absolute improvement of over 22% compared to the previous 70.4% state-of-the-art [12]. This model also sets a baseline for LRS3-TED at 58.9%.
Deep Audio-Visual Speech Recognition
In Figure 5 we show how the WER changes as a function of the number of words in a test sentence. Figure 6 shows the performance of the models on the 30 most common words. Figure 7 shows the effect of increasing the beam width for the video-only TM-seq2seq model when evaluating on LRS2-BBC. It is noteworthy that increasing the beam width is more beneficial when decoding with the external language model (+ extLM).
Deep Audio-Visual Speech Recognition
Decoding examples. The model learns to correctly predict complex unseen sentences from a wide range of content -examples are shown in Table 3.  
Deep Audio-Visual Speech Recognition
The visual information can be used to improve the performance of ASR, particularly in environments with background noise [36,38,40]. Here, we analyse the performance of the audio-visual models described in Section 3. Table 2 demonstrate that the mouth movements provide important cues in speech recognition when the audio signal is noisy; and give an improvement in performance Decoding examples. Table 4 shows some of the many examples where the model fails to predict the correct sentence from the lips or the audio alone, but successfully deciphers the words when both streams are present.
Deep Audio-Visual Speech Recognition
Alignment and attention visualisation. The encoder-decoder attention mechanism of the TM-seq2seq model generates explicit  alignment between the input video frames and the hypothesised character output. Figure 9 visualises the alignment of the characters "comes from one of the most beautiful parts of the world" and the corresponding video frames. Since the architecture contains multiple attention heads, we obtain the alignment by averaging the attention masks over all the decoder layers in the log domain.
Deep Audio-Visual Speech Recognition
Noisy audio. We perform the audio-only and audio-visual experiments with noisy audio, synthesized by adding babble noise to the original utterances. Speech recognition in a noisy environment is extremely challenging, as can be seen from the significantly lower performance of the off-the-shelf Google S2T ASR baseline (over 60% performance degradation compared to clean). This difficulty is also reflected on the performance of our audio-only models, that the word error rates similar to the ones obtained when only using the lips. However combining the two modalities provides a significant improvement, with the word error rate dropping significantly, by up to 30%. Notably, the audio-visual models perform much better than either the video-only, or audio-only ones under the presence of loud background noise.
Deep Audio-Visual Speech Recognition
AV attention visualization. In Figure 10 we compare the atten-tion masks of different TM-seq2seq models in the presence and absence of additive babble noise in the audio stream.
Deep Audio-Visual Speech Recognition
Here, we assess the performance of the audio-visual models when the audio and video inputs are not temporally aligned. Since the audio and video have been synchronised in our dataset, we synthetically shift the video frames to achieve an out-of-sync effect. We evaluate the performance on de-synchronised samples of the LRS2-BBC dataset. We consider the TM-CTC and TM-seq2seq architectures, with and without fine-tuning on randomly shifted samples. The results are shown in Figure 8. It is clear that the TM-seq2seq architecture is more resistant to these shifts. We only need to calibrate the model for one epoch for the out-ofsync effect to practically vanish. This showcases the advantage of employing independent encoder-decoder attention mechanisms for the two modalities. In contrast, TM-CTC, that concatenates the two encodings, struggles to deal with the shifts, even after several epochs of fine-tuning. 
Deep Audio-Visual Speech Recognition
The TM-seq2seq model performs significantly better for lipreading in terms of WER, when no audio is supplied. For audioonly or audio-visual tasks, the two methods perform similarly. However the CTC models appear to handle background noise better; in the presence of loud babble noise, both the audio-only and audio-visual TM-seq2seq models perform significantly worse that their TM-CTC counterparts.
Deep Audio-Visual Speech Recognition
Training time. The TM-seq2seq models have a more complex architecture and are harder to train, with the full audio-visual model taking approximately 8 days to complete the full curriculum for both datasets, on a single GeForce Titan X GPU with 12GB memory. In contrast, the audiovisual TM-CTC model trains faster i.e. in approximately 5 days on the same hardware. It should be noted however that since both architectures contain no recurrent modules and no batch normalization, their implementation can be heavily parallelized into multiple GPUs. Inference time. Decoding of the TM-CTC model does not require auto-regression and therefore the CTC probabilities need only be evaluated once, regardless of the beam width W . This is not the case for TM-seq2seq, where for every step of the beam search, the decoder subnetwork needs to be evaluated W times. This makes the decoding of the CTC model faster, which can be an important factor for deployment.
Deep Audio-Visual Speech Recognition
Language modelling. Both models perform better when an external language model is incorporated in the beam search, however the gains are much higher for TM-CTC, since no explicit language consistency is enforced by the visual model alone.
Deep Audio-Visual Speech Recognition
Generalization to longer sequences. We observed that the TM-CTC model generalizes better and adapts faster as the sequence lengths are increased during the curriculum learning. We believe this also affects the training time as the latter takes more epochs to converge.
Deep Audio-Visual Speech Recognition
In this paper, we introduced a large-scale, unconstrained audiovisual dataset, LRS2-BBC, formed by collecting and preprocessing thousands of videos from the British television.
Deep Audio-Visual Speech Recognition
We considered two models that can transcribe audio and video sequences of speech into characters and showed that the same architectures can also be used when only one of the modalities is present. Our best visual-only model surpasses the performance of the previous state-of-the-art on the LRS2-BBC lip reading dataset by a large margin, and sets a strong baseline for the recently released LRS3-TED. We finally demonstrate that visual information helps improve speech recognition performance even when the clean audio signal is available. Especially in the presence of noise in the audio, combining the two modalities leads to a significant improvement. Comparing (c) with (d), the attention of the audio-only models appears to be more spread around the area where the noise is applied, while the last frames are not attended upon. Similarly for the audio-visual model, the audio attention is more focused when the audio is clean (f) compared to when it is noisy (g). The ground truth transcription of the sentence is "one of the articles there is about the queen elizabeth". Observing the transcriptions, we see that the audio-only model (d) does not predict the central words correctly when noise is added, however the audio-visual model (g & h) successfully transcribes the sentence, by leveraging the visual cues. Interestingly, in this particular example, the transcription that the video-only model outputs (e) is completely wrong; the combination of both modalities however yields a correct prediction. Finally, the attention mask of the AV model on the video input (f) has a clear monotonic trend and is similar to the one of the video-only model (e); this also verifies that the model indeed learns to use the video modality even when audio is present.  [45]. The strides for the residual 2D convolutional blocks apply to the first layer of the block only (i.e. the total downsampling factor in the network is 32). A short cut connection is added after every pair of 2D convolutions [24]. The 2D convolutions are applied separately on every time-frame.
Deep Audio-Visual Speech Recognition
The details of the building blocks used by our models are outlined in Figure 11. The same multi-head attention block shown is used for both the self-attention and encoder-decoder attention layers of the models. A multi-head attention block, as described by Vaswani et al. [49], receives a query (Q), a key (K) and a value (V ) tensor as inputs and produces h context vectors, one for every attention head i:
Deep Audio-Visual Speech Recognition
where Q, K, and V have size d model and dk = d model h is the size of every attention head. The h context vectors are concatenated and propagated through a feedforward block that consists of two linear layers with ReLU non-linearities in between. For the selfattention layers it is always Q = K = V , while for the encoderdecoder attention of the TM-seq2seq model, K = V are the encoder outputs to be attended upon and Q is the decoder input, i.e. the network's output at the previous decoding step for the first layer and the output of the previous decoder layer for the rest. We use the same architecture hyperparameters as the original base model of Vaswani et al. [49] with d model = 512 and h = 8 attention heads everywhere. The sizes of the two linear layers in the feedforward block are F 1 = 2048, F 2 = 512.
Deep Audio-Visual Speech Recognition
For decoding with the TM-seq2seq model, we use a left-to right beam search with width W as in [26,52], with the hypotheses y being scored as follows:
Deep Audio-Visual Speech Recognition
score(x, y) = log p(y|x) + α log p LM (y) LP (y)  . We did not experiment with a coverage penalty. The best values for the hyperparameters were determined via grid search on the validation set: for decoding without the external language model they were set to W = 6, α = 0.0, β = 0.6 and for decoding with the external language model (+ extLM) to W = 35, α = 0.1 β = 0.7.
Deep Audio-Visual Speech Recognition
Algorithm 1 describes the CTC decoding procedure with an external language model. It is also a beam search with width W and hyperparameters α and β that control the relative weight given to the LM and the length penalty. The beam search is similar to the one described for seq2seq above, with some additional bookkeeping required to handle the emission of repeated and blank characters and normalization LP(y) = |y| β . We obtain the best results on the validation set with W = 100, α = 0.5, β = 0.1.
Deep Audio-Visual Speech Recognition
The F1, precision and recall rates shown in figure E, are calculated from the word-wise minimum edit distance operations. For every sample in the evaluation set we can calculate the fewest word substitution, insertion and deletion operations needed to get from the ground truth to the predicted transcription. After aggregating in B T those operations over the evaluation set for every word, we calculate the average measures per word as follows:
Deep Audio-Visual Speech Recognition
T P (w) = n m (w) F N (w) = j n s (j, w) + n i (w) F P (w) = j n s (w, j) + n d (w) P recision(w) = T P (w) T P (w) + F P (w)
Deep Audio-Visual Speech Recognition
Recall(w) = T P (w) T P (w) + F N (w) F 1(w) = 2 P recision(w)Recall(w) P recision(w) + Recall(w)
Deep Audio-Visual Speech Recognition
where n s (w,j) is the total count over the evaluation set of substitutions of word j with word w, and n m (w), n i (w) and n d (w) are the total matches, deletions and insertions respectively of word w.
LipNet: End-to-End Sentence-level Lipreading
Lipreading plays a crucial role in human communication and speech understanding, as highlighted by the McGurk effect (McGurk & MacDonald, 1976), where one phoneme's audio dubbed on top of a video of someone speaking a different phoneme results in a third phoneme being perceived.
LipNet: End-to-End Sentence-level Lipreading
Lipreading plays a crucial role in human communication and speech understanding, as highlighted by the McGurk effect (McGurk & MacDonald, 1976), where one phoneme's audio dubbed on top of a video of someone speaking a different phoneme results in a third phoneme being perceived.
LipNet: End-to-End Sentence-level Lipreading
Lipreading is a notoriously difficult task for humans, specially in the absence of context 1 . Most lipreading actuations, besides the lips and sometimes tongue and teeth, are latent and difficult to disambiguate without context (Fisher, 1968;Woodward & Barber, 1960). For example, Fisher (1968) gives 5 categories of visual phonemes (called visemes), out of a list of 23 initial consonant phonemes, that are commonly confused by people when viewing a speaker's mouth. Many of these were asymmetrically confused, and observations were similar for final consonant phonemes.
LipNet: End-to-End Sentence-level Lipreading
Lipreading is a notoriously difficult task for humans, specially in the absence of context 1 . Most lipreading actuations, besides the lips and sometimes tongue and teeth, are latent and difficult to disambiguate without context (Fisher, 1968;Woodward & Barber, 1960). For example, Fisher (1968) gives 5 categories of visual phonemes (called visemes), out of a list of 23 initial consonant phonemes, that are commonly confused by people when viewing a speaker's mouth. Many of these were asymmetrically confused, and observations were similar for final consonant phonemes.
LipNet: End-to-End Sentence-level Lipreading
Consequently, human lipreading performance is poor. Hearing-impaired people achieve an accuracy of only 17±12% even for a limited subset of 30 monosyllabic words and 21±11% for 30 compound words (Easton & Basala, 1982). An important goal, therefore, is to automate lipreading. Machine lipreaders have enormous practical potential, with applications in improved hearing aids, silent dictation in public spaces, security, speech recognition in noisy environments, biometric identification, and silent-movie processing.
LipNet: End-to-End Sentence-level Lipreading
Consequently, human lipreading performance is poor. Hearing-impaired people achieve an accuracy of only 17±12% even for a limited subset of 30 monosyllabic words and 21±11% for 30 compound words (Easton & Basala, 1982). An important goal, therefore, is to automate lipreading. Machine lipreaders have enormous practical potential, with applications in improved hearing aids, silent dictation in public spaces, security, speech recognition in noisy environments, biometric identification, and silent-movie processing.
LipNet: End-to-End Sentence-level Lipreading
Machine lipreading is difficult because it requires extracting spatiotemporal features from the video (since both position and motion are important). Recent deep learning approaches attempt to extract those features end-to-end. Most existing work, however, performs only word classification, not sentence-level sequence prediction.
LipNet: End-to-End Sentence-level Lipreading
In this paper, we present LipNet, which is to the best of our knowledge, the first end-to-end sentence-level lipreading model. As with modern deep learning based automatic speech recognition (ASR), LipNet is trained end-to-end to make sentence-level predictions. Our model operates at the character-level, using spatiotemporal convolutional neural networks (STCNNs), recurrent neural networks (RNNs), and the connectionist temporal classification loss (CTC) Graves et al. (2006).
LipNet: End-to-End Sentence-level Lipreading
In this paper, we present LipNet, which is to the best of our knowledge, the first end-to-end sentence-level lipreading model. As with modern deep learning based automatic speech recognition (ASR), LipNet is trained end-to-end to make sentence-level predictions. Our model operates at the character-level, using spatiotemporal convolutional neural networks (STCNNs), recurrent neural networks (RNNs), and the connectionist temporal classification loss (CTC) Graves et al. (2006).
LipNet: End-to-End Sentence-level Lipreading
Our empirical results on the GRID corpus (Cooke et al., 2006), one of the few public sentence-level datasets, show that LipNet attains a 95.2% sentence-level word accuracy, in a overlapped speakers split that is popular for benchmarking lipreading methods. The previous best accuracy reported on an aligned word classification version of this task was 86.4% (Gergen et al., 2016). Furthermore, LipNet can generalise across unseen speakers in the GRID corpus with an accuracy of 88.6%.
LipNet: End-to-End Sentence-level Lipreading
Our empirical results on the GRID corpus (Cooke et al., 2006), one of the few public sentence-level datasets, show that LipNet attains a 95.2% sentence-level word accuracy, in a overlapped speakers split that is popular for benchmarking lipreading methods. The previous best accuracy reported on an aligned word classification version of this task was 86.4% (Gergen et al., 2016). Furthermore, LipNet can generalise across unseen speakers in the GRID corpus with an accuracy of 88.6%.
LipNet: End-to-End Sentence-level Lipreading
We also compare the performance of LipNet with that of hearing-impaired people who can lipread on the GRID corpus task. On average, they achieve an accuracy of 52.3%, in contrast to LipNet's 1.69× higher accuracy in the same sentences.
LipNet: End-to-End Sentence-level Lipreading
Finally, by applying saliency visualisation techniques (Zeiler & Fergus, 2014;Simonyan et al., 2013), we interpret LipNet's learned behaviour, showing that the model attends to phonologically important regions in the video. Furthermore, by computing intra-viseme and inter-viseme confusion matrices at the phoneme level, we show that almost all of LipNet's few erroneous predictions occur within visemes, since context is sometimes insufficient for disambiguation.
LipNet: End-to-End Sentence-level Lipreading
Finally, by applying saliency visualisation techniques (Zeiler & Fergus, 2014;Simonyan et al., 2013), we interpret LipNet's learned behaviour, showing that the model attends to phonologically important regions in the video. Furthermore, by computing intra-viseme and inter-viseme confusion matrices at the phoneme level, we show that almost all of LipNet's few erroneous predictions occur within visemes, since context is sometimes insufficient for disambiguation.
LipNet: End-to-End Sentence-level Lipreading
In this section, we outline various existing approaches to automated lipreading.
LipNet: End-to-End Sentence-level Lipreading
Automated lipreading: Most existing work on lipreading does not employ deep learning. Such work requires either heavy preprocessing of frames to extract image features, temporal preprocessing of frames to extract video features (e.g., optical flow or movement detection), or other types of handcrafted vision pipelines (Matthews et al., 2002;Zhao et al., 2009;Gurban & Thiran, 2009;Papandreou et al., 2007;2009;Pitsikalis et al., 2006;Lucey & Sridharan, 2006;Papandreou et al., 2009). The automated lipreading literature is too vast to adequately cover, so we refer the reader to Zhou et al. (2014) for an extensive review.
LipNet: End-to-End Sentence-level Lipreading
Automated lipreading: Most existing work on lipreading does not employ deep learning. Such work requires either heavy preprocessing of frames to extract image features, temporal preprocessing of frames to extract video features (e.g., optical flow or movement detection), or other types of handcrafted vision pipelines (Matthews et al., 2002;Zhao et al., 2009;Gurban & Thiran, 2009;Papandreou et al., 2007;2009;Pitsikalis et al., 2006;Lucey & Sridharan, 2006;Papandreou et al., 2009). The automated lipreading literature is too vast to adequately cover, so we refer the reader to Zhou et al. (2014) for an extensive review.
LipNet: End-to-End Sentence-level Lipreading
Notably, Goldschen et al. (1997) were the first to do visual-only sentence-level lipreading using hidden Markov models (HMMs) in a limited dataset, using hand-segmented phones. Later, Neti et al. (2000) were the first to do sentence-level audiovisual speech recognition using an HMM combined with hand-engineered features, on the IBM ViaVoice (Neti et al., 2000) dataset. The authors improve speech recognition performance in noisy environments by fusing visual features with audio ones. The dataset contains 17111 utterances of 261 speakers for training (about 34.9 hours) and is not publicly available. As stated, their visual-only results cannot be interpreted as visual-only recognition, as they are used as rescoring of the noisy audio-only lattices. Using a similar approach, Potamianos et al. (2003) report speaker independent and speaker adapted 91.62%, 82.31% WER in the same dataset respectively, and 38.53%, 16.77% WER in the connected DIGIT corpus, which contains sentences of digits.
LipNet: End-to-End Sentence-level Lipreading
Notably, Goldschen et al. (1997) were the first to do visual-only sentence-level lipreading using hidden Markov models (HMMs) in a limited dataset, using hand-segmented phones. Later, Neti et al. (2000) were the first to do sentence-level audiovisual speech recognition using an HMM combined with hand-engineered features, on the IBM ViaVoice (Neti et al., 2000) dataset. The authors improve speech recognition performance in noisy environments by fusing visual features with audio ones. The dataset contains 17111 utterances of 261 speakers for training (about 34.9 hours) and is not publicly available. As stated, their visual-only results cannot be interpreted as visual-only recognition, as they are used as rescoring of the noisy audio-only lattices. Using a similar approach, Potamianos et al. (2003) report speaker independent and speaker adapted 91.62%, 82.31% WER in the same dataset respectively, and 38.53%, 16.77% WER in the connected DIGIT corpus, which contains sentences of digits.
LipNet: End-to-End Sentence-level Lipreading
Furthermore, Gergen et al. (2016) use speaker-dependent training on an LDA-transformed version of the Discrete Cosine Transforms of the mouth regions in an HMM/GMM system. This work holds the previous state-of-the-art on the GRID corpus with a speaker-dependent accuracy of 86.4%. Generalisation across speakers and extraction of motion features is considered an open problem, as noted in (Zhou et al., 2014). LipNet addresses both of these issues.
LipNet: End-to-End Sentence-level Lipreading
Furthermore, Gergen et al. (2016) use speaker-dependent training on an LDA-transformed version of the Discrete Cosine Transforms of the mouth regions in an HMM/GMM system. This work holds the previous state-of-the-art on the GRID corpus with a speaker-dependent accuracy of 86.4%. Generalisation across speakers and extraction of motion features is considered an open problem, as noted in (Zhou et al., 2014). LipNet addresses both of these issues.
LipNet: End-to-End Sentence-level Lipreading
Classification with deep learning: In recent years, there have been several attempts to apply deep learning to lipreading. However, all of these approaches perform only word or phoneme classification, whereas LipNet performs full sentence sequence prediction. Approaches include learning multimodal audio-visual representations (Ngiam et al., 2011;Sui et al., 2015;Ninomiya et al., 2015;Petridis & Pantic, 2016), learning visual features as part of a traditional speech-style processing pipeline (e.g. HMMs, GMM-HMMs, etc.) for classifying words and/or phonemes (Almajai et al., 2016;Takashima et al., 2016;Noda et al., 2014;Koller et al., 2015), or combinations thereof (Takashima et al., 2016). Many of these approaches mirror early progress in applying neural networks for acoustic processing in speech recognition . Chung & Zisserman (2016a) propose spatial and spatiotemporal convolutional neural networks, based on VGG, for word classification. The architectures are evaluated on a word-level dataset BBC TV (333 and 500 classes), but, as reported, their spatiotemporal models fall short of the spatial architectures by an average of around 14%. Additionally, their models cannot handle variable sequence lengths and they do not attempt sentence-level sequence prediction. Chung & Zisserman (2016b) train an audio-visual max-margin matching model for learning pretrained mouth features, which they use as inputs to an LSTM for 10-phrase classification on the OuluVS2 dataset, as well as a non-lipreading task. Wand et al. (2016) introduce LSTM recurrent neural networks for lipreading but address neither sentence-level sequence prediction nor speaker independence. Garg et al. (2016) apply a VGG pre-trained on faces to classifying words and phrases from the MIRACL-VC1 dataset, which has only 10 words and 10 phrases. However, their best recurrent model is trained by freezing the VGGNet parameters and then training the RNN, rather than training them jointly. Their best model achieves only 56.0% word classification accuracy, and 44.5% phrase classification accuracy, despite both of these being 10-class classification tasks.
LipNet: End-to-End Sentence-level Lipreading
Classification with deep learning: In recent years, there have been several attempts to apply deep learning to lipreading. However, all of these approaches perform only word or phoneme classification, whereas LipNet performs full sentence sequence prediction. Approaches include learning multimodal audio-visual representations (Ngiam et al., 2011;Sui et al., 2015;Ninomiya et al., 2015;Petridis & Pantic, 2016), learning visual features as part of a traditional speech-style processing pipeline (e.g. HMMs, GMM-HMMs, etc.) for classifying words and/or phonemes (Almajai et al., 2016;Takashima et al., 2016;Noda et al., 2014;Koller et al., 2015), or combinations thereof (Takashima et al., 2016). Many of these approaches mirror early progress in applying neural networks for acoustic processing in speech recognition . Chung & Zisserman (2016a) propose spatial and spatiotemporal convolutional neural networks, based on VGG, for word classification. The architectures are evaluated on a word-level dataset BBC TV (333 and 500 classes), but, as reported, their spatiotemporal models fall short of the spatial architectures by an average of around 14%. Additionally, their models cannot handle variable sequence lengths and they do not attempt sentence-level sequence prediction. Chung & Zisserman (2016b) train an audio-visual max-margin matching model for learning pretrained mouth features, which they use as inputs to an LSTM for 10-phrase classification on the OuluVS2 dataset, as well as a non-lipreading task. Wand et al. (2016) introduce LSTM recurrent neural networks for lipreading but address neither sentence-level sequence prediction nor speaker independence. Garg et al. (2016) apply a VGG pre-trained on faces to classifying words and phrases from the MIRACL-VC1 dataset, which has only 10 words and 10 phrases. However, their best recurrent model is trained by freezing the VGGNet parameters and then training the RNN, rather than training them jointly. Their best model achieves only 56.0% word classification accuracy, and 44.5% phrase classification accuracy, despite both of these being 10-class classification tasks.
LipNet: End-to-End Sentence-level Lipreading
Sequence prediction in speech recognition: The field of automatic speech recognition (ASR) would not be in the state it is today without modern advances in deep learning, many of which have occurred in the context of ASR (Graves et al., 2006;. The connectionist temporal classification loss (CTC) of Graves et al. (2006) drove the movement from deep learning as a component of ASR, to deep ASR systems trained end-to-end (Graves & Jaitly, 2014;Maas et al., 2015;Amodei et al., 2015). As mentioned earlier, much recent lipreading progress has mirrored early progress in ASR, but stopping short of sequence prediction.
LipNet: End-to-End Sentence-level Lipreading
Sequence prediction in speech recognition: The field of automatic speech recognition (ASR) would not be in the state it is today without modern advances in deep learning, many of which have occurred in the context of ASR (Graves et al., 2006;. The connectionist temporal classification loss (CTC) of Graves et al. (2006) drove the movement from deep learning as a component of ASR, to deep ASR systems trained end-to-end (Graves & Jaitly, 2014;Maas et al., 2015;Amodei et al., 2015). As mentioned earlier, much recent lipreading progress has mirrored early progress in ASR, but stopping short of sequence prediction.
LipNet: End-to-End Sentence-level Lipreading
LipNet is the first end-to-end model that performs sentence-level sequence prediction for visual speech recogntion. That is, we demonstrate the first work that takes as input as sequence of images and outputs a distribution over sequences of tokens; it is trained end-to-end using CTC and thus also does not require alignments.
LipNet: End-to-End Sentence-level Lipreading
Lipreading Datasets: Lipreading datasets (AVICar, AVLetters, AVLetters2, BBC TV, CUAVE, OuluVS1, OuluVS2) are plentiful (Zhou et al., 2014;Chung & Zisserman, 2016a), but most only contain single words or are too small. One exception is the GRID corpus (Cooke et al., 2006), which has audio and video recordings of 34 speakers who produced 1000 sentences each, for a total of 28 hours across 34000 sentences. Table 1 summarises state-of-the-art performance in each of the main lipreading datasets. Table 1: Existing lipreading datasets and the state-of-the-art accuracy reported on these. The size column represents the number of utterances used by the authors for training. Although the GRID corpus contains entire sentences, Gergen et al. (2016) consider only the simpler case of predicting isolated words. LipNet predicts sequences and hence can exploit temporal context to attain much higher accuracy. Phrase-level approaches were treated as plain classification. We use the GRID corpus to evaluate LipNet because it is sentence-level and has the most data. The sentences are drawn from the following simple grammar: command (4) + color (4) + preposition (4) + letter (25) + digit (10) + adverb ( {again, now, please, soon}, yielding 64000 possible sentences. For example, two sentences in the data are "set blue by A four please" and "place red at C zero again".
LipNet: End-to-End Sentence-level Lipreading
Lipreading Datasets: Lipreading datasets (AVICar, AVLetters, AVLetters2, BBC TV, CUAVE, OuluVS1, OuluVS2) are plentiful (Zhou et al., 2014;Chung & Zisserman, 2016a), but most only contain single words or are too small. One exception is the GRID corpus (Cooke et al., 2006), which has audio and video recordings of 34 speakers who produced 1000 sentences each, for a total of 28 hours across 34000 sentences. Table 1 summarises state-of-the-art performance in each of the main lipreading datasets. Table 1: Existing lipreading datasets and the state-of-the-art accuracy reported on these. The size column represents the number of utterances used by the authors for training. Although the GRID corpus contains entire sentences, Gergen et al. (2016) consider only the simpler case of predicting isolated words. LipNet predicts sequences and hence can exploit temporal context to attain much higher accuracy. Phrase-level approaches were treated as plain classification. We use the GRID corpus to evaluate LipNet because it is sentence-level and has the most data. The sentences are drawn from the following simple grammar: command (4) + color (4) + preposition (4) + letter (25) + digit (10) + adverb ( {again, now, please, soon}, yielding 64000 possible sentences. For example, two sentences in the data are "set blue by A four please" and "place red at C zero again".
LipNet: End-to-End Sentence-level Lipreading
LipNet is a neural network architecture for lipreading that maps variable-length sequences of video frames to text sequences, and is trained end-to-end. In this section, we describe LipNet's building blocks and architecture.
LipNet: End-to-End Sentence-level Lipreading
Convolutional neural networks (CNNs), containing stacked convolutions operating spatially over an image, have been instrumental in advancing performance in computer visions tasks such as object recognition that receive an image as input (Krizhevsky et al., 2012). A basic 2D convolution layer from C channels to C channels (without a bias and with unit stride) computes
LipNet: End-to-End Sentence-level Lipreading
Convolutional neural networks (CNNs), containing stacked convolutions operating spatially over an image, have been instrumental in advancing performance in computer visions tasks such as object recognition that receive an image as input (Krizhevsky et al., 2012). A basic 2D convolution layer from C channels to C channels (without a bias and with unit stride) computes
LipNet: End-to-End Sentence-level Lipreading
for input x and weights w ∈ R C ×C×kw×k h where we define x cij = 0 for i, j out of bounds. Spatiotemporal convolutional neural networks (STCNNs) can process video data by convolving across time, as well as the spatial dimensions (Karpathy et al., 2014;Ji et al., 2013). Hence similarly,
LipNet: End-to-End Sentence-level Lipreading
for input x and weights w ∈ R C ×C×kw×k h where we define x cij = 0 for i, j out of bounds. Spatiotemporal convolutional neural networks (STCNNs) can process video data by convolving across time, as well as the spatial dimensions (Karpathy et al., 2014;Ji et al., 2013). Hence similarly,
LipNet: End-to-End Sentence-level Lipreading
Gated Recurrent Unit (GRU) (Chung et al., 2014) is a type of recurrent neural network (RNN) that improves upon earlier RNNs by adding cells and gates for propagating information over more timesteps and learning to control this information flow. It is similar to the Long Short-Term Memory (LSTM) RNN (Hochreiter & Schmidhuber, 1997). We use the standard formulation:
LipNet: End-to-End Sentence-level Lipreading
Gated Recurrent Unit (GRU) (Chung et al., 2014) is a type of recurrent neural network (RNN) that improves upon earlier RNNs by adding cells and gates for propagating information over more timesteps and learning to control this information flow. It is similar to the Long Short-Term Memory (LSTM) RNN (Hochreiter & Schmidhuber, 1997). We use the standard formulation:
LipNet: End-to-End Sentence-level Lipreading
where z := {z 1 , . . . , z T } is the input sequence to the RNN, denotes element-wise multiplication, and sigm(r) = 1/(1 + exp(−r)). We use a bidirectional GRU (Bi-GRU) as introduced by Graves & Schmidhuber (2005) in the context of LSTMs: one RNN maps
LipNet: End-to-End Sentence-level Lipreading
where z := {z 1 , . . . , z T } is the input sequence to the RNN, denotes element-wise multiplication, and sigm(r) = 1/(1 + exp(−r)). We use a bidirectional GRU (Bi-GRU) as introduced by Graves & Schmidhuber (2005) in the context of LSTMs: one RNN maps
LipNet: End-to-End Sentence-level Lipreading
The Bi-GRU ensures that h t depends on z t for all t . To parameterise a distribution over sequences, at time-step t let p(u t |z) = softmax(mlp(h t ; W mlp )), where mlp is a feed-forward network with weights W mlp . Then we can define the distribution over length-T sequences as p(u 1 , . . . , u T |z) = 1≤t≤T p(u t |z), where T is determined by z, the input to the GRU. In LipNet, z is the output of the STCNN.
LipNet: End-to-End Sentence-level Lipreading
The connectionist temporal classification (CTC) loss (Graves et al., 2006) is widely used in modern speech recognition as it eliminates the need for training data that aligns inputs to target outputs (Amodei et al., 2015;Graves & Jaitly, 2014;Maas et al., 2015). Given a model that outputs a sequence of discrete distributions over the token classes (vocabulary) augmented with a special "blank" token, CTC computes the probability of a sequence by marginalising over all sequences that are defined as equivalent to this sequence. This simultaneously removes the need for alignments and addresses variable-length sequences. Let V denote the set of tokens that the model classifies at a single time-step of its output (vocabulary), and the blank-augmented vocabularyṼ = V ∪ { } where denotes the CTC blank symbol. Define the function B :Ṽ * → V * that, given a string over V , deletes adjacent duplicate characters and removes blank tokens. For a label sequence y ∈ V * , CTC defines p(y|x) = u∈B −1 (y) s.t. |u|=T p(u 1 , . . . , u T |x), where T is the number of time-steps in the sequence model. For example, if T = 3, CTC defines the probability of a string "am" as p(aam) + p(amm) + p( am) + p(a m) + p(am ). This sum is computed efficiently by dynamic programming, allowing us to perform maximum likelihood. Figure 1 illustrates the LipNet architecture, which starts with 3×(spatiotemporal convolutions, channel-wise dropout, spatial max-pooling). Subsequently, the features extracted are followed by two Bi-GRUs. The Bi-GRUs are crucial for efficient further aggregation of the STCNN output. Finally, a linear transformation is applied at each time-step, followed by a softmax over the vocabulary augmented with the CTC blank, and then the CTC loss. All layers use rectified linear unit (ReLU) activation functions. More details including hyperparameters can be found in Table 3 of Appendix A.
LipNet: End-to-End Sentence-level Lipreading
The connectionist temporal classification (CTC) loss (Graves et al., 2006) is widely used in modern speech recognition as it eliminates the need for training data that aligns inputs to target outputs (Amodei et al., 2015;Graves & Jaitly, 2014;Maas et al., 2015). Given a model that outputs a sequence of discrete distributions over the token classes (vocabulary) augmented with a special "blank" token, CTC computes the probability of a sequence by marginalising over all sequences that are defined as equivalent to this sequence. This simultaneously removes the need for alignments and addresses variable-length sequences. Let V denote the set of tokens that the model classifies at a single time-step of its output (vocabulary), and the blank-augmented vocabularyṼ = V ∪ { } where denotes the CTC blank symbol. Define the function B :Ṽ * → V * that, given a string over V , deletes adjacent duplicate characters and removes blank tokens. For a label sequence y ∈ V * , CTC defines p(y|x) = u∈B −1 (y) s.t. |u|=T p(u 1 , . . . , u T |x), where T is the number of time-steps in the sequence model. For example, if T = 3, CTC defines the probability of a string "am" as p(aam) + p(amm) + p( am) + p(a m) + p(am ). This sum is computed efficiently by dynamic programming, allowing us to perform maximum likelihood. Figure 1 illustrates the LipNet architecture, which starts with 3×(spatiotemporal convolutions, channel-wise dropout, spatial max-pooling). Subsequently, the features extracted are followed by two Bi-GRUs. The Bi-GRUs are crucial for efficient further aggregation of the STCNN output. Finally, a linear transformation is applied at each time-step, followed by a softmax over the vocabulary augmented with the CTC blank, and then the CTC loss. All layers use rectified linear unit (ReLU) activation functions. More details including hyperparameters can be found in Table 3 of Appendix A.
LipNet: End-to-End Sentence-level Lipreading
In this section, we evaluate LipNet on the GRID corpus. The augmentation methods employed don't make use of external data and rely purely on the GRID corpus.
LipNet: End-to-End Sentence-level Lipreading
Preprocessing: The GRID corpus consists of 34 subjects, each narrating 1000 sentences. The videos for speaker 21 are missing, and a few others are empty or corrupt, leaving 32746 usable videos. We employ a split (unseen speakers; not previously used in the literature) holding out the data of two male speakers (1 and 2) and two female speakers (20 and 22) for evaluation (3971 videos). The remainder is used for training (28775 videos). We also use a sentence-level variant of the split (overlapped speakers) similar to Wand et al. (2016), where 255 random sentences from each speaker are used for evaluation. All remaining data from all speakers is pooled together for training. All videos are 3 seconds long with a frame rate of 25fps. The videos were processed with the DLib face detector, and the iBug face landmark predictor (Sagonas et al., 2013) with 68 landmarks coupled with an online Kalman Filter. Using these landmarks, we apply an affine transformation to extract a mouth-centred crop of size 100 × 50 pixels per frame. We standardise the RGB channels over the whole training set to have zero mean and unit variance.
LipNet: End-to-End Sentence-level Lipreading
Preprocessing: The GRID corpus consists of 34 subjects, each narrating 1000 sentences. The videos for speaker 21 are missing, and a few others are empty or corrupt, leaving 32746 usable videos. We employ a split (unseen speakers; not previously used in the literature) holding out the data of two male speakers (1 and 2) and two female speakers (20 and 22) for evaluation (3971 videos). The remainder is used for training (28775 videos). We also use a sentence-level variant of the split (overlapped speakers) similar to Wand et al. (2016), where 255 random sentences from each speaker are used for evaluation. All remaining data from all speakers is pooled together for training. All videos are 3 seconds long with a frame rate of 25fps. The videos were processed with the DLib face detector, and the iBug face landmark predictor (Sagonas et al., 2013) with 68 landmarks coupled with an online Kalman Filter. Using these landmarks, we apply an affine transformation to extract a mouth-centred crop of size 100 × 50 pixels per frame. We standardise the RGB channels over the whole training set to have zero mean and unit variance.
LipNet: End-to-End Sentence-level Lipreading
Augmentation: We augment the dataset with simple transformations to reduce overfitting. First, we train on both the regular and the horizontally mirrored image sequence. Second, since the dataset provides word start and end timings for each sentence video, we augment the sentence-level training data with video clips of individual words as additional training instances. These instances have a decay rate of 0.925. Third, to encourage resilience to varying motion speeds by deletion and duplication of frames, this is performed with a per-frame probability of 0.05. The same augmentation methods were followed in all proposed baselines and models.
LipNet: End-to-End Sentence-level Lipreading
To evaluate LipNet, we compare its performance to that of three hearing-impaired people who can lipread, as well as three ablation models inspired by recent state-of-the-art work (Chung & Zisserman, 2016a;Wand et al., 2016).
LipNet: End-to-End Sentence-level Lipreading
To evaluate LipNet, we compare its performance to that of three hearing-impaired people who can lipread, as well as three ablation models inspired by recent state-of-the-art work (Chung & Zisserman, 2016a;Wand et al., 2016).
LipNet: End-to-End Sentence-level Lipreading
Hearing-Impaired People: This baseline was performed by three members of the Oxford Students' Disability Community. After being introduced to the grammar of the GRID corpus, they observed 10 minutes of annotated videos from the training dataset, then annotated 300 random videos from the evaluation dataset. When uncertain, they were asked to pick the most probable answer.
LipNet: End-to-End Sentence-level Lipreading
Baseline-LSTM: Using the sentence-level training setup of LipNet, we replicate the model architecture of the previous deep learning GRID corpus state-of-the-art (Wand et al., 2016). See Appendix A for more implementation details.
LipNet: End-to-End Sentence-level Lipreading
Baseline-LSTM: Using the sentence-level training setup of LipNet, we replicate the model architecture of the previous deep learning GRID corpus state-of-the-art (Wand et al., 2016). See Appendix A for more implementation details.
LipNet: End-to-End Sentence-level Lipreading
Baseline-2D: Based on the LipNet architecture, we replace the STCNN with spatial-only convolutions similar to those of Chung & Zisserman (2016a). Notably, contrary to the results we observe with LipNet, Chung & Zisserman (2016a) report 14% and 31% poorer performance of their STC-NNs compared to the 2D architectures in their two datasets.
LipNet: End-to-End Sentence-level Lipreading
Baseline-2D: Based on the LipNet architecture, we replace the STCNN with spatial-only convolutions similar to those of Chung & Zisserman (2016a). Notably, contrary to the results we observe with LipNet, Chung & Zisserman (2016a) report 14% and 31% poorer performance of their STC-NNs compared to the 2D architectures in their two datasets.
LipNet: End-to-End Sentence-level Lipreading
Baseline-NoLM: Identical to LipNet, but with the language model used in beam search disabled.
LipNet: End-to-End Sentence-level Lipreading
To measure the performance of LipNet and the baselines, we compute the word error rate (WER) and the character error rate (CER), standard metrics for the performance of ASR models. We produce approximate maximum-probability predictions from LipNet by performing CTC beam search. WER (or CER) is defined as the minimum number of word (or character) insertions, substitutions, and deletions required to transform the prediction into the ground truth, divided by the number of words (or characters) in the ground truth. Note that WER is usually equal to classification error when the predicted sentence has the same number of words as the ground truth, particularly in our case since almost all errors are substitution errors. Table 2 summarises the performance of LipNet compared to the baselines. According to the literature, the accuracy of human lipreaders is around 20% (Easton & Basala, 1982;Hilder et al., 2009). As expected, the fixed sentence structure and the limited subset of words for each position in the GRID corpus facilitate the use of context, increasing performance. On the unseen speakers split, the three hearing-impaired people achieve 57.3%, 50.4%, and 35.5% WER respectively, yielding an average of 47.7% WER. For both unseen and overlapped speakers evaluation, the highest performance is achieved by the architectures enhanced with convolutional stacks. LipNet exhibits a 2.3× higher performance in the overlapped compared to the unseen speakers split. For unseen speakers, Baseline-2D and LipNet achieve 1.8× and 4.2× lower WER, respectively, than hearing-impaired people.
LipNet: End-to-End Sentence-level Lipreading
To measure the performance of LipNet and the baselines, we compute the word error rate (WER) and the character error rate (CER), standard metrics for the performance of ASR models. We produce approximate maximum-probability predictions from LipNet by performing CTC beam search. WER (or CER) is defined as the minimum number of word (or character) insertions, substitutions, and deletions required to transform the prediction into the ground truth, divided by the number of words (or characters) in the ground truth. Note that WER is usually equal to classification error when the predicted sentence has the same number of words as the ground truth, particularly in our case since almost all errors are substitution errors. Table 2 summarises the performance of LipNet compared to the baselines. According to the literature, the accuracy of human lipreaders is around 20% (Easton & Basala, 1982;Hilder et al., 2009). As expected, the fixed sentence structure and the limited subset of words for each position in the GRID corpus facilitate the use of context, increasing performance. On the unseen speakers split, the three hearing-impaired people achieve 57.3%, 50.4%, and 35.5% WER respectively, yielding an average of 47.7% WER. For both unseen and overlapped speakers evaluation, the highest performance is achieved by the architectures enhanced with convolutional stacks. LipNet exhibits a 2.3× higher performance in the overlapped compared to the unseen speakers split. For unseen speakers, Baseline-2D and LipNet achieve 1.8× and 4.2× lower WER, respectively, than hearing-impaired people.
LipNet: End-to-End Sentence-level Lipreading
The WER for unseen speakers Baseline-2D is 26.7%, whereas for LipNet it is 2.3× lower, at 11.4%. Similarly, the error rate for overlapped speakers was 2.4× lower for LipNet compared to Baseline-2D. Both results demonstrate the importance of combining STCNNs with RNNs. This performance difference confirms the intuition that extracting spatiotemporal features using a STCNN is better than aggregating spatial-only features. This observation contrasts with the empirical observations of Chung & Zisserman (2016a). Furthermore, LipNet's use of STCNN, RNNs, and CTC cleanly allow processing both variable-length input and variable-length output sequences, whereas the architectures of Chung & Zisserman (2016a) and Chung & Zisserman (2016b) only handle the former.
LipNet: End-to-End Sentence-level Lipreading
The WER for unseen speakers Baseline-2D is 26.7%, whereas for LipNet it is 2.3× lower, at 11.4%. Similarly, the error rate for overlapped speakers was 2.4× lower for LipNet compared to Baseline-2D. Both results demonstrate the importance of combining STCNNs with RNNs. This performance difference confirms the intuition that extracting spatiotemporal features using a STCNN is better than aggregating spatial-only features. This observation contrasts with the empirical observations of Chung & Zisserman (2016a). Furthermore, LipNet's use of STCNN, RNNs, and CTC cleanly allow processing both variable-length input and variable-length output sequences, whereas the architectures of Chung & Zisserman (2016a) and Chung & Zisserman (2016b) only handle the former.
LipNet: End-to-End Sentence-level Lipreading
Baseline-LSTM exhibits the lowest performance, in both unseen and overlapped speakers, with 52.8% and 26.3% WER, respectively. Interestingly, although Baseline-LSTM replicates the architecture of Wand et al. (2016), and despite the numerous data augmentation methods, the model performs 1.3× lower than the reported 79.6% word-level accuracy illustrating the difficulty of a sentence-level task even in a restricted grammar.
LipNet: End-to-End Sentence-level Lipreading
Baseline-LSTM exhibits the lowest performance, in both unseen and overlapped speakers, with 52.8% and 26.3% WER, respectively. Interestingly, although Baseline-LSTM replicates the architecture of Wand et al. (2016), and despite the numerous data augmentation methods, the model performs 1.3× lower than the reported 79.6% word-level accuracy illustrating the difficulty of a sentence-level task even in a restricted grammar.
LipNet: End-to-End Sentence-level Lipreading
Finally, by disabling the language model, the Baseline-NoLM exhibits approximately 1.2× higher WER than our proposed model.
LipNet: End-to-End Sentence-level Lipreading
In this section, we analyse the learned representations of LipNet from a phonological perspective. First, we create saliency visualisations (Simonyan et al., 2013;Zeiler & Fergus, 2014) to illustrate where LipNet has learned to attend. In particular, we feed an input into the model and greedily decode an output sequence, yielding a CTC alignmentû ∈Ṽ * (following the notation of Sections 3.2 and 3.3). Then, we compute the gradient of t p(û t |x) with respect to the input video frame sequence, but unlike Simonyan et al. (2013), we use guided backpropagation (Springenberg et al., 2014). Second, we train LipNet to predict ARPAbet phonemes, instead of characters, to analyse visual phoneme similarities using intra-viseme and inter-viseme confusion matrices.
LipNet: End-to-End Sentence-level Lipreading
In this section, we analyse the learned representations of LipNet from a phonological perspective. First, we create saliency visualisations (Simonyan et al., 2013;Zeiler & Fergus, 2014) to illustrate where LipNet has learned to attend. In particular, we feed an input into the model and greedily decode an output sequence, yielding a CTC alignmentû ∈Ṽ * (following the notation of Sections 3.2 and 3.3). Then, we compute the gradient of t p(û t |x) with respect to the input video frame sequence, but unlike Simonyan et al. (2013), we use guided backpropagation (Springenberg et al., 2014). Second, we train LipNet to predict ARPAbet phonemes, instead of characters, to analyse visual phoneme similarities using intra-viseme and inter-viseme confusion matrices.
LipNet: End-to-End Sentence-level Lipreading
We apply saliency visualisation techniques to interpret LipNet's learned behaviour, showing that the model attends to phonologically important regions in the video. In particular, in Figure 2  The production of the word please requires a great deal of articulatory movement at the beginning: the lips are pressed firmly together for the bilabial plosive /p/ (frame 1). At the same time, the blade of the tongue comes in contact with the alveolar ridge in anticipation of the following lateral /l/. The lips then part, allowing the compressed air to escape between the lips (frame 2). The jaw and lips then open further, seen in the distance between the midpoints of the upper and lower lips, and the lips spread (increasing the distance between the corners of the mouth), for the close vowel /iy/ (frame 3-4). Since this is a relatively steady-state vowel, lip position remains unchanged for the rest of its duration (frames 4-8), where the attention level drops considerably. The jaw and the lips then close slightly, as the blade of the tongue needs to be brought close to the alveolar ridge, for /z/ (frames 9-10), where attention resumes.
LipNet: End-to-End Sentence-level Lipreading
Lay is interesting since the bulk of frontally visible articulatory movement involves the blade of the tongue coming into contact with the alveolar ridge for /l/ (frames 2-6), and then going down for the vowel /ey/ (frames 7-9). That is exactly where most of LipNet's attention is focused, as there is little change in lip position.
LipNet: End-to-End Sentence-level Lipreading
According to DeLand (1931) and Fisher (1968), Alexander Graham Bell first hypothesised that multiple phonemes may be visually identical on a given speaker. This was later verified, giving rise to the concept of a viseme, a visual equivalent of a phoneme (Woodward & Barber, 1960;Fisher, 1968). For our analysis, we use the phoneme-to-viseme mapping of Neti et al. (2000), clustering the phonemes into the following categories: Lip-rounding based vowels (V), Alveolar-semivowels (A), Alveolar-fricatives (B), Alveolar (C), Palato-alveolar (D), Bilabial (E), Dental (F), Labio-dental (G), and Velar (H). The full mapping can be found in Table 4 in Appendix A. The GRID corpus contain 31 out of the 39 phonemes in ARPAbet. We compute confusion matrices between phonemes and then group phonemes into viseme clusters, following Neti et al. (2000). Figure 3 shows the confusion matrices of the 3 most confused viseme categories, as well as the confusions between the viseme categories. The full phoneme confusion matrix is in Figure 4 in Appendix B.
LipNet: End-to-End Sentence-level Lipreading
According to DeLand (1931) and Fisher (1968), Alexander Graham Bell first hypothesised that multiple phonemes may be visually identical on a given speaker. This was later verified, giving rise to the concept of a viseme, a visual equivalent of a phoneme (Woodward & Barber, 1960;Fisher, 1968). For our analysis, we use the phoneme-to-viseme mapping of Neti et al. (2000), clustering the phonemes into the following categories: Lip-rounding based vowels (V), Alveolar-semivowels (A), Alveolar-fricatives (B), Alveolar (C), Palato-alveolar (D), Bilabial (E), Dental (F), Labio-dental (G), and Velar (H). The full mapping can be found in Table 4 in Appendix A. The GRID corpus contain 31 out of the 39 phonemes in ARPAbet. We compute confusion matrices between phonemes and then group phonemes into viseme clusters, following Neti et al. (2000). Figure 3 shows the confusion matrices of the 3 most confused viseme categories, as well as the confusions between the viseme categories. The full phoneme confusion matrix is in Figure 4 in Appendix B.
LipNet: End-to-End Sentence-level Lipreading
Given that the speakers are British, the confusion between /aa/ and /ay/ (Figure 3a) is most probably due to the fact that the first element, and the greater part, of the diphthong /ay/ is articulatorily identical with /aa/: an open back unrounded vowel (Ferragne & Pellegrino, 2010). The confusion of /ih/ (a rather close vowel) and /ae/ (a very open vowel) is at first glance surprising, but in fact in the sample /ae/ occurs only in the word at, which is a function word normally pronounced with a reduced, weak vowel /ah/. /ah/ and /ih/ are the most frequent unstressed vowels and there is a good deal of variation within and between them, e.g. private and watches (Cruttenden, 2014).
LipNet: End-to-End Sentence-level Lipreading
Given that the speakers are British, the confusion between /aa/ and /ay/ (Figure 3a) is most probably due to the fact that the first element, and the greater part, of the diphthong /ay/ is articulatorily identical with /aa/: an open back unrounded vowel (Ferragne & Pellegrino, 2010). The confusion of /ih/ (a rather close vowel) and /ae/ (a very open vowel) is at first glance surprising, but in fact in the sample /ae/ occurs only in the word at, which is a function word normally pronounced with a reduced, weak vowel /ah/. /ah/ and /ih/ are the most frequent unstressed vowels and there is a good deal of variation within and between them, e.g. private and watches (Cruttenden, 2014).
LipNet: End-to-End Sentence-level Lipreading
The confusion within the categories of bilabial stops /p b m/ and alveolar stops /t d n/ (Figures 3b-c) is unsurprising: complete closure at the same place of articulation makes them look practically identical. The differences of velum action and vocal fold vibration are unobservable from the front.
LipNet: End-to-End Sentence-level Lipreading
Finally, the quality of the viseme categorisation of Neti et al. (2000) is confirmed by the fact that the matrix in Figure 3d is diagonal, with only minor confusion between alveolar (C) and palatoalveolar (D) visemes. Articulatorily, alveolar /s z/ and palato-alveolar /sh zh/ fricatives are distinguished by only a small difference in tongue position: against the palate just behind the alveolar ridge, which is not easily observed from the front. The same can be said about dental /th/ and alveolar /t/.
LipNet: End-to-End Sentence-level Lipreading
Finally, the quality of the viseme categorisation of Neti et al. (2000) is confirmed by the fact that the matrix in Figure 3d is diagonal, with only minor confusion between alveolar (C) and palatoalveolar (D) visemes. Articulatorily, alveolar /s z/ and palato-alveolar /sh zh/ fricatives are distinguished by only a small difference in tongue position: against the palate just behind the alveolar ridge, which is not easily observed from the front. The same can be said about dental /th/ and alveolar /t/.
LipNet: End-to-End Sentence-level Lipreading
We proposed LipNet, the first model to apply deep learning to end-to-end learning of a model that maps sequences of image frames of a speaker's mouth to entire sentences. The end-to-end model eliminates the need to segment videos into words before predicting a sentence. LipNet requires neither hand-engineered spatiotemporal visual features nor a separately-trained sequence model.
LipNet: End-to-End Sentence-level Lipreading
Our empirical evaluation illustrates the importance of spatiotemporal feature extraction and efficient temporal aggregation, confirming the intuition of Easton & Basala (1982). Furthermore, LipNet greatly outperforms a human lipreading baseline, exhibiting 4.1× better performance, and 4.8% WER which is 2.8× lower than the word-level state-of-the-art (Gergen et al., 2016) in the GRID corpus.
LipNet: End-to-End Sentence-level Lipreading
Our empirical evaluation illustrates the importance of spatiotemporal feature extraction and efficient temporal aggregation, confirming the intuition of Easton & Basala (1982). Furthermore, LipNet greatly outperforms a human lipreading baseline, exhibiting 4.1× better performance, and 4.8% WER which is 2.8× lower than the word-level state-of-the-art (Gergen et al., 2016) in the GRID corpus.
LipNet: End-to-End Sentence-level Lipreading
While LipNet is already an empirical success, the deep speech recognition literature (Amodei et al., 2015) suggests that performance will only improve with more data. In future work, we hope to demonstrate this by applying LipNet to larger datasets, such as a sentence-level variant of that collected by Chung & Zisserman (2016a).
LipNet: End-to-End Sentence-level Lipreading
While LipNet is already an empirical success, the deep speech recognition literature (Amodei et al., 2015) suggests that performance will only improve with more data. In future work, we hope to demonstrate this by applying LipNet to larger datasets, such as a sentence-level variant of that collected by Chung & Zisserman (2016a).
LipNet: End-to-End Sentence-level Lipreading
Some applications, such as silent dictation, demand the use of video only. However, to extend the range of potential applications of LipNet, we aim to apply this approach to a jointly trained audiovisual speech recognition model, where visual input assists with robustness in noisy environments.
LipNet: End-to-End Sentence-level Lipreading
In this appendix, we provide additional details about the implementation and architecture.
LipNet: End-to-End Sentence-level Lipreading
A.1 IMPLEMENTATION LipNet is implemented using Torch, the warp-ctc CTC library (Amodei et al., 2015), and Stanford-CTC's decoder implementation. The network parameters were initialised using He initialisation (He et al., 2015), apart from the square GRU matrices that were orthogonally initialised, as described in (Chung et al., 2014). The models were trained with channel-wise dropout (dropout rate p = 0.5) after each pooling layer and mini-batches of size 50. We used the optimiser Adam (Kingma & Ba, 2014) with a learning rate of 10 −4 , and the default hyperparameters: a first-moment momentum coefficient of 0.9, a second-moment momentum coefficient of 0.999, and the numerical stability parameter = 10 −8 .
LipNet: End-to-End Sentence-level Lipreading
A.1 IMPLEMENTATION LipNet is implemented using Torch, the warp-ctc CTC library (Amodei et al., 2015), and Stanford-CTC's decoder implementation. The network parameters were initialised using He initialisation (He et al., 2015), apart from the square GRU matrices that were orthogonally initialised, as described in (Chung et al., 2014). The models were trained with channel-wise dropout (dropout rate p = 0.5) after each pooling layer and mini-batches of size 50. We used the optimiser Adam (Kingma & Ba, 2014) with a learning rate of 10 −4 , and the default hyperparameters: a first-moment momentum coefficient of 0.9, a second-moment momentum coefficient of 0.999, and the numerical stability parameter = 10 −8 .
LipNet: End-to-End Sentence-level Lipreading
The CER and WER scores were computed using CTC beam search with the following parameters for Stanford-CTC's decoder: beam width 200, α = 1, and β = 1.5. On top of that, we use a character 5-gram binarised language model, as suggested in (Graves & Jaitly, 2014).
LipNet: End-to-End Sentence-level Lipreading
The CER and WER scores were computed using CTC beam search with the following parameters for Stanford-CTC's decoder: beam width 200, α = 1, and β = 1.5. On top of that, we use a character 5-gram binarised language model, as suggested in (Graves & Jaitly, 2014).
LipNet: End-to-End Sentence-level Lipreading
The videos were processed with DLib face detector (King, 2009) and the iBug face shape predictor with 68 landmarks (Sagonas et al., 2013). The RGB input frames were normalised using the following per-channel means and standard deviations: [µ R = 0.7136, σ R = 0.1138, µ G = 0.4906, σ G = 0.1078, µ B = 0.3283, σ B = 0.0917]. Table 3 summarises the LipNet architecture hyperparameters, where T denotes time, C denotes channels, F denotes feature dimension, H and W denote height and width and V denotes the number of words in the vocabulary including the CTC blank symbol. 
LipNet: End-to-End Sentence-level Lipreading
The videos were processed with DLib face detector (King, 2009) and the iBug face shape predictor with 68 landmarks (Sagonas et al., 2013). The RGB input frames were normalised using the following per-channel means and standard deviations: [µ R = 0.7136, σ R = 0.1138, µ G = 0.4906, σ G = 0.1078, µ B = 0.3283, σ B = 0.0917]. Table 3 summarises the LipNet architecture hyperparameters, where T denotes time, C denotes channels, F denotes feature dimension, H and W denote height and width and V denotes the number of words in the vocabulary including the CTC blank symbol. 
LipNet: End-to-End Sentence-level Lipreading
Size / Stride / Pad Input size Dimension order STCNN 3 × 5 × 5 / 1, 2, 2 / 1, 2, 2 75 × 3 × 50 × 100 T × C × H × W Pool 1 × 2 × 2 / 1, 2, 2 75 × 32 × 25 × 50 T × C × H × W STCNN 3 × 5 × 5 / 1, 2, 2 / 1, 2, 2 75 × 32 × 12 × 25 T × C × H × W Pool 1 × 2 × 2 / 1, 2, 2 75 × 64 × 12 × 25 T × C × H × W STCNN 3 × 3 × 3 / 1, 2, 2 / 1, 1, 1 75 × 64 × 6 × 12 T × C × H × W Pool 1 × 2 × 2 / 1, 2, 2 75 × 96 × 6 × 12
LipNet: End-to-End Sentence-level Lipreading
Note that spatiotemporal convolution sizes depend on the number of channels, and the kernel's three dimensions. Spatiotemporal kernel sizes are specified in the same order as the input size dimensions. The input dimension orderings are given in parentheses in the input size column.
LipNet: End-to-End Sentence-level Lipreading
Layers after the Bi-GRU are applied per-timestep.
LipNet: End-to-End Sentence-level Lipreading
Baseline-LSTM replicates the setup of Wand et al. (2016), and is trained the same way as LipNet.
LipNet: End-to-End Sentence-level Lipreading
Baseline-LSTM replicates the setup of Wand et al. (2016), and is trained the same way as LipNet.
LipNet: End-to-End Sentence-level Lipreading
The model uses two LSTM layers with 128 neurons. The input frames were converted to grayscale and were down-sampled to 50 × 25px, dropout p = 0, and the parameters were initialised uniformly with values between [−0.05, 0.05]. Table 4 shows the phoneme to viseme clustering of Neti et al. (2000) and Figure 4 shows LipNet's full phoneme confusion matrix.  
LipNet: End-to-End Sentence-level Lipreading
The model uses two LSTM layers with 128 neurons. The input frames were converted to grayscale and were down-sampled to 50 × 25px, dropout p = 0, and the parameters were initialised uniformly with values between [−0.05, 0.05]. Table 4 shows the phoneme to viseme clustering of Neti et al. (2000) and Figure 4 shows LipNet's full phoneme confusion matrix.  
LiRA: Learning Visual Speech Representations from Audio through Self-supervision
Self-supervised learning aims to leverage unlabelled data by extracting the training objective directly from the input itself, in an attempt to model meaningful representations of the proposed modality which capture its content and structure. In works adopting this methodology, this task is usually known as the "pretext task" and this initial training procedure is known as the "pre-training" stage. After pre-training, the network is trained on the "downstream task", which generally involves a smaller set of manually labelled data. This methodology has received substantial attention in recent years within the computer vision community. Pretext tasks for visual self-supervision include image colourisation [42], jigsaw puzzle solving [21], as well as combinations of these and other tasks [12]. Self-supervised learning has also been explored in the speech community through works such as Contrastive Predicting Coding (CPC) [22] and wav2vec [34], which predict/discriminate future segments of audio samples; LIM (Local Info Max) [32], which maximises mutual information for the same speaker; and, more recently, PASE (Problem Agnostic Speech Encoder) [26,33], which predicts established audio features such as STFT and MFCC.
LiRA: Learning Visual Speech Representations from Audio through Self-supervision
Self-supervision has also been adopted in the audiovisual domain. Recent approaches include audiovisual fusion [27,28], clustering [4], and distillation [31]; cross-modal discrimination [23]; cyclic translation between modalities [30]; and permutative predictive coding [38]. Shukla et al. [35] focus * denotes equal contribution to this work. on learning audio representations by facial reconstruction from waveform speech. Conversely, [24] predict frequency-based summaries of ambient sound from video, while other recent works apply audio-visual synchronisation [5,7,13] to learn visual embeddings. A task that can benefit from self-supervised learning is lip-reading. Current state-of-the-art lip-reading models rely on annotating hundreds of hours of visual speech data [18], which is costly. To solve this issue, Afouras et al. [3] propose using a pre-trained Automatic Speech Recognition (ASR) model to produce machine-generated captions for unsupervised pre-training. This provides automatically labelled data but still relies on an ASR model trained on large amounts of labelled data.
LiRA: Learning Visual Speech Representations from Audio through Self-supervision
In this work, we aim to leverage the vast amount of available audiovisual speech data to learn generic visual speech features and improve state-of-the-art lip-reading models by predicting audio features from visual speech. The targeted audio features are extracted from waveform audio without the need for additional labels using an established speech encoder (PASE+ [33]). Using the proposed approach, the learnt visual features are explicitly guided by audio which contains rich information about speech. This in turn can lead to learning visual features which are more suitable for speech recognition. After this training procedure, we apply our model (Fig. 2) for lip-reading on a transcribed visual speech dataset.
LiRA: Learning Visual Speech Representations from Audio through Self-supervision
Our research contributions are as follows: 1) We present LiRA, which learns powerful visual speech representations by predicting acoustic features from raw video taken from large audio-visual datasets. 2) We demonstrate that LiRA provides a good initialisation for fine-tuning lip-reading models which consistently outperforms training from scratch, and that this method is particularly beneficial for smaller labelled datasets. 3) We show that LiRA outperforms previous selfsupervised methods for word-level lip-reading, achieving an accuracy of 88.1% on LRW by pre-training on unlabelled data. 4) Finally, we leverage our self-supervised approach towards sentence-level lip-reading, and find that our fine-tuned model achieves state-of-the-art performance for LRS2.
LiRA: Learning Visual Speech Representations from Audio through Self-supervision
LiRA predicts PASE+ features from raw video and is composed of three distinct components. The first is the spatial encoder, which is a traditional 2D ResNet-18 preceded by a 3D front-end layer. The second component is the temporal encoder -the conformer -which receives as input the frame-wise features produced by the spatial encoder and returns a set of features of the same size. The conformer encoder combines traditional attention-based transformer blocks, which excel at capturing global temporal dependencies, with convolutional layers, which model local patterns efficiently [10]. The final component is the projection head (based on the MLP -Multi-Layer Perceptronworkers presented in [26]), which projects these representations into the predicted PASE+ features. To train the model, we apply an L1 loss between the generated embeddings and the features extracted from the pre-trained (frozen) PASE+ model, as shown in Fig. 1. We would also like to mention that we have also experimented with predicting MFCC features but the results were worse than predicting PASE+ features.
LiRA: Learning Visual Speech Representations from Audio through Self-supervision
To evaluate the visual speech representations, we run three variations of end-to-end lip-reading experiments. The training procedure is illustrated in Fig. 2. LiRA-Supervised models are trained from scratch based on the same encoder as in the self-supervised training [16]. This serves as our baseline model since it is trained only with the labelled training data. LiRA-Frozen models are trained using LiRA features from the pre-trained encoder. This allows us to evaluate the visual representations learned during self-supervised learning. Finally, LiRA-FineTuned models use the same model as LiRA-Supervised but are initialised with the pre-trained encoder weights from the pretext task. By using this configuration, we can evaluate the model initialisation capabilities of the proposed self-supervised learning approach. For each of these methods, we adopt a separate model for each lipreading task -six models in total. For word-level lip-reading, we use a Multi-Scale Temporal Convolutional Network (MS-TCN) [19] on top of the encoder, followed by a linear classifier for classification. For sentence-level lip-reading, we follow the state-of-the-art lip-reading model [16] on LRS2 and build a hybrid CTC/attention model. We use the same conformer encoder architecture as in the pre-training phase, followed by the transformer decoder for sequence-to-sequence training [39]. We also perform fine-tuning experiments using the pre-trained model.
LiRA: Learning Visual Speech Representations from Audio through Self-supervision
In this work, we use an unlabelled version of Lip Reading Sentences 3 (LRS3) for pre-training and evaluate the performance of speech representations on LRW and LRS2. LRW [6] is comprised of approximately 500 000 1.16 second labelled utterances (173 hours in total) featuring a specific word from a 500 word vocabulary. It features hundreds of different speakers recorded in a variety of different backgrounds and head poses. LRS2 [1] is composed of approximately 150 000 transcribed utterances of varying lengths (224.5 hours in total). This corpus presents  LRS3 [2] similarly contains approximately 150 000 utterances of varying lengths (438.9 hours in total) taken from TED talks. However, these utterances are substantially longer than the ones featured in LRS2, resulting in effectively double the total amount of hours of video and a larger vocabulary. This dataset guarantees no overlap between the speakers featured in the train and test sets, meaning that the test set is entirely comprised of speakers that were not seen in other sets.
LiRA: Learning Visual Speech Representations from Audio through Self-supervision
To crop the mouth Regions of Interest (ROIs), we start by detecting the 68-point facial landmarks using dlib [11]. We then normalise each frame using a neutral reference frame to remove rotation and size differences.
LiRA: Learning Visual Speech Representations from Audio through Self-supervision
Given the transformed facial landmarks, a fixed bounding box is used to crop mouth ROIs with a size of 96 × 96.
LiRA: Learning Visual Speech Representations from Audio through Self-supervision
Following [15], we produce augmented visual streams by applying the techniques of horizontal flipping with a probability of 0.5 and random cropping to a size of 88 × 88. During the testing phase, instead of randomly cropping, we crop a patch of size 88 × 88 from the centre of the image. For the word-level classification, mixup with a weight of 0.4 is employed. Self-supervised 71.6 PT-CDDL [8] Self-supervised 75.9 AV-PPC [38] Self-supervised 84.8
LiRA: Learning Visual Speech Representations from Audio through Self-supervision
LiRA-Supervised [15] Supervised 87.4 LiRA-Frozen Self-supervised 83.1 LiRA-FineTuned Self-supervised 88.1
LiRA: Learning Visual Speech Representations from Audio through Self-supervision
The 3D front-end module preceding our ResNet consists of a convolutional layer with kernel size (5, 7, 7) followed by a max pooling layer. The conformer, on the other hand, is comprised of an initial embedding module -feed forward layer combined with layer normalisation, dropout (0.1), activation (ReLU -Rectified Linear Unit) and relative positional encoding (as proposed in [9]) -followed by a set of conformer [10] blocks which varies according to the dataset used for the downstream task (6 blocks for LRW, 12 blocks for LRS2). The conformer blocks feature the following parameters: d ff = 2048, n head = 4, d q = 256, d k = 256, d v = 256; where d ff is the hidden dimension of the feed-forward modules, n head is the number of self-attention heads, and d q , d k , d v are the dimensions of the key (K), query (Q), and value (V) in the self-attention layers respectively. The MLP consists of a linear layer with a hidden dimension of 256 units, ReLU activation, dropout, and a linear layer to project the representation to 256-dimensional latent space. For prediction, we average the PASE+ features, which are computed at 100 frames per second (fps), over time to match the frame rate of the input visual features (25 fps). We optimise our model using Adam (β1 = 0.9, β2 = 0.98, = 10 −9 ) combined with the Noam scheduler [39] (25 000 warm-up steps). The model is trained on LRS3 with a batch size of 32. For simplicity, we randomly sample 1 second from each clip and use it as the input to our network, discarding any utterances with less than 1 second in length.
LiRA: Learning Visual Speech Representations from Audio through Self-supervision
LiRA-Supervised In LiRA-Supervised, we train word-level (Fig. 2a) and sentence-level lip-reading models (Fig. 2d) from scratch. In particular, for the task of word-level lip-reading, we add a MS-TCN followed by a linear classifier with an output dimension of 500 on top of the encoder like [17]. A crossentropy loss is employed to optimise the whole model using Adam with decoupled Weight decay (AdamW) [14] with β1 = 0.9, β2 = 0.999, = 10 −8 and a weight decay of 0.01 for 80 epochs with a batch size of 32. The initial learning rate is set to 0.0003. For the task of sentence-level lip-reading, we use 12 multi-head attention blocks (d ff = 2048, n head = 4, d q = 256, d k = 256, d v = 256) together with a linear layer on the top of conformer blocks like [16]. Following [20], we use a combination of CTC and cross-entropy loss to train a hybrid CTC/Attention architecture for 50 epochs with a batch size of 8. In this case, we use Adam with β1 = 0.9, β2 = 0.98 and = 10 −9 with the first 25 000 steps for warm-up. The initial learning rate is set to 0.0004. At the decoding phase, we use a res-b3 res-b4 ce-b1 ce-b2 ce-b3 ce-b4 ce-b5 ce-b6
LiRA: Learning Visual Speech Representations from Audio through Self-supervision
Features from Different Layers in Self-supervised Training   70   75   80   85 Accuracy (%) LiRA -Frozen Figure 3: Accuracy of feature classification (LiRA-Frozen) on LRW based on features extracted from different layers after pre-training on LRS3 via self-supervision. "res-b3" and "res-b4" refer to the output of blocks 3 and 4 from the ResNet-18 respectively; and "ce-b2" to "ce-b12" refer to the layers from every two conformer blocks from bottom to top.
LiRA: Learning Visual Speech Representations from Audio through Self-supervision
beam size of 20 for beam search. During decoding, we also apply a transformer-based language model trained on LRS2, LRS3, and Librispeech 960h [25] (16.2 million words in total). Due to graphic memory limitations, we exclude utterances with more than 600 frames during training.
LiRA: Learning Visual Speech Representations from Audio through Self-supervision
LiRA-Frozen At the end of self-supervised training, the features extracted from the pre-trained frozen encoder are fed to a classifier for evaluation. For word-level lip-reading, we use a MS-TCN, followed by a linear layer with an output size of 500 for classification (Fig. 2b). For the sentence-level lip-reading, the LiRA features are first fed to 12 conformer blocks, and then the encoded representations are used for CTC/attention joint training (Fig. 2e).
LiRA: Learning Visual Speech Representations from Audio through Self-supervision
LiRA-FineTuned We follow the same hyperparameter setting as LiRA-Supervised, but instead of training from scratch, we initialise the encoder with the pre-trained weights from the pretext task and then fine-tune the entire model for word-level lip-reading (Fig. 2c) and sentence-level lip-reading (Fig. 2f).
LiRA: Learning Visual Speech Representations from Audio through Self-supervision
We first evaluate the performance of LiRA-Supervised by training the model from scratch. This leads to an accuracy of 87.4% on LRW which is very close to the state-of-art performance. For LiRA-Frozen, which is pre-trained on LRS3, the learnt visual speech representations are evaluated on word-level lip-reading by training a MS-TCN classifier on top of the frozen representations, as illustrated in Fig. 2b. Feature extraction performance (LiRA-Frozen) for different layers is portrayed in Fig. 3. We observe that the representations extracted from the last layer of the ResNet-18 achieve a maximum accuracy of 83.1 % as seen in Table 1. It is clear that the performance generally decreases as the layer becomes deeper, which may indicate that the features extracted in deeper layers are further tuned towards the pretext task and therefore fail to generalise as well for other tasks. The performance of the 3 downstream scenarios while varying the amount of training data on LRW is shown in Fig. 4a. We use LRS3 for self-supervised pre-training. We observe that the feature extraction approach leads to superior performance compared to LiRA-Supervised when using smaller fractions of the labelled training set (1-2 %). This indicates that the pre-trained model learns useful visual features which also work well on LRW. By adopting this methodology, we can simply train the classification layers while the encoder remains frozen, and hence   Fig. 4a. We also observe that the gap between the performance of LiRA-FineTuned and LiRA-Supervised becomes smaller when we increase the amount of labelled data for training. This demonstrates that pre-training using the proposed self-supervised task is particularly beneficial when the labelled training set is very small. In the extreme case, where only 1 % of the labelled training data is used, LiRA-Supervised achieves an accuracy of 3.6 %. In contrast, we obtain 24.5 % accuracy when LiRA-FineTuned is trained using the same amount of data. This is mainly due to the fact that the self-supervised training provides a good initialisation for network training. We also show that LiRA-FineTuned provides an absolute improvement of 0.8 % in accuracy over LiRA-Supervised when both are trained on full LRW. This demonstrates that LiRA-FineTuned consistently outperforms LiRA-Supervised, even for larger labelled training sets.
LiRA: Learning Visual Speech Representations from Audio through Self-supervision
To investigate the performance of visual speech representations in a more challenging task, we run training from scratch (Fig. 2d) and fine-tuning (Fig. 2f) experiments on LRS2 after pre-training on LRS3. We present our results as a function of the fraction of labelled data used during training.
LiRA: Learning Visual Speech Representations from Audio through Self-supervision
Results are shown in Fig. 4b. It is evident that the performance of LiRA-FineTuned significantly outperforms the su-pervised baseline. We also observe that the performance of LiRA-Supervised is hard to optimise without a good initialisation. The performance becomes worse and worse as the training set increases beyond 18 % of the total amount of labelled data. This is likely due to the large amount of very long utterances featured in LRS2, which makes training from scratch especially difficult. To overcome this problem, we use curriculum learning. In particular, we first train the model using 11 % of the labelled training set, which is composed of videos with less than 155 frames in length and then use this model for initialisation when training on the entire training set. This curriculum learning strategy results in a substantially more effective training procedure, achieving 39.1 % WER for the full dataset.
LiRA: Learning Visual Speech Representations from Audio through Self-supervision
Fine-tuning the self-supervised model leads to a small improvement over the curriculum learning strategy resulting in a 38.8 % WER. This is the new state-of-the-art performance on the LRS2 dataset when no external labelled datasets are used for training. We also observe that it leads to a 9.5 % absolute improvement compared to the previous state-of-the-art model [1], as reported in Table 2. Furthermore, as displayed in Fig. 4, we are able to outperform the previous state-of-the-art of 48.3 % WER using 18× fewer labelled data -76 hours (36 % of LRS2) vs 1 362 hours (MVLRS, LRS2, and LRS3).
LiRA: Learning Visual Speech Representations from Audio through Self-supervision
We present LiRA, which learns visual speech representations by cross-modal self-supervised learning. We train a visual model by predicting acoustic features from visual speech, and observe that it can be adapted for lip-reading with remarkable success. By fine-tuning our models for this new task, we achieve an accuracy of 88.1 % on LRW and report a WER of 38.8 % on LRS2. Given the extent of modern audiovisual corpora, we believe it would be promising to leverage this method towards other visual tasks such as emotion recognition and speaker recognition in the future.
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
Human perception of speech is intrinsically multimodal, involving audition and vision. The speech production is accompanied by the movement of lips and teeth, which can be visually interpreted to understand speech. Visual cues of speech not only play an essential role in language learning for pre-lingual children (Meltzoff & Moore, 1977;Davies et al., 2008), but also improve speech understanding in noisy environment (Sumby & Pollack, 1954) and provide patients of speech impairment with means of communication. Furthermore, perceptual studies (McGurk & MacDonald, 1976) have shown that such visual cues can alter the perceived sound.
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
For machine learning models, the tight coupling between audio and visual lip movement information emerges as a natural source for supervision to learn speech representations, which has not been extensively utilized yet in the self-supervised speech representation learning literature. Recent successful representation learning frameworks for speech (e.g., APC (Chung et al., 2019), CPC (Oord et al., 2018;Kharitonov et al., 2021), wav2vec 2.0 (Baevski et al., 2020;Hsu et al., 2021b), De-CoAR2.0 (Ling & Liu, 2020), HuBERT (Hsu et al., 2021c;a)) are mostly built entirely on audio. The fundamental research question addressed in this paper is whether a self-supervised audio-visual speech representation learned from the lip movement information, alongside the audio signal in video recordings, captures cross-modal correlations and improves downstream performance for visual speech recognition (i.e., lip reading) and automatic speech recognition (ASR) tasks. Existing ML models for lip-reading rely heavily on text transcriptions to achieve an acceptable level of accuracy. The state-of-the-art lip-reading model (Makino et al., 2019) requires 31K hours of transcribed video data for training. Such large amounts of labeled data are expensive and hard to obtain for most of the world's 7,000 languages. The benefit from a robust visual speech representation learning framework goes beyond lip-reading. Additionally, it can benefit a vast range of applications, 2 RELATED WORK The strong correlation between video modalities provides an effective means for self-supervised representation learning on videos, which has been explored in many prior works and is still an active research area. This work draws inspiration from two lines of previous research:
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
Multimodal general video representation focuses on learning self-supervised audio-visual representations of general videos to solve high-level semantic tasks, e.g., action recognition and audio event detection (Arandjelović & Zisserman, 2017;Bruno et al., 2018;Morgado et al., 2021;Chen et al., 2020;Lee et al., 2021). Owens & Efros (2018) learns a multimodal network to predict whether the audio and visual streams of a video are temporally synchronized while Pham et al. (2019) applies a cyclic translation between different modalities. Piergiovanni et al. (2020) learns the visual representation through a multi-tasking framework that incorporates a series of pretext tasks such as reconstruction and temporal ordering prediction. Sharing our work's inspiration of DeepClustering (Caron et al., 2018), XDC (Alwassel et al., 2020) and AV-BERT (Chan et al., 2021) learn cross-modal representations through predicting cross-modal or cluster assignments. In contrast to XDC, AV-HuBERT is trained with a BERT-like masked prediction loss, which forces the model to learn the structure within the multimodal input and was shown in Hsu et al. (2021c) to be more resilient to bad cluster assignments compared to unmasked cluster prediction. On the other hand, AV-BERT focuses on learning utterance-level multimodal environment embeddings that serves as the global context for ASR, while our objective is to learn frame-level audio-visual speech representations and pre-train a model that can be fine-tuned for downstream tasks with either modality.
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
Semi-and self-supervised audio-visual speech representation learning focuses on improving lip-reading with untranscribed audio-visual speech data. To solve isolated visual word recognition,  learns visual embeddings using a contrastive loss based on audio-visual synchronization. Ma et al. (2021a) learns visual speech representations by minimizing the distance between its latent features and off-the-shelf audio embeddings. Using an external supervised ASR to transcribe unlabeled audio,  trains their lip-reading model on the augmented labeled and pseudo-labeled data. Unlike Ma et al. (2021a) and , our model is trained from scratch and encouraged to learn contextualized representations with a masked prediction task. Moreover, our method does not rely on any external pre-trained models.
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
Our research builds on Audio HuBERT (Hsu et al., 2021a) which is a self-supervised learning framework for speech and audio. It alternates between two steps: feature clustering and masked prediction. In the first step, a discrete latent variable model (e.g., k-means) is applied to a sequence of acoustic frames A 1:T producing a sequence of frame-level assignments z a 1:T . Clusters of signalprocessing-based acoustic features, e.g., Mel-frequency cepstral coefficients (MFCC), exhibit nontrivial correlations with the inherent acoustic units of speech inputs. Using (A 1:T , z a 1:T ) pairs, the second step learns new feature representations by minimizing a masked prediction loss, similar to masked language modeling in BERT (Devlin et al., 2019). The pressure to predict cluster assignments of masked audio regions forces the model to learn good local acoustic representations for unmasked regions and long-range temporal dependencies between latent features. Repeating these two steps improves the cluster quality and consequently the quality of the learned representations.
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
Single-modal Visual HuBERT: The most naïve way to extend HuBERT to the visual domain is by generating targets using visual features. Formally, given an image sequence I 1:T , we first cluster the image features into a sequence of discrete units z i 1:T via k-means: z i t = k-means(G(I t )) ∈ {1, 2, ..., V }, where G is an visual feature extractor and V is the codebook size. The cluster assignments z i 1:T serve as the prediction targets of the model. Initially, G can be an engineered image feature extractor such as Histogram of Oriented Gradients (HoG), analogous to MFCC in audio HuBERT. The intermediate layers of the HuBERT model are used as G in later iterations.
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
To perform the masked prediction task, the model first encodes I 1:T using a ResNet into an intermediate visual feature sequence f v 1:T , which is then corrupted intof v 1:T via a binary mask M . Specifically, ∀t ∈ M ,f v t is replaced with a learned masked embedding. We adopt the same strategy in HuBERT to generate span masks. The masked visual featuresf v 1:T are encoded into a sequence of contextualized features e 1:T via a transformer encoder followed by a linear projection layer. The loss is computed over the masked regions and optionally over unmasked ones (when α ≥ 0):
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
Where (W ∈ R d×V , b ∈ R V ) are parameters of the projection layer which maps features into logits predicting the cluster assignments.
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
Cross-modal Visual HuBERT: The single-modal visual HuBERT aims to learn visual speech representation through gradually refined image features. However, it does not employ the audio stream of the video. Presumably, audio features, e.g., MFCC or a pre-trained audio HuBERT model, correlate with phones better than vanilla image features (e.g., HoG) do. To this end, we train an audio encoder based on the aligned audio frame sequence A 1:T in parallel to the visual encoder. The iterative training alternates between the two encoders. In each iteration, an audio encoder E a is utilized to generate target cluster assignments z a 1:T . The visual encoder E v is trained subsequently with (I 1:T , z a 1:T ). The z a 1:T is also used to train the next iteration of the audio encoder E a for refinement. The cross-modality visual HuBERT can be seen as modeling visual inputs by distilling knowledge from the audio stream, where z a 1:T represents the audio-side knowledge. We hypothesize that the audio feature is more favorable to speech representation learning than the visual feature, which is validated in the Section E.1. Critical for the lip-reading downstream task, the masked prediction objective used by HuBERT forces the model to capture temporal relationships, which facilitates prediction of homophemes, which are groups of sounds with identical visual shapes (e.g., 'p'-'b', 'f'-'v', 'sh'-'ch') that are impossible to distinguish using a single image frame.  Figure 1: Illustration of AV-HuBERT. Masked prediction losses are only computed for the three middle frames, because at least one modality is masked for those frames. See section A for its comparison between single-modal and cross-modal visual HuBERT.
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
Our primary model in this work is Audio-Visual HuBERT (AV-HuBERT), shown in figure 1, which is trained iteratively by alternating between feature clustering and masked prediction in a similar way to the Visual HuBERT but with four main improvements:
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
Audio-visual input: The AV-HuBERT model consumes both acoustic and image frames for the masked prediction training, which enables better modeling and distillation of the correlations between the two modalities. Specifically, image sequences and acoustic features pass through their light-weight modality-specific encoders to produce intermediate features, which are then fused and fed into a shared backbone transformer encoder to predict masked cluster assignments. The targets are generated from clustering audio features or features extracted from the previous iteration of the AV-HuBERT model. When fine-tuned for lip-reading, we drop the audio input to work solely with the visual input. The input discrepancy is addressed by modality dropout described next.
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
Modality dropout: Audio-visual speech recognition models can relate audio input to lexical output more effortlessly than the visual input stream, as observed in the literature (Afouras et al., 2018a;Ma et al., 2021b). This causes the audio modality to dominate model decisions. The problem is aggravated in our setting because the target cluster assignments are initially generated from acoustic features. To prevent the model's over-reliance on the audio stream in our joint model, we only use a linear layer to encode acoustic input to force the audio encoder to learn simple features.
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
Additionally, before fusing audio and visual inputs into the backbone transformer encoder, dropout is applied to mask the full features of one modality; we refer to it as modality dropout. With a probability p m , both modalities are used as input. When only one modality is used, the audio stream is selected with a probability of p a . Formally, given the encoded audio and visual feature sequence f a 1:T and f v 1:T , equation 2 shows feature fusion equipped with modality dropout:
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
where concat denotes channel-wise concatenation. Note that modality drop out is applied at the sequence level instead of at the frame-level, which effectively tasks AV-HuBERT to perform masked prediction with visual-only, audio-only, or audio-visual input. Modality dropout prevents the model from ignoring video input and encourages the model to produce the prediction regardless of what modalities are used as input. Furthermore, since the fine-tuning and inference phases use the visual stream alone (no audio input), this modality dropout mechanism bridges the gap between pretraining (multimodal) and fine-tuning/inference (single-modality). A similar dropout mechanism is used in prior work (Zhang et al., 2019a;Makino et al., 2019;Neverova et al., 2014;Abdelaziz et al., 2020) to increase the robustness in multi-modal settings. We verify the modality dropout effectiveness in Section D.
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
Audio-visual clustering: One benefit of pre-training on both modalities is the ability to generate multimodal cluster assignments that serve as target labels for the masked prediction task of the next iteration. In contrast to the Cross-modal Visual HuBERT where targets are generated from audio-based features or a prior Audio HuBERT model, the targets for AV-HuBERT are naturally multimodal after the first iteration. Lip movement sequences provide complementary information to the audio stream. Combining both modalities produces cluster assignments of higher quality for AV-HuBERT, as shown in Section E.1.
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
We propose a novel masking strategy for AV-HuBERT that masks segments in the visual stream by substituting them with random segments from the same video. More formally, given an input video I v 1:T , an imposter video I v,f 1:T f and a mask consisting of n intervals M = {(s i , t i )} 1≤i≤n , we corrupted I v 1:T intoĨ v 1:T by setting:
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
where p i is a sampled integer offset from the interval [0, T f − t i + s i ]. Now, to solve the task, the model needs to first identify the fake frames and then infer the labels belonging to the original frames. Since the "filled-in" segments are from real video segments and temporally smooth, the fake segment detection sub-task becomes less trivial compared to when using vanilla masking or substitution with non-consecutive frames. We show an ablation confirming the advantage of the proposed masking strategy in Section D.
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
The audio and visual segments are masked independently using two different masking probabilities m a and m v . We hypothesize that the difficulty of the masked prediction task differs for each modality: inferring the masked targets given the audio stream is more straightforward than using the lip movement stream. Setting a high masking probability for acoustic frames is essential to help the whole model capture the language characteristics. On the contrary, setting a high masking probability for the visual input provide the model with more imposter segments than the original ones, hurting its ability to learn meaningful features (studied in Section D of the appendix). Given the output probability p 1:T and target cluster assignments z 1:T , the AV-HuBERT pre-training loss is:
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
where M a and M v denotes the frames that are masked for the audio and the visual stream. α is a hyperparameter weighting the contribution of the unmasked regions in the overall objective.
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
The proposed pre-training approaches can be fine-tuned for visual speech recognition using any sequence classification loss. In this paper, we focus on fine-tuning with the connectionist temporal classification (CTC) (Graves et al., 2006) and attention-based sequence-to-sequence cross-entropy loss (Bahdanau et al., 2016) (S2S for brevity), which are the most popular choices. Assume that the feature sequence output of our pre-trained model is e 1:T and the ground-truth transcription is w = w 1 , w 2 , ..., w s . For CTC, a projection layer is used to map the visual feature sequence into the output probabilities:
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
and U is the output vocabulary size (+1: plus blank symbol). The model is trained with CTC loss:
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
where B maps an alignment sequence π to w. For S2S, a tranformer decoder is appended to the pre-trained encoder to autoregressively decode the feature sequence e 1:T into target probabilities p(w t |w 1:t−1 , e 1:T ). The whole model is trained with cross entropy loss: L s2s = − s t=1 log p(w t |w 1:t−1 , e 1:T ).
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
We conduct experiments on two datasets: LRS3 (Afouras et al., 2018b) with 433 hours of transcribed English videos and VoxCeleb2 (Chung et al., 2018) with 2442 hours of unlabeled multilingual videos. We only use the English portion of VoxCeleb2, which amounts to 1,326 hours of content. The inputs to our backbone model are lip Regions-Of-Interest (ROIs) for the visual stream and log filterbank energy feature for the audio stream. The image encoder is a modified ResNet-18, which has been used in prior work (Ma et al., 2021b;Martinez et al., 2020;Stafylakis & Tzimiropoulos, 2017). The audio encoder is simply a linear projection layer. We consider two model configurations: BASE with 12 transformer blocks and LARGE with 24 transformer blocks. For BASE and LARGE, the embedding dimension/feed-forward dimension/attention heads in each transformer block are 768/3072/12 and 1024/4096/16 respectively. The number of parameters in BASE and LARGE are 103M and 325M respectively. α in equation 4 is set to 0.
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
The model uses five iterations of feature clustering and masked prediction during pre-training. See Section B.4, E for details on clustering. For fine-tuning, we use phone targets for the CTC loss and unigram-based subword units (Kudo, 2018) for the S2S loss. For decoding the CTC-trained model, we use a 4-gram language model trained on LRS3 training text. For the S2S fine-tuned model, we rely only on its own decoder module to incorporate language information, with no external language model employed during inference. To further show the complementary relationship between AV-HuBERT and existing approaches of using unlabeled data, we also experiment on combining AV-HuBERT with self-training. Specifically, we generate pseudo labels for unlabeled data using a finetuned HuBERT, and fine-tune the pre-trained AV-HuBERT model with the combination of pseudolabeled videos and original labeled videos. Note that no additional data is used when combined with self-training. More details about the used datasets, data pre-processing, and model training are in Section B. Table 1 compares the performance of our AV-HuBERT pre-training approach to previously published supervised, semi-supervised, and self-supervised lip-reading systems using different amounts of labeled and unlabeled data. Since the CTC and S2S fine-tuning approaches have similar trends, only S2S results are shown in table 1. Complete results of CTC fine-tuning are in Table C shows that AV-HuBERT and self-training are complementary to each other. Note the overall gain is attributed mainly to AV-HuBERT as self-training alone leads to much worse performance (> 50% WER). More details can be found in section C.3. Many prior models pre-train their visual front-end, e.g., ResNet-18, using word-level annotated lip-reading videos, which is costly to collect since they require word boundary information. In contrast to these models, our models are fully pre-trained from scratch using the proposed approach.
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
Compared to the semi-supervised approach Jasper-KD , which transcribed 334 hours of the English data in VoxCeleb2 using a pre-trained ASR system, 2 our best model achieves 29% lower absolute WER benefiting from VoxCeleb2 for pre-training. Even when limiting our model to the LRS3 data for pre-training and fine-tuning, our model surpasses their semi-supervised system by 18%. Compared to LiRA (Ma et al., 2021a), a recently proposed self-supervised model for lip-reading, AV-HuBERT-BASE provides 17.5% lower absolute WER on average for low-resource and high-resource settings. The implementation details of LiRA are provided in Section B.5.
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
With the same network architecture, our pre-training approach significantly reduces WER compared to training from scratch, in both low-resource (92.3% → 32.5%, LARGE) and high-resource (62.3% → 28.6%, LARGE) settings. A qualitative view of the improvement can be found in Section F. Additionally, we notice that our AV-HuBERT pre-training helps under a fully supervised setting. Using the LRS3 data only (433 hours), pre-training followed by fine-tuning (41.6%, LARGE) outperforms training a lip-reading model from scratch (62.3%, LARGE) to predict the output text. AV-HuBERT, pre-trained on video-audio pairs, learns better fine-grained visual representation than the scratch model trained on video-text pairs. The benefits of AV-HuBERT pre-training in various labeled data setups can be found in Section C.1.
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
We compare AV-HuBERT against a suite of alternatives, including the Single-modal and Crossmodal Visual HuBERT in Table 2. All the models are BASE pretrained on 433 hours of unlabeled data and fine-tuned on 30 hours of labeled data. For this comparison, we use CTC fine-tuning due to its computational efficiency and given their similarity in results trends to S2S. 
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
Iter-2
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
As shown in table 2, target cluster assignments driven from a single modality, either audio or visual, do not provide much WER reduction beyond the second iteration. Training the AV-HuBERT model using targets driven from audio-visual features keeps improving for more iterations and achieves better final performance. As mentioned in Section 3, visual information is complementary to audio input; hence, using both produces higher quality target clusters. Measuring the quality of different target labels using cluster quality metrics such as purity and NMI shows the same trend observed from lip-reading WER (See appendix E.1).
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
Fixing target labels used for the masked-prediction pre-training, AV-HuBERT (AV/MFCC→A) outperforms the Cross-modal Visual HuBERT (V/MFCC→A) by a large margin. AV-HuBERT effectively transfers knowledge from audio into the visual encoder and the backbone transformer model to benefit visual-only fine-tuning and inference. In contrast to AV-HuBERT, iterative pre-training brings much smaller gains to single-modality visual HuBERT ("V/MFCC→V", "V/HoG→V").
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
Starting with audio features is critical for learning effective target labels for the masked-prediction task. Phonetic information, which is crucial for lip-reading, is primarily present in the audio stream. All the models considered so far are based on audio feature MFCC clustering in their initial iteration. As is shown in the "V/HoG→V" row, using hand-engineered visual features provides a lousy starting point for iterative learning. Visual features such as HoG mainly incorporate low-level visual cues such as edges and luminance, which is irrelevant to the downstream recognition task. 
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
Since the correlation between lip movements and the produced sound is governed by the vocal apparatus that is language-agnostic, the proposed AV-HuBERT can utilize multi-lingual data for such learning. This would be particularly beneficial for low-resource languages. Nevertheless, the masked language modeling aspect in AV-HuBERT is still language-dependent, implying that mixing other languages would increase the language domain discrepancy. To study how these two factors affect AV-HuBERT, we compare using monolingual English data only versus multilingual videos in the pre-training phase. In particular, we vary the amount of English data we use in pre-training while the number of non-English utterances is fixed to 1,116 hours in all the settings. For simplicity, we train AV-HuBERT (BASE) for one iteration with MFCC clusters and fine-tune it with CTC using 30 hours of labeled data. As is shown in table 3, using non-English data in pre-training significantly reduces WER when there are no or very little English data in pre-training (≤ 100 hours). As we increase the amount of English data, the gain diminishes because the out-of-domain effect brought by non-English data outweighs the benefit of the overall increase in pre-training data. Using the whole English data only for pre-training is better than combining it with other languages (59.9% vs. 64.3%). Training with 5 iterations leads to similar results (47.3% vs. 48.9%). This experiment highlights the importance of the domain match between pre-training and fine-tuning data. For zero/low-resource settings, merging data from other languages in pre-training benefits the downstream task. When the unlabeled data from the target language is abundant, limiting the pre-training data to one language is beneficial.
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
The multimodal clusters produced by AV-HuBERT, which have higher quality than audio-HuBERT targets, can also benefit speech pre-training. To test our hypothesis, we trained an audio-HuBERT, with only audio input during the masked-prediction pre-training, for one iteration with cluster assignments driven from AV-HuBERT features. We also pre-trained an audio-HuBERT from scratch using clusters driven from the MFCC features for three iterations. The two pre-trained models are evaluated on a downstream ASR task. Table 4 shows the performance of different models fine-tuned on the ASR task. We only include the performance of S2S fine-tuning for our models as it consistently outperforms the CTC fine-tuning. An audio-HuBERT pre-trained using targets generated by AV-HuBERT features outperforms the vanilla audio-HuBERT in low-resource (30h) and high-resource settings (433h) fine-tuning settings across different model architectures. With the same amount of labeled data, our best model (1.4%) outperforms the prior SOTA (2.3%) even without an external language model during inference.
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
Given that the AV-HuBERT model utilizes both modalities at its input, it can be fine-tuned, in principle, for the ASR downstream task. In practice, we notice pre-training an audio-HuBERT with audio-visual cluster leads to better ASR performance (3.8%) than a pre-trained AV-HuBERT (4.6%), potentially due to its hyperparameters being selected based on lip reading rather than ASR. In fact, audio-HuBERT can be treated as a special case of AV-HuBERT with p m = 0, p a = 1. 
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
We presented multiple pre-training models for visual speech recognition. Our AV-HuBERT model leverages the strong correlation between the audio and lip movement streams for self-supervised audio-visual speech representation learning. Our pre-training approaches iteratively alternate between feature clustering and learning new features through a masked-prediction loss. The AV-HuBERT model consumes masked image and audio frames to predict target cluster assignments. The targets are initially generated from MFCC features and gradually refined through iterative training. Experiments on visual speech recognition show that AV-HuBERT achieves SOTA using 433 hours of text transcriptions, two orders of magnitude less than the 31,000 hours of labeled data used in the prior best approach. When using only one-thousandth of labeled data, the lip-reading performance outperforms the prior SOTA by more than 10% (relative). AV-HuBERT also improves the representation for the ASR downstream task. An audio-HuBERT model trained with targets generated by an AV-HuBERT model shows superior performance, achieving the SOTA in the audio-based speech recognition in the LRS3 dataset. As future work, AV-HuBERT can be applied for multilingual lip-reading in low-resource languages. Additionally, our approach can be extended to other applications of visual speech representation, such as speech enhancement and generation.
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
All the data used in this paper are publicly available and are used under the following three licenses: the TED terms of use, the Creative Commons BY-NC-ND 4.0 license and Creative Commons Attribution 4.0 International License. Through spot-checking, we find the datasets are gender balanced and cover a wide range of races and ages. However, the distribution of speakers in the data may not be representative of the global human population. Please be cautious of unintended societal, gender, racial and other biases caused by the fact. To maintain anonymity, only the mouth area of a speaker is visualized wherever used in the paper. The proposed method can be applied in several areas including security and crime investigations. However it can also be used for malicious purposes such as surveillance and wiretapping. We are committed to distributing our code and model carefully, with special attention to any potential security and privacy concerns.
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
Our code and models are publicly available. In the meantime, we include as many implementation details as we can in the paper.  YouTube. In the original dataset, the training data is split into two partitions: pretrain (403 hours) and trainval (30 hours). Both parts are transcribed at the sentence level and come from the same source as the test set. The pretrain set differs from trainval in that the duration of its video clips are of a much wider range and can be shorter or longer than a full sentence. In the low-resource setup, we only use trainval as the labeled data. As no official development set is provided, we randomly select 1,200 sequences from trainval as the validation set (about 1 hour) for early stopping and hyperparameter tuning.
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
VoxCeleb2 (Chung et al., 2018) is originally created for multilingual audio-visual speaker recognition and it contains over 2,442 hours of utterances of over 6,000 speakers extracted from YouTube videos. The dataset naturally serves our purpose as it does not contain ground-truth transcriptions. The VoxCeleb2 has substantial domain discrepency to the LRS3 data as its utterances are from multiple languages and includes videos in a larger variety of domains including interviews, talks, excerpts under indoor and outdoor environments. In VoxCeleb2, by default we only use the English portion for pre-training. As no ground-truth language label is given in the VoxCeleb2, we use a simple heuristic to choose the English samples. Specifically, we use an off-the-shelf character-based ASR model (Hsu et al., 2021a) trained on Librispeech which achieves 1.9%/3.5% WER on clean/other test set. We run greedy decoding on the VoxCeleb2 and use the proportion of valid English words to determine if a target utterance is English or not. An utterance is only selected if the proportion of valid English words is higher than 60%. The total amount of unlabeled data after filtering is 1,326 hours.
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
For each video clip, we detect the 68 facial keypoints using dlib (King, 2009) and align each frame to a reference face frame via affine transformation. We crop a 96×96 region-of-interest (ROI) centered on the mouth. Each image frame is converted to grayscale. We randomly cropped 88 × 88 from the whole ROI and randomly flipped it horizontally with probablity of 0.5 during training. At test time, 88 × 88 ROI is center cropped and does not go through horizontal flipping. The preprocessing steps remain same as prior works in lip reading (Martinez et al., 2020;Ma et al., 2021b). For the associated audio, we extract the 26-dimensional log filterbank energy feature at a stride of 10 ms from the raw waveform and use it as input to the model. As the image frames are sampled at 25Hz, we stack the 4 neighboring acoustic frames to synchronize the two modalities.
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
In the modified ResNet-18 (Ma et al., 2021b;Martinez et al., 2020;Stafylakis & Tzimiropoulos, 2017), the first convolutional layer is substituted by a 3D convolutional layer with kernel size 5 × 7 × 7. The visual feature tensor is flattened into a single-dimensional vector through a 2D average pooling layer in the end. We use one linear projection layer as the audio encoding module. The acoustic features are normalized by per-frame statistics before feeding into the network (Ba et al., 2016). We use a dropout of p = 0.1 after the self-attention block within each transformer layer, and each transformer layer is dropped (Fan et al., 2020) at a rate of 0.1.
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
Pretraining Our models are implemented with fairseq (Ott et al., 2019). The whole network is randomly initialized before pre-training. In pre-training, the model is trained for five iterations in total. For the initial iteration, we generate the targets by running k-means clustering algorithm on 39-dimensional MFCC feature (13 coefficients with its first-and second-order derivatives) extracted from raw audio waveform. For the subsequent iterations, the feature from an intermediate layer of the model in the previous iteration is used for clustering. The layer index (one-based) used for clustering in iteration 1-4 are {9, 12, 12, 12}. The number of features are clustered to {100, 100, 500, 1000, 2000} respectively for the 5 iterations. See section E.3 for analysis. To save training time, we always use the BASE model to generate clusters and LARGE model is only trained in the 5th iteration.
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
We set both p m and p a to 0.5 for modality dropout at training time. To extract features for clustering, both modalities are used. We adopt the strategy used in wav2vec 2.0 (Baevski et al., 2020) to generate masks, where p% of all frames are randomly selected as start and subsequent l frames are masked. In iteration 1-4, we mask the fused features and set p/l to be 8/10 respectively as we observe such practice generates higher quality cluster assignments (see section E.2). In the last iteration, we set p/l to be 6/5 for video and 8/10 for audio (see section D).
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
We train the model with Adam (Kingma & Ba, 2015), warming up the learning rate for the first 8% of updates to a peak of 0.002 and then linearly decay it. Videos are batched together to not exceed 1,000 image frames (40 seconds) per GPU. Both BASE and LARGE models are updated for 400K and 600K steps at each iteration, respectively in 433h/1759h unlabeled settings. We train on 32 and 64 V100-GPUs for BASE and LARGE. On average, each iteration takes ∼ 2.0/3.0 days for BASE and ∼ 2.4/3.6 days for LARGE in using 433h/1759h unlabeled data for pre-training.
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
Fine-tuning After pre-training, we fine-tune the AV-HuBERT model on labeled (video, text) pairs. The audio encoder is removed and its output is replaced by a zero-vector. In CTC fine-tuning, a randomly initialized projection layer is added on top of the transformer to map features into phonemes. The lexicon is constructed with CMUDict (cmu). In fine-tuning with S2S, we use a 6-layer/9-layer transformer decoder on BASE and LARGE model to decode features into unigram-based subword units (Kudo, 2018). The vocabulary size in CTC and S2S are 46 and 1000 respectively.
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
In CTC, the pre-trained model is updated from the initial iteration without any freezing. The model is fine-tuned for 30K/100K steps respectively in 30h/433h setting. In S2S, the pre-trained model (i.e., encoder) is frozen for the first N % updates . N is 100 and 50 for 30h and 433h labeled setting respectively. The entire model is trained for 18K/45K steps in the 30h/433h setting. Both models are trained with Adam, with the learning rate being warmed up for the first P % of updates to a peak of 0.001 and linearly decayed. P is tuned among {10, 30, 50}. All hyperparamters are tuned on the validation set.
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
Decoding For CTC, we use a 4-gram language model trained on text data in the LRS3 training set. The perplexity of the 4-gram LM on test set is 110.5. No language model is used for S2S decoding. For CTC, we tune the beam width among {5, 10, 20, 50, 100, 150}, the language model weight among {0, 1, 2, 4, 8} and word insertion penalty among {±4, ±2, ±1, 0}. For S2S, The beam width and length penalty are tuned among {5, 10, 20, 50} and {0, ±1}. The tuning is done on the validation set.
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
We apply self-training on LARGE AV-HuBERT. Specifically, the fine-tuned LARGE HuBERT model(A/MFCC→AV, Table 4) is used to assign pseudo-labels to the unlabeled audiovisual data. In 30h/433h setting, the amount of data for fine-tuning A/MFCC→AV are 30h and 433h respectively. The pre-trained AV-HuBERT LARGE is fine-tuned with the pseudo-labeled videos and videos with ground-truth text labels (30h/433h). Note the data used here is exactly same with the case of using AV-HuBERT only.
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
We re-implemented the LiRA (Ma et al., 2021a) training objective in our framework, as there does not exist publicly available implementations and we aim to focus the comparison on the pre-training objective rather than the architectural difference. We use the same backbone architecture as the BASE AV-Hubert except the output layer being a linear project layer with an output dimension of 256. The 256-dimensional frame PASE+ feature is extracted from the audio with its official implementation in (Ravanelli et al., 2020). The original PASE+ feature is downsampled to 25Hz for synchronization with the visual stream. The pre-trained model is fine-tuned in both CTC and S2S. The optimizer and learning rate schedule in pre-training, hyperparameter search in fine-tuning and decoding remain the same as AV-HUBERT. Note with our implementation, the WER is reduced by 23% (94.3% → 71.9%) using LiRA when the percentage of labeled data is 6.9% (30h labeled, 433h in total), while Ma et al. (2021a) achieves ∼ 10% improvement in a similar setting.
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
C.1 AMOUNT OF LABELED DATA Table C.1 shows the effect of pre-training on different amount of labeled data for fine-tuning. We use 433 hours of unlabeled data (LRS3 only) and randomly selected 1, 10 and 100 hours of labeled data for fine-tuning. Overall pre-training brings large and consistent gains across different amount of labeled data. Specifically CTC-based fine-tuning outperforms S2S-based fine-tuning in lowresource settings (1-hour and 10-hour). The larger number of parameters as well as the lack of a language model for decoding makes S2S model more likely to overfit especially when the amount of fine-tuning data is small.   Table C.3 shows the performance of only applying self-training. The WER of a self-training only model is significantly higher than AV-HuBERT and self-trained AV-HuBERT, which suggests that the gain of the combined approach is primarily from AV-HuBERT.
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
C.4 FULL RESULTS WITH CTC FINE-TUNING Table C.4 shows the full results on LRS3, which includes the CTC fine-tuning performance for all the models we implemented. In general, the conclusions we draw from S2S (e.g., the benefits of our pre-training approach in different settings, the improvement over LiRA) in section 4.2 holds for CTC as well. 
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
The ablation studies in this section are done in the last iteration of the AV-HuBERT, pre-trained with 433 hours of unlabeled data. The model is fine-tuned with 30 hours of labeled data using CTC. Masking Strategy In the first part of table D.1, we compare the proposed masking strategy against several alternatives. Feature masking applies span mask at feature level and leads to the worst performance, which is due to the leakage of information to ResNet. Directly masking image sequence with random Gaussian noise or a learned embedding slightly improves the performance by preventing the prior issue. However, those artificial frames also corrupts the raw image sequence and enlarges the domain gap in videos between pre-training and fine-tuning. On the other hand, our proposed method achieves better performance. Specifically, using segments from the same utterance ("Sub, (same, seg)") as the imposter leads to the best result compared to sampling from a different utterance ("Sub, (diff, seg)") or sampling non-consecutive frames from the same utterance ("Sub, (same, frm)"). 3 The filled-in fake images are visually similar to the raw images and substitution with a segment keeps the temporal smoothness making the replaced segment more realistic, which enforces the ResNet to encode more fine-grained visual details.
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
Masking probability We set two different masking probabilities for audio and visual stream in the last iteration. The probabilities of an acoustic frame and image frame being masked are 0.8 and 0.3 respectively. As is shown in the second part of table D.1, setting masks for audio and visual stream independently is essential because the optimal masking probability for audio and visual stream are different. Audio encoder tends to deteriorate into a simple acoustic feature extractor when mask length is small. On the other hand, long visual mask will lead to the model lacking context to distinguish between fake and real images.
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
Modality dropout The third part of table D.1 compares the model performance with and without modality dropout. Randomly dropping audio sequence prevents the model from over-relying on audio for masked prediction and helps the visual representation learning.
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
Where to compute prediction loss In the last part of table D.1, we compare the choice of masked prediction vs. prediction. The loss weight on unmasked region does not have a large impact on the fine-tuning performance. This is different from the findings in Audio HuBERT Hsu et al. (2021a), where masked prediction leads to much better performance. Given image frames as input, the prediction of cluster assignments, which are mostly determined by the accompanied audio stream, helps encode phonetic information into the visual representation. The task is much less trivial than a single-modal model (audio-HuBERT), where setting non-zero weight on unmasked prediction can easily make the model deteriorate into an acoustic feature extractor. In addition, the high quality of targets in the last iteration also makes such prediction more helpful.
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
For analysis, we use frame-level phonetic labels as the ground-truth and match its correlation between cluster assignments. The phonetic labels are obtained via forced alignement from a monophone based HMM-GMM ASR model trained on LRS3. In particular, we use clustering purity and Normalize Mutual Information (NMI) as the evaluation metrics. Table E.1 shows that (1) The quality of cluster assignments is consistent with fine-tuning performance across different models (2) Hand-engineered audio feature (MFCC, NMI: 21.5%) has much stronger correlation with phonetic labels than the visual feature (HoG, NMI: 1.6%) (3) Audio-visual clusters (NMI: 44.2%) are of better quality than pure audio-based clusters (NMI: 39.7%). (4) In single-modality visual Hubert (V/HoG→V), feature quality is improved negligibly through iteration training. 
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
In iteration 1-4, we apply the mask in the fused feature. We observe such practice generates targets of higher quality, thus helping future iterations more. Table E.2 shows a comparison between such two different masking strategies. Input-level masking enhances the learning of visual representation (see table D.1) while produces worse audio-visual feature (NMI: 27.2%). In contrast, the two streams of original input are better aligned in feature-level masking which is consistent with the cluster generation process, thus leading to better audio-visual clusters (NMI: 37.7%).  The cluster assignment quality generally improves with more iterations. In the first iteration, features in the middle layers show higher quality than the other layers. The target for the first iteration (MFCC clusters) is of worse quality, thus later layers that are more correlated with targets do not yield the best cluster. Target quality improves with more training iterations, thus the best feature layer shifts towards the end. Setting a larger number of clusters increases the clustering quality as can be seen from the comparison between "varied clusters" and "2K clusters". In terms of the 12th layer which we choose, the highest NMI (44.2%) is achieved in the last iteration. In addition, more iterations of training improves the overall quality of clusters produced by a model though the highest NMI/purity among all layers does not necessarily increase in later iterations. Therefore, setting a larger number of iterations brings stable gains which are more robust to the index of layer chosen for clustering. It is important as the purity/NMI, whose measurement rely on a supervised model, are not used for hyperparameter search in practice. Our self-supervised model is the LARGE AV-HUBERT pre-trained with 1,759 hours unlabeled data and fine-tuned with 433 hours labeled data. The baseline model is the supervised baseline trained with 433 hours labeled data. Both models use the S2S criterion for supervised training and have the same number of parameters. Qualitatively, our approach provides transcriptions with much higher quality. The baseline approach confuses among words of similar sound while our model output more semantically sound sentences. Figure F.2 also shows typical errors made by our model. We noticed many errors are on short sentences. This is mainly because lip reading relies heavily on the context for recognition due to the existence of homophomes. Thus the error rates in lip reading are notably higher in short utterances, which differs from ASR, as can be seen from figure F.1. Substitution among words with homophemes ('fiction' vs. 'vision' in a.4, 'part' vs. 'bunk' in b.5) is another source of error made by the model. 