How has ASR been used to improve lip-reading models?
2.2 Transfer Learning from Audio to Video
Although the two parallel speech modalities, audio
and video, remain aligned at the temporal level,
ASR is still far more accurate than lip reading,
benefiting from its easy access to labeled audio
utterance and fine-grained phoneme audio repre-
sentation. Ma et al. (2021) hopes to take advantage
of this natural temporal alignment and use audio to
assist with lip-reading training. Ren et al. (2021)
proposes different distillation strategies (from ASR
to lip-reading) which enables lip-reading to learn
complementary and discriminant clues from ASR.
Shi et al. (2022) has adopted audio assisted pre-
training methods, that regard audio as the auxiliary
supervision for visual utterances in order to obtain
fine-grained phoneme visual representations. How-
ever, in previous methods (Ren et al., 2021; Huang
et al., 2021), audio only played the role of auxiliary
supervision, and the lip-reading models could not
be trained using non-visual utterances (i.e., audio)
alone. In our work, we attempt to maintain the
alignment of audio and video in phoneme space,
and employ only labeled audio utterances instead
of visual utterances to train the lip-reading model.
OpenSR: Open-Modality Speech Recognition via Maintaining Multi-Modality Alignment
Speech Recognition builds a bridge between the multimedia streaming (audio-only, visual-only or audio-visual) and the corresponding text transcription. However, when training the specific model of new domain, it often gets stuck in the lack of new-domain utterances, especially the labeled visual utterances. To break through this restriction, we attempt to achieve zero-shot modality transfer by maintaining the multi-modality alignment in phoneme space learned with unlabeled multimedia utterances in the high resource domain during the pre-training, and propose a training system Open-modality Speech Recognition (OpenSR) that enables the models trained on a single modality (e.g., audio-only) applicable to more modalities (e.g., visual-only and audio-visual). Furthermore, we employ a cluster-based prompt tuning strategy to handle the domain shift for the scenarios with only common words in the new domain utterances. We demonstrate that OpenSR enables modality transfer from one to any in three different settings (zero-, few- and full-shot), and achieves highly competitive zero-shot performance compared to the existing few-shot and full-shot lip-reading methods. To the best of our knowledge, OpenSR achieves the state-of-the-art performance of word error rate in LRS2 on audio-visual speech recognition and lip-reading with 2.7% and 25.0%, respectively.
[{"first":"Xize","last":"Cheng","middle":[]},{"first":"Tao","last":"Jin","middle":[]},{"first":"Linjun","last":"Li","middle":[]},{"first":"Wang","last":"Lin","middle":[]},{"first":"Xinyu","last":"Duan","middle":[]},{"first":"Zhou","last":"Zhao","middle":[]}]
LiRA: Learning Visual Speech Representations from Audio through Self-supervision
The large amount of audiovisual content being shared online today has drawn substantial attention to the prospect of audiovisual self-supervised learning. Recent works have focused on each of these modalities separately, while others have attempted to model both simultaneously in a cross-modal fashion. However, comparatively little attention has been given to leveraging one modality as a training objective to learn from the other. In this work, we propose Learning visual speech Representations from Audio via self-supervision (LiRA). Specifically, we train a ResNet+Conformer model to predict acoustic features from unlabelled visual speech. We find that this pre-trained model can be leveraged towards word-level and sentence-level lip-reading through feature extraction and fine-tuning experiments. We show that our approach significantly outperforms other self-supervised methods on the Lip Reading in the Wild (LRW) dataset and achieves state-of-the-art performance on Lip Reading Sentences 2 (LRS2) using only a fraction of the total labelled data.
[{"first":"Pingchuan","last":"Ma","middle":[]},{"first":"Rodrigo","last":"Mira","middle":[]},{"first":"Stavros","last":"Petridis","middle":[]},{"first":"Bjorn","last":"Schuller","middle":["W."]},{"first":"Maja","last":"Pantic","middle":[]}]
Learning from the Master: Distilling Cross-modal Advanced Knowledge for Lip Reading
Lip reading aims to predict the spoken sentences from silent lip videos. Due to the fact that such a vision task usually performs worse than its counterpart speech recognition, one potential scheme is to distill knowledge from a teacher pretrained by audio signals. However, the latent domain gap between the cross-modal data could lead to a learning ambiguity and thus limits the performance of lip reading. In this paper, we propose a novel collaborative framework for lip reading, and two aspects of issues are considered: 1) the teacher should understand bi-modal knowledge to possibly bridge the inherent cross-modal gap; 2) the teacher should adjust teaching contents adaptively with the evolution of the student. To these ends, we introduce a trainable "master" network which ingests both audio signals and silent lip videos instead of a pretrained teacher. The master produces logits from three modalities of features: audio modality, video modality, and their combination. To further provide an interactive strategy to fuse these knowledge organically, we regularize the master with the task-specific feedback from the student, in which the requirement of the student is implicitly embedded. Meanwhile, we involve a couple of "tutor" networks into our system as guidance for emphasizing the fruitful knowledge flexibly. In addition, we incorporate a curriculum learning design to ensure a better convergence. Extensive experiments demonstrate that the proposed network outperforms the state-of-the-art methods on several benchmarks, including in both word-level and sentence-level scenarios.
[{"first":"Sucheng","last":"Ren","middle":[]},{"first":"Yong","last":"Du","middle":[]},{"first":"Jianming","last":"Lv","middle":[]},{"first":"Guoqiang","last":"Han","middle":[]},{"first":"Shengfeng","last":"He","middle":[]}]
Multi-Singer: Fast Multi-Singer Singing Voice Vocoder With A Large-Scale Corpus
High-fidelity multi-singer singing voice synthesis is challenging for neural vocoder due to the singing voice data shortage, limited singer generalization, and large computational cost. Existing open corpora could not meet requirements for high-fidelity singing voice synthesis because of the scale and quality weaknesses. Previous vocoders have difficulty in multi-singer modeling, and a distinct degradation emerges when conducting unseen singer singing voice generation. To accelerate singing voice researches in the community, we release a large-scale, multi-singer Chinese singing voice dataset OpenSinger. To tackle the difficulty in unseen singer modeling, we propose Multi-Singer, a fast multi-singer vocoder with generative adversarial networks. Specifically, 1) Multi-Singer uses a multi-band generator to speed up both training and inference procedure. 2) to capture and rebuild singer identity from the acoustic feature (i.e., mel-spectrogram), Multi-Singer adopts a singer conditional discriminator and conditional adversarial training objective. 3) to supervise the reconstruction of singer identity in the spectrum envelopes in frequency domain, we propose an auxiliary singer perceptual loss. The joint training approach effectively works in GANs for multi-singer voices modeling. Experimental results verify the effectiveness of OpenSinger and show that Multi-Singer improves unseen singer singing voices modeling in both speed and quality over previous methods. The further experiment proves that combined with FastSpeech 2 as the acoustic model, Multi-Singer achieves strong robustness in the multi-singer singing voice synthesis pipeline. Samples are available at https://Multi-Singer.github.io/
[{"first":"Rongjie","last":"Huang","middle":[]},{"first":"Feiyang","last":"Chen","middle":[]},{"first":"Yi","last":"Ren","middle":[]},{"first":"Jinglin","last":"Liu","middle":[]},{"first":"Chenye","last":"Cui","middle":[]},{"first":"Zhou","last":"Zhao","middle":[]}]
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
Video recordings of speech contain correlated audio and visual information, providing a strong signal for speech representation learning from the speaker's lip movements and the produced sound. We introduce Audio-Visual Hidden Unit BERT (AV-HuBERT), a self-supervised representation learning framework for audio-visual speech, which masks multi-stream video input and predicts automatically discovered and iteratively refined multimodal hidden units. AV-HuBERT learns powerful audio-visual speech representation benefiting both lip-reading and automatic speech recognition. On the largest public lip-reading benchmark LRS3 (433 hours), AV-HuBERT achieves 32.5% WER with only 30 hours of labeled data, outperforming the former state-of-the-art approach (33.6%) trained with a thousand times more transcribed video data (31K hours). The lip-reading WER is further reduced to 26.9% when using all 433 hours of labeled data from LRS3 and combined with self-training. Using our audio-visual representation on the same benchmark for audio-only speech recognition leads to a 40% relative WER reduction over the state-of-the-art performance (1.3% vs 2.3%). Our code and models are available at https://github.com/facebookresearch/av_hubert
[{"first":"Bowen","last":"Shi","middle":[]},{"first":"Wei-Ning","last":"Hsu","middle":[]},{"first":"Kushal","last":"Lakhotia","middle":[]},{"first":"Abdelrahman","last":"Mohamed","middle":[]}]
OpenSR: Open-Modality Speech Recognition via Maintaining Multi-Modality Alignment
Speech Recognition (Afouras et al., 2018a;Zhao et al., 2020) (e.g., Audio-Visual Speech Recognition known as AVSR) transcribs visual and audio data into text form, building a bridge between multi-media speech (Cheng et al., 2023;Huang et al., 2023a,b; and natural language (Yin et   2022, 2021; Jin et al., 2022a,b;Jin, 2021). Among them, ASR (Automatic Speech Recognition) and lip-reading (VSR, Visual Speech Recognition) are twin tasks transcribed using only audio and only vision, respectively. Audio utterances with clear pronunciation are sufficient for ASR training, most of which can be easily collected from recordings of telephone conversations and audiobooks (Korvas et al., 2014). While current lip-reading training systems require mostly-frontal and high-resolution videos with a sufficiently high frame rate, such that motions around the lip area are clearly captured (Prajwal et al., 2022). The significant difficulty of collecting labeled visual utterances hinders the training of lip-reading models suitable for new domains or low-resource domains, resulting in the relatively low-speed development of domain-specific lip-reading models, compared with ASR models. Since audio speech is easier to obtain, can we use audio utterance alone to train lip-reading models for target domains? Humans, once they have mastered a language, can immediately predict the lip movements of a phoneme (Meltzoff and Moore, 1977). Similarly, if we align the phoneme represen-tations of acoustic and image fragments with each other, we can then apply the term distribution and syntax specification of the target domain learned from the audio utterance to lip-reading. Building on this novel idea, we employ the audio-visual aligned encoder, such as AV-Hubert, co-trained on a large number of multi-modality high resource domain utterances, to align the multi-modality utterances in the same phoneme space, and train a domain-specific decoder from phoneme space to text using the labeled audio-only utterances. Since the audio-visual encoder is trained to embed different modalities of the same phoneme near one another (for instance, the visual phoneme feature of /tu:/ is aligned with its audio phoneme feature), we can flexibly transfer the knowledge of the target domain (e.g., the mapping from phoneme /tu:/ to homophones two and too, and the syntax used to distinguish them) from a single modality (i.e., audio) to more modalities (i.e., visual-only and audio-visual). As shown in Figure 1, the zero-shot lip-reading performance of our method is even better than most full-shot methods.
OpenSR: Open-Modality Speech Recognition via Maintaining Multi-Modality Alignment
Furthermore, we propose to enhance the discrimination of visually similar words (e.g., pet and bet, which are audio distinguishable but visually similar (Kim et al., 2022)) with more readily available utterances containing only common words. However, the uneven word distribution causes the sequence modeling to be easily corrupted by domain shift during fine-tuning . We propose a cluster-based prompt tuning strategy, Cluster Prompt, only tuning the limited parameters of the fine-grained prompt-embedding of each visual phoneme cluster, to prevent excessive impact on sequence modeling.
OpenSR: Open-Modality Speech Recognition via Maintaining Multi-Modality Alignment
The main contributions are as follows:
OpenSR: Open-Modality Speech Recognition via Maintaining Multi-Modality Alignment
• To the best of our knowledge, OpenSR is the first to achieve zero-shot lip-reading, which fully considers the modality transfer.
OpenSR: Open-Modality Speech Recognition via Maintaining Multi-Modality Alignment
• OpenSR is the first low-resource lip-reading method that attempts to leverage common words, adopting the Cluster Prompt strategy to overcome the ensuing domain shift problem and improve the accuracy of lipreading by 19.1% to 36.1%.
OpenSR: Open-Modality Speech Recognition via Maintaining Multi-Modality Alignment
• OpenSR achieves modality transfer and the state-of-the-art performance in all three settings (zero-shot, few-shot and full-shot). In particular, it achieves 2.7% and 25% word error rate on LRS2 in AVSR and lip-reading.
OpenSR: Open-Modality Speech Recognition via Maintaining Multi-Modality Alignment
2 Related Work
OpenSR: Open-Modality Speech Recognition via Maintaining Multi-Modality Alignment
The lip-reading task has attracted many researchers (Cooke et al., 2006;Afouras et al., 2018a), aiming to recognize spoken sentences according to the given video of lip movements without relying on the audio stream. With the support of a large amount of visual speech utterances (Afouras et al., 2018a,b), Assael et al. (2016) first proposes to use neural networks for lip reading. (Ma et al., 2021) and (Shi et al., 2022) adopt different pre-training strategies, attempting to obtain fine-grained lip representation by pre-training on a large amount of additional unlabeled audio-visual utterances (Chung et al., 2018). Some works (Makino et al., 2019;Serdyuk et al., 2021) adopt massive labeled visual utterances (even more than 90,000h) for training and promote the generalization of the model. However, none of these methods can train lip-reading models without large amounts of labeled visual utterances, making them unusable in low-resource domains where labeled visual utterances are scarce or unavailable. This is also the original intention of our work, for which we propose a training system that can employ only labeled audio utterances to train the lip-reading model.
OpenSR: Open-Modality Speech Recognition via Maintaining Multi-Modality Alignment
Although the two parallel speech modalities, audio and video, remain aligned at the temporal level, ASR is still far more accurate than lip reading, benefiting from its easy access to labeled audio utterance and fine-grained phoneme audio representation. Ma et al. (2021) hopes to take advantage of this natural temporal alignment and use audio to assist with lip-reading training.  proposes different distillation strategies (from ASR to lip-reading) which enables lip-reading to learn complementary and discriminant clues from ASR. Shi et al. (2022) has adopted audio assisted pretraining methods, that regard audio as the auxiliary supervision for visual utterances in order to obtain fine-grained phoneme visual representations. However, in previous methods Huang et al., 2021), audio only played the role of auxiliary supervision, and the lip-reading models could not be trained using non-visual utterances (i.e., audio) alone. In our work, we attempt to maintain the alignment of audio and video in phoneme space, and employ only labeled audio utterances instead of visual utterances to train the lip-reading model. Figure 2: Illustration of OpenSR training system. In the second stage, parameters other than those in the transformer decoder are frozen to maintain the cross modality alignment state achieved in the first stage. We propose three different tuning and inference strategies in the third stage for scenarios with different scales of labeled visual utterances in the target domain.
OpenSR: Open-Modality Speech Recognition via Maintaining Multi-Modality Alignment
AV-Hubert (Shi et al., 2022) is a self-supervised representation learning method for audio-visual speech, alternating between feature clustering and mask prediction, as shown in the first stage of Figure 2. During the feature clustering, the audiovisual speech is labeled as a sequence of framelevel assignments z={z a t } T t=1 based on the sequence of image sequences V ={V t } T t=1 and audio acoustic frames A={A t } T t=1 (MFCC or audiovisual features from the previous encoder) with a discrete latent variable model (e.g., k-means). Subsequently, with the paired data (A, V, z), the model learns a better audio-visual representation in phoneme space f p ={f p t } T t=1 ∈ R T ×D , where T is the length of the sequence and D is the dimension of the embedding, by reducing the mask prediction loss, just like the mask language modeling in BERT (Devlin et al., 2018). The above two steps are repeated during training to improve the quality of audio-visual speech clustering and representation.
OpenSR: Open-Modality Speech Recognition via Maintaining Multi-Modality Alignment
Furthermore, the random dropout of modalities in the framework maps the speech features of different modalities into the same phoneme space, which not only optimizes the representation quality of the uni-modality, but also achieves cross-modality representation alignment with a large amount of source domain (i.e., high resource domain) unlabeled audio-visual utterances. OpenSR is the first attempt to take full advantage of this cross-modality alignment of speech representations.
OpenSR: Open-Modality Speech Recognition via Maintaining Multi-Modality Alignment
Now that the audio and visual representations have been mapped into the same phoneme space, we can use audio of the target domain as an alternative to video when labeled visual utterance is scarce. As shown in the second stage in Figure 2, we adopt the AV-Fusion and the pre-trained Transformer Encoder in the first stage to obtain the features of phoneme space f p ={f p t } training stage. With the parameters of AV-Fusion and the Transformer Encoder being frozen, we obtain the fusion features f m ={f m t } T t=1 ∈ R T ×D :
OpenSR: Open-Modality Speech Recognition via Maintaining Multi-Modality Alignment
and then encode them into a common phoneme space,
OpenSR: Open-Modality Speech Recognition via Maintaining Multi-Modality Alignment
The freezing parameter allows speech recognition training to focus on the decoder (from the phoneme space to the text space), while avoiding the missing visual modality utterances from destroying the cross-modality alignment relationship in the phoneme space. A tunable Transformer Decoder is appended to autoregressively decode the phoneme feature f p into the target probabilities:
OpenSR: Open-Modality Speech Recognition via Maintaining Multi-Modality Alignment
is the ground-truth transcription. In the second and third stages of training, the overall model is trained with cross-entropy loss
OpenSR: Open-Modality Speech Recognition via Maintaining Multi-Modality Alignment
Lip-reading
OpenSR: Open-Modality Speech Recognition via Maintaining Multi-Modality Alignment
The decoder trained with the target domain audio utterances fully learns the word distribution and syntactic characteristics of the target domain. Furthermore, the OpenSR training system can further tune the lip-reading model with the labeled visual utterances. Depending on the amount of visual utterances, it can be divided into three types: zeroshot, few-shot and full-shot.
OpenSR: Open-Modality Speech Recognition via Maintaining Multi-Modality Alignment
Zero-Shot In OpenSR, the target domain decoder trained in the ASR can be directly transferred to the target domain lip-reading. Both audio and video are aligned in the same phoneme space, and the co-usable decoder for multi-modality speech recognition only needs to map from phoneme space to text. In inferring on lip-reading, only the visual utterances f v ={f v t } T t=1 ∈ R T ×D inputs into AV-Fusion, and the fusion feature can be formally expressed as:
OpenSR: Open-Modality Speech Recognition via Maintaining Multi-Modality Alignment
. Full-Shot When sufficient labeled visual utterance is available for training, the model parameters can be further fine-tuned. With the word distribution and specific syntax of the target domain, the decoder can steadily boost the discrimination and representation of the pre-trained encoder for visual utterance. Few-Shot More commonly, when we can only use visual utterances containing only common words for training, the training of the model is likely to be affected by the data distribution, and the model will be more inclined to recognize common words. We adopt a prompt tuning strategy called Cluster Prompt to make the model pay more attention to local phoneme features, as shown in the Figure 3.
OpenSR: Open-Modality Speech Recognition via Maintaining Multi-Modality Alignment
In the first stage, the pre-training process uses kmeans to put cluster labels on audio-visual features during training. We further explore this clusterbased approach by tuning the learnable clustering embeddings c j ={c i j } N i=1 ∈ R N ×D for each cluster in each layer layer j of the encoder, where N is the number of the clusters. Cluster embedding is the cluster-specific fine-grained bias from audiobased to visual-based phoneme features, which is used to further enhance the visual discriminability between visually similar clusters. The input x j ={x t j } T t=1 ∈ R T ×D to each layer is first fed into a cluster network Meta, which is consisted of a linear projection layer and a cluster-wise softmax layer, to extract the cluster weights u j ∈ R T ×N for each phoneme features x j :
OpenSR: Open-Modality Speech Recognition via Maintaining Multi-Modality Alignment
and then combine the cluster embedding vector to update each audio-visual fusion feature:
OpenSR: Open-Modality Speech Recognition via Maintaining Multi-Modality Alignment
At the same time, we freeze the parameters of the encoder and the decoder to maintain the learned syntactic knowledge and reduce the tuning computational resources:  195H) and Train (29H), both of which are transcribed from videos to text at the sentence level. The only difference is that the video clips in the Pretrain partition is not strictly trimmed and sometimes longer than the corresponding text.
OpenSR: Open-Modality Speech Recognition via Maintaining Multi-Modality Alignment
We conducted experiments on LRS2 with different training data amounts (i.e., Pretrain+Train(224h) and Train (29h)). Note that since the video of the LRS3 dataset has already been used in the pretraining process of AV-Hubert, we do not conduct experiments on it.
OpenSR: Open-Modality Speech Recognition via Maintaining Multi-Modality Alignment
LRS2-COMMON Based on the LRS2 dataset, we further propose the LRS2-COMMON to verify the lip-reading performance of the few-shot model trained with labeled visual utterances containing only common words. We counted the word frequency of each word in the Train partition of LRS2, and extracted new training sets with only common words according to the word frequency. Note that during the inference, we use the complete Test and Validation that contain not only common words.
OpenSR: Open-Modality Speech Recognition via Maintaining Multi-Modality Alignment
For all experiments on LRS2, we use the word error rate (WER) as the evaluation index of speech recognition (both lip-reading and AVSR). WER can be defined as WER=(S + D + I)/M , where S, D, I, M represent the number of words replaced, deleted, inserted and referenced respectively. During validation, the inference is only performed when all of the validation utterances are of the same modality as the training utterances. For ex- ample, zero-shot lip-reading trained on labeled audio utterances should also be validated with audio utterances (the inference is performed on the visual utterances, during the testing). In Section A, we present more implement details.
OpenSR: Open-Modality Speech Recognition via Maintaining Multi-Modality Alignment
As shown in Table 2, we compare our method with the previous methods in LRS2 to highlight the effect of our proposed training system OpenSR. As the first training system that achieves zero-shot lipreading, OpenSR not only achieves the state-of-theart zero-shot and few-shot performance, but even outperforms most full-shot methods. This demonstrates that our training system can effectively train lip-reading models suitable for domains lacking labeled visual utterances. Furthermore, we demonstrate that OpenSR can improve the lip-reading capability of the full-shot lip-reading model. Since the features of audio and video are projected in the  
OpenSR: Open-Modality Speech Recognition via Maintaining Multi-Modality Alignment
same phoneme space after pre-training, a decoder suitable for both ASR and VSR can be trained using only labeled audio utterances. Benefiting from this well-trained decoder, the performance is improved by 2.1% compared to (Shi et al., 2022) using a similar framework.
OpenSR: Open-Modality Speech Recognition via Maintaining Multi-Modality Alignment
For the domains without labeled visual utterances, there are two ways to train lip-reading models using knowledge transfer: cross-domain and crossmodality. The experiments in Table 3 provide answers to the following questions about knowledge transfer:
OpenSR: Open-Modality Speech Recognition via Maintaining Multi-Modality Alignment
• How much does the domain shift affect the lip-reading model? • Is OpenSR training with cross-modality better than cross-domain transferring? When there is no labeled visual utterances for lip-reading model training in the target domain (Here with LRS2), most of the current methods train on labeled visual utterances from other high resource domains (Here with LRS3). Compared with the model trained with in-domain utterances (ID: 2,7,12,17), the performance of models trained with other domains (ID: 1,6,11,16) utterances decreased by 3.7% -11.6%, mainly because there may be terminological and syntactic differences among domains. In Section B, we thoroughly discuss the domain differences between two datasets LRS2&3 of the similar domain from the perspective of term distribution.
OpenSR: Open-Modality Speech Recognition via Maintaining Multi-Modality Alignment
In contrast, OpenSR training system learns the word distribution and syntax of the target domain from audio, which effectively avoids domain shift that affects the lip-reading model transferring between domains. As shown in Table 3, even training without labeled visual utterances, the performance of OpenSR training with only audio utterances (ID: 4,9,14,19) can achieve considerable improvement (2.8% -8.9%) in varying degrees comparing with cross-domain methods (ID: 1,6,11,16). In Section C, we further present qualitative results on crossdomain and cross-modality.
OpenSR: Open-Modality Speech Recognition via Maintaining Multi-Modality Alignment
4.5 Pre-training alone cannot achieve zero-shot modality transfer.
OpenSR: Open-Modality Speech Recognition via Maintaining Multi-Modality Alignment
With the extraordinary performance of AV-Hubert on the uni-modality speech recognition, there might be some doubt that this zero-shot modality transfer benefits entirely from AV-Hubert. Indeed, pre-training does align different modalities into the phoneme space and enhance the feature representation, but as shown in the experiment (ID: 3,8,13,18) in Table 3, the model with simple fine-tuning cannot achieve zero-shot modality transfer yet. The idea of parameter freezing in the second stage of OpenSR makes the model better maintain the alignment state between audio and visual features in phoneme space. Maintaining the multi-modality alignment is the key to zero-shot modality transfer, and OpenSR consistently achieves performance commensurate with the scale of the training utterances and model parameters with this property.  Table 4 compares the performance and tuning parameter scale with the state-of-the-art method in LRS2-COMMONs with different word frequencies. The domain shift seriously affects the training of the decoder in AV-Hubert, when the word frequency threshold of the tuning dataset (from LRS2-224H to LRS2-COMMON TF>100 ) gradually increased, the WER of lip-reading increases sharply, from 28.6% to 68.8%. By learning the word distribution and specific syntax in the target domain from audio, OpenSR ensures that the model will not overfit the common words even if only the video utterances containing a small number of common words is used, as shown in training strategy 'OpenSR + Finetune'. Furthermore, our proposed Cluster Prompt training strategy in the few-shot scenario shows a further improvement compared to training strategy 'OpenSR + Finetune' in terms of common words with a high word frequency threshold ('>100', '>50' and '>20'). Also, note that the amount of the tuning parameters (×0.02) of the Cluster Prompt is significantly smaller than the amount (×1.00) in the other strategies. In particular, with the decrease of the frequency threshold of common words, the amount of tuning utterances keeps increasing, and the influence of domain shift gradually disappears. Compared to Cluster Prompt, the strategy of fine-tuning learns more about the distribution of the target domain that is applicable to lip-reading from the common word utterances, such as sequence modeling in the decoder specifically for the visual utterances.
OpenSR: Open-Modality Speech Recognition via Maintaining Multi-Modality Alignment
We conducted ablation experiments on modules participating in fine-tuning to explore why OpenSR could optimize the upper accuracy limit of the fullshot lip-reading, as shown in Table 5. Only 3.473% (from 35.995% to 32.522%) improvement comes from tuning the decoder, while fine-tuning the pretrained encoder can achieve additional 9.569% improvement (from 35.995% to 26.426%). The limited improvement achieved by continuing to tune the decoder demonstrates that the word distribution and syntax learned in the audio modality can be zero-shot transferred to the visual modality. Meanwhile, this is why OpenSR can increase the accuracy ceiling, as the encoder pre-trained on large amounts of utterances and the decoder with knowledge of the target domain can steadily boost each other. Furthermore, we also attempt to determine the parameters that need to be tuned most during modality transfer. When tuning the last few layers in the encoder, the performance boost is the most pronounced, increasing the performance by 7.722%  needs to be further tuned while modality transfer is the target-modality representation of the encoder, so that it can distinguish visually similar words (e.g., PET and BET). By maintaining multimodality feature alignment, the decoder trained with a single modality (i.e., audio) utterance can directly apply the knowledge of the target domain to lip-reading models.
OpenSR: Open-Modality Speech Recognition via Maintaining Multi-Modality Alignment
The Cluster Prompt enables the model to provide prompt embeddings according to the clusters of different phoneme features. In Figure 4, we show the influence of the number of clusters on the performance of accuracy. Obviously, when there is only one cluster, it can be considered as providing the prompt embedding only for the visual modality, which can also be called Modality Prompt. With the increase of the number of clusters (N), the cluster division of the model becomes more refined, so that the prompt embedding also becomes more consistent with each representation. In particularly, we noticed that the accuracy decreases when the number of clusters increases to inf (the performance of inf in the figure is simply represented by n=500). This is because there is no enough data to support the training of over-differentiated cluster specific prompt embedding.
OpenSR: Open-Modality Speech Recognition via Maintaining Multi-Modality Alignment
OpenSR can also achieve zero-shot modality transfer from a single modality to multi-modality, by maintaining both audio and visual modality features aligned at the same time. As shown in Table   6, OpenSR outperforms other methods in audiovisual speech recognition. Note that, even in the zero-shot setting, OpenSR performes better than the full-shot performance previous of most previous methods. Furthermore, only using 29h visual utterance to participate in training is enough to comprehensively surpass the previous methods.
OpenSR: Open-Modality Speech Recognition via Maintaining Multi-Modality Alignment
Comparing with AV-Hubert of the same backbone, the performance of OpenSR is 0.4% better, demonstrating the significance of the term distribution and syntax specification learned during a single modality training for full-shot speech recognition in other-modality. 
OpenSR: Open-Modality Speech Recognition via Maintaining Multi-Modality Alignment
The bottleneck of domain-specific models is the lack of target domain data, such as the challenge of visual utterance collection in lip-reading. We propose OpenSR, a training system that can train target domain lip-reading models without using labeled visual utterance. The other modality models can directly utilize the target domain knowledge obtained from the single modality (e.g., audio) utterance, via maintaining the multi-modality feature alignment state learned during pre-training. This zero-shot modality transfer idea can alleviate the training problem caused by the severely scarce labeled data of some modalities. For example, despite there is no labeled visual utterance of the target domain, we can still use OpenSR to train the lip-reading model with the labeled audio utterance for the target-domain specifically. Furthermore, our training system achieves zero-shot modality transfer in a tuning-based manner, bringing a new perspective utilizing the pre-trained models, which can be transferred to other multi-modality pre-training models such as CLIP (Radford et al., 2021).
OpenSR: Open-Modality Speech Recognition via Maintaining Multi-Modality Alignment
Lip-reading has many applications, including instruction dictation in public areas or a noisy environment and information accessibility for the hearing impaired. OpenSR makes it possible to quickly build domain-specific lip-reading models for low-resource domains (lack of labeled visual utterances), which greatly enhances the fairness of lip-reading technology across the domains and languages.
OpenSR: Open-Modality Speech Recognition via Maintaining Multi-Modality Alignment
For speech recognition, there may be concerns about the risk of information leakage. But in fact, as mentioned above, the lip-reading model has a relatively high requirement on the visual utterances used for lip-reading, requiring mostly-frontal and high-resolution videos with a sufficiently high frame rate, such that motions around the lip area are clearly captured. In general, only close-range cameras or online meetings have similar video conditions, which ensures that the lip-reading model will not be abused in potentially privacy revealing scenarios such as surveillance videos.
OpenSR: Open-Modality Speech Recognition via Maintaining Multi-Modality Alignment
Audio and Visual Utterance Preprocessing. For the visual utterance, we only intercept the lip region for lip-reading. As the previous methods (Shi et al., 2022;Afouras et al., 2018a,b), we adopt dlib (King, 2009) to detect the 68 facial keypoints and align each face with its neighbors. We crop a 96 × 96 region-of-interest (ROI) talking head video centered on the mouth from each visual utterance. For the audio utterance, we also remain same preprocessing steps as the prior works (Ma et al., 2021;Shi et al., 2022). We extract the 26-dimensional log filterbank energy feature from the raw waveform and stack the 4 neighboring acoustic frames for synchronization. During training, for data enhancement, we randomly crop 88 × 88 from the whole ROI and flipped it horizontally with 0.5 probability. To improve noise robustness, we apply noise with a probability of 0.25 to each audio utterance from (Snyder et al., 2015) as steps in (Afouras et al., 2018a).
OpenSR: Open-Modality Speech Recognition via Maintaining Multi-Modality Alignment
Pre-training Setup. OpenSR builds on pretraining process of AV-Hubert (Shi et al., 2022), directly utilizing its checkpoint for the subsequent stages. During pre-training, a modified ResNet-18 used in prior works (Ma et al., 2021;Martinez, 2021) and a linear projection layer are adopted as visual and audio encoders, respectively. It considers two models with different configurations: Transformer-BASE and Transformer-LARGE have 12/24 Transformer layers with the embedding dimension/feed-forward dimension/attention heads of 768/3072/12 and 1024/4096/16. We simply adopted the pre-trained model obtained by training on LRS3 (Afouras et al., 2018b) and VoxCeleb2 (Chung et al., 2018).
OpenSR: Open-Modality Speech Recognition via Maintaining Multi-Modality Alignment
OpenSR Tuning Setup in a Single Modality. In the second stage of OpenSR, we fine-tune the decoder with the labeled audio utterance and the absent modality (visual) feature is replaced by a zerovector. For comparison, we adopt the same decoder configuration as (Shi et al., 2022), with 6 and 9 Transformer layers in Transformer-BASE and Transformer-LARGE, respectively. With the encoder parameters frozen, we fine-tune the decoder on a single 3090 GPU for 45K/120K steps in the 29h/224h setting. Note that, also only audio can be used while tuning the hyperparamters on the validation set during the second stage. Conversely, during the inference or further tuning with the visual utterance (the third stage), we only adopt the visual utterance as input and replace the audio feature with the zero-vector 0 D . Each stage of OpenSR is trained with Adam, with the learning rate being warmed up for the first 50% of updates to 0.0005.
OpenSR: Open-Modality Speech Recognition via Maintaining Multi-Modality Alignment
In this section, starting with LRS2 and LRS3, we explore the differences in word distribution in different domains. Although both datasets are extracted from television shows (BBC and TED respectively), we still find a huge difference in word distribution between them, perhaps due to the different content of the shows.
OpenSR: Open-Modality Speech Recognition via Maintaining Multi-Modality Alignment
Different frequencies of the same word in different domains. In Figure 5, we visually show the word frequency of each word in LRS2&3, each domain has more than 4,000 words that do not appear in the other's dictionary, as shown by the red dots. There are also a number of words that vary widely in word frequency from domain to domain, as shown by the points off the diagonal. In general, words that are far off the diagonal tend to be terms specific to the domain. Except for a few words with high frequencies that are common to all domains, most words have different word frequencies in different domains. From the perspective of common terms in LRS2, we further quantitatively reveal the differences of the common terms distribution in Figure 6. There are 834 words (2.02% of the LRS2 vocabulary size) with word frequency differences (greater than and less than 10, respectively), demonstrating that a large part of the terms is domain-specific. At the same time, Figure 5: Different frequencies of the same word in LRS2&3. The color shows the number of words at that position. Figure 6: The number of terms whose TF lrs2 ≥ K lrs2 and TF lrs3 ≤ K lrs3 . TF lrs2&3 and K lrs2&3 are the word frequency of each word and the counting threshold on LRS2&3, respectively. there are a number of terms that varied greatly between domains: 88 words with frequency over 10 (e.g., bargain, crafts and saxon) and 2 words with frequency over 80 (cos and antiques) never appears in LRS3.
OpenSR: Open-Modality Speech Recognition via Maintaining Multi-Modality Alignment
The IoU of words in LRS2 and LRS3 In Figure  7, we show the IoU (Intersection over Union) of the words with the highest @T op k word frequency in two datasets. Among the top 100 words in the two datasets, there are still 21 different words in addition to non-domain-specific generic words such as 'The', 'A' and 'I' et al. From the perspective of the whole dataset, the IoU of the dictionaries in the two datasets is only 36.925%, which means that there are a large number of words that occur only in their respective domains. Even between near-domain datasets LRS2 and LRS3, there are differences both in the most commonly used words and in the whole dictionary. In certain domains, such as biomedicine and electronic information, the greater difference in word distribution between 
OpenSR: Open-Modality Speech Recognition via Maintaining Multi-Modality Alignment
Performance on visually confusing words. Zero-shot OpenSR trained using only audio performed worse on similar-sounding words. We show several video clips of visually confusable words in Table 7, and show the performance comparison of the model trained with different scale visual utterances in Table 8. With the gradual introduction of visual utterances, the lip-reading performance of the model for visually confusable words is significantly enhanced, demonstrating the significance of our proposed further training using the utterances of common words.
OpenSR: Open-Modality Speech Recognition via Maintaining Multi-Modality Alignment
The distribution of terms will seriously affect the recognition results, while the training speech recognition models. If a term appears never or only a few times, it is essentially unrecognized, as shown in the Table 9. We notice that the models are more likely to come up more common answers (words with high frequency in the in-domain dictionary), probably because the model needs to be trained to fit the word distribution in the training utterances.  OpenSR zero-shot (LRS2): let's take a look behind the scenes (at) how it all came scenes 14 2 OpenSR full-shot (LRS2): let's take a look behind the scenes (at) how it all game seeds 3 7 We further present the performance on the LRS3 dataset in Table 10 (although it has been used in pre-training). During pre-training, the encoder has fully mastered the domain-specific knowledge (the word distribution and syntax) in LRS3, resulting in the performance of 30.6% under the zero-shot setting that is very close to the performance of 28.5% under full-shot (+2.1%). Note that, on the LRS2 dataset, which is not used during the pretraining, the performance of zero-shot is 11% worse than that of full-shot (36.0%->25.0%). In fact, the difference between the further tuning effects on LRS2 and LRS3 (11% and 2.1%) effectively can also effectively demonstrate the domain shift between the LRS2 and LRS3 datasets.
OpenSR: Open-Modality Speech Recognition via Maintaining Multi-Modality Alignment
A For every submission:
OpenSR: Open-Modality Speech Recognition via Maintaining Multi-Modality Alignment
A1. Did you describe the limitations of your work?
OpenSR: Open-Modality Speech Recognition via Maintaining Multi-Modality Alignment
Section 6.Ethical Discussion A2. Did you discuss any potential risks of your work?
OpenSR: Open-Modality Speech Recognition via Maintaining Multi-Modality Alignment
Section 6.Ethical Discussion A3. Do the abstract and introduction summarize the paper's main claims?
OpenSR: Open-Modality Speech Recognition via Maintaining Multi-Modality Alignment
In abstract and introduction A4. Have you used AI writing assistants when working on this paper?
OpenSR: Open-Modality Speech Recognition via Maintaining Multi-Modality Alignment
Left blank.
OpenSR: Open-Modality Speech Recognition via Maintaining Multi-Modality Alignment
B Did you use or create scientific artifacts?
OpenSR: Open-Modality Speech Recognition via Maintaining Multi-Modality Alignment
In reference B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
OpenSR: Open-Modality Speech Recognition via Maintaining Multi-Modality Alignment
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Not applicable. Left blank.
OpenSR: Open-Modality Speech Recognition via Maintaining Multi-Modality Alignment
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank. C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used?
OpenSR: Open-Modality Speech Recognition via Maintaining Multi-Modality Alignment
In appendix.A
OpenSR: Open-Modality Speech Recognition via Maintaining Multi-Modality Alignment
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
LiRA: Learning Visual Speech Representations from Audio through Self-supervision
Self-supervised learning aims to leverage unlabelled data by extracting the training objective directly from the input itself, in an attempt to model meaningful representations of the proposed modality which capture its content and structure. In works adopting this methodology, this task is usually known as the "pretext task" and this initial training procedure is known as the "pre-training" stage. After pre-training, the network is trained on the "downstream task", which generally involves a smaller set of manually labelled data. This methodology has received substantial attention in recent years within the computer vision community. Pretext tasks for visual self-supervision include image colourisation [42], jigsaw puzzle solving [21], as well as combinations of these and other tasks [12]. Self-supervised learning has also been explored in the speech community through works such as Contrastive Predicting Coding (CPC) [22] and wav2vec [34], which predict/discriminate future segments of audio samples; LIM (Local Info Max) [32], which maximises mutual information for the same speaker; and, more recently, PASE (Problem Agnostic Speech Encoder) [26,33], which predicts established audio features such as STFT and MFCC.
LiRA: Learning Visual Speech Representations from Audio through Self-supervision
Self-supervision has also been adopted in the audiovisual domain. Recent approaches include audiovisual fusion [27,28], clustering [4], and distillation [31]; cross-modal discrimination [23]; cyclic translation between modalities [30]; and permutative predictive coding [38]. Shukla et al. [35] focus * denotes equal contribution to this work. on learning audio representations by facial reconstruction from waveform speech. Conversely, [24] predict frequency-based summaries of ambient sound from video, while other recent works apply audio-visual synchronisation [5,7,13] to learn visual embeddings. A task that can benefit from self-supervised learning is lip-reading. Current state-of-the-art lip-reading models rely on annotating hundreds of hours of visual speech data [18], which is costly. To solve this issue, Afouras et al. [3] propose using a pre-trained Automatic Speech Recognition (ASR) model to produce machine-generated captions for unsupervised pre-training. This provides automatically labelled data but still relies on an ASR model trained on large amounts of labelled data.
LiRA: Learning Visual Speech Representations from Audio through Self-supervision
In this work, we aim to leverage the vast amount of available audiovisual speech data to learn generic visual speech features and improve state-of-the-art lip-reading models by predicting audio features from visual speech. The targeted audio features are extracted from waveform audio without the need for additional labels using an established speech encoder (PASE+ [33]). Using the proposed approach, the learnt visual features are explicitly guided by audio which contains rich information about speech. This in turn can lead to learning visual features which are more suitable for speech recognition. After this training procedure, we apply our model (Fig. 2) for lip-reading on a transcribed visual speech dataset.
LiRA: Learning Visual Speech Representations from Audio through Self-supervision
Our research contributions are as follows: 1) We present LiRA, which learns powerful visual speech representations by predicting acoustic features from raw video taken from large audio-visual datasets. 2) We demonstrate that LiRA provides a good initialisation for fine-tuning lip-reading models which consistently outperforms training from scratch, and that this method is particularly beneficial for smaller labelled datasets. 3) We show that LiRA outperforms previous selfsupervised methods for word-level lip-reading, achieving an accuracy of 88.1% on LRW by pre-training on unlabelled data. 4) Finally, we leverage our self-supervised approach towards sentence-level lip-reading, and find that our fine-tuned model achieves state-of-the-art performance for LRS2.
LiRA: Learning Visual Speech Representations from Audio through Self-supervision
LiRA predicts PASE+ features from raw video and is composed of three distinct components. The first is the spatial encoder, which is a traditional 2D ResNet-18 preceded by a 3D front-end layer. The second component is the temporal encoder -the conformer -which receives as input the frame-wise features produced by the spatial encoder and returns a set of features of the same size. The conformer encoder combines traditional attention-based transformer blocks, which excel at capturing global temporal dependencies, with convolutional layers, which model local patterns efficiently [10]. The final component is the projection head (based on the MLP -Multi-Layer Perceptronworkers presented in [26]), which projects these representations into the predicted PASE+ features. To train the model, we apply an L1 loss between the generated embeddings and the features extracted from the pre-trained (frozen) PASE+ model, as shown in Fig. 1. We would also like to mention that we have also experimented with predicting MFCC features but the results were worse than predicting PASE+ features.
LiRA: Learning Visual Speech Representations from Audio through Self-supervision
To evaluate the visual speech representations, we run three variations of end-to-end lip-reading experiments. The training procedure is illustrated in Fig. 2. LiRA-Supervised models are trained from scratch based on the same encoder as in the self-supervised training [16]. This serves as our baseline model since it is trained only with the labelled training data. LiRA-Frozen models are trained using LiRA features from the pre-trained encoder. This allows us to evaluate the visual representations learned during self-supervised learning. Finally, LiRA-FineTuned models use the same model as LiRA-Supervised but are initialised with the pre-trained encoder weights from the pretext task. By using this configuration, we can evaluate the model initialisation capabilities of the proposed self-supervised learning approach. For each of these methods, we adopt a separate model for each lipreading task -six models in total. For word-level lip-reading, we use a Multi-Scale Temporal Convolutional Network (MS-TCN) [19] on top of the encoder, followed by a linear classifier for classification. For sentence-level lip-reading, we follow the state-of-the-art lip-reading model [16] on LRS2 and build a hybrid CTC/attention model. We use the same conformer encoder architecture as in the pre-training phase, followed by the transformer decoder for sequence-to-sequence training [39]. We also perform fine-tuning experiments using the pre-trained model.
LiRA: Learning Visual Speech Representations from Audio through Self-supervision
In this work, we use an unlabelled version of Lip Reading Sentences 3 (LRS3) for pre-training and evaluate the performance of speech representations on LRW and LRS2. LRW [6] is comprised of approximately 500 000 1.16 second labelled utterances (173 hours in total) featuring a specific word from a 500 word vocabulary. It features hundreds of different speakers recorded in a variety of different backgrounds and head poses. LRS2 [1] is composed of approximately 150 000 transcribed utterances of varying lengths (224.5 hours in total). This corpus presents  LRS3 [2] similarly contains approximately 150 000 utterances of varying lengths (438.9 hours in total) taken from TED talks. However, these utterances are substantially longer than the ones featured in LRS2, resulting in effectively double the total amount of hours of video and a larger vocabulary. This dataset guarantees no overlap between the speakers featured in the train and test sets, meaning that the test set is entirely comprised of speakers that were not seen in other sets.
LiRA: Learning Visual Speech Representations from Audio through Self-supervision
To crop the mouth Regions of Interest (ROIs), we start by detecting the 68-point facial landmarks using dlib [11]. We then normalise each frame using a neutral reference frame to remove rotation and size differences.
LiRA: Learning Visual Speech Representations from Audio through Self-supervision
Given the transformed facial landmarks, a fixed bounding box is used to crop mouth ROIs with a size of 96 × 96.
LiRA: Learning Visual Speech Representations from Audio through Self-supervision
Following [15], we produce augmented visual streams by applying the techniques of horizontal flipping with a probability of 0.5 and random cropping to a size of 88 × 88. During the testing phase, instead of randomly cropping, we crop a patch of size 88 × 88 from the centre of the image. For the word-level classification, mixup with a weight of 0.4 is employed. Self-supervised 71.6 PT-CDDL [8] Self-supervised 75.9 AV-PPC [38] Self-supervised 84.8
LiRA: Learning Visual Speech Representations from Audio through Self-supervision
LiRA-Supervised [15] Supervised 87.4 LiRA-Frozen Self-supervised 83.1 LiRA-FineTuned Self-supervised 88.1
LiRA: Learning Visual Speech Representations from Audio through Self-supervision
The 3D front-end module preceding our ResNet consists of a convolutional layer with kernel size (5, 7, 7) followed by a max pooling layer. The conformer, on the other hand, is comprised of an initial embedding module -feed forward layer combined with layer normalisation, dropout (0.1), activation (ReLU -Rectified Linear Unit) and relative positional encoding (as proposed in [9]) -followed by a set of conformer [10] blocks which varies according to the dataset used for the downstream task (6 blocks for LRW, 12 blocks for LRS2). The conformer blocks feature the following parameters: d ff = 2048, n head = 4, d q = 256, d k = 256, d v = 256; where d ff is the hidden dimension of the feed-forward modules, n head is the number of self-attention heads, and d q , d k , d v are the dimensions of the key (K), query (Q), and value (V) in the self-attention layers respectively. The MLP consists of a linear layer with a hidden dimension of 256 units, ReLU activation, dropout, and a linear layer to project the representation to 256-dimensional latent space. For prediction, we average the PASE+ features, which are computed at 100 frames per second (fps), over time to match the frame rate of the input visual features (25 fps). We optimise our model using Adam (β1 = 0.9, β2 = 0.98, = 10 −9 ) combined with the Noam scheduler [39] (25 000 warm-up steps). The model is trained on LRS3 with a batch size of 32. For simplicity, we randomly sample 1 second from each clip and use it as the input to our network, discarding any utterances with less than 1 second in length.
LiRA: Learning Visual Speech Representations from Audio through Self-supervision
LiRA-Supervised In LiRA-Supervised, we train word-level (Fig. 2a) and sentence-level lip-reading models (Fig. 2d) from scratch. In particular, for the task of word-level lip-reading, we add a MS-TCN followed by a linear classifier with an output dimension of 500 on top of the encoder like [17]. A crossentropy loss is employed to optimise the whole model using Adam with decoupled Weight decay (AdamW) [14] with β1 = 0.9, β2 = 0.999, = 10 −8 and a weight decay of 0.01 for 80 epochs with a batch size of 32. The initial learning rate is set to 0.0003. For the task of sentence-level lip-reading, we use 12 multi-head attention blocks (d ff = 2048, n head = 4, d q = 256, d k = 256, d v = 256) together with a linear layer on the top of conformer blocks like [16]. Following [20], we use a combination of CTC and cross-entropy loss to train a hybrid CTC/Attention architecture for 50 epochs with a batch size of 8. In this case, we use Adam with β1 = 0.9, β2 = 0.98 and = 10 −9 with the first 25 000 steps for warm-up. The initial learning rate is set to 0.0004. At the decoding phase, we use a res-b3 res-b4 ce-b1 ce-b2 ce-b3 ce-b4 ce-b5 ce-b6
LiRA: Learning Visual Speech Representations from Audio through Self-supervision
Features from Different Layers in Self-supervised Training   70   75   80   85 Accuracy (%) LiRA -Frozen Figure 3: Accuracy of feature classification (LiRA-Frozen) on LRW based on features extracted from different layers after pre-training on LRS3 via self-supervision. "res-b3" and "res-b4" refer to the output of blocks 3 and 4 from the ResNet-18 respectively; and "ce-b2" to "ce-b12" refer to the layers from every two conformer blocks from bottom to top.
LiRA: Learning Visual Speech Representations from Audio through Self-supervision
beam size of 20 for beam search. During decoding, we also apply a transformer-based language model trained on LRS2, LRS3, and Librispeech 960h [25] (16.2 million words in total). Due to graphic memory limitations, we exclude utterances with more than 600 frames during training.
LiRA: Learning Visual Speech Representations from Audio through Self-supervision
LiRA-Frozen At the end of self-supervised training, the features extracted from the pre-trained frozen encoder are fed to a classifier for evaluation. For word-level lip-reading, we use a MS-TCN, followed by a linear layer with an output size of 500 for classification (Fig. 2b). For the sentence-level lip-reading, the LiRA features are first fed to 12 conformer blocks, and then the encoded representations are used for CTC/attention joint training (Fig. 2e).
LiRA: Learning Visual Speech Representations from Audio through Self-supervision
LiRA-FineTuned We follow the same hyperparameter setting as LiRA-Supervised, but instead of training from scratch, we initialise the encoder with the pre-trained weights from the pretext task and then fine-tune the entire model for word-level lip-reading (Fig. 2c) and sentence-level lip-reading (Fig. 2f).
LiRA: Learning Visual Speech Representations from Audio through Self-supervision
We first evaluate the performance of LiRA-Supervised by training the model from scratch. This leads to an accuracy of 87.4% on LRW which is very close to the state-of-art performance. For LiRA-Frozen, which is pre-trained on LRS3, the learnt visual speech representations are evaluated on word-level lip-reading by training a MS-TCN classifier on top of the frozen representations, as illustrated in Fig. 2b. Feature extraction performance (LiRA-Frozen) for different layers is portrayed in Fig. 3. We observe that the representations extracted from the last layer of the ResNet-18 achieve a maximum accuracy of 83.1 % as seen in Table 1. It is clear that the performance generally decreases as the layer becomes deeper, which may indicate that the features extracted in deeper layers are further tuned towards the pretext task and therefore fail to generalise as well for other tasks. The performance of the 3 downstream scenarios while varying the amount of training data on LRW is shown in Fig. 4a. We use LRS3 for self-supervised pre-training. We observe that the feature extraction approach leads to superior performance compared to LiRA-Supervised when using smaller fractions of the labelled training set (1-2 %). This indicates that the pre-trained model learns useful visual features which also work well on LRW. By adopting this methodology, we can simply train the classification layers while the encoder remains frozen, and hence   Fig. 4a. We also observe that the gap between the performance of LiRA-FineTuned and LiRA-Supervised becomes smaller when we increase the amount of labelled data for training. This demonstrates that pre-training using the proposed self-supervised task is particularly beneficial when the labelled training set is very small. In the extreme case, where only 1 % of the labelled training data is used, LiRA-Supervised achieves an accuracy of 3.6 %. In contrast, we obtain 24.5 % accuracy when LiRA-FineTuned is trained using the same amount of data. This is mainly due to the fact that the self-supervised training provides a good initialisation for network training. We also show that LiRA-FineTuned provides an absolute improvement of 0.8 % in accuracy over LiRA-Supervised when both are trained on full LRW. This demonstrates that LiRA-FineTuned consistently outperforms LiRA-Supervised, even for larger labelled training sets.
LiRA: Learning Visual Speech Representations from Audio through Self-supervision
To investigate the performance of visual speech representations in a more challenging task, we run training from scratch (Fig. 2d) and fine-tuning (Fig. 2f) experiments on LRS2 after pre-training on LRS3. We present our results as a function of the fraction of labelled data used during training.
LiRA: Learning Visual Speech Representations from Audio through Self-supervision
Results are shown in Fig. 4b. It is evident that the performance of LiRA-FineTuned significantly outperforms the su-pervised baseline. We also observe that the performance of LiRA-Supervised is hard to optimise without a good initialisation. The performance becomes worse and worse as the training set increases beyond 18 % of the total amount of labelled data. This is likely due to the large amount of very long utterances featured in LRS2, which makes training from scratch especially difficult. To overcome this problem, we use curriculum learning. In particular, we first train the model using 11 % of the labelled training set, which is composed of videos with less than 155 frames in length and then use this model for initialisation when training on the entire training set. This curriculum learning strategy results in a substantially more effective training procedure, achieving 39.1 % WER for the full dataset.
LiRA: Learning Visual Speech Representations from Audio through Self-supervision
Fine-tuning the self-supervised model leads to a small improvement over the curriculum learning strategy resulting in a 38.8 % WER. This is the new state-of-the-art performance on the LRS2 dataset when no external labelled datasets are used for training. We also observe that it leads to a 9.5 % absolute improvement compared to the previous state-of-the-art model [1], as reported in Table 2. Furthermore, as displayed in Fig. 4, we are able to outperform the previous state-of-the-art of 48.3 % WER using 18× fewer labelled data -76 hours (36 % of LRS2) vs 1 362 hours (MVLRS, LRS2, and LRS3).
LiRA: Learning Visual Speech Representations from Audio through Self-supervision
We present LiRA, which learns visual speech representations by cross-modal self-supervised learning. We train a visual model by predicting acoustic features from visual speech, and observe that it can be adapted for lip-reading with remarkable success. By fine-tuning our models for this new task, we achieve an accuracy of 88.1 % on LRW and report a WER of 38.8 % on LRS2. Given the extent of modern audiovisual corpora, we believe it would be promising to leverage this method towards other visual tasks such as emotion recognition and speaker recognition in the future.
Learning from the Master: Distilling Cross-modal Advanced Knowledge for Lip Reading
Lip reading, which is also referred to as visual speech recognition, aims at predicting words or sentences being spoken from muted lip videos. This vision task enables to switch speech to text without relying on hearing, and therefore, it could apply to many practical scenarios, such as dubbling for silent films, creating a voice for aphonia patients, and serving for security systems. To tackle this problem, early researches usually adopt HMM with designed hand- * The first two authors contribute equally.
Learning from the Master: Distilling Cross-modal Advanced Knowledge for Lip Reading
† Corresponding author (hesfe@scut.edu.cn).  Figure 1: (a) Traditional knowledge distillation from pretrained audio teacher to a video student. (b) Our method distills advanced knowledge from the master that trained not only with both audio and video data, but also the feedback from the student, leading to a more compatible knowledge transfer. Furthermore, the introduced audio and video tutors provide additional cues to the student for further bringing the cross-modal gap.
Learning from the Master: Distilling Cross-modal Advanced Knowledge for Lip Reading
crafted features [6,10,19], whereas recent works exploit deep neural networks [18,29,23] for lip reading.
Learning from the Master: Distilling Cross-modal Advanced Knowledge for Lip Reading
Notwithstanding the tremendous success of deep learning and large benchmark construction [8,1,2], the admitted most critical challenge is the inherent limitation of visual information, which severely impedes a good performance of lip reading models. For example, different characters "p" and "b" share a similar lip shape so that they are hard to be distinguished in video clips. In contrast, such an uncertainty can be uniquely identified by audio information, and audio based speech recognition would not be affected by ambiguities caused by the intrinsic limitation of visual information. Specifically, the counterpart of lip reading task, i.e., speech recognition, could achieve a much more accurate translation of speech to text. The performance gap can even reach 40% [1] on the metric of word error rate (WER).
Learning from the Master: Distilling Cross-modal Advanced Knowledge for Lip Reading
Based on this fact, one potential solution is to transfer knowledge from audio data to video data via knowledge distillation (KD) [15] (Fig. 1a). Several previous approaches [29,31] attempt to build KD-based models, which consist of a teacher network pretrained by audio signals and a student network utilized for lip reading. The audio information is introduced in this way and supposed to be Lip reading (KD) Audio 54.2 a complementary clue for facilitating the performance of the student. Due to the existed heterogeneity between two modalities, however, such a general audio teacher may only provide limited hidden knowledge to the student for promotion. This observation is examined in Tab. 1, in which we conduct an experiment on LRS2-BBC [1] dataset to test the impact of the teachers which share a same structure but pretrained on different modalities. Note that except for the teachers, the structures of both student networks are also the same. Several interesting facts can be observed: (i) The performance gap (about 40% in WER) between lip reading and speech recognition tasks is similar as reported in previous research [1]. (ii) When distilling knowledge from an audio teacher, the performance of the student is even worse than that of the student with distilled knowledge from the video teacher. Combining (i) and (ii), we can conclude that: a "master" that merely has an advanced accuracy (audio teacher in this case) does not act as a good teacher to the video student; while the video teacher which shares the same data domain with the student, can supervise the student for learning more distilled knowledge representations. Obviously, cross-modal gap is the reason of why this phenomenon occurs. Then we spontaneously raise a question: In lip reading, how can the visual student learn more comprehensible knowledge from the audio "master"? We consider the above problem from two aspects. First, the teacher should understand bi-modal knowledge to possibly bridge the inherent cross-modal gap. Second, the teacher should dynamically adjust teaching contents in consistent with the evolution of the -student. In this way, the changing requirement of the student would help the teacher to regulate and emphasize the important knowledge from different modalities. Therefore, the cross-modal gap could be fused with a clear aim. In this paper, we propose an innovative deep lip reading model (see in Fig. 1b). Instead of using a pretrained teacher network, we design a trainable and much more powerful network named "master". To produce a more compatible knowledge with regard to visual student, the master takes not only speech audio data but also lip video data as inputs, and produces three types of probabilities respectively based on audio modality, video modality, and their combination. Then to fuse these knowledge adap-tively, we incorporate a couple of "tutor" networks into our framework as knowledge fusion guidance, which are pretrained from audio and video data respectively. Based on the interactions among the "master", the "tutors", and the requirement (feedback) of the student, we design a dynamic fusion loss to balance yet fuse different types of knowledge. We also propose a curriculum learning strategy to mitigate the learning ambiguity during training, ensuring a better convergence.
Learning from the Master: Distilling Cross-modal Advanced Knowledge for Lip Reading
In summary, our key contributions are as follows:
Learning from the Master: Distilling Cross-modal Advanced Knowledge for Lip Reading
• We propose a collaborative learning based framework for lip-reading. Unlike most other existing methods directly using a pretrained teacher to distill knowledge, we embed an advanced trainable master network into our system. The master could be adjusted according to the feedback of the student, and thus provides bimodal knowledge dynamically for the student to learn in a better way. • We incorporate a couple of tutor networks into our system, which are respectively pretrained by audio and video data. To get the master, the tutors, and the student to cooperate, we particularly tailor a dynamic fusion loss to guide the student to learn audio-visual probabilities, which alleviates ambiguities caused by the cross-domain gap. • We present a curriculum learning strategy for lipreading. By measuring and sorting the difficulty of samples, it could enhance the effectiveness of model training as well as ensuring a better convergence. • We outperform state-of-the-art lip-reading methods on three benchmarks, indicating the effectiveness of the proposed method.
Learning from the Master: Distilling Cross-modal Advanced Knowledge for Lip Reading
Lip reading. Lip reading have drawn increasing attention in recent years. Most deep learning based lip reading methods include a pair of frontend and backend, and focus on the corresponding architecture design. The frontend is used to extract visual features, and usually is ResNet [14] or VGG [21] with some modifications. In particular, Chung et al. [8] evaluate the performance of different frontends with multiple transform. While the backend is used to map visual features to natural language, mainly including temporal convolution [23], recurrent neural network (RNN) [23], and transformer [1]. Specifically, Stafylakis et al. [23] propose to combine residual network as frontend with long short term memory network (LSTM) [16] as backend. Afouras et al. [1] introduce transformer to replace RNN as backend. Zhang et al. [27] focus on short-range temporal information with temporal focal block and local self-attention.
Learning from the Master: Distilling Cross-modal Advanced Knowledge for Lip Reading
To deal with the problem that one output word is corresponded to several frames in the lip-reading task, two architectures are designed for the alignment purpose. The first one is sequence to sequence (Seq2Seq) [24] model, which reads all visual features before prediction. The other one is connectionist temporal classification (CTC) [11], which is an emission model that predicts the results for each frame and searches the optimal alignment for final prediction. Lip-Net [3] is based on this architecture.
Learning from the Master: Distilling Cross-modal Advanced Knowledge for Lip Reading
As multi-modal learning develops, lip reading methods attempt to extract information from audio data as a complement for a better performance. Afouras et al. [31] propose to distill knowledge from a pretrained audio teacher to guide lip reading. Zhao et al. [29] propose frame-, sequence-and context-level distillation methods into the prediction. However, the cross-modal gap between the two modalities is completely ignored, which seriously limits the lip reading performance.
Learning from the Master: Distilling Cross-modal Advanced Knowledge for Lip Reading
Knowledge distillation. Knowledge Distillation [15] (KD) aims at transferring knowledge from teachers to students. There are two main factors that may affect the performance of KD. The first is the type of knowledge, and lots of previous works propose to discover more useful knowledge as guidance. For example, FitNets [20] utilizes hidden layer features rather the logits for distillation. Zagoruyko et al. [26] introduce an attention mechanism to selectively transfer knowledge. Yang et al. [25] propose a more tolerant teacher with softer logits, which is easy for the student to mimic. Furlanello et al. [9] integrate multiple teachers to boost the utility of knowledge. However, these methods are designed from a perspective of model capacity rather than data capacity, and thus limited in distilling knowledge from a different data modality due to the cross-modal gap.
Learning from the Master: Distilling Cross-modal Advanced Knowledge for Lip Reading
The other factor is the strategies of distillation. Online distillation, also known as collaborative distillation, is of great interest recently. It aims to alleviate the model capacity gap between the student and the teacher. By treating all the students as teacher, Zhang et al. [28] propose DML to make the students learn from each other. ONE [30] constructs the teacher using a multiple branch design. CLNN [22] establishes multiple head branches in the teacher with a corresponding scaling gate for the branch diversity. KDCL [12] ensembles the predictions of students, generating soft targets as supervision. Note that collaborative distillation methods transfer knowledge between students without considering the feedback of each student. It means that when different students learn for different tasks, the potential task gap may distract the learning objective.
Learning from the Master: Distilling Cross-modal Advanced Knowledge for Lip Reading
Given a sequence of lip frames {X n V ∈ R H×W ×3 |n = 1, 2, ..., N V }, our system aims at predicting the spoken word U ∈ R K or sentence {Z q ∈ R K |q = 1, 2, ..., Q} from lip movements, where Q indicates the length of the sentence in the sentence-level scenarios and K represents the size of vocabulary. It means that a sentence is predicted character by character, while a word is directly recognized in the word-level scenarios. Fig.2 illustrates the pipeline of the proposed method. Our network includes four modules: master, tutor A , tutor V and student, here the subscripts A and V respectively denotes the audio modality and the video modality. The master f m (·) takes audio signals X A and video clips X V as inputs, and provides three types of knowledge:
Learning from the Master: Distilling Cross-modal Advanced Knowledge for Lip Reading
generated from audio-visual combination. The student takes X V as input, and outputs probabilites f s (X V ; θ s ). The pretrained tutor A outputs logits f t A (X A ) in regard to speech recognition, whereas the pretrained tutor V produces logits f t V (X V ) for lip reading. Note that the pretrained tutors in our model are to provide the information for balancing knowledges from different modalities as well as evaluating the training data by difficulty, not for directly transferring knowledge to the student like pretrained teachers in most existing approaches.
Learning from the Master: Distilling Cross-modal Advanced Knowledge for Lip Reading
We alternatively train the master and the student, and thus the whole training procedure includes two stages. (i) In the student training stage, the student is optimized by both the distilled cross-modal knowledge from the master and the supervision with lip-reading labels. (ii) In the master training stage, the master is optimized not only by the supervision of labels, but more importantly, the feedback from the temporarily updated student.
Learning from the Master: Distilling Cross-modal Advanced Knowledge for Lip Reading
Let us start with the definitions of audio stream, visual stream, and audio-visual combination. As the names suggest, (i) audio stream is a stream that produces the logits based on audio data, (ii) visual stream is a stream that produces the logits based on video data, and (iii) audio-visual combination aims to combine audio and video features for the logits prediction. Each stream is comprised of a feature extraction frontend, a feature mapping backend and a final classifier. The tutor A is an audio stream based model, the tutor V and the student are visual stream based models, and the master consists of both audio stream and visual stream. Next, we will introduce the architectures of these modules in detail.
Learning from the Master: Distilling Cross-modal Advanced Knowledge for Lip Reading
Audio stream. Audio stream adopts a ResNet-18 [14] as a frontend. Regarding the deployed backend, we have two model variants: (a) Temporal convolution (TC) backend [23] is utilized in word-level lip reading, and (b) transformer sequence to sequence (TM-Seq2Seq) backend [1] is used in both word-level and sentence-level scenarios.
Learning from the Master: Distilling Cross-modal Advanced Knowledge for Lip Reading
Different from original ResNet, we replace all 2D convolution kernels in the frontend of audio stream with 1D 
Learning from the Master: Distilling Cross-modal Advanced Knowledge for Lip Reading
Task-Specific Supervision Figure 2: Illustration of our method. The whole training process is split into the student training stage and the master training stage. During the student training stage, the student absorbs task-specific (lip reading) supervision and knowledge distilled from the master with cross-modal guidance provided by video and audio tutors. During the master training stage, a temporary student is introduced as an auxiliary network, feeding task-specific supervision into the master for its update.
Learning from the Master: Distilling Cross-modal Advanced Knowledge for Lip Reading
filters since a waveform lies in 1D space, and the filter size of the first convolution layer is changed to 5ms with a stride of 0.25ms. As for the TM-Seq2Seq backend, we follow the same setting as that of Afouras et al. [1], to include the multi-head attention and the feedforward block. Visual stream. The structure of visual stream is almost the same as audio stream. The only difference is that, we replace the first 2D convolution layer in the frontend with kernel size of 7 × 7 by a 3D convolution layer with kernel size of 5 × 7 × 7 , because of an extra time dimension.
Learning from the Master: Distilling Cross-modal Advanced Knowledge for Lip Reading
Audio-visual combination. Audio-visual combination is only set up in the master for generating logits of merged features, which are derived from both audio stream and visual stream. Specifically, in the case of model (a), we directly concatenate the output feature vectors, which are respectively produced by the backends of audio and visual streams, into a new vector. In the case of model (b), the audio and video feature vectors are separately attended by the context vectors and then concatenated into a new vector before feeding in the classifier.
Learning from the Master: Distilling Cross-modal Advanced Knowledge for Lip Reading
Existing teacher networks are usually pretrained, or lacking of considering to be trained based on that how well the students perform on their tasks. The teachers thus tend to be inflexible in adjusting their teaching contents due to the neglect of the needs of the students. On the other hand, the students require more comprehensive knowledge distilled from the teachers to achieve a better performance. Our method is implemented to address these problems, of which the training procedure is divided into two stages: the student training stage and the master training stage. Next, we will take word-level recognition as an example to explain the proposed method.
Learning from the Master: Distilling Cross-modal Advanced Knowledge for Lip Reading
Student Training Stage. During the student training stage, we only update the student parameters θ s . The training objective contains two terms: cross entropy loss L CE and dynamic fusion loss L DF . L CE controls the classification accuracy, and L DF is used to match the logits between the teacher and the student.
Learning from the Master: Distilling Cross-modal Advanced Knowledge for Lip Reading
Given a one-hot label y = [y 1 , y 2 , ..., y K ], here K indicates the size of vocabulary as defined in Sec 3.1, the cross-entropy loss L CE between the predictionŷ = [ŷ 1 ,ŷ 2 , ...,ŷ K ] and label y is defined as follows:
Learning from the Master: Distilling Cross-modal Advanced Knowledge for Lip Reading
Note that in sentence-level prediction, Eq. (1) is corresponding to the loss generated by a character, and the same is true for subsequent other losses.
Learning from the Master: Distilling Cross-modal Advanced Knowledge for Lip Reading
Regarding the dynamic fusion loss L DF which is different from traditional distillation loss [15], extra cross-modal guidance is affiliated within its design. We will concretely discuss it soon. Therefore, the overall objective function L s is formulated as follows:
Learning from the Master: Distilling Cross-modal Advanced Knowledge for Lip Reading
where λ s indicates a balance factor for regularization, and the optimized parameters θ * s is then calculated by θ * s = arg min θs L s .
Learning from the Master: Distilling Cross-modal Advanced Knowledge for Lip Reading
Cross-Modal Gap Fusion. Distilling knowledge from audio modality to video modality for lip reading is arguably necessary, since the ambiguities caused by different phonemes with almost identical visemes could be avoided. However, a derived trouble is the existed cross-modal gap, which is verified in Table 1. On the other hand, we observe from Table 1 that a video teacher can also promote the performance of the video student. This phenomenon may be attributed to a transferring of more distilled knowledge. Based on these facts, we drive the teacher outputs different types of knowledge, i.e., audio knowledge, video knowledge, and audio-visual knowledge, for further distillation. This design makes it possible for the student to make its own trade-offs on which modality to learn more from. Then the problem turns into that, how to fuse the cross-modal knowledge.
Learning from the Master: Distilling Cross-modal Advanced Knowledge for Lip Reading
We tackle this issue by introducing two pretrained audio and video tutors, i.e., tutor A and tutor V . We exploit the fixed features generated by the tutors as guidance, and encode them into weighting factors w A and w V for measuring the preference of different types of knowledge from audio or video modality. Note that we use the output feature vectors {H A , H V } of the backends of the tutors here for richer representations, instead of the logits produced by the classifiers. We further present a dynamic fusion loss L DF as a regularizer during the student training stage, that is
Learning from the Master: Distilling Cross-modal Advanced Knowledge for Lip Reading
Here we use the focal loss L F (·) [17] to ease the difficultyimbalance and class-imbalance (in the sentence-level scenario) problems. We calculate two weighting factors {w A , w V } by
Learning from the Master: Distilling Cross-modal Advanced Knowledge for Lip Reading
where FC(·; θ * ) denotes a fully connected layer with parameters θ * , ⊕ denotes the concatenation operation, and φ(·) indicates the sigmoid function. All the parameters of the FCs are trained as part of the master.
Learning from the Master: Distilling Cross-modal Advanced Knowledge for Lip Reading
Master Training Stage. During the master training stage, the values of the parameters θ s are fixed. Nevertheless, we introduce a temporarily updated student network f ts (·; θ ts ) as an auxiliary module. The rationale behind this setting is that, the master needs to receive a task-specific (i.e., lip reading task) feedback for adaptively adjusting its teaching contents, meanwhile the feedback from the student should be renewed with the update of the master. In other words, f ts (·; θ ts ) and f m (·; θ m ) are alternatively updated during this stage, where θ m = {θ A , θ V , θ F A , θ F V , θ F AV }.
Learning from the Master: Distilling Cross-modal Advanced Knowledge for Lip Reading
Input: Training audio, video, label pairs {X n A , X n V , y n }; The initialization of parameters θ m , θ s , α, λ s , λ m , G 0 , P , ξ; The pretained tutor A and tutor V . Output: The master f (·; θ * m ); The student f (·; θ * s ). 1: repeat 2: while training the student do Calculate the logits of the student f s (X V ; θ s ); 5: Calculate the logits of the master f m (
Learning from the Master: Distilling Cross-modal Advanced Knowledge for Lip Reading
Update the parameters θ s by Eq. (3). Update the parameters θ ts by Eq. (6); 11: Resample training audio, video, label pairs {X A , X V , y }; 12: Update the parameters θ m by Eq. (7). 13: end while 14: until converged Specifically, the temporary student first learns from the master, and is governed by the same loss L s . The updated parameters θ ts is given by
Learning from the Master: Distilling Cross-modal Advanced Knowledge for Lip Reading
where α represents the learning rate. Then the master is trained according to both data labels and the requirement of the updated student. We define the optimization of the master as follows:
Learning from the Master: Distilling Cross-modal Advanced Knowledge for Lip Reading
where λ m denotes a balancing factor. Note that the parameters of the FCs are updated by the first term, and here we omit the derivation based on the chain rule.
Learning from the Master: Distilling Cross-modal Advanced Knowledge for Lip Reading
In practice, the updates of the temporary student and the master adopt different training samples. Because when updating the master, the student should utilize a validation set instead of a training set. Finally, the algorithm of "Learning from the Master" is summarized in Algorithm 1.
Learning from the Master: Distilling Cross-modal Advanced Knowledge for Lip Reading
Previous lip reading methods [1,7,27] are proposed to randomly sample training data first and then feed them into the network. However, the order of training samples is ignored, which may influence the effectiveness of training process. A more desired scheme is to make the framework learn from an easy start and gradually increase the difficulty to facilitate a better convergence. Therefore, we incorporate a curriculum learning [4,13] based strategy into the training stage of our approach. It is implemented by two steps.
Learning from the Master: Distilling Cross-modal Advanced Knowledge for Lip Reading
First, we build a rating function R(·) to evaluate the difficulty of each sample pair X n A , X n V (with the same label), that is
Learning from the Master: Distilling Cross-modal Advanced Knowledge for Lip Reading
where C(·) denotes the confidence, sort(·) indicates a sorting operation. It is easy to understand that the higher the function value, the easier the sample pair. Note that when several sample pairs have the same rating value, we resort them according to the video modality based confidence
Learning from the Master: Distilling Cross-modal Advanced Knowledge for Lip Reading
. Given a series of training data sorted by their difficulties, we secondly introduce a fixed exponential pacing function [13] to establish the increment of samples during training, which is given by
Learning from the Master: Distilling Cross-modal Advanced Knowledge for Lip Reading
where G i indicates the input percentage of training data in the ith iteration (G 0 is the percentage at the start), P is an exponential factor, and ξ is the number of iterations in each step. Based on the rating function and the pacing function, the order and the increment of the training samples are determined more sensibly. Such a strategy could reduce the learning ambiguities at the beginning of training and also enable a better convergence. More discussions will be presented in Sec. 4.4.
Learning from the Master: Distilling Cross-modal Advanced Knowledge for Lip Reading
To evaluate the proposed method, we utilize three benchmark datasets, i.e., one word-level dataset LRW [8] and two sentence-level datasets LRS2-BBC [1], LRS3-TED [2].
Learning from the Master: Distilling Cross-modal Advanced Knowledge for Lip Reading
LRW. Lip Reading in the Wild dataset is a large wordlevel dataset with 500 words and 450000 utterances. Each video is 1.16 second long with 29 frames.
Learning from the Master: Distilling Cross-modal Advanced Knowledge for Lip Reading
LRS2-BBC. The Lip Reading Sentence dataset comes from BBC talks including 143 kilo utterances and 2.3 million words. The dataset is split into pretrain/train-val/test data, while train-val and pretrain data comes from the same date and pretrain data contain a word-level boundary alignment.
Learning from the Master: Distilling Cross-modal Advanced Knowledge for Lip Reading
LRS3-TED. LRS3-TED dataset comes from TED talks with 150 kilo utterances and over 4.2 million words. This is the most difficult audio-video lip reading dataset. The dataset construction is also divided into the same setting as LRS2-BBC.
Learning from the Master: Distilling Cross-modal Advanced Knowledge for Lip Reading
Preprocessing. To crop aligned center mouth areas of the video data, we first use dlib [5] to detect the facial landmarks. Then we randomly crop and interpolate the results to obtain 112×112 lip-centered images. Lastly, we transform all faces for removing rotations and different scalings.
Learning from the Master: Distilling Cross-modal Advanced Knowledge for Lip Reading
Vocabulary size. In the word-level prediction, the size of the vocabulary is set to 500, which is in line with the number of words in LRW. Regarding the sentence-level scenarios, i.e., LRS2-BBC and LRS3-TED, we set the size of vocabulary as 40, including 26 letters, 10 digits, and 4 special tokens ([SPACE], [PAD], [EOS] and punctuation mark).
Learning from the Master: Distilling Cross-modal Advanced Knowledge for Lip Reading
Training protocol. The student and the master are trained alternatively using SGD optimizer with a momentum of 0.9 and a weight decay of 1e-4. In audio stream, we take the raw wave as input. While in visual stream, the input videos are sampled at 25fps. We follow the same data augmentation and audio-subtitle forced alignment strategies as in Afouras et al. [1], and training together with the proposed curriculum learning described in Sec.3.4.
Learning from the Master: Distilling Cross-modal Advanced Knowledge for Lip Reading
The whole training process includes both pretraining and fine-tuning steps. Specifically, we first pretrain our model with a backend of TC at word level, using LRW and the pretrain sets of LRS2-BBS and LRS3-TED. In regard to the word-level prediction, we then fine-tune the pretrained model with LRW. While for the sentence-level prediction, we replace TC with TM-Seq2Seq as the backend in the pretrained model, and continue training with the pretrain set of LRS2-BBS or LRS3-TED. The new pretrained model is then fine-tuned with the related train-val set.
Learning from the Master: Distilling Cross-modal Advanced Knowledge for Lip Reading
The learning rate α is set to 10 −3 during pretraining. During fine-tuning, α is initialized to 10 −4 and decreased by half every time the validation error flats, down to a final learning rate 10 −6 . The other hyperparameters in Algorithm 1 are set as follows: λ s = 10, λ m = 10, G 0 = 0.25, P = 1.75, ξ = 10 7 .
Learning from the Master: Distilling Cross-modal Advanced Knowledge for Lip Reading
Evaluation metric. For all the experiments, we adopt the word error rate (WER) as the measurement, which is defined as WER = S+D+I N U M , where S, D, I is the number of substituted, deleted, and inserted words respectively to get from the reference to the inference, and N U M is the total number of words in the reference.
Learning from the Master: Distilling Cross-modal Advanced Knowledge for Lip Reading
We compare our method with several state-of-the-art methods, including MT [8], Temporal Convolution [23], WAS [7], Bi-LSTM [23], TM-CTC [1], TM-Seq2Seq [1], Conv-Seq2Seq [27], LIBS [29], and TM-CTC-KD [31]. Unless stated otherwise, all the reported results are directly copied from the original papers.
Learning from the Master: Distilling Cross-modal Advanced Knowledge for Lip Reading
Word-Level Lip Reading. Table 2 reports the quantitative comparisons with existing approaches from LRW dataset, regarding word-level lip reading. It can be seen that, Ours-TC significantly outperforms its baseline Temporal Convolution which is without the master module, with a WER improvement of 6.7%. Besides, Ours-TM achieves the best performance compared to all the other methods. In particular, it acquires an increase of 2% in WER compared to that of the second best method Conv-Seq2Seq.
Learning from the Master: Distilling Cross-modal Advanced Knowledge for Lip Reading
Sentence-Level Lip Reading. We also performed experiments on sentence-level lip reading, and the results are listed in the last two columns of Table 2. It can be observed that Our-TM performs the best on both LRS2-BBC and LRS3-TED in comparison with the other methods. More importantly, compared to TM-Seq2Seq, which incorporates a same backend as Ours-TM and is trained on an extra nonpublic dataset MV-LRS, our method achieves a better performance with WER improvements 0.6% on LRS2-BBC and 0.9% on LRS3-TED in the case of using much less training data. Furthermore, compared to Conv-Seq2Seq, which uses a more advanced structure than that of our taskspecific network (the student), Ours-TM still achieves better performance, with WER improvements 2.5% on LRS2-BBC, 1.1% on LRS3-TED.
Learning from the Master: Distilling Cross-modal Advanced Knowledge for Lip Reading
Tendency of misrecognition. We further investigate the top-4 cases on LRW with the highest error rate and list the comparison results of Ours-TC w/o KD and Ours-TC in Table 3. It can be observed that when multiple phonemes are mapped to one viseme, e.g., phonemes TH and DH vs. viseme /t/, our method can reach an average improvement of nearly 6% in accuracy with the comparison.
Learning from the Master: Distilling Cross-modal Advanced Knowledge for Lip Reading
Combining all the above observations, it implies that: (i)  Introducing the proposed master can effectively promote the performance of our task-specific network. (ii) Though our model mainly focuses on the benefits over standard distillation methods, it has the potential to achieve better performance when replacing the architecture of the task-specific network with a more advanced one.
Learning from the Master: Distilling Cross-modal Advanced Knowledge for Lip Reading
In this section, we investigate the effectiveness of our proposed modules, including the master network, the crossmodal gap fusion, and the curriculum learning strategy. We use a single-modal lip-reading network as baseline, i.e., Video Teacher in Table 4.
Learning from the Master: Distilling Cross-modal Advanced Knowledge for Lip Reading
Effectiveness of the Master. To explore the efficacy of the master, we examined 6 different pairs of teacher-student or master-student designs based on different modalities, and tested their performances on LRS2-BBC. The results are summarized in Table 4. Note that the reported performance of the Audio-Visual Master comes from its AV branch, and the architectures of the pretrained teachers are totally the same as those of their counterpart trainable masters. Besides, the curriculum learning strategy is not used here. Then we have the following observations and analyses. (i) In the case of single models without KD, no matter whether the models are trainable (i.e., the masters) or not (i.e., the teachers), the descending order of their performance in different modalities is always {audio-visual modality (AV), audio modality (A) and video modality (V)}. This verifies the importance of learning from cross-modal data instead of single-modal data. (ii) In the cases of distilling knowledge from the teachers and the masters, the descending orders of the performance of the students in different modalities are always {V, AV, A} and {AV, A, V}, respectively. The first sorting order implies that, compared to audio modality, audio-visual modality could provide additional information and thus help to alleviate the ambiguities caused by the cross-modal gap, but using a simple fusion strategy (concatenation) is limited. Whereas the other sorting order reveals the effectiveness of the master that could reduce the cross-modal gap to some extent, due to its dynamic regulation based on the task-specific feedback of the student. (iii) No matter which modality is used, the teachers always perform better than their counterpart masters. While no matter which modality is used, the students learned from the masters perform better than those learned from the teachers all the time. These facts demonstrate that the proposed master is more effective than a pretrained teacher because of its adaptability to the student, although a sacrifice in its own performance is obtained.
Learning from the Master: Distilling Cross-modal Advanced Knowledge for Lip Reading
Effectiveness of Cross-Modal Gap Fusion. A simple fusion strategy by concatenating the features from cross modalities shows a limitted improvement, which is verified in Table 4. This motivates us to come up with the design of cross-modal gap fusion. Further, to evaluate the effectiveness of this module, we examined 7 different and representative weight settings and reported the results in Table 5. Note that "T" and "M" respectively indicate that the student is distilled from a teacher or a master, and the curriculum learning strategy is also not used here. We have the following observations and analyses: (i) T1 performs worse than T2, and T2 performs similarly to T3. It seems that introducing AV modality is not a better choice compared to incorporate {A,V}; (ii) However, both M1 and M3 perform better than M2. This reveals the importance of the interaction between audio stream and visual stream in the master, which confirms the necessity of AV. Furthermore, A and V are also useful but should be treated dynamically rather than equally. As a result, M4 achieves the best performance compared to other methods, proving the effectiveness of the proposed cross-modal gap fusion module. Note that we fix the weight of AV to avoid increasing the system complexity.
Learning from the Master: Distilling Cross-modal Advanced Knowledge for Lip Reading
Effectiveness of Curriculum Learning on various backends. The proposed method has a specific focus on distillation performance and thus a lightweight structure (2 TC layers in the backend) is used. However, our work can be used as a plug-in for any state-of-the-art method, like [18] which uses a more advanced backend. To demonstrate the effectiveness of our curriculum learning (CL) strategy, as well as the superiority of our framework, Table 6 reports the corresponding comparisons that replacing our backend from 2 TC layers to 8 TC layers and the one used in [18]. Note that here we only use LRW as training data, which is the same training setting as in [18]. It can be seen that, when using a lightweight model, CL would be helpful to improve the performance of our method as well as acceralating convergence. Nonetheless, when replacing with an advanced model, the improvement of CL degrades, and it mainly helps to accelerate the training. Specifically, our method can achieve better performance against the comparisons, and still outperforms [18] when without CL. This reveals the power of our CL strategy and tailored distillation approach.
Learning from the Master: Distilling Cross-modal Advanced Knowledge for Lip Reading
In this paper, we propose a novel framework for lip reading based on knowledge distillation. Instead of distilling knowledge from a pretrained audio teacher, we particularly introduce a trainable master into our model. With the aid of the task-specific feedback from the student, the cross-modal gap fusion module, and the curriculum learning strategy, the master could adjust its teaching contents to the student adaptively. Extensive experiments demonstrate the validity of the proposed method.
Multi-Singer: Fast Multi-Singer Singing Voice Vocoder With A Large-Scale Corpus
Singing voice synthesis (SVS) aims to synthesize high-quality and expressive singing voices based on musical score information. Singing voice synthesis (SVS) systems [2,14,22] take music score and lyric information as input to generate singing voices, and these systems have been widely deployed in music softwares, music boxes, and so on. SVS systems could generate singing voices with comparable quality to reference songs, which attract widespread research interest.
Multi-Singer: Fast Multi-Singer Singing Voice Vocoder With A Large-Scale Corpus
Following the essential components similar to TTS systems, SVS systems generally adopt an acoustic model [5,23] to convert the musical scores into acoustic features, and a vocoder [2,21] to generate audio waveform from acoustic features. Neural vocoders can synthesize natural-sounding speech, which generally determines the upper bound of generated sound quality. In this paper, we concentrate on waveform modeling in vocoder.
Multi-Singer: Fast Multi-Singer Singing Voice Vocoder With A Large-Scale Corpus
Unlike traditional TTS [3,[32][33][34]44], there are several challenges to build a multi-singer SVS system: 1) Open source and high-quality singing voice data. Unlike speech, high-quality singing voices are commonly produced by professional singers. Because of the high cost of recording and labeling songs, researchers hardly have access to large and open-source singing voice corpora. 2) Fast audio synthesis against limited computation resource. For application deployment, generation speed and computational cost need further consideration. 3) Multi-singer modeling. Timbres could be wildly different among groups while singing voices vary from expression and style. When applying SVS systems for unseen singer modeling, there comes distinct degradation in synthetic singing voice quality.
Multi-Singer: Fast Multi-Singer Singing Voice Vocoder With A Large-Scale Corpus
In the past few years, researchers work to address the challenges above in singing voices modeling, while some problems emerge: 1) SVS systems like DeepSinger [35] mine data from the web, but processed data with noise still could not meet requirements for high-fidelity SVS synthesis. Further, although several singing voice datasets such as MIR-1K dataset [15] and JukeBox [7] have been released for research purposes, but the corpora are not so large as expected for multiple tasks. 2) Several parallel generation methods [40,42] have been proposed to speed up waveforms synthesis. However, existing multi-band architectures do not consider characteristic differences among frequency bands, so a powerful frequency-adapted multi-band technique is required. 3) Researchers investigate ways to generate high-quality waveforms during multisinger modeling. Previous multi-speaker data training strategy [8] increases model generalization. Unfortunately, without explicitly adopting architecture for singer identity reconstruction, vocoders would be data-hungry, and generalization restriction still comes. Giving additional information during generation would be another strategy. Speaker Conditional WaveRNN [28] takes speaker embeddings as additional input, while extra embedding would be sometimes hard-earned during inference procedure and slows down generation as well.
Multi-Singer: Fast Multi-Singer Singing Voice Vocoder With A Large-Scale Corpus
To accelerate SVS research, we assemble an open-source, largescale, and multi-singer singing voice corpus OpenSinger. To the best of our knowledge, OpenSinger is the first open-source Chinese singing voice dataset. We have attached part of OpenSinger to the supplementary materials, and we will release the entire dataset after paper publication. To overcome afore problems in this paper, we propose Multi-Singer, which achieves computational efficiency and keeps powerful capability for multi-singer singing voices modeling. To be more specific, 1) we introduce a novel multi-band generator, which speeds up singing voice generation and improves the audio quality of different frequency bands. 2) Then we introduce the singer conditional discriminator with conditional loss function for high quality and similarity multi-singer singing voice generation.
Multi-Singer: Fast Multi-Singer Singing Voice Vocoder With A Large-Scale Corpus
3) To further reconstruct singer representations in the frequency domain, we propose an auxiliary singer perceptual loss based on embeddings extracted from a pre-trained speaker encoder. The proposed training method effectively works in GANs for multisinger singing voice modeling.
Multi-Singer: Fast Multi-Singer Singing Voice Vocoder With A Large-Scale Corpus
Experimental results show that Multi-Singer can generate highfidelity multi-singer singing voices and achieve the best mean opinion score (MOS) among existing neural vocoders. Combined with FastSpeech 2 as an acoustic model, Multi-Singer shows strong robustness in singing voice synthesis systems. Multi-Singer is substantially faster than most neural vocoders, and it samples 125 times faster than real-time on single NVIDIA 2080Ti GPU with comparable quality to an autoregressive counterpart.
Multi-Singer: Fast Multi-Singer Singing Voice Vocoder With A Large-Scale Corpus
In this section, we overview existing datasets, provide a singing voice synthesis background and briefly review several variations of vocoders.
Multi-Singer: Fast Multi-Singer Singing Voice Vocoder With A Large-Scale Corpus
Training TTS and SVS systems both require a significant amount of annotated data [9,10,15]. The rapid increase in the amount of multimedia content on the Internet in recent years makes data much more important. Researchers have released speech and singing voice corpora, varying from languages, speakers, and so on.
Multi-Singer: Fast Multi-Singer Singing Voice Vocoder With A Large-Scale Corpus
For speech synthesis, various datasets are available for different tasks. LSSED [11] is a challenging large-scale English speech emotion dataset, which has data collected from 820 subjects to simulate real-world distribution. AISHELL-3 [36] contains roughly 85 hours of emotion-neutral recordings spoken by 218 native Chinese speakers, which could be applied for multi-speaker speech synthesis. Data could be collected with different methods, including mining from webs automatically, recording manually, and so on. The read English speech LibriSpeech corpus [27] is derived from audiobooks. Japanese multi-speaker singing-voice corpus JVS-MuSiC [37] is produced in a recording studio, and the recordings were controlled by a professional sound director.
Multi-Singer: Fast Multi-Singer Singing Voice Vocoder With A Large-Scale Corpus
Singing voice data differs greatly from speech. Our preliminary research concludes that the main differences between singing voices and speech lie in phoneme duration and pitch (i.e., fundamental frequency), which we would discuss in appendix A in the supplementary materials. Unlike TTS with sufficient transcribed data, SVS suffers from data shortage due to its high recording and annotation cost and stricter copyright issues in the music domain. Limited singing datasets of different sizes and annotated contents are available for research purposes, and here we introduce a few singing voice datasets for comparison in Table 1.
Multi-Singer: Fast Multi-Singer Singing Voice Vocoder With A Large-Scale Corpus
The MIR-1K dataset [15] establish the first comprehensive and publicly available dataset for singing voice separation, which does not contain segmentation on the word level. The proposal of multisinger NUS-48E corpus [10] is an ongoing effort toward a comprehensive, well-annotated dataset for singing voice related research. But NUS-48E dataset includes 48 songs with reasonably balanced phoneme distribution, which is not large enough for SVS systems training. JukeBox [7] contains 467 hours of singing audio data sampled at 16 kHz, which has been downloaded from the Internet Archive (IA) with a wide variety of acoustic environments and recording apparatus. Hence, it is not suitable for high-quality singing voice synthesis.
Multi-Singer: Fast Multi-Singer Singing Voice Vocoder With A Large-Scale Corpus
To conclude, the above data could not meet our requirements for singing voice synthesis research in terms of quality and quantity. Here in this paper, we propose an open-source, large-scale, and multi-singer singing voice corpus OpenSinger.
Multi-Singer: Fast Multi-Singer Singing Voice Vocoder With A Large-Scale Corpus
Singing voice synthesis (SVS) is a generative task that produces acoustic waveforms of singing given lyrics and music score input. A typical singing voice synthesis system consists of an acoustic model to convert musical scores into acoustic features and a vocoder to generate audio waveforms from acoustic features. Previous works have conducted studies on SVS from multiple aspects. DeepSinger [35] is a multi-lingual SVS system built from scratch using singing training data mined from music websites. Choi at all [6] build a Korean singing voice synthesis system using an autoregressive algorithm that generates spectrogram with the boundary equilibrium GAN objective. Chen at all [2] introduce multi-scale adversarial training in both the acoustic model and vocoder to improve singing modeling. As the papers say, these previous SVS systems could generate natural singing voices. However, because vocoders in such SVS systems are not designed towards multiple singers, there would be a distinct degradation in quality when generating unseen singers' voices.
Multi-Singer: Fast Multi-Singer Singing Voice Vocoder With A Large-Scale Corpus
With the powerful model assumption and the solid theoretical ground, autoregressive vocoders have dominated singing voice modeling for a long time. WaveRNN [18]   regularly adopted to synthesize waveform in SVS systems. Nonautoregressive vocoder Parallel WaveNet [25] provides a fast waveform generation method based on a teacher-student framework with probability density distillation. Besides, WaveGlow [29] is another non-autoregressive vocoder, which consumes an enormous computation cost during training. More recently, researchers propose diffusion-based models WaveGrad [4] and DiffWave [20] for waveform generation, which are built on prior work of score matching and diffusion probabilistic models.
Multi-Singer: Fast Multi-Singer Singing Voice Vocoder With A Large-Scale Corpus
To avoid the sample-by-sample causal inference or the use of teacher models, researchers have adopted the idea of the generative adversarial network (GAN) to train neural vocoders. MelGAN [21] is a light non-autoregressive vocoder pioneering based on a generative adversarial network, which is free from distillation. HiFi-GAN [19] consists of small sub-discriminators obtaining specific periodic parts of raw waveforms, achieving higher computational efficiency and sample quality. VocGAN [41] applies the joint conditional and unconditional objective, which is inspired by successful results in high-resolution image synthesis. Although these vocoders could be applied in SVS systems, distinct degradations occur when generalizing those systems to unseen singers.
Multi-Singer: Fast Multi-Singer Singing Voice Vocoder With A Large-Scale Corpus
OpenSinger contains pop songs collected from 93 singers, and singing utterances are saved in wav format, sampled at 24 kHz, and quantized by 16 bits. OpenSinger consists of 50 hours of singing voices recorded in a professional recording studio, including 30 hours from 41 females and 20 hours from 25 males apart from the person-of-interest (POI). Figures in appendix A in the supplementary materials summarize the distribution of pitch, sentence-level duration, and phoneme-level duration of utterances. The major features of OpenSinger include:
Multi-Singer: Fast Multi-Singer Singing Voice Vocoder With A Large-Scale Corpus
• Open source. Lack of data could obstruct the construction of SVS systems, so we release our corpus to accelerate research in the community. • Large scale. Data-hungry singing voice systems need a significant amount of data in the training process. To our best knowledge, OpenSinger is the most extensive Chinese multisinger singing voice corpus. • High quality. Similar to text-to-speech, high-quality audios without noise or background sound are essential for highfidelity singing voice synthesis. Professional singers and studios both ensure high-quality utterances in OpenSinger.
Multi-Singer: Fast Multi-Singer Singing Voice Vocoder With A Large-Scale Corpus
Collection Procedure In the data collection procedure, we select Chinese traditional and pop songs, and organize a group of 93 professional singers to record 80 hours of singing voices. The recording takes place in a private recording studio. The songs are saved in wav format, sampled at 24 kHz, and quantized by 16 bits.
Multi-Singer: Fast Multi-Singer Singing Voice Vocoder With A Large-Scale Corpus
Data labeling A professional annotation team annotates the utterances in each song. Each utterance is annotated with the name of the song, singer, and reference text, which is the official Latinized notation for marking Chinese pronunciations. Further, We use open source tools pypinyin 1 to convert Chinese lyrics into phonemes.
Multi-Singer: Fast Multi-Singer Singing Voice Vocoder With A Large-Scale Corpus
After data collection, the singing voice corpus still could not meet high-quality singing voice generation requirements for the following aspects: 1) Most vocoders need data in the form of relatively short utterances usually up to a few seconds in length, each with corresponding text during training procedures. 2) Pauses and resulting silences commonly remain in raw songs, which would hurt adversarial stability and bring unnecessary calculations. Our processing procedure consists of several stages to fix the issues we listed above for high-quality singing voice generation, including segmentation, silence trimming, and alignment.
Multi-Singer: Fast Multi-Singer Singing Voice Vocoder With A Large-Scale Corpus
Silence trimming After data division, there still usually be some long-term silences in singing voices. Voice Activation Detection (VAD) has been used to remove silent segments, and thus we detect and discard the non-vocal segments using VAD. During silence trimming, each utterance has been manually verified to discard audio samples that do not contain singing vocals every 100ms. Cutting these silence segments could significantly shorten the audios as exceedingly speeds up the alignment formation.
Multi-Singer: Fast Multi-Singer Singing Voice Vocoder With A Large-Scale Corpus
Segmentation Obviously, singing voices with lengthy audio clips are not suitable for memory-limited GPU computation, so we fragment them into many small files. Following the Lyrics-to-Singing alignment in [35], we split the whole song into aligned lyrics and audio. We segment an audio by the frames that are aligned to the separation marks in raw lyrics, making sure each processed sentence's duration is limited to 0-11 seconds.
Multi-Singer: Fast Multi-Singer Singing Voice Vocoder With A Large-Scale Corpus
Alignment For more precise alignment, we adopt Montreal Forced Aligner 2 and take an orthographic transcription of an audio file and generate a time-aligned version. In Montreal Forced Aligner, an annotation for phonemes is generated by aligning the manuallylabeled phone strings of the sung lyrics using training conventional Gaussian Mixture Model (GMM) -Hidden Markov Model (HMM) system. The annotated phonemes support us for broad and preliminary observations about the phoneme-level duration of the corpus.
Multi-Singer: Fast Multi-Singer Singing Voice Vocoder With A Large-Scale Corpus
After the data collection and processing procedure, we check for audio quality and conduct statistical evaluation, including sentencelevel and phoneme-level duration distribution, pitch distribution and speaker similarity.
Multi-Singer: Fast Multi-Singer Singing Voice Vocoder With A Large-Scale Corpus
Sentence-Level Duration Distribution We have segmented the songs into sentence-level singing voices during audio processing. We plot the sentence-level duration distribution of the male and female singing voices separately. As shown in appendix A in the supplementary materials, sentence durations between genders are similar with some small variations due to our manual segmentation in advance.
Multi-Singer: Fast Multi-Singer Singing Voice Vocoder With A Large-Scale Corpus
We align the phoneme sequence to the singing audio frames using MFA mentioned above and visualize distribution in appendix A in the supplementary materials. Phoneme durations of male and female singing voices have a similar distribution, mainly scatters around 10 ms to 500 ms.
Multi-Singer: Fast Multi-Singer Singing Voice Vocoder With A Large-Scale Corpus
Pitch Distribution We extract pitch from the audio using Parselmouth 3 and show statistical distribution. Obviously, due to the diversity of timbre and pronunciation habits among genders, the drawn pitch distribution figures differ greatly. Female singing voices exhibit a larger pitch range, trending toward high frequency more significantly. In other words, there are more high-frequency parts in female singing voices.
Multi-Singer: Fast Multi-Singer Singing Voice Vocoder With A Large-Scale Corpus
Speaker Similarity To learn about speaker identity differences among singers in OpenSinger, we visualize the speaker representations in Figure 1 using Resemblyzer 4 . 10 utterances of 10 randomly sampled speakers are chosen, each is converted into a 256dimensional embedding and reduced to 2-dimensional with Uniform Manifold Approximation and Projection (UMAP). It has presented a large inter-speaker distance among singers and demonstrated the singer diversity in OpenSinger.
Multi-Singer: Fast Multi-Singer Singing Voice Vocoder With A Large-Scale Corpus
Several characteristic differences exist between singing voice and speech generation: 1) Singing voices include richer emotional information, which varies from singing expression and style. 2) Singing voices have longer continuous pronunciations and contain more high-frequency parts. To verify our thinking, we conduct statistical analysis on the Chinese speech dataset CSMSC 5 and plot the phoneme-level duration as well as pitch distribution. Because sentences are cut and segmented manually, we haven't considered sentence-level discrepancies. Table 2 come a few interesting phenomena: 1) Since singing voices are more various and diverse, statistic distributions of singing voices become broader with large variance. 2) Pitch and phoneme-level duration in speech corpus are lower than that in singing voices on average, demonstrating the longer continuous pronunciation and more high-frequency parts in singing voices.  To conclude, the characteristic differences between speech and singing voice we analyze above have brought additional challenges for singing voices synthesis researches, and reduce researchers' enthusiasm to assemble singing voice corpora. To our best knowledge, OpenSinger is the largest open-source, multi-singer, Chinese singing voice corpus, and we hope that the release of OpenSinger could contribute to the community.
Multi-Singer: Fast Multi-Singer Singing Voice Vocoder With A Large-Scale Corpus
In recent years, state-of-the-art vocoders have significantly improved the singing voice quality and spawned SVS systems deployments. However, due to the high computational cost and timeconsumed generation, real-time applications could become challenges. Researchers have adopted multi-band generation such as Multiband-WaveRNN [42], Multiband-MelGAN [40] to speed up waveform modeling. Related multi-band vocoders generate each sub-band of waveform, and then conduct bands splicing using Pseudo Quadrature Mirror Filter Bank (PQMF) [24]. Nevertheless, previous multi-band techniques are designed towards fast generation and lack considering the characteristic differences among frequency bands (e.g., short-period high-frequency band and longperiod low-frequency band), usually resulting in the limitation of synthetic singing voice quality. This paper proposes the frequencyadaptive multi-band generation technique to adjust singing voice synthesis in both speed and quality.
Multi-Singer: Fast Multi-Singer Singing Voice Vocoder With A Large-Scale Corpus
Sufficient singer generalization means that the vocoder could generate high-fidelity audio in various singer domains, regardless of whether the input has been encountered during training or has come from an out-of-domain singer. Researchers have investigated ways to supervise vocoders for learning singer identity in multisinger modeling.
Multi-Singer: Fast Multi-Singer Singing Voice Vocoder With A Large-Scale Corpus
Multi-speaker data Training on data of multiple speakers could probably improve model generalization. Using multi-speaker data, researchers [8] demonstrate better performance in speaker similarity for multi-speaker speech synthesis. However, without explicitly adopting architecture for speaker identity reconstruction, vocoders would be data-hungry and encounter generalization restriction. A distinct degradation emerges when we adapt these vocoders to the unseen speakers modeling.
Multi-Singer: Fast Multi-Singer Singing Voice Vocoder With A Large-Scale Corpus
Extra embedding input Taking embedding as additional inputs could be another way to handle singer generalization. Transfer learning from speaker verification to multi-speaker text-to-speech synthesis [17] takes the speaker verification network as the speaker encoder and concatenates the generated speaker embedding to each encoder time step. Speaker Conditional WaveRNN [28] takes speaker embeddings extracted from a pre-trained speaker verification model as additional input and exploits extra speaker information. However, extra embeddings would sometimes be hard-earned in SVS systems during the inference process. Worse still, it takes a higher computation cost and reduces inference speed intolerably.
Multi-Singer: Fast Multi-Singer Singing Voice Vocoder With A Large-Scale Corpus
To maintain feasibility and improve generalization towards the unseen singers, we have better explore how to teach vocoders explicitly capture singer identity embed in the acoustic feature (i.e., mel-spectrogram) without additional computational cost during singing voice synthesis.
Multi-Singer: Fast Multi-Singer Singing Voice Vocoder With A Large-Scale Corpus
Generative adversarial network based vocoder jointly trains a powerful generator G, and convolutional neural network (CNN) discriminator D, to generate time-domain waveform from the corresponding input mel-spectrogram. We have introduced a technique to improve inference speed and quality of synthetic waveform in adaptation to the unseen singer: 1) Firstly, to speed up waveforms modeling, we introduce a multi-band generator, which synthesizes sub-band signals adapted to the frequencies. 2) For multi-singer singing voice generation, Multi-Singer adopts a singer conditional discriminator to judge whether the singer identity of input voices has been properly constructed. 3) For training objective, we introduce joint adversarial training of conditional and unconditional loss and propose singer perceptual loss to penalize the generator for synthesizing singing voices with singer identification bias. The joint training method effectively works in GANs for raw audio generation.
Multi-Singer: Fast Multi-Singer Singing Voice Vocoder With A Large-Scale Corpus
To develop high-fidelity waveform generation in parallel, we introduce a multi-band generator. It is acknowledged that waveform signals' characteristics vary in different frequency bands, so we model them separately. The multi-band generator transforms the input noise drawn from a Gaussian distribution to the output waveform in parallel, and the multi-band generation process has been shown in Figure 2. We divide waveform into four frequency bands including two high-frequency bands and two low-frequency bands, and the waveform generation in high and low frequency bands is conducted separately using two distinct frequency-adapted models, respectively. The synthetic sub-band waveforms are merged into the final singing voices through the PQMF filter.
Multi-Singer: Fast Multi-Singer Singing Voice Vocoder With A Large-Scale Corpus
As shown in Figure 2, the generator consists of WaveNet blocks, whose architecture has been discussed in appendix E in the supplementary materials. Specifically, receptive fields and the number of WaveNet blocks vary in these two frequency-adapted models, and they are carefully designed towards different acoustic characteristics among frequency bands. The generator is non-autoregressive and capable of adjusting singing voice synthesis in both speed and quality.
Multi-Singer: Fast Multi-Singer Singing Voice Vocoder With A Large-Scale Corpus
The architecture of Discriminators has been shown in Figure 2(c), where p1 and p2 denotes the possibility of the sample generated by conditional and unconditional discriminator, respectively. Conditional input has been applied for high-resolution image synthesis tasks and produced successful results [43], and conditional adversarial networks have been widely acknowledged for stable performance in image-to-image translation [16]. Previous vocoder studies have further demonstrated the efficiency of conditional input in discriminators. VocGAN [41] introduces the hierarchically-nested JCU discriminator, which learns intermediate representations directly conditioned on the input mel-spectrogram. GAN-TTS [1] proposes conditional DBlock, where the embedding of the linguistic features are added after the first convolution.
Multi-Singer: Fast Multi-Singer Singing Voice Vocoder With A Large-Scale Corpus
To better supervise the singer identity reconstruction in multisinger singing voice generation, we adopt a singer conditional discriminator. The singer conditional discriminator judges whether the singer identity of input voices has been properly constructed, and the generator is trained to fool the singer conditional discriminator by increasing the real possibility of the generated sample.
Multi-Singer: Fast Multi-Singer Singing Voice Vocoder With A Large-Scale Corpus
The singer conditional discriminator(p1) first adopts a 256x downsampled block, which is performed using strided average pooling with kernel size 4 and done in 4 stages of 8x, 8x, 2x, and 2x downsampling. After passing through the downsampled block, raw waveforms are converted into 256-dimension vectors. Because singer identifications keep stable in the long-term waveforms, we feed these 256-dimension representations in LSTM layers and obtain steady identity in the final output. We conduct element-wise addition operation between high-level singer identities and reference singer embeddings, and project them to possibility with a linear layer and ReLU activation function.
Multi-Singer: Fast Multi-Singer Singing Voice Vocoder With A Large-Scale Corpus
The unconditional discriminator(p2) consists of ten layers of non-causal dilated 1-D convolutions. The strides are set to one and linearly increasing dilations are applied for the 1-D convolutions starting from one to eight except for the first and last layers. Channels and kernel sizes are set to 64 and 5, respectively.
Multi-Singer: Fast Multi-Singer Singing Voice Vocoder With A Large-Scale Corpus
Training objectives should be carefully designed for stable training and faster convergence. In this section, we first describe Joint adversarial conditional and unconditional loss, and then we propose an auxiliary training loss named Singer Perceptual Loss. Finally, we adopt multi-resolution STFT loss as additional auxiliary loss and decide our final loss function.
Multi-Singer: Fast Multi-Singer Singing Voice Vocoder With A Large-Scale Corpus
Joint adversarial conditional and unconditional (JCU) loss In contrast to conventional adversarial loss, JCU loss combines the conditional and unconditional adversarial losses as Eq. 1. Previous work like VocGAN [41] has demonstrated the efficiency of JCU loss for adversarial training. Here our adversarial conditional loss is significantly different from that in VocGAN. For multi-singer modeling, our proposed conditional loss is conducted by combating between generator and singer conditional discriminator, leading generator to better capture singer identity embed in the acoustic feature input (i.e., mel-spectrogram). The adversarial unconditional loss enhances the generator to synthesize more natural singing voices by classifying ground truth samples to 1 and the synthetic samples to 0. The JCU loss is defined in Eq. 1 and Eq. 2.  
Multi-Singer: Fast Multi-Singer Singing Voice Vocoder With A Large-Scale Corpus
where , and denote the ground truth and synthetic singing voices, and the singer embedding respectively. is the multi-band generator, is the unconditional discriminator, is the singer conditional discriminator.
Multi-Singer: Fast Multi-Singer Singing Voice Vocoder With A Large-Scale Corpus
Singer Perceptual Loss In this work, we propose an auxiliary loss function that could improve the vocoder performance for multisinger singing voice modeling. Perceptual loss is introduced in style reconstruction by Gratys et al. [12] for the first time, and many studies have taken the ideas behind perceptual loss to improve the quality of the outputs generated by a deep-learning-based model. Here we introduce a new loss objective named singer perceptual loss, which enables the generator to sense singer bias and optimize speaker similarity between ground truth and synthetic waveforms in the frequency domain during training.
Multi-Singer: Fast Multi-Singer Singing Voice Vocoder With A Large-Scale Corpus
Singer perceptual loss depends on high-level hidden states extracted from the pre-trained speaker encoder, which we will discuss in appendix C in the supplementary materials. We judge melspectrogram in the frequency domain to calculate singer perceptual loss. It is acknowledged that the spectrum envelope of waveforms could provide singer representations, and hence envelope reconstruction in the frequency domain is also the procedure of singer identity reconstruction. Singer perceptual loss supervises singer identification reconstruction more efficiently, and it is defined as follows:
Multi-Singer: Fast Multi-Singer Singing Voice Vocoder With A Large-Scale Corpus
where ( , ) denotes singer perceptual loss, ∥ · ∥ 2 denotes the L2 norms, and let (Mel( )) be the -th layer hidden state of LSTM in the pre-trained speaker encoder when processing the mel-spectrogram of signal .
Multi-Singer: Fast Multi-Singer Singing Voice Vocoder With A Large-Scale Corpus
Multi-resolution STFT Loss To further stabilize the adversarial training process, Multi-Singer adopts a multi-resolution STFT(Short Time Fourier Transform) auxiliary loss. Similar to the previous work [39], we define the STFT loss as follows:
Multi-Singer: Fast Multi-Singer Singing Voice Vocoder With A Large-Scale Corpus
where ∥ · ∥ and ∥ · ∥ 1 denote the Frobenius and L1 norms;
Multi-Singer: Fast Multi-Singer Singing Voice Vocoder With A Large-Scale Corpus
(·) and denote the STFT magnitude of the -th STFT parameter set and the number of elements in the magnitude, respectively. − and − denote the spectral convergence and log STFT magnitude, respectively.
Multi-Singer: Fast Multi-Singer Singing Voice Vocoder With A Large-Scale Corpus
The final multi-resolution STFT loss is the sum of M losses with different analysis parameters (i.e., FFT size, window size, and hop size), which is represented as follows:
Multi-Singer: Fast Multi-Singer Singing Voice Vocoder With A Large-Scale Corpus
Final loss As mentioned above, our auxiliary loss consists of the singer perceptual loss and multi-resolution STFT loss as follows:
Multi-Singer: Fast Multi-Singer Singing Voice Vocoder With A Large-Scale Corpus
where ( , ) denotes the multi-resolution STFT loss, and ( ) denotes the auxiliary loss of generator.
Multi-Singer: Fast Multi-Singer Singing Voice Vocoder With A Large-Scale Corpus
To conclude, our final loss function for the generator is defined as a linear combination of the auxiliary loss, adversarial loss:
Multi-Singer: Fast Multi-Singer Singing Voice Vocoder With A Large-Scale Corpus
where denotes the hyperparameter balancing loss terms and we set = 10. By jointly optimizing the waveform-domain adversarial loss and auxiliary loss including singer perceptual loss and multiresolution STFT loss, the generator can learn the distribution of the realistic speech waveform effectively.
Multi-Singer: Fast Multi-Singer Singing Voice Vocoder With A Large-Scale Corpus
In this section, we first describe the experimental setup including dataset and model configurations. Then we report experimental results and conduct some analyses.
Multi-Singer: Fast Multi-Singer Singing Voice Vocoder With A Large-Scale Corpus
We randomly choose 340 utterances for validation and 60 utterances from 6 singers as the seen singer test set. To evaluate the model generalization to unseen singers, we prepare 5 utterances from each singer including five males and five females as the additional unseen singer test set. The seen singer test set helps judge if the neural vocoder could synthesize high-fidelity singing voices, and the unseen singer test set helps measure the model's generalization to singer out of domain. At the same time, we take Chinese speech dataset CSMSC for comparison and choose 60 utterances as the test set.
Multi-Singer: Fast Multi-Singer Singing Voice Vocoder With A Large-Scale Corpus
Speaker encoder We train the speaker encoder with a few large scale multi-speaker datasets following the guidance in [17]: 1) Lib-riSpeech Other, which contains 461 hours of speech from a set of 1,166 speakers; 2) VoxCeleb and VoxCeleb2 which contain 139K utterances from 1,211 speakers, and 1.09M utterances from 5,994 speakers, respectively. 3) OpenSinger. We further fine-tune the speaker encoder on the singing voice corpus to ensure it has learned about the distribution of singing voices. Speaker encoder has been trained until convergence (around 50k iterations) as shown in appendix C in the supplementary materials.
Multi-Singer: Fast Multi-Singer Singing Voice Vocoder With A Large-Scale Corpus
Multi-Singer The generator in Multi-Singer takes 80-band melspectrograms as input, which are extracted in params (FFT:512, hop size:128, window size:512). At the training stage, a multi-resolution STFT loss would be computed by the sum of three different STFT losses as described in appendix D in the supplementary materials. Multi-Singer is trained for 40k steps with RAdam optimizer to stabilize training. Note that we apply pre-training on the generator for the first 100k steps, after which generator and discriminators are jointly trained. When training previous vocoders such as WaveRNN, MelGAN, and Parallel WaveGAN from scratch, GitHub implementations are used for reproducibility and the configurations follow their original papers.
Multi-Singer: Fast Multi-Singer Singing Voice Vocoder With A Large-Scale Corpus
In order to validate the audio quality of OpenSinger, we train several state-of-the-art neural vocoders such as WaveRNN [18], Mel-GAN [21] and Parallel WaveGAN [39]. We then conduct evaluations on synthetic seen singer samples 6 , and the evaluation matrix has been discussed in appendix B in the supplementary materials.
Multi-Singer: Fast Multi-Singer Singing Voice Vocoder With A Large-Scale Corpus
Our further evaluation lies on the corpus scale. It's well-known that high-fidelity TTS is data-hungry and hence model pre-training is essential in practice. Researchers [8] used to perform the warmstart strategy to overcome data shortage. To explore the effectiveness of additional speech data in SVS training, we pre-train vocoder on a large speech corpus CSMSC (200k steps) and fine-tune the model on OpenSinger until convergence (500k steps).
Multi-Singer: Fast Multi-Singer Singing Voice Vocoder With A Large-Scale Corpus
Corpus verification results have been introduced in Table 3, and we come to the following conclusions:
Multi-Singer: Fast Multi-Singer Singing Voice Vocoder With A Large-Scale Corpus
• OpenSinger simulates the real-world singing voice distribution, and state-of-the-art neural vocoders perform as well as previous papers say. Robust models could be trained using OpenSinger, demonstrating the effectiveness of this dataset. • A slight performance drop appears when using the speech pre-train strategy, which indicates that additional speech data is not needed since OpenSinger is large enough for high-quality singing voice modeling. 6 Table 3: MOS results with 95% confidence intervals and FDSD for corpus verification.
Multi-Singer: Fast Multi-Singer Singing Voice Vocoder With A Large-Scale Corpus
To verify the effectiveness of the proposed multi-band generator in Multi-Singer, we conduct comparison with competing multi-band architectures such as multiband-MelGAN and multiband-WaveRNN. For fair comparison, we train these models from scratch under the setting in section 5.1 and implement MOS assessments and real-time factor (RTF) evaluation. From the experimental results in Table 4, we draw the following conclusions: 1) Due to the autoregressive architecture, Multi-band WaveRNN achieves the best performance and generates the most natural sounds, which limits overall generation speed on the other hand. 2) Multi-band MelGAN could achieve fast singing voice synthesis, while a distinct quality degradation in audio comes. 3) as for Multi-Singer, because of the multi-band architecture adapted towards characteristic differences among frequency bands, the non-autoregressive generator could adjust singing voice synthesis in both speed and quality  Table 4: MOS results with 95% confidence intervals and realtime factor (RTF) of each multi-band generation methods.
Multi-Singer: Fast Multi-Singer Singing Voice Vocoder With A Large-Scale Corpus
To perform Multi-Singer's better generalization to multiple unseen singers, we evaluate on the synthetic singing voices and compare them with competing architectures. For a fair comparison, we train these vocoders on our proposed multi-singer dataset OpenSinger and use the same experiment setup described in Section 5.1. We implement MOS assessments and present objective evaluations such as Fréchet Deep Speech Distances (FDSD), and speaker cosine similarity as shown in Table 5. We have attached the concrete matrix in appendix B in the supplementary materials. Note that the quality of the added out-of-domain unseen singer recordings is lower than that of seen speaker recordings. Therefore we do not conduct meaningless evaluation across seen and unseen singer test set. Experimental results represent the robustness of Multi-Singer and its outperform capability of unseen singer modeling. We come to conclusions as follows: 1) Architectures such as MelGAN and Parallel WaveGAN haven't explicitly introduced methods for multisinger adaptation, thus an unavoidable degradation occurs when modeling singing voices of unseen singers. 2) Singer Conditional WaveRNN (SC-WaveRNN) introduces singer-embedding as additional information to control the singer identity during inferences,  Table 5: MOS results with 95% confidence intervals, FDSD and cosine similarity for multi-singer modeling. Note that the quality of the added out-of-domain unseen singer recordings is lower than that of seen speaker recordings. Therefore we do not conduct evaluation across seen and unseen singer test set.  
Multi-Singer: Fast Multi-Singer Singing Voice Vocoder With A Large-Scale Corpus
We conduct ablation studies under the settings in Section 5.1 to verify the effectiveness of several components in Multi-Singer, including 1) multi-band generator; 2) singer conditional discriminator; and 3) singer perceptual loss. Table 6 shows the mean opinion score of audio quality as assessed via human listening tests and objective evaluation results. Our analysis leads to the following conclusions: Replacing multi-band generator with a full band generator causes a significant decrease in generation speed, and we observe that the quality of the synthetic singing voices drops slightly at the same time. The absence of Singer Conditional Discriminator (SCD) results in decreased cosine similarity scores on unseen singers, which suggests that the modified vocoder has difficulties in capturing singer identity. Because abundant singer representations are embedded in the spectrum envelope of singing voices, removing the frequency-domain auxiliary objective Singer Perceptual Loss (SPL) could weaken vocoder in the reconstruction of singer representations. Our ablation study shows that the baseline model Multi-Singer can speed up waveforms generation and efficiently reestablish the singer identity of unseen singers in singing voices.
Multi-Singer: Fast Multi-Singer Singing Voice Vocoder With A Large-Scale Corpus
To verify the effectiveness of Multi-Singer in singing voice synthesis systems, we adopt a modified FastSpeech 2 as an acoustic model to convert the words of songs into acoustic features and build an overall system. We have attached the modified FastSpeech 2 in appendix F in the supplementary materials. During training, the configuration follows prior work [31]. Since the F0 and duration are usually known in singing voice synthesis, we remove the  
Multi-Singer: Fast Multi-Singer Singing Voice Vocoder With A Large-Scale Corpus
As shown in Github pages 7 , there come differences between speech data and singing voice data. To be more specific, 1) Singing voices vary from expression and style, including richer emotional information. 2) Singing voices usually have a high sampling rate, which causes a wider spectrogram band in the frequency domain and more high-frequency parts.
Multi-Singer: Fast Multi-Singer Singing Voice Vocoder With A Large-Scale Corpus
All our Mean Opinion Score (MOS) tests are crowdsourced and conducted by native speakers. We refer to the rubric for MOS scores in [30], and the scoring criteria has been included in Table 8 
Multi-Singer: Fast Multi-Singer Singing Voice Vocoder With A Large-Scale Corpus
Fréchet Deep Speech Distances (FDSD) judges the quality of synthetic audio samples based on their distance to a reference set. These distances are conceptually similar to the FID (Fréchet Inception Distance). The energy distance can be combined with GAN-based learning, further improving on either individual technique. As the paper [13] says, FDSD is a proper scoring rule with respect to the distribution over spectrograms of the generated waveform audio.
Multi-Singer: Fast Multi-Singer Singing Voice Vocoder With A Large-Scale Corpus
Cosine similarity is an objective metric that measures speaker similarity among multi-singer audio. cos_sim( , ) = · | | | | | | | | . We also compute the average cosine similarity between embeddings extracted from synthetic speech and the ground truth embeddings to measure the speaker similarity performance objectively. Embeddings of utterances from the same speaker have high cosine similarity, while those from different speakers are far apart in the embedding space.
Multi-Singer: Fast Multi-Singer Singing Voice Vocoder With A Large-Scale Corpus
Speaker verification verifies the speaker identity and tells if the representations come from a related speaker. A speaker-discriminative neural encoder [38] on a speaker verification (SV) task using a state-of-the-art generalized end-to-end loss. After training on a large amount of data, the speaker encoder could attain robust representations that capture an ample singer identity space. As a result, speaker encoders are usually used for feature extraction, which effectively captures the audio's long-term speaker identity. 7   Inspired by previous work that has reported the effectiveness of combining a well-trained d-vector model with a TTS model, we build a speaker encoder that projects the mel-spectrogram from the speech utterance to a 256-dimensional. Generalized end-toend loss (GE2E) makes the training of speaker verification models more efficient than the previous tuple-based end-to-end (TE2E) loss function. During the training process as shown in appendix C in the supplementary materials, the speaker encoder could reduce intra-speaker variance and separate different speakers apart.
Multi-Singer: Fast Multi-Singer Singing Voice Vocoder With A Large-Scale Corpus
Here we introduce details of the multi-resolution STFT loss. Window size   1024  600  120  2048  120  250  512 240 50 Table 9: The details of the multi-resolution STFT loss. A hanning window was applied before the FFT process.
Multi-Singer: Fast Multi-Singer Singing Voice Vocoder With A Large-Scale Corpus
The architecture of WaveNet block [26] in generator has been shown in Figure 3. X and Mel denote noise and mel-spectrogram, respectively. Noise passes through the dilated convolution layers, and X, Mel are divided into xa, xb and sa, sb, respectively. After the sigmoid-tanh calculation, the processed feature passes through two fully-connected networks and output H and X, which would be fed into the next dilated convolution layer.
Multi-Singer: Fast Multi-Singer Singing Voice Vocoder With A Large-Scale Corpus
As shown in Figure 3, our modified FastSpeech 2 converts lyrics, F0, duration, and singer embedding into the mel-spectrogram, after which Multi-Singer converts the mel-spectrogram into singing voices.
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
Human perception of speech is intrinsically multimodal, involving audition and vision. The speech production is accompanied by the movement of lips and teeth, which can be visually interpreted to understand speech. Visual cues of speech not only play an essential role in language learning for pre-lingual children (Meltzoff & Moore, 1977;Davies et al., 2008), but also improve speech understanding in noisy environment (Sumby & Pollack, 1954) and provide patients of speech impairment with means of communication. Furthermore, perceptual studies (McGurk & MacDonald, 1976) have shown that such visual cues can alter the perceived sound.
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
For machine learning models, the tight coupling between audio and visual lip movement information emerges as a natural source for supervision to learn speech representations, which has not been extensively utilized yet in the self-supervised speech representation learning literature. Recent successful representation learning frameworks for speech (e.g., APC (Chung et al., 2019), CPC (Oord et al., 2018;Kharitonov et al., 2021), wav2vec 2.0 (Baevski et al., 2020;Hsu et al., 2021b), De-CoAR2.0 (Ling & Liu, 2020), HuBERT (Hsu et al., 2021c;a)) are mostly built entirely on audio. The fundamental research question addressed in this paper is whether a self-supervised audio-visual speech representation learned from the lip movement information, alongside the audio signal in video recordings, captures cross-modal correlations and improves downstream performance for visual speech recognition (i.e., lip reading) and automatic speech recognition (ASR) tasks. Existing ML models for lip-reading rely heavily on text transcriptions to achieve an acceptable level of accuracy. The state-of-the-art lip-reading model (Makino et al., 2019) requires 31K hours of transcribed video data for training. Such large amounts of labeled data are expensive and hard to obtain for most of the world's 7,000 languages. The benefit from a robust visual speech representation learning framework goes beyond lip-reading. Additionally, it can benefit a vast range of applications, 2 RELATED WORK The strong correlation between video modalities provides an effective means for self-supervised representation learning on videos, which has been explored in many prior works and is still an active research area. This work draws inspiration from two lines of previous research:
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
Multimodal general video representation focuses on learning self-supervised audio-visual representations of general videos to solve high-level semantic tasks, e.g., action recognition and audio event detection (Arandjelović & Zisserman, 2017;Bruno et al., 2018;Morgado et al., 2021;Chen et al., 2020;Lee et al., 2021). Owens & Efros (2018) learns a multimodal network to predict whether the audio and visual streams of a video are temporally synchronized while Pham et al. (2019) applies a cyclic translation between different modalities. Piergiovanni et al. (2020) learns the visual representation through a multi-tasking framework that incorporates a series of pretext tasks such as reconstruction and temporal ordering prediction. Sharing our work's inspiration of DeepClustering (Caron et al., 2018), XDC (Alwassel et al., 2020) and AV-BERT (Chan et al., 2021) learn cross-modal representations through predicting cross-modal or cluster assignments. In contrast to XDC, AV-HuBERT is trained with a BERT-like masked prediction loss, which forces the model to learn the structure within the multimodal input and was shown in Hsu et al. (2021c) to be more resilient to bad cluster assignments compared to unmasked cluster prediction. On the other hand, AV-BERT focuses on learning utterance-level multimodal environment embeddings that serves as the global context for ASR, while our objective is to learn frame-level audio-visual speech representations and pre-train a model that can be fine-tuned for downstream tasks with either modality.
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
Semi-and self-supervised audio-visual speech representation learning focuses on improving lip-reading with untranscribed audio-visual speech data. To solve isolated visual word recognition,  learns visual embeddings using a contrastive loss based on audio-visual synchronization. Ma et al. (2021a) learns visual speech representations by minimizing the distance between its latent features and off-the-shelf audio embeddings. Using an external supervised ASR to transcribe unlabeled audio,  trains their lip-reading model on the augmented labeled and pseudo-labeled data. Unlike Ma et al. (2021a) and , our model is trained from scratch and encouraged to learn contextualized representations with a masked prediction task. Moreover, our method does not rely on any external pre-trained models.
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
Our research builds on Audio HuBERT (Hsu et al., 2021a) which is a self-supervised learning framework for speech and audio. It alternates between two steps: feature clustering and masked prediction. In the first step, a discrete latent variable model (e.g., k-means) is applied to a sequence of acoustic frames A 1:T producing a sequence of frame-level assignments z a 1:T . Clusters of signalprocessing-based acoustic features, e.g., Mel-frequency cepstral coefficients (MFCC), exhibit nontrivial correlations with the inherent acoustic units of speech inputs. Using (A 1:T , z a 1:T ) pairs, the second step learns new feature representations by minimizing a masked prediction loss, similar to masked language modeling in BERT (Devlin et al., 2019). The pressure to predict cluster assignments of masked audio regions forces the model to learn good local acoustic representations for unmasked regions and long-range temporal dependencies between latent features. Repeating these two steps improves the cluster quality and consequently the quality of the learned representations.
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
Single-modal Visual HuBERT: The most naïve way to extend HuBERT to the visual domain is by generating targets using visual features. Formally, given an image sequence I 1:T , we first cluster the image features into a sequence of discrete units z i 1:T via k-means: z i t = k-means(G(I t )) ∈ {1, 2, ..., V }, where G is an visual feature extractor and V is the codebook size. The cluster assignments z i 1:T serve as the prediction targets of the model. Initially, G can be an engineered image feature extractor such as Histogram of Oriented Gradients (HoG), analogous to MFCC in audio HuBERT. The intermediate layers of the HuBERT model are used as G in later iterations.
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
To perform the masked prediction task, the model first encodes I 1:T using a ResNet into an intermediate visual feature sequence f v 1:T , which is then corrupted intof v 1:T via a binary mask M . Specifically, ∀t ∈ M ,f v t is replaced with a learned masked embedding. We adopt the same strategy in HuBERT to generate span masks. The masked visual featuresf v 1:T are encoded into a sequence of contextualized features e 1:T via a transformer encoder followed by a linear projection layer. The loss is computed over the masked regions and optionally over unmasked ones (when α ≥ 0):
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
Where (W ∈ R d×V , b ∈ R V ) are parameters of the projection layer which maps features into logits predicting the cluster assignments.
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
Cross-modal Visual HuBERT: The single-modal visual HuBERT aims to learn visual speech representation through gradually refined image features. However, it does not employ the audio stream of the video. Presumably, audio features, e.g., MFCC or a pre-trained audio HuBERT model, correlate with phones better than vanilla image features (e.g., HoG) do. To this end, we train an audio encoder based on the aligned audio frame sequence A 1:T in parallel to the visual encoder. The iterative training alternates between the two encoders. In each iteration, an audio encoder E a is utilized to generate target cluster assignments z a 1:T . The visual encoder E v is trained subsequently with (I 1:T , z a 1:T ). The z a 1:T is also used to train the next iteration of the audio encoder E a for refinement. The cross-modality visual HuBERT can be seen as modeling visual inputs by distilling knowledge from the audio stream, where z a 1:T represents the audio-side knowledge. We hypothesize that the audio feature is more favorable to speech representation learning than the visual feature, which is validated in the Section E.1. Critical for the lip-reading downstream task, the masked prediction objective used by HuBERT forces the model to capture temporal relationships, which facilitates prediction of homophemes, which are groups of sounds with identical visual shapes (e.g., 'p'-'b', 'f'-'v', 'sh'-'ch') that are impossible to distinguish using a single image frame.  Figure 1: Illustration of AV-HuBERT. Masked prediction losses are only computed for the three middle frames, because at least one modality is masked for those frames. See section A for its comparison between single-modal and cross-modal visual HuBERT.
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
Our primary model in this work is Audio-Visual HuBERT (AV-HuBERT), shown in figure 1, which is trained iteratively by alternating between feature clustering and masked prediction in a similar way to the Visual HuBERT but with four main improvements:
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
Audio-visual input: The AV-HuBERT model consumes both acoustic and image frames for the masked prediction training, which enables better modeling and distillation of the correlations between the two modalities. Specifically, image sequences and acoustic features pass through their light-weight modality-specific encoders to produce intermediate features, which are then fused and fed into a shared backbone transformer encoder to predict masked cluster assignments. The targets are generated from clustering audio features or features extracted from the previous iteration of the AV-HuBERT model. When fine-tuned for lip-reading, we drop the audio input to work solely with the visual input. The input discrepancy is addressed by modality dropout described next.
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
Modality dropout: Audio-visual speech recognition models can relate audio input to lexical output more effortlessly than the visual input stream, as observed in the literature (Afouras et al., 2018a;Ma et al., 2021b). This causes the audio modality to dominate model decisions. The problem is aggravated in our setting because the target cluster assignments are initially generated from acoustic features. To prevent the model's over-reliance on the audio stream in our joint model, we only use a linear layer to encode acoustic input to force the audio encoder to learn simple features.
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
Additionally, before fusing audio and visual inputs into the backbone transformer encoder, dropout is applied to mask the full features of one modality; we refer to it as modality dropout. With a probability p m , both modalities are used as input. When only one modality is used, the audio stream is selected with a probability of p a . Formally, given the encoded audio and visual feature sequence f a 1:T and f v 1:T , equation 2 shows feature fusion equipped with modality dropout:
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
where concat denotes channel-wise concatenation. Note that modality drop out is applied at the sequence level instead of at the frame-level, which effectively tasks AV-HuBERT to perform masked prediction with visual-only, audio-only, or audio-visual input. Modality dropout prevents the model from ignoring video input and encourages the model to produce the prediction regardless of what modalities are used as input. Furthermore, since the fine-tuning and inference phases use the visual stream alone (no audio input), this modality dropout mechanism bridges the gap between pretraining (multimodal) and fine-tuning/inference (single-modality). A similar dropout mechanism is used in prior work (Zhang et al., 2019a;Makino et al., 2019;Neverova et al., 2014;Abdelaziz et al., 2020) to increase the robustness in multi-modal settings. We verify the modality dropout effectiveness in Section D.
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
Audio-visual clustering: One benefit of pre-training on both modalities is the ability to generate multimodal cluster assignments that serve as target labels for the masked prediction task of the next iteration. In contrast to the Cross-modal Visual HuBERT where targets are generated from audio-based features or a prior Audio HuBERT model, the targets for AV-HuBERT are naturally multimodal after the first iteration. Lip movement sequences provide complementary information to the audio stream. Combining both modalities produces cluster assignments of higher quality for AV-HuBERT, as shown in Section E.1.
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
We propose a novel masking strategy for AV-HuBERT that masks segments in the visual stream by substituting them with random segments from the same video. More formally, given an input video I v 1:T , an imposter video I v,f 1:T f and a mask consisting of n intervals M = {(s i , t i )} 1≤i≤n , we corrupted I v 1:T intoĨ v 1:T by setting:
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
where p i is a sampled integer offset from the interval [0, T f − t i + s i ]. Now, to solve the task, the model needs to first identify the fake frames and then infer the labels belonging to the original frames. Since the "filled-in" segments are from real video segments and temporally smooth, the fake segment detection sub-task becomes less trivial compared to when using vanilla masking or substitution with non-consecutive frames. We show an ablation confirming the advantage of the proposed masking strategy in Section D.
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
The audio and visual segments are masked independently using two different masking probabilities m a and m v . We hypothesize that the difficulty of the masked prediction task differs for each modality: inferring the masked targets given the audio stream is more straightforward than using the lip movement stream. Setting a high masking probability for acoustic frames is essential to help the whole model capture the language characteristics. On the contrary, setting a high masking probability for the visual input provide the model with more imposter segments than the original ones, hurting its ability to learn meaningful features (studied in Section D of the appendix). Given the output probability p 1:T and target cluster assignments z 1:T , the AV-HuBERT pre-training loss is:
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
where M a and M v denotes the frames that are masked for the audio and the visual stream. α is a hyperparameter weighting the contribution of the unmasked regions in the overall objective.
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
The proposed pre-training approaches can be fine-tuned for visual speech recognition using any sequence classification loss. In this paper, we focus on fine-tuning with the connectionist temporal classification (CTC) (Graves et al., 2006) and attention-based sequence-to-sequence cross-entropy loss (Bahdanau et al., 2016) (S2S for brevity), which are the most popular choices. Assume that the feature sequence output of our pre-trained model is e 1:T and the ground-truth transcription is w = w 1 , w 2 , ..., w s . For CTC, a projection layer is used to map the visual feature sequence into the output probabilities:
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
and U is the output vocabulary size (+1: plus blank symbol). The model is trained with CTC loss:
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
where B maps an alignment sequence π to w. For S2S, a tranformer decoder is appended to the pre-trained encoder to autoregressively decode the feature sequence e 1:T into target probabilities p(w t |w 1:t−1 , e 1:T ). The whole model is trained with cross entropy loss: L s2s = − s t=1 log p(w t |w 1:t−1 , e 1:T ).
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
We conduct experiments on two datasets: LRS3 (Afouras et al., 2018b) with 433 hours of transcribed English videos and VoxCeleb2 (Chung et al., 2018) with 2442 hours of unlabeled multilingual videos. We only use the English portion of VoxCeleb2, which amounts to 1,326 hours of content. The inputs to our backbone model are lip Regions-Of-Interest (ROIs) for the visual stream and log filterbank energy feature for the audio stream. The image encoder is a modified ResNet-18, which has been used in prior work (Ma et al., 2021b;Martinez et al., 2020;Stafylakis & Tzimiropoulos, 2017). The audio encoder is simply a linear projection layer. We consider two model configurations: BASE with 12 transformer blocks and LARGE with 24 transformer blocks. For BASE and LARGE, the embedding dimension/feed-forward dimension/attention heads in each transformer block are 768/3072/12 and 1024/4096/16 respectively. The number of parameters in BASE and LARGE are 103M and 325M respectively. α in equation 4 is set to 0.
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
The model uses five iterations of feature clustering and masked prediction during pre-training. See Section B.4, E for details on clustering. For fine-tuning, we use phone targets for the CTC loss and unigram-based subword units (Kudo, 2018) for the S2S loss. For decoding the CTC-trained model, we use a 4-gram language model trained on LRS3 training text. For the S2S fine-tuned model, we rely only on its own decoder module to incorporate language information, with no external language model employed during inference. To further show the complementary relationship between AV-HuBERT and existing approaches of using unlabeled data, we also experiment on combining AV-HuBERT with self-training. Specifically, we generate pseudo labels for unlabeled data using a finetuned HuBERT, and fine-tune the pre-trained AV-HuBERT model with the combination of pseudolabeled videos and original labeled videos. Note that no additional data is used when combined with self-training. More details about the used datasets, data pre-processing, and model training are in Section B. Table 1 compares the performance of our AV-HuBERT pre-training approach to previously published supervised, semi-supervised, and self-supervised lip-reading systems using different amounts of labeled and unlabeled data. Since the CTC and S2S fine-tuning approaches have similar trends, only S2S results are shown in table 1. Complete results of CTC fine-tuning are in Table C shows that AV-HuBERT and self-training are complementary to each other. Note the overall gain is attributed mainly to AV-HuBERT as self-training alone leads to much worse performance (> 50% WER). More details can be found in section C.3. Many prior models pre-train their visual front-end, e.g., ResNet-18, using word-level annotated lip-reading videos, which is costly to collect since they require word boundary information. In contrast to these models, our models are fully pre-trained from scratch using the proposed approach.
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
Compared to the semi-supervised approach Jasper-KD , which transcribed 334 hours of the English data in VoxCeleb2 using a pre-trained ASR system, 2 our best model achieves 29% lower absolute WER benefiting from VoxCeleb2 for pre-training. Even when limiting our model to the LRS3 data for pre-training and fine-tuning, our model surpasses their semi-supervised system by 18%. Compared to LiRA (Ma et al., 2021a), a recently proposed self-supervised model for lip-reading, AV-HuBERT-BASE provides 17.5% lower absolute WER on average for low-resource and high-resource settings. The implementation details of LiRA are provided in Section B.5.
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
With the same network architecture, our pre-training approach significantly reduces WER compared to training from scratch, in both low-resource (92.3% → 32.5%, LARGE) and high-resource (62.3% → 28.6%, LARGE) settings. A qualitative view of the improvement can be found in Section F. Additionally, we notice that our AV-HuBERT pre-training helps under a fully supervised setting. Using the LRS3 data only (433 hours), pre-training followed by fine-tuning (41.6%, LARGE) outperforms training a lip-reading model from scratch (62.3%, LARGE) to predict the output text. AV-HuBERT, pre-trained on video-audio pairs, learns better fine-grained visual representation than the scratch model trained on video-text pairs. The benefits of AV-HuBERT pre-training in various labeled data setups can be found in Section C.1.
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
We compare AV-HuBERT against a suite of alternatives, including the Single-modal and Crossmodal Visual HuBERT in Table 2. All the models are BASE pretrained on 433 hours of unlabeled data and fine-tuned on 30 hours of labeled data. For this comparison, we use CTC fine-tuning due to its computational efficiency and given their similarity in results trends to S2S. 
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
Iter-2
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
As shown in table 2, target cluster assignments driven from a single modality, either audio or visual, do not provide much WER reduction beyond the second iteration. Training the AV-HuBERT model using targets driven from audio-visual features keeps improving for more iterations and achieves better final performance. As mentioned in Section 3, visual information is complementary to audio input; hence, using both produces higher quality target clusters. Measuring the quality of different target labels using cluster quality metrics such as purity and NMI shows the same trend observed from lip-reading WER (See appendix E.1).
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
Fixing target labels used for the masked-prediction pre-training, AV-HuBERT (AV/MFCC→A) outperforms the Cross-modal Visual HuBERT (V/MFCC→A) by a large margin. AV-HuBERT effectively transfers knowledge from audio into the visual encoder and the backbone transformer model to benefit visual-only fine-tuning and inference. In contrast to AV-HuBERT, iterative pre-training brings much smaller gains to single-modality visual HuBERT ("V/MFCC→V", "V/HoG→V").
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
Starting with audio features is critical for learning effective target labels for the masked-prediction task. Phonetic information, which is crucial for lip-reading, is primarily present in the audio stream. All the models considered so far are based on audio feature MFCC clustering in their initial iteration. As is shown in the "V/HoG→V" row, using hand-engineered visual features provides a lousy starting point for iterative learning. Visual features such as HoG mainly incorporate low-level visual cues such as edges and luminance, which is irrelevant to the downstream recognition task. 
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
Since the correlation between lip movements and the produced sound is governed by the vocal apparatus that is language-agnostic, the proposed AV-HuBERT can utilize multi-lingual data for such learning. This would be particularly beneficial for low-resource languages. Nevertheless, the masked language modeling aspect in AV-HuBERT is still language-dependent, implying that mixing other languages would increase the language domain discrepancy. To study how these two factors affect AV-HuBERT, we compare using monolingual English data only versus multilingual videos in the pre-training phase. In particular, we vary the amount of English data we use in pre-training while the number of non-English utterances is fixed to 1,116 hours in all the settings. For simplicity, we train AV-HuBERT (BASE) for one iteration with MFCC clusters and fine-tune it with CTC using 30 hours of labeled data. As is shown in table 3, using non-English data in pre-training significantly reduces WER when there are no or very little English data in pre-training (≤ 100 hours). As we increase the amount of English data, the gain diminishes because the out-of-domain effect brought by non-English data outweighs the benefit of the overall increase in pre-training data. Using the whole English data only for pre-training is better than combining it with other languages (59.9% vs. 64.3%). Training with 5 iterations leads to similar results (47.3% vs. 48.9%). This experiment highlights the importance of the domain match between pre-training and fine-tuning data. For zero/low-resource settings, merging data from other languages in pre-training benefits the downstream task. When the unlabeled data from the target language is abundant, limiting the pre-training data to one language is beneficial.
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
The multimodal clusters produced by AV-HuBERT, which have higher quality than audio-HuBERT targets, can also benefit speech pre-training. To test our hypothesis, we trained an audio-HuBERT, with only audio input during the masked-prediction pre-training, for one iteration with cluster assignments driven from AV-HuBERT features. We also pre-trained an audio-HuBERT from scratch using clusters driven from the MFCC features for three iterations. The two pre-trained models are evaluated on a downstream ASR task. Table 4 shows the performance of different models fine-tuned on the ASR task. We only include the performance of S2S fine-tuning for our models as it consistently outperforms the CTC fine-tuning. An audio-HuBERT pre-trained using targets generated by AV-HuBERT features outperforms the vanilla audio-HuBERT in low-resource (30h) and high-resource settings (433h) fine-tuning settings across different model architectures. With the same amount of labeled data, our best model (1.4%) outperforms the prior SOTA (2.3%) even without an external language model during inference.
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
Given that the AV-HuBERT model utilizes both modalities at its input, it can be fine-tuned, in principle, for the ASR downstream task. In practice, we notice pre-training an audio-HuBERT with audio-visual cluster leads to better ASR performance (3.8%) than a pre-trained AV-HuBERT (4.6%), potentially due to its hyperparameters being selected based on lip reading rather than ASR. In fact, audio-HuBERT can be treated as a special case of AV-HuBERT with p m = 0, p a = 1. 
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
We presented multiple pre-training models for visual speech recognition. Our AV-HuBERT model leverages the strong correlation between the audio and lip movement streams for self-supervised audio-visual speech representation learning. Our pre-training approaches iteratively alternate between feature clustering and learning new features through a masked-prediction loss. The AV-HuBERT model consumes masked image and audio frames to predict target cluster assignments. The targets are initially generated from MFCC features and gradually refined through iterative training. Experiments on visual speech recognition show that AV-HuBERT achieves SOTA using 433 hours of text transcriptions, two orders of magnitude less than the 31,000 hours of labeled data used in the prior best approach. When using only one-thousandth of labeled data, the lip-reading performance outperforms the prior SOTA by more than 10% (relative). AV-HuBERT also improves the representation for the ASR downstream task. An audio-HuBERT model trained with targets generated by an AV-HuBERT model shows superior performance, achieving the SOTA in the audio-based speech recognition in the LRS3 dataset. As future work, AV-HuBERT can be applied for multilingual lip-reading in low-resource languages. Additionally, our approach can be extended to other applications of visual speech representation, such as speech enhancement and generation.
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
All the data used in this paper are publicly available and are used under the following three licenses: the TED terms of use, the Creative Commons BY-NC-ND 4.0 license and Creative Commons Attribution 4.0 International License. Through spot-checking, we find the datasets are gender balanced and cover a wide range of races and ages. However, the distribution of speakers in the data may not be representative of the global human population. Please be cautious of unintended societal, gender, racial and other biases caused by the fact. To maintain anonymity, only the mouth area of a speaker is visualized wherever used in the paper. The proposed method can be applied in several areas including security and crime investigations. However it can also be used for malicious purposes such as surveillance and wiretapping. We are committed to distributing our code and model carefully, with special attention to any potential security and privacy concerns.
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
Our code and models are publicly available. In the meantime, we include as many implementation details as we can in the paper.  YouTube. In the original dataset, the training data is split into two partitions: pretrain (403 hours) and trainval (30 hours). Both parts are transcribed at the sentence level and come from the same source as the test set. The pretrain set differs from trainval in that the duration of its video clips are of a much wider range and can be shorter or longer than a full sentence. In the low-resource setup, we only use trainval as the labeled data. As no official development set is provided, we randomly select 1,200 sequences from trainval as the validation set (about 1 hour) for early stopping and hyperparameter tuning.
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
VoxCeleb2 (Chung et al., 2018) is originally created for multilingual audio-visual speaker recognition and it contains over 2,442 hours of utterances of over 6,000 speakers extracted from YouTube videos. The dataset naturally serves our purpose as it does not contain ground-truth transcriptions. The VoxCeleb2 has substantial domain discrepency to the LRS3 data as its utterances are from multiple languages and includes videos in a larger variety of domains including interviews, talks, excerpts under indoor and outdoor environments. In VoxCeleb2, by default we only use the English portion for pre-training. As no ground-truth language label is given in the VoxCeleb2, we use a simple heuristic to choose the English samples. Specifically, we use an off-the-shelf character-based ASR model (Hsu et al., 2021a) trained on Librispeech which achieves 1.9%/3.5% WER on clean/other test set. We run greedy decoding on the VoxCeleb2 and use the proportion of valid English words to determine if a target utterance is English or not. An utterance is only selected if the proportion of valid English words is higher than 60%. The total amount of unlabeled data after filtering is 1,326 hours.
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
For each video clip, we detect the 68 facial keypoints using dlib (King, 2009) and align each frame to a reference face frame via affine transformation. We crop a 96×96 region-of-interest (ROI) centered on the mouth. Each image frame is converted to grayscale. We randomly cropped 88 × 88 from the whole ROI and randomly flipped it horizontally with probablity of 0.5 during training. At test time, 88 × 88 ROI is center cropped and does not go through horizontal flipping. The preprocessing steps remain same as prior works in lip reading (Martinez et al., 2020;Ma et al., 2021b). For the associated audio, we extract the 26-dimensional log filterbank energy feature at a stride of 10 ms from the raw waveform and use it as input to the model. As the image frames are sampled at 25Hz, we stack the 4 neighboring acoustic frames to synchronize the two modalities.
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
In the modified ResNet-18 (Ma et al., 2021b;Martinez et al., 2020;Stafylakis & Tzimiropoulos, 2017), the first convolutional layer is substituted by a 3D convolutional layer with kernel size 5 × 7 × 7. The visual feature tensor is flattened into a single-dimensional vector through a 2D average pooling layer in the end. We use one linear projection layer as the audio encoding module. The acoustic features are normalized by per-frame statistics before feeding into the network (Ba et al., 2016). We use a dropout of p = 0.1 after the self-attention block within each transformer layer, and each transformer layer is dropped (Fan et al., 2020) at a rate of 0.1.
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
Pretraining Our models are implemented with fairseq (Ott et al., 2019). The whole network is randomly initialized before pre-training. In pre-training, the model is trained for five iterations in total. For the initial iteration, we generate the targets by running k-means clustering algorithm on 39-dimensional MFCC feature (13 coefficients with its first-and second-order derivatives) extracted from raw audio waveform. For the subsequent iterations, the feature from an intermediate layer of the model in the previous iteration is used for clustering. The layer index (one-based) used for clustering in iteration 1-4 are {9, 12, 12, 12}. The number of features are clustered to {100, 100, 500, 1000, 2000} respectively for the 5 iterations. See section E.3 for analysis. To save training time, we always use the BASE model to generate clusters and LARGE model is only trained in the 5th iteration.
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
We set both p m and p a to 0.5 for modality dropout at training time. To extract features for clustering, both modalities are used. We adopt the strategy used in wav2vec 2.0 (Baevski et al., 2020) to generate masks, where p% of all frames are randomly selected as start and subsequent l frames are masked. In iteration 1-4, we mask the fused features and set p/l to be 8/10 respectively as we observe such practice generates higher quality cluster assignments (see section E.2). In the last iteration, we set p/l to be 6/5 for video and 8/10 for audio (see section D).
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
We train the model with Adam (Kingma & Ba, 2015), warming up the learning rate for the first 8% of updates to a peak of 0.002 and then linearly decay it. Videos are batched together to not exceed 1,000 image frames (40 seconds) per GPU. Both BASE and LARGE models are updated for 400K and 600K steps at each iteration, respectively in 433h/1759h unlabeled settings. We train on 32 and 64 V100-GPUs for BASE and LARGE. On average, each iteration takes ∼ 2.0/3.0 days for BASE and ∼ 2.4/3.6 days for LARGE in using 433h/1759h unlabeled data for pre-training.
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
Fine-tuning After pre-training, we fine-tune the AV-HuBERT model on labeled (video, text) pairs. The audio encoder is removed and its output is replaced by a zero-vector. In CTC fine-tuning, a randomly initialized projection layer is added on top of the transformer to map features into phonemes. The lexicon is constructed with CMUDict (cmu). In fine-tuning with S2S, we use a 6-layer/9-layer transformer decoder on BASE and LARGE model to decode features into unigram-based subword units (Kudo, 2018). The vocabulary size in CTC and S2S are 46 and 1000 respectively.
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
In CTC, the pre-trained model is updated from the initial iteration without any freezing. The model is fine-tuned for 30K/100K steps respectively in 30h/433h setting. In S2S, the pre-trained model (i.e., encoder) is frozen for the first N % updates . N is 100 and 50 for 30h and 433h labeled setting respectively. The entire model is trained for 18K/45K steps in the 30h/433h setting. Both models are trained with Adam, with the learning rate being warmed up for the first P % of updates to a peak of 0.001 and linearly decayed. P is tuned among {10, 30, 50}. All hyperparamters are tuned on the validation set.
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
Decoding For CTC, we use a 4-gram language model trained on text data in the LRS3 training set. The perplexity of the 4-gram LM on test set is 110.5. No language model is used for S2S decoding. For CTC, we tune the beam width among {5, 10, 20, 50, 100, 150}, the language model weight among {0, 1, 2, 4, 8} and word insertion penalty among {±4, ±2, ±1, 0}. For S2S, The beam width and length penalty are tuned among {5, 10, 20, 50} and {0, ±1}. The tuning is done on the validation set.
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
We apply self-training on LARGE AV-HuBERT. Specifically, the fine-tuned LARGE HuBERT model(A/MFCC→AV, Table 4) is used to assign pseudo-labels to the unlabeled audiovisual data. In 30h/433h setting, the amount of data for fine-tuning A/MFCC→AV are 30h and 433h respectively. The pre-trained AV-HuBERT LARGE is fine-tuned with the pseudo-labeled videos and videos with ground-truth text labels (30h/433h). Note the data used here is exactly same with the case of using AV-HuBERT only.
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
We re-implemented the LiRA (Ma et al., 2021a) training objective in our framework, as there does not exist publicly available implementations and we aim to focus the comparison on the pre-training objective rather than the architectural difference. We use the same backbone architecture as the BASE AV-Hubert except the output layer being a linear project layer with an output dimension of 256. The 256-dimensional frame PASE+ feature is extracted from the audio with its official implementation in (Ravanelli et al., 2020). The original PASE+ feature is downsampled to 25Hz for synchronization with the visual stream. The pre-trained model is fine-tuned in both CTC and S2S. The optimizer and learning rate schedule in pre-training, hyperparameter search in fine-tuning and decoding remain the same as AV-HUBERT. Note with our implementation, the WER is reduced by 23% (94.3% → 71.9%) using LiRA when the percentage of labeled data is 6.9% (30h labeled, 433h in total), while Ma et al. (2021a) achieves ∼ 10% improvement in a similar setting.
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
C.1 AMOUNT OF LABELED DATA Table C.1 shows the effect of pre-training on different amount of labeled data for fine-tuning. We use 433 hours of unlabeled data (LRS3 only) and randomly selected 1, 10 and 100 hours of labeled data for fine-tuning. Overall pre-training brings large and consistent gains across different amount of labeled data. Specifically CTC-based fine-tuning outperforms S2S-based fine-tuning in lowresource settings (1-hour and 10-hour). The larger number of parameters as well as the lack of a language model for decoding makes S2S model more likely to overfit especially when the amount of fine-tuning data is small.   Table C.3 shows the performance of only applying self-training. The WER of a self-training only model is significantly higher than AV-HuBERT and self-trained AV-HuBERT, which suggests that the gain of the combined approach is primarily from AV-HuBERT.
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
C.4 FULL RESULTS WITH CTC FINE-TUNING Table C.4 shows the full results on LRS3, which includes the CTC fine-tuning performance for all the models we implemented. In general, the conclusions we draw from S2S (e.g., the benefits of our pre-training approach in different settings, the improvement over LiRA) in section 4.2 holds for CTC as well. 
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
The ablation studies in this section are done in the last iteration of the AV-HuBERT, pre-trained with 433 hours of unlabeled data. The model is fine-tuned with 30 hours of labeled data using CTC. Masking Strategy In the first part of table D.1, we compare the proposed masking strategy against several alternatives. Feature masking applies span mask at feature level and leads to the worst performance, which is due to the leakage of information to ResNet. Directly masking image sequence with random Gaussian noise or a learned embedding slightly improves the performance by preventing the prior issue. However, those artificial frames also corrupts the raw image sequence and enlarges the domain gap in videos between pre-training and fine-tuning. On the other hand, our proposed method achieves better performance. Specifically, using segments from the same utterance ("Sub, (same, seg)") as the imposter leads to the best result compared to sampling from a different utterance ("Sub, (diff, seg)") or sampling non-consecutive frames from the same utterance ("Sub, (same, frm)"). 3 The filled-in fake images are visually similar to the raw images and substitution with a segment keeps the temporal smoothness making the replaced segment more realistic, which enforces the ResNet to encode more fine-grained visual details.
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
Masking probability We set two different masking probabilities for audio and visual stream in the last iteration. The probabilities of an acoustic frame and image frame being masked are 0.8 and 0.3 respectively. As is shown in the second part of table D.1, setting masks for audio and visual stream independently is essential because the optimal masking probability for audio and visual stream are different. Audio encoder tends to deteriorate into a simple acoustic feature extractor when mask length is small. On the other hand, long visual mask will lead to the model lacking context to distinguish between fake and real images.
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
Modality dropout The third part of table D.1 compares the model performance with and without modality dropout. Randomly dropping audio sequence prevents the model from over-relying on audio for masked prediction and helps the visual representation learning.
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
Where to compute prediction loss In the last part of table D.1, we compare the choice of masked prediction vs. prediction. The loss weight on unmasked region does not have a large impact on the fine-tuning performance. This is different from the findings in Audio HuBERT Hsu et al. (2021a), where masked prediction leads to much better performance. Given image frames as input, the prediction of cluster assignments, which are mostly determined by the accompanied audio stream, helps encode phonetic information into the visual representation. The task is much less trivial than a single-modal model (audio-HuBERT), where setting non-zero weight on unmasked prediction can easily make the model deteriorate into an acoustic feature extractor. In addition, the high quality of targets in the last iteration also makes such prediction more helpful.
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
For analysis, we use frame-level phonetic labels as the ground-truth and match its correlation between cluster assignments. The phonetic labels are obtained via forced alignement from a monophone based HMM-GMM ASR model trained on LRS3. In particular, we use clustering purity and Normalize Mutual Information (NMI) as the evaluation metrics. Table E.1 shows that (1) The quality of cluster assignments is consistent with fine-tuning performance across different models (2) Hand-engineered audio feature (MFCC, NMI: 21.5%) has much stronger correlation with phonetic labels than the visual feature (HoG, NMI: 1.6%) (3) Audio-visual clusters (NMI: 44.2%) are of better quality than pure audio-based clusters (NMI: 39.7%). (4) In single-modality visual Hubert (V/HoG→V), feature quality is improved negligibly through iteration training. 
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
In iteration 1-4, we apply the mask in the fused feature. We observe such practice generates targets of higher quality, thus helping future iterations more. Table E.2 shows a comparison between such two different masking strategies. Input-level masking enhances the learning of visual representation (see table D.1) while produces worse audio-visual feature (NMI: 27.2%). In contrast, the two streams of original input are better aligned in feature-level masking which is consistent with the cluster generation process, thus leading to better audio-visual clusters (NMI: 37.7%).  The cluster assignment quality generally improves with more iterations. In the first iteration, features in the middle layers show higher quality than the other layers. The target for the first iteration (MFCC clusters) is of worse quality, thus later layers that are more correlated with targets do not yield the best cluster. Target quality improves with more training iterations, thus the best feature layer shifts towards the end. Setting a larger number of clusters increases the clustering quality as can be seen from the comparison between "varied clusters" and "2K clusters". In terms of the 12th layer which we choose, the highest NMI (44.2%) is achieved in the last iteration. In addition, more iterations of training improves the overall quality of clusters produced by a model though the highest NMI/purity among all layers does not necessarily increase in later iterations. Therefore, setting a larger number of iterations brings stable gains which are more robust to the index of layer chosen for clustering. It is important as the purity/NMI, whose measurement rely on a supervised model, are not used for hyperparameter search in practice. Our self-supervised model is the LARGE AV-HUBERT pre-trained with 1,759 hours unlabeled data and fine-tuned with 433 hours labeled data. The baseline model is the supervised baseline trained with 433 hours labeled data. Both models use the S2S criterion for supervised training and have the same number of parameters. Qualitatively, our approach provides transcriptions with much higher quality. The baseline approach confuses among words of similar sound while our model output more semantically sound sentences. Figure F.2 also shows typical errors made by our model. We noticed many errors are on short sentences. This is mainly because lip reading relies heavily on the context for recognition due to the existence of homophomes. Thus the error rates in lip reading are notably higher in short utterances, which differs from ASR, as can be seen from figure F.1. Substitution among words with homophemes ('fiction' vs. 'vision' in a.4, 'part' vs. 'bunk' in b.5) is another source of error made by the model. 