What are existing approaches for unsupervised speech recognition?
Recent work show the viability of unsupervised
speech recognition. W2V2-U [2] accomplished this
by running Principal Component Analysis (PCA),
k-means clustering, and mean pooling to convert
W2V2 [3] features into phoneme-granularity fea-
tures, then trains a GAN model to output phoneme
text from the post-processed model [2]. The state-
of-the-art method for phoneme-level unsupervised
ASR is W2V2-U2.0 [23] which directly trains a
CNN to output phonemes from W2V2 features
and uses a reconstruction loss to tie the input
audio with corresponding generated text. Both
methods use WFSTs to decode the phonemes into
raw text. While there have been preliminary at-
tempts [11, 27] to use W2V2-U2.0 with phoneme
language models1, we are the first to combine it
with semantically-rich subword-based LLMs
On Compressing Sequences for Self-Supervised Speech Models
Compressing self-supervised models has become increasingly necessary, as self-supervised models become larger. While previous approaches have primarily focused on compressing the model size, shortening sequences is also effective in reducing the computational cost. In this work, we study fixed-length and variable-length subsampling along the time axis in self-supervised learning. We explore how individual downstream tasks are sensitive to input frame rates. Subsampling while training self-supervised models not only improves the overall performance on downstream tasks under certain frame rates, but also brings significant speed-up in inference. Variable-length subsampling performs particularly well under low frame rates. In addition, if we have access to phonetic boundaries, we find no degradation in performance for an average frame rate as low as 10 Hz.
[{"first":"Yen","last":"Meng","middle":[]},{"first":"Hsuan-Jui","last":"Chen","middle":[]},{"first":"Jiatong","last":"Shi","middle":[]},{"first":"Shinji","last":"Watanabe","middle":[]},{"first":"Paola","last":"Garcia","middle":[]},{"first":"Hung-yi","last":"Lee","middle":[]},{"first":"Hao","last":"Tang","middle":[]}]
Unsupervised Speech Recognition
Despite rapid progress in the recent past, current speech recognition systems still require labeled training data which limits this technology to a small fraction of the languages spoken around the globe. This paper describes wav2vec-U, short for wav2vec Unsupervised, a method to train speech recognition models without any labeled data. We leverage self-supervised speech representations to segment unlabeled audio and learn a mapping from these representations to phonemes via adversarial training. The right representations are key to the success of our method. Compared to the best previous unsupervised work, wav2vec-U reduces the phoneme error rate on the TIMIT benchmark from 26.1 to 11.3. On the larger English Librispeech benchmark, wav2vec-U achieves a word error rate of 5.9 on test-other, rivaling some of the best published systems trained on 960 hours of labeled data from only two years ago. We also experiment on nine other languages, including low-resource languages such as Kyrgyz, Swahili and Tatar.
[{"first":"Alexei","last":"Baevski","middle":[]},{"first":"Wei-Ning","last":"Hsu","middle":[]},{"first":"Alexis","last":"Conneau","middle":[]},{"first":"Michael","last":"Auli","middle":[]}]
SUPERB @ SLT 2022: Challenge on Generalization and Efficiency of Self-Supervised Speech Representation Learning
We present the SUPERB challenge at SLT 2022, which aims at learning self-supervised speech representation for better performance, generalization, and efficiency. The challenge builds upon the SUPERB benchmark and implements metrics to measure the computation requirements of self-supervised learning (SSL) representation and to evaluate its generalizability and performance across the diverse SUPERB tasks. The SUPERB benchmark provides comprehensive coverage of popular speech processing tasks, from speech and speaker recognition to audio generation and semantic understanding. As SSL has gained interest in the speech community and showed promising outcomes, we envision the challenge to uplevel the impact of SSL techniques by motivating more practical designs of techniques beyond task performance. We summarize the results of 14 submitted models in this paper. We also discuss the main findings from those submissions and the future directions of SSL research.
[{"first":"Tzu-hsun","last":"Feng","middle":[]},{"first":"Annie","last":"Dong","middle":[]},{"first":"Ching-Feng","last":"Yeh","middle":[]},{"first":"Shu-wen","last":"Yang","middle":[]},{"first":"Tzu-Quan","last":"Lin","middle":[]},{"first":"Jiatong","last":"Shi","middle":[]},{"first":"Kai-Wei","last":"Chang","middle":[]},{"first":"Zili","last":"Huang","middle":[]},{"first":"Haibin","last":"Wu","middle":[]},{"first":"Xuankai","last":"Chang","middle":[]},{"first":"Shinji","last":"Watanabe","middle":[]},{"first":"Abdelrahman","last":"Mohamed","middle":[]},{"first":"Shang-Wen","last":"Li","middle":[]},{"first":"Hung-yi","last":"Lee","middle":[]}]
Towards End-to-end Unsupervised Speech Recognition
Unsupervised speech recognition has shown great potential to make Automatic Speech Recognition (ASR) systems accessible to every language. However, existing methods still heavily rely on hand-crafted pre-processing. Similar to the trend of making supervised speech recognition end-to-end, we introduce wav2vec-U 2.0 which does away with all audio-side pre-processing and improves accuracy through better architecture. In addition, we introduce an auxiliary self-supervised objective that ties model predictions back to the input. Experiments show that wav2vec-U 2.0 improves unsupervised recognition results across different languages while being conceptually simpler.
[{"first":"Alexander","last":"Liu","middle":["H."]},{"first":"Wei-Ning","last":"Hsu","middle":[]},{"first":"Michael","last":"Auli","middle":[]},{"first":"Alexei","last":"Baevski","middle":[]}]
On Compressing Sequences for Self-Supervised Speech Models
Large-scale self-supervised speech models [1,2,3,4,5], have proven their generality, serving various downstream tasks, such as speech recognition, speaker recognition, and emotion recognition [6]. Given the success of large-scale models (Transformers with 12 layers or more) [1,2,3,4,5], it is natural to believe that large model capacity is necessary for the models to be general purpose. To what extent the large capacity is needed is an open question, but the high computation cost and memory usage pose a high technical barrier for the adoption, and more importantly, training of these models. In this work, we aim to tackle the high computational cost of self-supervised learning.
On Compressing Sequences for Self-Supervised Speech Models
To reduce the technical requirement of using these models, several studies have explored model compression techniques, such as weight pruning, knowledge distillation, and architectural modification [7,8,9]. However, in speech processing, the number of frames, along the time axis, is typically the dominating factor at runtime. For Transformers in particular, though quadratic memory consumption of self-attention * Equal contribution.
On Compressing Sequences for Self-Supervised Speech Models
is possible to avoid [10], most implementations still require quadratic memory and quadratic runtime. Sub-quadratic attention mechanisms are actively being developed [11,12,13], but the overhead typically makes these attention mechanisms less useful in practice [14]. Instead of reducing the runtime and memory complexity of self-attention, we study subsampling techniques to reduce the sequence length directly.
On Compressing Sequences for Self-Supervised Speech Models
Subsampling (and sometimes concatenating) contiguous frames is a common technique for speeding up training and inference of automatic speech recognizers [15,16,17]. This approach can be as simple as concatenating every two frames [17] or dropping every even frames [18,19]. Subsampling can also be achieved with pooling or convolution with a stride larger than 1 [20,21,22,23]. In this paper, we term this family of approaches fixed-length subsampling. Fixed-length subsampling has been explored in self-supervised learning [24,25,26]. Both Wu et al. [24] and Vyas et al. [25] use fixedlength subsampling, paired with upsampling, during the optimization of self-supervised losses. Lee et al. [26] is the most similar to our work in that it also uses fixed-length subsampling to reduce the time resolution paired with upsampling in a knowledge distillation setting. The subsampled representation is then upsampled when evaluated on downstream tasks. Though this approach achieves the desired computational speed-up, how subsampling affects the learned representation largely remains unclear and is the central focus of this paper. Another difference is that we directly use the subsampled representation in downstream, and explored a framework that works better without upsampling involved.
On Compressing Sequences for Self-Supervised Speech Models
Since phones in speech come in varying duration, it is widely accepted that self-supervised models enforcing piecewise constant constraints would learn better representations [27,28,29]. Piecewise constant constraints can have various forms, but they mostly involve finding boundaries of segments and pooling frame representations into segment representations. We term this approach variable-length subsampling. In this work, we study both the computational speedup and the impact on learned representations with variablelength subsampling. Our approach involves adding a layer of Continuous Integrate-and-Fire (CIF) [30] to learn subsampled representation while optimizing loss functions.
On Compressing Sequences for Self-Supervised Speech Models
To push the boundary of small, yet competitive selfsupervised models, we build our work on top of DistilHu-BERT [31], a model consisting of two Transformer layers, trained with knowledge distillation from HuBERT [3]. We evaluate our approach on phone recognition, automatic speech recognition (ASR) with word pieces of different sizes, keyword spotting, intent classification, speaker identification, and emotion recognition. We find that phone recognition and ASR are the tasks most impacted by the subsampled representation. Matching the frame rate of downstream tasks gives the best performance. Variable-length subsampling has particularly strong performance for the settings with a low frame rate. For analysis, we also show that given the phonetic segmentation, variable-length sampling can be as good as, if not better than, the DistilHuBERT baseline while having a frame rate as low as 10 Hz.
On Compressing Sequences for Self-Supervised Speech Models
The model we study in this paper is based on DistilHuBERT.
On Compressing Sequences for Self-Supervised Speech Models
For an input waveform s, there is a convolutional neural network (CNN) that converts the waveform s into a sequence of feature vectors x 1 , . . . , x T , or simply 
On Compressing Sequences for Self-Supervised Speech Models
where W is a trainable projection and the distance d can be cosine, dot product, Euclidean distance, or a combination of them. In DistilHuBERT, the distillation is applied to multiple layers of HuBERT via multitask learning. In particular, we minimize the sum of several loss terms L(u is the -th hidden layer of HuBERT and each loss has a learnable projection W ( ) (also called a prediction head).
On Compressing Sequences for Self-Supervised Speech Models
To subsample frames in time while maintaining the use of the loss from DistilHuBERT, we have two options, one with subsampling followed by upsampling, and another by subsampling the targets. Specifically, the first option minimizes
On Compressing Sequences for Self-Supervised Speech Models
by having upsampling after the output of the student, while the second option optimizes
On Compressing Sequences for Self-Supervised Speech Models
by subsampling the output of the teacher. Figure 1 shows the two options. For the rest of the sections, we will discuss the choices of subsampling and upsampling functions.
On Compressing Sequences for Self-Supervised Speech Models
For fixed-length subsampling, we adopt two commonly used approaches, convolution subsampling and average pooling with a stride larger than 1. We can control the frame rates by changing the strides. For convolution subsampling, the kernel size is set to the same as the stride size.
On Compressing Sequences for Self-Supervised Speech Models
For variable-length subsampling, we follow a two-step approach. A segmentation (a sequence of boundaries) is first proposed; vectors within each segment are pooled (for example with weighted averaging) and passed to the subsequent layers. We use Continuous Integrate-and-Fire (CIF) [30] to produce segmentation proposals. At a high level, for an input sequence of length T , the CIF module takes input from the previous layer and produces a sequence α 1 , α 2 , ..., α T of nonnegative numbers (using a combination of convolution, feedforward layer, and sigmoid function). Whenever t i=1 α i , i.e., the accumulation up to time t crosses an integer boundary, a segment boundary at time t is proposed (or fired). In other words, the sequence α 1:T controls both where and how many boundaries should be present. In fact, fixed-length subsampling can be seen as a special case with α t = 1/F for all t = 1, . . . , T where F can be set based on the desired subsampling rate.
On Compressing Sequences for Self-Supervised Speech Models
The CIF module can be trained end to end without any supervision. In practice, however, the segmentation seldom corresponds well with any actual boundaries in speech. Below we discuss several options of boundary guidance to help CIF learn more meaningful boundaries.
On Compressing Sequences for Self-Supervised Speech Models
Cardinality Guidance The cardinality guidance encourages the CIF module to produce the desired number of segments at training time; hence the name. Specifically, we add the term
On Compressing Sequences for Self-Supervised Speech Models
to our loss function, where K is the desired number of segments. The normalization T (absent in the original CIF formulation) is added to make sure the term is comparable across utterances of different lengths. Though the guidance is used at training time, at test time, we use the α's produced by the CIF module as is.
On Compressing Sequences for Self-Supervised Speech Models
Segmentation Guidance In some cases where we have access to phonetic boundaries, such as through forced alignments or other unsupervised approaches, the segmentation can be used as a source of supervision for the CIF module. Suppose a segmentation of K segments is provided as a sequence of boundaries in time indices t 1 , . . . , t K . To encourage the boundaries to be placed in accordance with the provided target, we introduce a segment-based loss The loss only focuses on where the boundaries are proposed.
On Compressing Sequences for Self-Supervised Speech Models
We also introduce a more stringent constraint, constructing a target
On Compressing Sequences for Self-Supervised Speech Models
to make sure the sequence α produced by the CIF module is close to the target α sup at every frame; hence a frame-based loss.
On Compressing Sequences for Self-Supervised Speech Models
Each of these losses can be added to our loss function with an interpolation factor. The segmentations for supervision are only used at training time, while at test time, we use the α's produced by the CIF module as is.
On Compressing Sequences for Self-Supervised Speech Models
Our experiment is based on the original DistilHuBERT implementation with S3PRL [6] and fairseq [32]. The pre-training setting and hyperparameters, including learning rate schedule, are the same as the original DistilHuBERT implementation except that we introduce the subsampling and upsampling modules. We have three prediction heads, targeting the representations of the 4 th , 8 th , and 12 th layers of a frozen Hu-BERT model. We use the 960-hour LibriSpeech dataset [33] for pre-training, and all models are trained for 200,000 updates with a batch size of 24.
On Compressing Sequences for Self-Supervised Speech Models
The CIF module for variable-length subsampling consists of a single one-dimensional 512-channel convolution with a stride of 1 and kernel width of 5, followed by a feedforward layer of 512 dimensions and an output dimension of 1.
On Compressing Sequences for Self-Supervised Speech Models
The experiment is evaluated on a subset of the SUPERB benchmark 1 , including phone recognition (PR), automatic 1 https://superbbenchmark.org speech recognition (ASR), keyword spotting (KS), intent classification (IC), speaker identification (SID), and emotion recognition (ER). We further evaluate on an additional task, automatic speech recognition with word pieces (ASR-5k).
On Compressing Sequences for Self-Supervised Speech Models
The ASR-5k model is trained with 5000 sentencepiece [34] targets (trained with byte-pair-encoding [35]). Both ASR and ASR-5k are evaluated without a language model (LM). To compare the performance with the DistilHuBERT paper, only the representation of the last layer (without the prediction heads) is used for downstream evaluation.
On Compressing Sequences for Self-Supervised Speech Models
Before we train our models with subsampling, we explore a range of frame rates for several tasks. We concatenate contiguous hidden vectors produced by the vanilla DistilHuBERT to achieve a target frame rate. This approach preserves the information sent to the downstream tasks while only changing the frame rate. We also explore averaging contiguous hidden vectors as an alternative. Since KS, IC, SID, and ER are utterance-level tasks, subsampling the hidden vectors does not affect the performance, so we only focus on PR, ASR, and ASR-5k.
On Compressing Sequences for Self-Supervised Speech Models
Results are shown in Table 1, where two subsampling methods are tested: concatenating (cat) and average pooling (avg). Phone recognition starts to fail for frame rates lower than 12.5 Hz. ASR with characters completely fails for frame rates lower than 25 Hz, while ASR with word pieces can sustain subsampling to a frame rate as low as 6.25 Hz. Figure 2 shows the average frame rates of different units, such as phones, characters, and word pieces. When we increase the number of word pieces learned, the size of the word pieces increases. In particular, when the number of word  pieces is large, many of the word pieces are actual words, and the average frame rate decreases accordingly. The results in Table 1 are consistent with the average frame rates in Figure 2, as these tasks rely on a CTC layer that can only produce labels as many as it has frames for. We also find that averaging is on par with concatenation, and decide to use averaging as the pooling method for the rest of the experiments.
On Compressing Sequences for Self-Supervised Speech Models
Unlike the previous section, here, we study models pretrained along with subsampling. Models trained with subsampling could potentially learn to represent the input speech differently from the vanilla DistilHuBERT.
On Compressing Sequences for Self-Supervised Speech Models
In this section, we explore the two options for fixed-length subsampling: subsampling paired with upsampling shown in Figure 1 (a), and subsampling of both the teacher's and the student's output shown in Figure 1 (b). For subsampling and upsampling pairs, we have convolution paired with deconvolution and averaging paired with repeating (duplicating frames to the desired frame rate).
On Compressing Sequences for Self-Supervised Speech Models
In Figure 3 (Left), we show the pretraining losses for different subsampling and upsampling pairs, and in Table 2 (I), we have their respective downstream performance. In Figure 3 (Left), we first note that convolution paired with deconvolution is able to recover the DistilHuBERT loss while averaging paired with repeating is worse. In Figure 3 (Middle), we also find that more aggressive sampling makes it more difficult to match the training loss of the vanilla DistilHu-BERT. However, in Table 2 (I), the downstream performance for averaging paired with repeating is on par with convolution paired with deconvolution for the two frame rates we explored.
On Compressing Sequences for Self-Supervised Speech Models
Due to this finding and the simplicity, we use averaging to conduct the experiments for subsampling both the teacher and the student. The training loss is shown in Figure 3 (Left) and the downstream performance is in Table 2 (II). We do find the training loss to be lower (perhaps due to having fewer frames), and the downstream performance is generally on par, if not better than, with upsampling. These results are also comparable to the ones in Table 1, including the failed case for phone recognition at the frame rate of 6.25 Hz.
On Compressing Sequences for Self-Supervised Speech Models
Based on the findings in the previous section, we decide to perform variable-length subsampling on both the teacher and the student. Recall that variable-length subsampling is achieved by producing a sequence α that decides where a boundary should be placed. Here, we choose a weighted sum of the vectors within a segment to produce a segment representation. In particular, suppose α t , . . . , α t are the α's in between two boundaries. The weights used for weighted averaging are , α t , . . . , α t − , where is the leftover weight that exceeds the integer from the previous boundary and is the leftover weight that exceeds the integer boundary and gets carried over to the next boundary. In order to match the subsampling exactly for both the teacher and the student, we decide to reuse the α produced by the student on the teacher, as indicated in Figure 1 (b).
On Compressing Sequences for Self-Supervised Speech Models
We first explore cardinality guidance, denote as card, where we set the number of segments to match the desired frame rate. For example, the number of segments K is set to T /4 if the desired frame rate is 12.5 Hz. We tune the weight of the added loss term L card and find that a weight of 0.5 works the best. The downstream performance is shown in the first two rows of Table 2 (III). The results are generally worse than fixed-length subsampling. We suspect for the cardinality guidance is too weak for constraining the model, so we turn to the stronger segmentation guidance.
On Compressing Sequences for Self-Supervised Speech Models
Below we explore two options, using smoothed HuBERT codes and unsupervised ASR, to obtain segmentation for guiding the CIF module.
On Compressing Sequences for Self-Supervised Speech Models
The amount of repeated HuBERT codes tend to correlate well with the duration of phones [36], so we can make use of the codes produced by the teacher as a proxy to segmentation. Here we run k-means on the output of HuBERT (layer 6, following [36]) to obtain the codes. We then use the dynamic programming algorithm proposed by Kamper and van Niekerk [37] to obtain the segmentation. Their algorithm generally smooths the frequent changes of a HuBERT code sequence, while also providing a hyperparameter λ penalizing short segments. In other words, larger λ produces larger segments, leading to a lower frame rate. There is no clear correspondence between λ and frame rate (except that they are positively correlated), so we simply sweep λ to find the desired frame rates. Recall that there are a frame-based approach and a segment-based approach to incorporate segmentation guidance. We tune the weight of the two additional loss terms, and find that 0.005 and 0.25 work best for L seg and L frame , respectively. Despite the weight of L seg being deceptively small, it is crucial to prevent CIF from producing degenerate solutions.
On Compressing Sequences for Self-Supervised Speech Models
Results of this approach are listed in the second block of Table 2 (III). Note that we tune λ to find a frame rate closest to the desirable one. For readability, we round the frame rates to the desirable ones in Table 2 (III). This approach performs significantly better than others for the 12.5 Hz frame rate, while matching the result of others for the 25 Hz frame rate. It shows the great potential of variable-length subsampling and calls for better segmentation.
On Compressing Sequences for Self-Supervised Speech Models
Another approach to obtain a segmentation of speech is through unsupervised ASR [38,39]. Our pre-trained unsupervised ASR model is a simplified version of wav2vec-U 2.0 [39] that is trained without the auxiliary k-means cluster loss. The model is trained on audio from the 100-hour LibriSpeech and texts from the LibriSpeech language modeling text. Once the model is trained, it simply acts as a frame classifier, producing posterior probabilities of phones at a frame rate of 16.7 Hz (60 ms frame period). The segmentation can be obtained by finding the maximum of each frame as predictions, and merging the same contiguous predictions. Since voice activity detection is involved during training, we overwrite segments where the silence is detected. Once the boundaries are extracted, we add frame-based and segment-based losses with the same weights as in Section 5.2.1.
On Compressing Sequences for Self-Supervised Speech Models
The downstream performance of using the segmentation of wav2vec-U 2.0 is in the last row of Table 2 (III). We also conduct a similar experiment with the smoothed HuBERT codes for comparison, matching the frame rate of the segmentation produced by the wav2vec-U 2.0 model. We find that this approach is particularly strong at phone recognition and ASR with word pieces, giving the best performance and frame rate tradeoff.
On Compressing Sequences for Self-Supervised Speech Models
The training loss of various variable-length subsampling is shown in Figure 3 (Right). Though the downstream performance of using wav2vec-U 2.0 segmentation provides a better tradeoff, the loss is slightly higher than the approach of smoothing HuBERT codes.
On Compressing Sequences for Self-Supervised Speech Models
Given the results from previous sections, an accurate segmentation could prove to be sufficient for an aggressive subsampling frame rate. As an analysis, we study forced alignments as a variable-length subsampling approach of its own. Forced alignments (of phones) are obtained from the Montreal Forced Aligner (MFA) [40]. Once the segmentation are obtained, we simply use average pooling to obtain a segment representation from the teacher and the student. We follow DistilHuBERT training without adding any other terms. For the downstream tasks, we also use forced alignments for subsampling. Note that the CIF module is not used in this setting, as segmentation is always provided.
On Compressing Sequences for Self-Supervised Speech Models
The result is shown in Table 2 (IV) and serves as our topline result. We do not report downstream performance for all tasks as transcripts are not available for computing forced alignments on those datasets. This approach achieves an average frame rate of 10 Hz (i.e, the average frame rate of phones), and has the best phone recognition and ASR results. The result suggests that there is still room for better segmentations.
On Compressing Sequences for Self-Supervised Speech Models
To measure the impact of subsampling on runtime, we report average multiply-accumulate operations (MACs) on a subset of LibriSpeech test-clean consisting of utterances between 1 and 20 seconds. MACs are measured in inference mode without counting prediction heads and upsampling. MACs of various subsampling approaches are reported in Table 2. The improvement in MACs is obscured by the MACs of the seven CNN layers, as the MACs of CNN dominate everything else. We therefore also report MACs without the CNN layers (denoted as MACs-C). The improvement is then clear and consistent with the reduction in frame rates. In summary, we study effect of subsampling in selfsupervised models, exploring different input frame rates for various downstream tasks. ASR with larger word pieces is more resistant to more aggressive subsampling, while ASR with characters can only sustain subsampling up to approximately 25 Hz. Simple averaging performs the best among all fixed-length subsampling options. Variable-length subsampling performs slightly behind fixed-length subsampling in high frame-rate settings but performs much better in low frame-rate settings. Segmentation can be used to guide variable-length subsampling, and heavily impacts the learned representations.
On Compressing Sequences for Self-Supervised Speech Models
For future work, the effect of subsampling on large self-supervised models, particularly, by training them with subsampling from random initialization, is largely unexplored. Aggressive subsampling in both supervised and selfsupervised settings are also still open. In fact, it is even possible to encode an utterance into a single vector [41], independent of the input length. Much research is needed in this direction.
Unsupervised Speech Recognition
Speech recognition performance on the much studied English Librispeech benchmark (Panayotov et al., 2015) has seen rapid improvement over the last few years due to advances in model architectures (Dong et al., 2018;, semisupervised learning (Xu et al., 2020b;Park et al., 2020) and self-supervised learning Chung et al., 2019b;Baevski et al., 2020c). However, all of these techniques require transcribed speech data which is not available for the vast majority of the nearly 7,000 languages of the world (Lewis et al., 2016). As a result, speech recognition technology is only available for about 125 different languages (Google, 2021). On the other hand, humans learn a lot about speech simply by listening to others around them and without explicit supervision (Werker and Tees, 1984;Hirsh-Pasek et al., 1987;Polka and Werker, 1994;Jusczyk et al., 1999;Johnson and Jusczyk, 2001).
Unsupervised Speech Recognition
Unsupervised learning has been very successful in machine translation resulting in systems that obtain remarkable accuracy given no labeled training data at all (Conneau et al., 2018;Artetxe et al., 2018). Inspired by this, there has been some work on unsupervised speech recognition based on learning to align unlabeled text and audio (Yeh et al., 2019) or adversarial learning (Liu et al., 2018;. These approaches showed promising initial results but their error rates are still high, with evaluation being limited to the small-scale and clean TIMIT benchmark.
Unsupervised Speech Recognition
In this work, we introduce a framework for unsupervised learning of speech recognition models. Wav2vec-U, or wav2vec Unsupervised, leverages self-supervised representations from wav2vec 2.0 (Baevski et al., 2020c) to embed the speech audio and to segment the Figure 1: Illustration of wav2vec Unsupervised: we learn self-supervised representations with wav2vec 2.0 on unlabeled speech audio (Step 1), then identify clusters in the representations with k-means (Step 2) to segment the audio data (Step 3). Next, we build segment representations by mean pooling the wav2vec 2.0 representations, performing PCA and a second mean pooling step between adjacent segments (Step 4). This is input to the generator which outputs a phoneme sequence (Step 5) that is fed to the discriminator, similar to phonemized unlabeled text (Step 6) to perform adversarial training (Step 7). audio into units with a simple k-means clustering method (see Figure 1 for an illustration of our approach). We find that the quality of the audio representations is key to the success of unsupervised speech recognition. Similar to Liu et al. (2018) and , we learn a mapping between segments and phonemes using adversarial training but different to their work, we also enable the algorithm to label segments as silences. We also introduce an unsupervised cross-validation metric to enable model development without labeled development data. Our unsupervised speech recognition model, the generator, is very lightweight: it consists of a single temporal convolution comprising only about 90k parameters to which we input frozen wav2vec 2.0 representations.
Unsupervised Speech Recognition
Experimental results demonstrate the viability of the framework for a variety of settings and languages. wav2vec-U improves the phone error rate (PER) on the small-scale TIMIT benchmark from 26.1 to 11.3 compared to the next best known unsupervised approach. To get a better sense of the performance compared to the best supervised methods, we measure performance on the larger Librispeech benchmark where our method achieves word error rate (WER) 5.9 on test-other. We also evaluate on six other European languages of the multilingual Librispsech benchmark  and on three non-European lowresource languages.
Unsupervised Speech Recognition
Automatic speech recognition (ASR) is the task of transcribing a speech waveform into a text transcript. In a supervised setup, a dataset D l = {(X i , Y i )} N i=1 of N speech X and text Y pairs are provided. Specifically, speech is usually represented as a sequence of feature vectors X = [x 1 , · · · x T ] ∈ (R m ) * , where each feature frame x t is an m-dimensional continuous vector such as Mel-frequency cepstral coefficients (MFCC). Text is usually represented as a sequence of discrete units Y = [y 1 , · · · , y L ] ∈ B * , where B denotes the vocabulary of text sequences (e.g., letters or words). The goal is to build a model, typically a probabilistic one such as p θ (Y | X), to predict the transcription given speech audio. Such models are often classified into two categories: hybrid (Juang et al., 1986;Young, 1996;Povey, 2005;Hinton et al., 2012;Abdel-Hamid et al., 2012;Bourlard and Morgan, 2012) or end-to-end (Graves et al., 2006;Graves, 2012;Chorowski et al., 2015;Rao et al., 2017).
Unsupervised Speech Recognition
Hybrid systems parameterize the joint distribution of speech and text p θ (X, Y ) with three components: an acoustic model (AM), a pronunciation model (PM), and a language model (LM) (Young, 1996). A language model describes the distribution over text sequences: p LM (Y ) = L i=1 p(y i | y 1 , · · · , y i−1 ). A pronunciation model is a lexicon mapping each word to its pronunciation represented as a sequence of phonemes, often created by linguists. For simplicity, we consider a deterministic lexicon here where each word only has one pronunciation, and therefore each word sequence Y can be transcribed into one phoneme sequence P = [p 1 , · · · , p M ] = f P M (Y ) ∈ O * , where O is the phoneme inventory.
Unsupervised Speech Recognition
The last component, the acoustic model p AM (X | P ), parameterizes the distribution over speech X given a phoneme sequence P . Since each phoneme p ∈ O can generate a speech feature sequence of variable length, this component is often formulated as a linear hidden Markov model (HMM), where each phoneme has its corresponding HMM model and each state can transition to itself or the next state. The probability of a phoneme sequence is described by concatenating the corresponding HMM models (linking the forward transition from the last state of a phoneme to the first state of the next phoneme in that sequence). Let an alignment A = [a 1 , · · · , a T ] denote a sequence of states from the concatenated HMM each speech frame x t corresponds to, the joint probability of speech and alignment can be written as
Unsupervised Speech Recognition
where p tran denotes the transition probability derived from the concatenated HMM model, and p emit denotes the emission probability, which is often parameterized by a Gaussian mixture model (GMM).
Unsupervised Speech Recognition
To compute the acoustic model probability, one can marginalize over all alignments: p AM (X | P ) = A p emit (X | A) p tran (A | P ) efficiently with the forward-backward algorithm. Oftentimes, the probability is approximated with p AM (X | P ) ≈ max A p emit (X | A) p tran (A | P ), which can also be computed efficiently with the Viterbi algorithm. The resulting alignment, called the forced alignment, can be used to segment the speech by phone identities  or to create frame-level targets for training neural network acoustic model that parameterizes p emit (a t | x t ) (Bourlard and Morgan, 2012).
Unsupervised Speech Recognition
Combining AM, PM and LM, a hybrid system computes the joint probability of speech and text as
Unsupervised Speech Recognition
The acoustic and the language model are often learned separately with a maximal likelihood objective (Bahl et al., 1983;Juang et al., 1986):
Unsupervised Speech Recognition
but alternative discriminative objectives for AMs taking LMs into account have also been explored (Bahl et al., 1986;Povey, 2005). Note that since training of language models requires only text data, one can leverage not only text from the paired data in D l , but also additional unpaired text data for training.
Unsupervised Speech Recognition
Once the hybrid system is trained, decoding amounts to finding the most probable text sequence for a given speech sequence arg max Y p θ (Y | X), which is equivalent to finding the sequence with the highest joint probability arg max Y p θ (X, Y ). In practice, decoding is often carried out by searching the minimal distortion path on a weighted finite state transducer (WFST; Mohri 1997; Mohri et al. 2002), obtained by composing WFSTs representing HMM emission, HMM transition, pronunciation model, and language model, respectively. Moreover, it is common to introduce a weighting factor α to balance the importance of acoustic and language model by re-writing the joint probability as
Unsupervised Speech Recognition
where α is determined by minimizing the edit distance ED on a development set
Unsupervised Speech Recognition
In our setting, we use either phoneme error rate (PER) or word error rate (WER) but other choices are possible. After α is determined, we decode an utterance X by finding
Unsupervised Speech Recognition
More recently, end-to-end approaches that directly parametrize p θ (Y | X) have gained increasing interest. This line of approaches includes but is not limited to connectionist temporal classification (CTC; Graves et al. 2006), recurrent neural network transducer (RNN-T; Graves 2012), sequence-to-sequence models (seq2seq; Chorowski et al. 2015). We focus our discussion on CTC, which is most relevant to this work.
Unsupervised Speech Recognition
As mentioned above, the emission model in the hybrid system can be replaced with a neural network predicting the posterior probability over HMM states for each frame, but it requires a seed HMM-GMM model to derive the forced-alignment. Instead of maximizing the probability of the derived forced alignment to HMM states, CTC parameterizes the distribution over alignment to text sequences directly, and marginalizes over all possible alignments for the target text sequence during training to compute the posterior directly. Formally speaking, for an input speech sequence X of T frames, CTC predicts a distribution over B = B ∪ { } for each input step, where B is the text alphabet and is a special blank symbol, representing empty output. The probability of an alignment A = [a 1 , · · · , a T ] is defined as T t=1 p θ (a t | X). Each alignment is mapped to a text sequence with a function g, which first removes consecutive repeating units and then removes all . For example, an alignment "c aa abb" is mapped to a text sequence "caab". The posterior probability of a text sequence Y given speech X of length T is therefore defined as:
Unsupervised Speech Recognition
and the marginalization on the right hand side can be computed efficiently with dynamic programming. Training of CTC optimizes the likelihood of the posterior distribution:
Unsupervised Speech Recognition
and decoding is approximated with finding the most probable alignment and mapping that alignment to a text sequence:
Unsupervised Speech Recognition
arg max at p θ (a t | X) .
Unsupervised Speech Recognition
While CTC is motivated by end-to-end modeling of speech recognition, it can also fit nicely into the hybrid framework. The HMM associated with CTC has one state for each letter and a special state for the blank symbol (Zeyer et al., 2017). Each state can transit to any state including itself with the same transition probability. See Hannun (2017) and   Figure 3 for an illustration. As a result, one can also compose a CTC acoustic model with a letter-PM (mapping words to letters instead of phonemes) and an LM into a single WFST for decoding, or train a CTC model that predicts phonemes to compose with regular PM and LM for decoding. Likewise, a weighting factor α can be introduced to balance the CTC acoustic model and the language model. By re-arranging Equation 3 as
Unsupervised Speech Recognition
we can observe that this is equivalent to the formulation of language model fusion for end-toend systems that combines posterior and prior probabilities of text for decoding (Chorowski and Jaitly, 2017).
Unsupervised Speech Recognition
In the semi-supervised setup, one is provided with unpaired speech D s u = {X j } Ns j=1 and text D t u = {Y j } Nt k=1 in addition to a labeled dataset D l . Most of the semi-supervised ASR studies focus on how to utilize unpaired speech, because it is straightforward to utilize additional text by training a better language model with both paired and unpaired text for decoding. Self-training is one of the simplest but most effective approaches to leveraging unpaired speech (Kahn et al., 2020a). It first trains a seed ASR model using labeled data D l , and then transcribes unpaired speech D s u into text {Ŷ j } Ns j=1 using the trained model to generate a pseudo-labeled datasetD l = {(X j ,Ŷ j } Ns j=1 . The transcribed text is often called the pseudo labels. Combining the real and the generated paired data, an ASR model can be trained with any supervised objective (Kahn et al., 2020a), or with separate objectives and weights Manohar et al., 2018;. This process can also can be repeated for multiple iterations to further improve performance (Xu et al., 2020b;Park et al., 2020).
Unsupervised Speech Recognition
In the following, we describe how we build suitable speech and text representations for unsupervised learning. Good representations are essential to learning a mapping from unlabeled speech audio to unlabeled text in a completely unsupervised fashion.
Unsupervised Speech Recognition
In the first step, we learn representations of the speech audio signal using self-supervised learning. There has been a lot of recent work in this direction which has shown strong performance in extremely low-labeled data setups across a range of languages (Conneau et al., 2020) and tasks (Fan et al., 2021;Pepino et al., 2021;Wang et al., 2021).
Unsupervised Speech Recognition
Wav2vec 2.0 consists of a convolutional feature encoder f : X → Z that maps a raw audio sequence X to latent speech representations z 1 , . . . , z T , which a Transformer g : Z → C then turns into context representations c 1 , . . . , c T (Baevski et al., 2020b,a). Each z t represents about 25ms of audio strided by 20ms and the Transformer architecture follows BERT (Vaswani et al., 2017;Devlin et al., 2019). During training, latent representations are discretized to q 1 , . . . , q T with a quantization module Z → Q to represent the targets in the objective. The quantization module uses a Gumbel softmax to choose entries from E = 2 codebooks with V = 320 entries each and the chosen entries are concatenated to obtain q (Jegou et al., 2011;Jang et al., 2016;Baevski et al., 2020b).
Unsupervised Speech Recognition
The model is trained to identify the true quantized latent q t for each masked time-step within a set of distractors Q t sampled from other masked time steps:
Unsupervised Speech Recognition
where c t is the output of the Transformer, and sim(a, b) denotes cosine similarity. This is augmented by a codebook diversity penalty to encourage the model to use all codebook entries (Dieleman et al., 2018). The codebook diversity penalty maximizes the entropy of the averaged softmax distribution over the codebook entries for each groupp g across a batch of utterances:
Unsupervised Speech Recognition
In our experiments, we use the publicly available English model pre-trained on 53k hours of Libri-Light  as well as XLSR-53 which was pre-trained on nearly 60k hours of speech audio in 53 languages (Conneau et al., 2020). Both use Large transformers with 24 blocks, model dimension 1,024, inner-dimension 4,096 and 16 attention heads (Baevski et al., 2020c).
Unsupervised Speech Recognition
Next, we embed the speech audio using self-supervised representations from wav2vec 2.0. Before that, we also remove silences from the speech audio using an off-the-shelf unsupervised method. One exception is the TIMIT benchmark, where silences are part of the transcription. Silence removal is important to learning a better mapping between speech audio representations and transcriptions.
Unsupervised Speech Recognition
Removing Silences. Most datasets we use for our experiments have audio data with silences. However, these parts of the audio do not correspond to any transcription and we therefore remove silences as much as possible. To remove silences, we apply rVAD, an unsupervised voice activity detection (VAD) model which determines the segments in the audio data corresponding to silences, and we remove these sections (Tan et al., 2020). We ablate this choice in § 3.4.
Unsupervised Speech Recognition
Speech Audio Representations. After silence removal, we embed the unlabeled speech audio with wav2vec 2.0 to obtain speech representations. Specifically, we use the representations of the context Transformer network c 1 , . . . , c T ( § 3.1). Each c t represents features computed from the entire utterance using self-attention and centered over the current timestep t; there is a 20ms gap between subsequent context representations ( § 3.1). The context network contains 24 Transformer blocks and we denote the output of block l at time-step t as c l t .  Figure 2). We consider English, German, Spanish, French, Italian, Dutch, Polish and Portuguese.
Unsupervised Speech Recognition
Our goal is to learn a model which can map from audio representations c l t to phonemes using no supervision. However, the representations of the uppermost block of wav2vec 2.0 may not be well suited for this task. These features are trained to directly predict masked latent representations spanning 25ms of speech audio which is much shorter than the typical duration of a phoneme. The uppermost block is therefore likely not well suited to learning a good mapping from the current time-step to phonemes. Similarly in natural language understanding, probing Transformer layers of the BERT models showed that blocks behave differently, with earlier blocks encoding syntactic information, while high-level semantic information appears at higher blocks (Tenney et al., 2019;Jawahar et al., 2019).
Unsupervised Speech Recognition
To get a better sense of this, we train supervised phoneme recognizers with a CTC loss on top of the frozen representations of each of the 24 blocks of the English wav2vec 2.0 Large model pre-trained on Libri-Light. We then evaluate phoneme error rate (PER) with respect to the phonemized transcriptions of Librispeech dev-other. The classifier takes as input c l t and contains a single softmax-normalized linear layer mapping to the phoneme inventory. Figure 2 shows that most of the first ten blocks as well as the final blocks provide very poor performance, while blocks 15-19 provide error rates below 9 PER. Block 15 achieves Figure 4: Example of segmenting the audio signal of the utterance jane may earn more. The top line shows the segmentation by the k-means segmentation method, the second line is the segmentation of Viterbi decoding with the GAN, and the third line shows the gold segmentation of human annotators from the TIMIT dataset. At the bottom, we show the corresponding spectogram, although, the input to our method is raw audio. The utterance has TIMIT ID MGLB0 SX4.
Unsupervised Speech Recognition
the best error rate of 7.5 PER. A similar insight has been used in the concurrent work of Hsu et al. (2021b). Does this choice of representation generalize across languages? To get a sense of this, we measure PER for eight different languages of the MLS dataset ( § 5.1) when inputting representations of the multilingual wav2vec 2.0 XLSR-53 model. Figure 3 confirms that block 15 provides good performance across a range of languages and we will use block 15 as the representation for speech audio in all subsequent experiments. For brevity we drop the superscript l and refer to block 15 representations simply as c 1 , . . . , c T .
Unsupervised Speech Recognition
Once the speech signal is embedded, we identify segments corresponding to meaningful units that can be mapped to phonemes. Segmentation has been shown to be crucial in prior work  since the right boundaries in the input representations make it easier to predict the output sequence.
Unsupervised Speech Recognition
Identifying Speech Audio Segments. There has been a lot of prior work in unsupervised speech segmentation (Kamper et al., 2017a,b;Rasanen et al., 2015;Kreuk et al., 2020) but here we simply use a method based on clustering the wav2vec 2.0 speech representations c 1 , . . . , c T . In a first step, we collect all the speech representations for the unlabeled speech data and perform k-means clustering to identify K = 128 clusters. We use the FAISS library to do fast clustering on GPUs (Johnson et al., 2019). Next, each c t is labeled with the corresponding cluster ID i t ∈ {1, . . . , K} and we introduce speech segment boundaries whenever the cluster ID changes.
Unsupervised Speech Recognition
DAVEnet + peak detection (Harwath and Glass, 2019) .893 .712 .792 CPC + peak detection (Kreuk et al., 2020) .839 .836 .837 k-means on wav2vec 2.0 features .935 .379 .539 wav2vec-U Viterbi prediction .598 .662 .629 Table 1: Quantitative evaluation of segment boundaries with respect to human labeled segment boundaries. We report precision, recall and f-measure using a 20ms tolerance.
Unsupervised Speech Recognition
Building Segment Representations. Once the speech audio representations are segmented, we compute a 512-dimensional PCA over all speech representations output by wav2vec 2.0 for the training set. Next, we mean-pool the PCA representations for a particular segment to obtain an average representation of the segment. The PCA retains only the most important features and we found this to be effective. Segment boundaries are noisy due to the lack of supervision and we therefore found it useful to also mean-pool pairs of adjacent segment representations to increase robustness. This results in sequences of speech segment representation S = s 1 , . . . , s T , S ∼ S for a given utterance. Figure 4 illustrates how the k-means segmentation strategy results in very granular units compared to the gold segments corresponding to phonemes. Based on the k-means units, the unsupervised model can then recover segments that correspond very closely to phonemic units identified by humans. Table 1 shows that k-means clustering results in very high precision but low recall when recovering gold phoneme boundaries on TIMIT. The Viterbi outputs of our model (wav2vec-U) result in more balanced, and better, accuracy because neighboring segments with the same predicted label are combined into a larger segment.
Unsupervised Speech Recognition
Similar to how we segment the unlabeled speech audio data into suitable units for unsupervised learning, we do the same for the unlabeled text data. We apply two pre-processing steps to the text data: phonemization and silence token insertion.
Unsupervised Speech Recognition
Phonemization. Phonemes characterize the different sounds which distinguish words from each other, e.g., for the word cat there are three phonemes corresponding to the three distinct sounds in the pronunciation of the word: /K/, /AE/, /T/. We phonemize the text data because we found it easier to learn a mapping between speech audio and the different sounds of a word rather than between audio and words or letters. Phonemization converts a sequence of words Y into a sequence of phonemes P = [p 1 , · · · , p M ], where p m ∈ O and O is the phoneme inventory. We use off-the-shelf tools for this step which we detail in § 5.2.
Unsupervised Speech Recognition
Silence token insertion. The unlabeled speech audio data is pre-processed by applying unsupervised silence removal. However, this process is not always accurate and many silences in the speech audio remain. To deal with this, we enable the unsupervised model to label some segments with a phonemic silence token (SIL; § 4.1). However, the phonemized unlabeled text data does not contain any silence tokens and this may pose difficulties for  Figure 5: Unsupervised performance when augmenting the unlabeled text data with silence tokens. We add silence tokens to the unlabeled text to better resemble the speech audio which does contain silences. Silence tokens surrounding sentences and removing silences from the audio results in better performance (left), and we show different rates of silence token insertion in the unlabeled text data (right). We report mean PER and standard deviation over 20 random seeds of unsupervised training on Librispeech dev-other.
Unsupervised Speech Recognition
adversarial learning ( § 4). We remedy this by inserting silence markers into the phonemized unlabeled text data. First, we add a SIL token to the beginning and the end of all phonemized unlabeled text sentences. Figure 5 (left) shows that this improves accuracy. The same table shows the detrimental effect of not removing silences from the audio data ( § 3.2). Second, we randomly insert SIL between words, or groups of phonemes corresponding to words. Figure 5 (right) shows that inserting the silence token at a rate of 0.25 yields the best end accuracy.
Unsupervised Speech Recognition
We use adversarial training to train an unsupervised speech recognition model using the representations of the unlabeled speech audio data and the unlabeled phonemized text data (Liu et al., 2018;. In the following, we detail the model architecture, the training objective as well as the unsupervised cross-validation metric we developed.
Unsupervised Speech Recognition
Generative adversarial networks (GAN; Goodfellow et al. 2014) train a generator network G and a discriminator/critic network C where the generator produces samples which are then judged by the discriminator. The discriminator is trained to classify whether samples are from the generator or from the real data distribution. The objective of the generator is to produce samples that are indistinguishable by the discriminator.
Unsupervised Speech Recognition
Concretely, G takes as input a sequence of T segment representations S = [s 1 , . . . , s T ] ( § 3.3) which are then mapped to a sequence of M phonemes G(S) = [p 1 , . . . , p M ]. The generator predicts a distribution over the phoneme set O for each segment and outputs the phoneme with the highest probability. If the argmax prediction of consecutive segments result in the same phoneme, then we sample one of these segments, therefore M ≤ T .
Unsupervised Speech Recognition
The phoneme set O includes a silence label SIL to enable labeling silences in the speech audio as such. Without a silence label, we noticed that the model was repurposing a particular phoneme to label silences which resulted in much lower performance since it inter- fered with subsequent language model decoding. In the backward pass, we back-propagate through segments sampled at the generator output. We do not modify the segment representations S during unsupervised training. The generator is parameterized as a single layer convolutional neural network (CNN; § 5.3).
Unsupervised Speech Recognition
The discriminator takes as input either a sequence P r ∼ P r of one-hot vectors denoting phonemized text from the real data distribution P r or a sequence of output distributions from the generator G(S). Each input vector has |O| dimensions to represent the distribution over phonemes for each segment (see Figure 6 for an illustration). The discriminator is also a CNN which outputs a probability indicating how likely the sample is to be from the data distribution ( § 5.3).
Unsupervised Speech Recognition
In our setup we use the original GAN objective with a gradient penalty (Goodfellow et al., 2014;, a segment smoothness penalty and a phoneme diversity penalty:
Unsupervised Speech Recognition
where P r ∼ P r is phonemized unlabeled text, G(S) is the transcription output by the generator of input segment representations S for some unlabeled speech audio. The first term trains the discriminator to assign high probability to real transcriptions, the second term encourages the discriminator to assign low probability to generator outputs, L gp is a gradient penalty, L sp is a smoothness penalty and L pd is a phoneme diversity loss which we detail next. During training we alternate updates for the discriminator and the generator. We also alternate batches of predicted transcriptions from the generator and phonemized unlabeled text.
Unsupervised Speech Recognition
Gradient penalty. To stabilize training, we penalize the gradient norm of the discriminator with respect to the input (Gulrajani et al., 2017). The penalty is computed for random samplesP ∼P which are a linear combination of the activations of pairs of real and fake samples. 1
Unsupervised Speech Recognition
Segment smoothness penalty. The k-means segmentation of the speech audio is more granular than a typical phonemized transcription and neighboring representations are highly correlated. We therefore found it useful to add a penalty which encourages the generator to produce similar outputs for adjacent segments:
Unsupervised Speech Recognition
Phoneme diversity loss. We also found it helpful to penalize low usage of the phoneme vocabulary by the generator network on the batch level. In particular, we maximize the entropy of the averaged softmax distribution H G (G(S)) of the generator over the phoneme vocabulary across a batch B of utterances:
Unsupervised Speech Recognition
Our goal is to build speech recognition models without any supervision. To this end, we developed the following cross-validation metric which does not require labeled data. We use the metric for early stopping, selecting a random seed, and hyper-parameter selection (λ, γ, η). We consider two quantities in our metric: LM negative log-likelihood (NLL) and vocabulary usage. N LL LM serves as an indicator of fluency for a given transcription and it is measured with a language model p LM trained on phonemized text data ( § 3.4). Vocabulary usage is the proportion of the phoneme vocabulary being output by the model via Viterbi decoding. Measuring vocabulary usage identifies degenerate models which output fluent but trivial transcriptions.
Unsupervised Speech Recognition
We denote Viterbi phoneme transcriptions for a given generator configuration G and unlabeled speech audio In the first step, we generate phoneme transcriptions for different training checkpoints or hyper-parameter settings and denote the transcriptions of the configuration with the lowest vocabulary-usage adjusted NLL asP = arg min P N LL LM (P) − log U (P). 3 Next, we discard model configurations which do not satisfy the following, usingP as the anchor:
Unsupervised Speech Recognition
The second term on the right hand side introduces a margin over the NLL of the anchor transcription N LL LM (P) based on the vocabulary usage of P andP: If U (P) is much lower compared to U (P), then we allow model configurations which produce transcriptions with higher NLL compared toP. However, if U (P) is a lot higher than U (P), then the model configuration will not satisfy the constraint. The log 1.2 factor serves as another margin allowing checkpoints with slightly worse vocabulary-usage adjusted NLL to be included. In the final step, we take into account the length of the transcriptions: out of the configurations P which satisfy the above constraint, we select the one which has the highest sum of log probability without normalizing the length:
Unsupervised Speech Recognition
This selects model configurations which produce phoneme sequences that score high under the language model but are not too long. How effective is this metric? To get a sense of this, we compare the performance of crossvalidation with a labeled development to cross-validation with our unsupervised metric on the TIMIT benchmark. We cross-validate GAN hyper-parameters, different checkpoints for early stopping, language model decoding hyper-parameters ( § 5.4) and HMM decoding hyper-parameters for self-training ( § 5.5). Figure 7 shows that the unsupervised metric has only between 0.3-1.2 higher PER compared to using a labeled development set. This enables model development of unsupervised speech recognition systems without labeled data at only a small drop in accuracy compared to the ideal setting where labeled development data is available.
Unsupervised Speech Recognition
We consider several corpora and a variety of languages to evaluate our approach. TIMIT is a small English dataset on which previous unsupervised speech recognitition work was conducted. Librispeech is a standard benchmark with nearly 1,000 hours of labeled English speech audio and MLS is a multilingual benchmark with eight European languages. In addition, we also consider non-European languages from the ALFFA and CommonVoice corpora.
Unsupervised Speech Recognition
TIMIT. This dataset contains about five hours of audio recordings with time-aligned phonetic transcripts (Garofolo et al., 1993). To compare to prior work, we consider two setups: the matched setting uses text and speech from the same set of utterances to train the model while the unmatched setting ensures that the unlabeled text data does not contain the transcriptions of the audio data. For the matched setup, we follow the standard train/dev/test split of TIMIT as done in Yeh et al. (2019). This is 3,696/400/192 train/dev/test utterances which contains only SX (compact) and SI (diverse) sentences. For the unmatched setting, we follow Chen et al. (2019) by training on 3,000 speech utterances and 1,000 transcriptions from the training portion of the complete dataset split. We use the remaining 620 training utterances for validation, and test on 1,680 sentences for testing. The complete dataset split contains 4,620 training and 1,680 testing utterances, with additional SA (dialect) sentences.
Unsupervised Speech Recognition
Librispeech and Libri-Light. The Librispeech corpus contains 960 hours of transcribed speech audio (LS-960) for training. The data is based on read English audio books. For unsupervised training, we only use the speech audio data but not the transcriptions. We use the official Librispeech language modeling data as unlabeled text data with the Libri-Light data removed . 4 This is a large text corpus of 635m words but we show that much smaller amounts of text and speech data still result in the same performance ( § 6.6). We evaluate on the standard dev-other/clean and test-clean/other sets. For development, we compute the unsupervised metric ( § 4.3) on dev-other. We also experiment with the audio data from Libri-Light (LL-60k) for which we follow the pre-processing of  resulting in 53.2k hours of speech audio.
Unsupervised Speech Recognition
Multilingual LibriSpeech (MLS). The Multilingual Librispeech dataset  is a large corpus of read audiobooks from Librivox in eight languages and we experiment with the following six languages: Dutch (du), French (fr), German (de), Italian (it), Portuguese (pt), Spanish (es). The latest version of this corpus contains around 50k hours including 44k hours in English. However, for unsupervised learning we only use 100 hours of speech audio for each language. As unlabeled text data, we use the LM data provided by MLS.
Unsupervised Speech Recognition
ALFFA. We experiment with the Swahili data from the ALFFA project (Gelas et al., 2012;Abate et al., 2005;Tachbelie et al., 2014) which is read speech. There are 9.2 hours of speech audio training data and we use the language modeling data provided by ALFFA as unlabeled text data as well as newscrawl 2008-2014, 2018-2020. 5 CommonVoice. This is a multilingual corpus of read speech for 38 languages (Ardila et al., 2020). We focus on two low-resource languages Kyrgyz (ky) and Tatar (tt) and use 1.8 hours and 4.6 hours of speech audio, respectively. As unlabeled text data for Kyrgyz we use the Kyrgyz community corpus 2017, 6 and newscrawl 2008-2014 and 2018-2020. 7 For Tatar we use the Tatar community corpus 2017, 8 and newscrawl 2005-2011. 9 We evaluate on the dev and test split of Rivière et al. (2020).
Unsupervised Speech Recognition
TIMIT provides time-aligned phonetic transcriptions annotated with an inventory of 60 phones adapted from the ARPAbet system, which treats silence as a phoneme. In addition, it includes a mapping from the 60 phoneme inventory to 48-and 39-phoneme inventories. Phoneme error rates are typically computed on the 39-phoneme inventory (Povey et al., 2011), which we map the phonetic transcripts to for training.
Unsupervised Speech Recognition
For Librispeech, we use the G2P phonemizer (Park and Kim, 2019) which uses the CMU dictionary to look up English word pronunciations, falling back to a neural network trained to output a phoneme sequence given a word. We phonemize the Librispeech LM corpus, with Librispeech and Librivox text data removed . We convert the full phoneme set to a reduced set containing 39 phonemes by removing the numerical stress markers from the vowels.
Unsupervised Speech Recognition
For other corpora, including English in the MLS dataset, we use Phonemizer which supports a large number of various languages, but is less accurate than G2P for English. 10 We disable the language-switching labels and prune phonemes that appear fewer than 1000 times in the text corpus.
Unsupervised Speech Recognition
Models are implemented in fairseq (Ott et al., 2019). The generator and discriminator are optimized with Adam (Kingma and Ba, 2015) using β 1 = 0.5 and β 2 = 0.98. The discriminator has a weight decay of 1e − 4 while the generator does not use weight decay. The discriminator is trained with a learning rate of 1e − 5 and the generator with 1e − 4, which are held constant throughout the training. We train for a total of 150k steps, during 5. http://data.statmt.org/news-crawl/sw 6. https://corpora.uni-leipzig.de?corpusId=kir_community_2017 7. http://data.statmt.org/news-crawl/ky 8. https://corpora.uni-leipzig.de?corpusId=tat_community_2017 9. https://corpora.uni-leipzig.de/en?corpusId=tat_news_2005-2011 10. https://github.com/bootphon/phonemizer which we alternate optimizing the discriminator and the generator (so each are updated 75k times in total). Each training step is performed using a batch of 160 randomly chosen samples from the unlabeled audio data and 160 randomly chosen text samples from the unlabeled text data. Training takes about 12 hours on a single V100 GPU.
Unsupervised Speech Recognition
The discriminator is composed of three causal convolution blocks with a hidden size of 384 and a kernel size of 6, resulting in a receptive field size of 16 segments. The input into the discriminator is an |O| dimensional vector representing the probability distribution over the phoneme vocabulary, and the output is a single logit for each time-step, indicating how likely the sample is to be from the data distribution. The first layer serves as embedding for the |O| phonemes.
Unsupervised Speech Recognition
The generator is a single non-causal convolution with kernel size 4. The input to the generator are the segment representations S of dimension 512 and the output is an |O| dimensional vector. The generator contains about 90k parameters and we do not backpropagate to the segment representations. We combine subsequent generator predictions prior to feeding them into the discriminator as described in § 4.1 and apply a softmax normalization. During training, we use dropout with p = 0.1 to the input of the generator (Srivastava et al., 2014).
Unsupervised Speech Recognition
For each language, we tune the following hyper-parameters using the unsupervised crossvalidation metric ( § 4.3): the gradient penalty weight λ is selected from the range [1.5, 2.0], the smoothness penalty weight γ from [0.5, 0.75], the phoneme diversity loss weight η from [2, 4], and we train 5 seeds for each configuration for a total of 40 models.
Unsupervised Speech Recognition
We wish to decode the output of either phoneme-based models, resulting from unsupervised training, or letter-based models, resulting from subsequent self-training ( § 5.5) using a language model.
Unsupervised Speech Recognition
To do so we build WFSTs ( § 2.2.2) using PyKaldi (Can et al., 2018), a Python port of Kaldi (Povey et al., 2011). The WFST takes as input the model emissions and if we decode to words, then we use the same phonemizer with which we pre-processed the unlabeled text data to build a mapping between phonemes to words. The WFST is composed with a 4-gram language model (Heafield, 2011) pruned to keep only 4-grams occurring more than 3 times. We add self-loops that mimic CTC behavior (Zhang et al., 2020a) where blank symbols (silence for the GAN or actual blank symbol for letter models) are mapped to epsilons and consecutive predictions of the same symbol are collapsed.
Unsupervised Speech Recognition
During decoding, we average the predicted phoneme distributions for segments which have the same argmax prediction. We provide acoustic scale as a parameter to the Kaldi decoder and we also add a scalar ν to the blank token emission (silence for GAN models and blank for others). We tune the optimal weights for these two parameters by minimizing a quantity that measures fluency of the output as well as faithfulness to the model output. In particular we minimize the following quantity on an unlabeled development set -assuming a phoneme-based model which we wish to decode to words:
Unsupervised Speech Recognition
where {P j } Ns j=1 are the Viterbi model outputs, {P j } Ns j=1 are the word-based outputs of the WFST converted to phonemes (or simply phoneme-based outputs if decoded to phonemes), H LM (P j ) is the entropy of a language model, ED is an edit distance such as PER for phonemes or character error rate for letter-based model models, and µ = 0.03. In practice, trivial solutions may achieve very low entropy. We counteract this by replacing H LM (P j ) by the average entropy of the language model training data if H LM (P j ) is lower than the entropy of the training data. We tune acoustic scale in the interval [0,8], and ν in [−3, 8].
Unsupervised Speech Recognition
For Librispeech experiments, we also decode with a Transformer language model (Baevski and Auli, 2018) trained on the Librispeech LM corpus using the beam search decoder of Pratap et al. (2019). The Transformer LM is identical to  and contains 20 blocks, model dimension 1,280, inner dimension 6,144 and 16 attention heads. We tune acoustic scale with a beam of 50 and test performance is measured with beam 500.
Unsupervised Speech Recognition
For self-training, we perform two iterations: first, we pseudo-label the training data with the unsupervised GAN model and train an HMM on the pseudo-labels ( § 2.2). Second, we relabel the training data with the HMM and then fine-tune the original wav2vec 2.0 model using the HMM pseudo-labels with a CTC loss ( § 2.3). HMM models use phonemes as output, while as wav2vec 2.0 models use letters. Both are decoded using WFST decoders into words. wav2vec 2.0 self-training for Librispeech uses the same fine-tuning parameters as the original wav2vec 2.0 model fine-tuned on 100 hours of Librispeech data, but we reduce masking probability to 0.25, reduce the batch size to 800k frames and train on 8 GPUs for 80k updates. 11 We use the last checkpoint instead of early stopping. For TIMIT selftraining, we use the one hour fine-tuning parameters of the original wav2vec 2.0 model. 12 This performs 13k updates on 4 GPUs.
Unsupervised Speech Recognition
We first evaluate our approach on the Librispeech benchmark to compare to competitive systems trained on 960 hours of labeled data ( § 6.1). This is followed by a comparison to prior unsupervised work on the small-scale TIMIT benchmark ( § 6.2). To get a sense of how well our approach works on non-English languages we evaluate on several European languages of the MLS benchmark ( § 6.3) as well as non-European low-resource languages ( § 6.4). We evaluate the effectiveness of different self-training strategies ( § 6.5) and show that good performance can be achieved with very little unlabeled data ( § 6.6).
Unsupervised Speech Recognition
We first test our approach on Librispeech to get a sense of how viable unsupervised speech recognition can be compared to the best supervised systems trained on a large amount of labeled data. Librispeech is a standard benchmark in the speech recognition commu- Table 2: WER on the Librispeech dev/test sets when using 960 hours of unlabeled audio data from Librispeech (LS-960) or 53.2k hours from Libri-Light (LL-60k) using representations from wav2vec 2.0 Large. Librispeech provides clean dev/test sets which are less challenging than the other sets. We report results for GAN training only (wav2vec-U) and with subsequent self-training (wav2vec-U + ST).
Unsupervised Speech Recognition
nity which provides about 960 hours of transcribed read audiobooks. We use the language modeling data of Librispeech as unlabeled text data for unsupervised training. 13 We experiment with the frozen representations of a wav2vec 2.0 Large model trained on the 53.2k hours of Libri-Light (LL-60k) which we denote as wav2vec-U Large and we also consider self-training ( § 5.5). wav2vec-U Large with self-training (wav2vec-U + ST) and a Transformer language model achieves WER 5.9 on test-other, the noisy test set. This shows that unsupervised speech recognition can perform remarkably well compared to the best supervised systems of the recent past on this much studied benchmark. Also, self-training is effective even when the teacher model is unsupervised as per the improvement over GAN training (wav2vec-U). Interestingly, self-training on just Librispeech, or 960 hours of unlabeled speech audio, achieves already very good performance of WER 6.4 on dev-other compared to self-training on all of Libri-Light (53.2k hours) which compares at 6.0 WER. We note that the number of parameters trained during adversarial training is very small: the generator contains only about 90k parameters for a single temporal convolution mapping to the phoneme set from frozen wav2vec 2.0 representations.
Unsupervised Speech Recognition
13. Below we show that much less unlabeled text and speech audio are sufficient to reach a similar level of performance ( § 6.6).
Unsupervised Speech Recognition
LM core-dev core-test all-test  
Unsupervised Speech Recognition
Prior work on unsupervised speech recognition focused on the TIMIT benchmark. In order to perform a direct comparison to these approaches, we report results on this benchmark as well. We consider two setups to compare to previous work: in the matched setting, the unlabeled text data is simply the transcriptions of the unlabeled audio data but unpaired.
Unsupervised Speech Recognition
In the unmatched setup, the unlabeled text data does not contain the transcriptions for the audio data which is a more realistic setting. We measure performance on the standard Kaldi dev and test sets (core-dev/core-test) as well as a slightly larger version of the test set (all-test) to be able to compare to Liu et al. (2018) and . Further details of the two setups can be found in § 5.1. We report performance for wav2vec-U with a 4-gram language model trained on the language modeling data of TIMIT and we also consider self-training (wav2vec-U + ST). Table 3 shows that wav2vec-U outperforms prior unsupervised work in both the matched and unmatched settings, reducing PER on all-test in the matched setup by 57% relative compared to . Our method has lower performance than the best supervised methods but it performs still very well at PER 12 on core-test in the matched setup compared to PER 8.3 for the state of the art (Baevski et al., 2020c).   (Rivière et al., 2020) 42.0 41.2 XLSR-53 (Conneau et al., 2020) 5.1 6.1
Unsupervised Speech Recognition
Unsupervised learning wav2vec-U 25.7 24.1 wav2vec-U + HMM 13.7 14.9 Table 5: PER for low-resource languages, Tatar (tt) and Kyrgyz (ky).
Unsupervised Speech Recognition
Supervised learning Besacier et al. (2015) 27.36
Unsupervised Speech Recognition
Unsupervised learning wav2vec-U 52.6 wav2vec-U + ST 32.2 Table 6: WER for Swahili from the ALFFA corpus. We compare to the supervised baseline of the ALFFA project.
Unsupervised Speech Recognition
To get a sense of how well the method works on non-English data, we experiment on six languages of the multilingual Librispeech corpus (MLS; . As baseline we consider the supervised systems of  trained on between 2k and 161 hours of labeled data, depending on the language. For adversarial learning we use 100 hours of unlabeled audio data from MLS for every language as well as the MLS language modeling data. As input to wav2vec-U we use the representations from XLSR-53 (Conneau et al., 2020), a wav2vec 2.0 model pre-trained on 53 languages. Table 4 shows that wav2vec-U generalizes across a range of languages. Performance is lower than supervised systems but it shows the viability for other languages.
Unsupervised Speech Recognition
Experiments so far focused on European languages, for most of which relatively large amounts of labeled data exist. Next, we turn to three low-resource languages, Swahili, Kyrgyz, and Tatar Table 7: PER on TIMIT for various self-training strategies. We compare the performance of just the GAN output (wav2vec-U) to one or two iterations of subsequent self-training with an HMM. We contrast this to using the HMM for re-segmenting the audio data as done in prior work . We also consider self-training based on fine-tuning the original wav2vec 2.0 model (fine-tune) in or two self-training iterations (Xu et al., 2020a) as well as a combination of HMM and fine-tuning-based self-training.
Unsupervised Speech Recognition
with only about 4.3m and 5.2m speakers, respectively. 14 We use between 1.8 hours (Kyrgyz) and 9.2 hours of unlabeled audio (Swahili), see § 5.1. To compare to prior work, we measure WER for Swahili and PER for Kyrgyz and Tatar. For Tatar and Kyrgyz we opted to use a reduced self-training regime for faster experimental turn-around where we only perform HMM self-training and we expect better performance with the full self-training setup ( § 6.5). Table 5 and Table 6 show that wav2vec-U achieves good performance on these low-resource languages compared to previous work that utilized labeled data. We note that for Tatar and Kyrgyz we use a much smaller amount of speech audio than prior work: compared to XLSR-53 we use 1.8h unlabeled data vs 17h of labeled data for Kyrgyz and 4.6h vs. 17h for Tatar.
Unsupervised Speech Recognition
The self-training strategy we use is as follows: once the GAN is trained, we use it together with a language model to pseudo-label the unlabeled audio, then we train an HMM on the labels and repeat pseudo-labeling with the HMM in order to fine-tune the wav2vec 2.0 model whose representation were originally fed to the GAN. Finally, we use the fine-tuned wav2vec 2.0 model to decode the test set. Table 7 shows that fine-tuning with an HMM (wav2vec-U + HMM) leads to substantial improvements, however, a second iteration of HMM self-training leads to much smaller additional gains (wav2vec-U + HMM + HMM). Using the HMM to re-segment the speech audio followed by repeated GAN training (wav2vec-U + HMM resegment + GAN), similar to , does not improve performance over just HMM self-training.
Unsupervised Speech Recognition
Another option is to directly fine-tune wav2vec 2.0 on the labels assigned by the GAN model (wav2vec-U + fine-tune) and this performs very well. However, another round of self- training based on a fine-tuned wav2vec 2.0 model does not improve performance (wav2vec 2.0 + fine-tune + fine-tune). We believe that this is due to overfitting since the fine-tuned model has over 300m parameters. This is in line with recent observations about overfitting in self-training for speech recognition (Likhomanenko et al., 2021). The HMM is less likely to overfit in the way the Large wav2vec 2.0 model does. We therefore found it effective to perform a single round of HMM self-training followed by wav2vec 2.0 fine-tuning (wav2vec-U + HMM + fine-tune). After two rounds of self-training, we do not require a language model anymore for this benchmark which is likely because the language model has been distilled into the model to a large degree.
Unsupervised Speech Recognition
For the experiments on Librispeech we used large amounts of unlabeled speech audio and text data for adversarial learning (960 hours of unlabeled speech audio and nearly 31m sentences of text data). For TIMIT we used much less data, only about 3.15h of speech audio and 140k phonemes of text data for the matched setup. Next, we perform controlled experiments on Librispeech to get a sense of how much data is sufficient to achieve good performance. Figure 8 (left) shows that 9.6h of speech audio data still achieves excellent performance. Similarly, Figure 8 (right) shows that only about 3,000 sentences of text data are sufficient to achieve a similar level of accuracy as using all of the text data. We show further ablations in Appendix A.
Unsupervised Speech Recognition
Phonemization. Our approach requires tools to phonemize text for the language of interest. Even for existing tools, we found that the quality of phonemization differs, e.g., G2P vs Phonemizer ( § 5.2) on the same English data results in a performance difference. Moreover, phonemizers are not available for all languages and this presents a bottleneck. To address this, future work may develop phonemizers for more languages, explore phonemization approaches that generalize across languages, or unsupervised training with graphemic text units such as letters.
Unsupervised Speech Recognition
Segmentation. In our work, we explored a simple segmentation technique based on selfsupervised representations, however, there is a large body of work on segmentation and some of these techniques may lead to improvements over our simple approach. Also, wav2vec 2.0 learns representations for fixed size units with a fixed stride, however, phonemic units are of variable size. Another direction is to learn variable sized representations during pre-training.
Unsupervised Speech Recognition
wav2vec-U is a framework which enables building speech recognition models without labeled data. It embeds and segments the speech audio with self-supervised representations from wav2vec 2.0, learns a mapping to phonemes with adversarial learning, and cross-validates hyper-parameter choices as well as early stopping with an unsupervised metric. Experiments on the standard Librispeech benchmark show performance close to the state of the art models from only a few years ago, even though these models relied on nearly 1,000 hours of labeled data. Compared to the previous best unsupervised speech recognition approach, wav2vec-U reduces TIMIT phoneme error rate from 26.1 to 11.3. We also demonstrate the viability of our approach on several languages other than English, some of which are low-resource. The ability to build speech recognition models solely from unlabeled speech audio and unlabeled text drastically lowers the effort to build speech technology for many more languages of the world.
SUPERB @ SLT 2022: Challenge on Generalization and Efficiency of Self-Supervised Speech Representation Learning
Today's commercial speech recognition systems in both academic and industry fields require ever-increasing volumes of text-annotated speech signals for training. The need for massive data in supervised learning hinders the fast advancement of speech processing research. To tackle this issue, self-supervised learning (SSL) [1] has emerged to reduce the dependency on large labeled data sets. SSL utilizes proxy supervised learning tasks (also called pretext tasks) to obtain training data from the tremendous amount of unlabeled corpora available on the web. Researchers explore various pretext tasks to pre-train large neural networks without labels and transfer the pretrained networks to solve complicated downstream tasks. Recently, SSL has become one of the research mainstreams in speech processing as well as other machine learning communities such as Natural Language Processing (NLP) [2,3] and Computer Vision (CV) [4,5].
SUPERB @ SLT 2022: Challenge on Generalization and Efficiency of Self-Supervised Speech Representation Learning
Existing SSL research in the speech area centers around learning more powerful pre-trained networks that yield higher accuracy in downstream tasks. SSL techniques have been shown critical to advance research with state-of-the-art (SOTA) results in speech tasks such as automatic speech recognition (ASR) [6], automatic speaker verification (ASV) [7], query by examples (QbE), and intent classification (IC) [8]. Pre-trained networks also show promising performance in low or zero resource scenarios [9,10]. These studies often evaluate SSL techniques in different benchmarking tasks and datasets, downstream model architectures, and fine-tuning techniques (e.g., fine-tuning entire models or freezing the pre-trained networks). To encourage comparable experiments and a comprehensive understanding of SSL techniques, SUPERB [11,12] was introduced to the speech community. SUPERB aims to provide a standard framework to train, evaluate, and compare the efficacy and generalizability of SSL speech networks on 13 speech tasks in recognition, detection, semantics, speaker, paralinguistics, and generation domains.
SUPERB @ SLT 2022: Challenge on Generalization and Efficiency of Self-Supervised Speech Representation Learning
We observed a growing interest in SUPERB from the speech community. For example, a SUPERB session was organized in The 2nd Workshop on Self-supervised Learning for Audio and Speech Processing @ AAAI 2022 1 and tutorials about SSL methodologies and their evaluation using SUPERB, were given at ICASSP and NAACL 2022 [13]. Despite the abundant interest, research still focuses on getting better accuracy, and this focus inevitably leads to using more model parameters, pre-training data, and computation resources. The demand for computation resources is prohibitive for SSL getting wider adoption in academia and production where realtime inference is critical and for more researchers to participate and advance the SSL technology.
SUPERB @ SLT 2022: Challenge on Generalization and Efficiency of Self-Supervised Speech Representation Learning
The SUPERB @ SLT 2022: challenge on generalization and efficiency of self-supervised speech representation learning (denoted as the challenge in the following) is then organized to motivate more and diverse SSL innovation. In the challenge, we establish multiple groups of downstream tasks based on the SUPERB and extended SUPERB-SG [11,12] tasks for participants to analyze model capability from different aspects. Diverse metrics beyond accuracy, including memory usage and number of operations, are built to encourage the exploration of efficient SSL techniques with lower memory footprint and computation requirements. Besides, we open two tracks of submission, the public-set track and the hidden-set track, for tuning developed techniques and evaluating the generalizability of techniques, respectively. The goal of the challenge is to 1) continue existing momentum in SSL innovation, and 2) motivate the community to rethink the design criteria and evaluation metrics, and encourage generalizable and efficient SSL techniques beyond task performance. In this paper, we summarize the challenge organization, participating SSL techniques, and the outcomes. After the challenge was open, we have gathered 5 different models submitted to the public-set leaderboard and 9 different models submitted to the hidden-set leaderboard. Among the models on the hidden-set leaderboard, one has been accepted by Interspeech 2022, 4 submitted papers to SLT 2022, and the other four have not yet been pub-lished. They have enormously different model sizes (from 22.50M to 619.82M) and design targets (achieving SOTA, model compression, more robustness, etc.). We hope the challenge evangelizes SSL for the speech community.
SUPERB @ SLT 2022: Challenge on Generalization and Efficiency of Self-Supervised Speech Representation Learning
2. BACKGROUND 2.1. Self-supervised learning (SSL) for speech Self-supervised learning (SSL) is getting popular in the speech area nowadays, after its wide success in Natural Language Processing (NLP) [2,3] and Computer Vision (CV) [4,5]. Based on the design of pretext tasks, which are used to obtain proxy supervision tasks for pre-training networks in SSL, SSL techniques in speech can be classified into three categories: generative approaches, contrastive approaches, and predictive approaches.
SUPERB @ SLT 2022: Challenge on Generalization and Efficiency of Self-Supervised Speech Representation Learning
For generative approaches, the pretext task is to generate data, i.e., reconstruct the input, based on the masked or corrupted input or to predict future input from the past. Approaches in this category consist of APC [14][15][16][17], DeCoAR [18,19], Mockingjay [20,21], TERA [22], MPC [23][24][25] , speech-XLNet [26], NPC [27], and PASE+ [28,29]. Pretext tasks for contrastive approaches are designed to learn a latent space representation, where distance is minimized between the anchor and positive examples while maximized between the anchor and negative. Positive and negative examples are often created by utilizing the distance between input frames and data augmentation. Representative works in contrastive approaches include CPC [30], wav2vec [31,32], and wav2vec 2.0 [10]. Inspired by BERT in NLP, predictive models, such as HuBERT and WavLM [9,33], aim good representations from unmasked input and infer discrete tokens at masked positions. Given the growing popularity of SSL in the speech area, an overview paper [1] and a tutorial [13] have been put to review recent works in this direction and attract more researchers to contribute.
SUPERB @ SLT 2022: Challenge on Generalization and Efficiency of Self-Supervised Speech Representation Learning
The efficacy of SSL techniques is usually evaluated in the following two-step setting. First, networks are pre-trained with pretext tasks designed in SSL techniques. Then the pre-trained networks are appended with optional prediction layers and fine-tuned with various downstream tasks. The fine-tuned results are evaluated towards downstream tasks to measure the efficacy of SSL techniques and the quality of their learned networks. There are established benchmarks for evaluating SSL in common settings, such as GLUE [34] in NLP, VTAB [35] in CV, and CH-MARL in Multimodal [36]. However, researchers in the speech area often adopted more diverse settings [1]. The design of prediction layers and fine-tuning methods are different across works and downstream tasks ranging from ASR and ASV to speech translation (ST) and Speech enhancement (SE).
SUPERB @ SLT 2022: Challenge on Generalization and Efficiency of Self-Supervised Speech Representation Learning
Due to the diversity in evaluation settings, there are increasing efforts to establish common benchmarking settings with shared downstream tasks and datasets. SUPERB [11,12] consolidates settings of prediction layer architectures, hyperparameter spaces, and tunable parameters to provide the SSL community with a benchmarking platform. SUPERB allows researchers to plug in their pre-trained networks and evaluate on 13 different speech tasks by leveraging s3prl toolkit 2 . LeBenchmark [37] provides a benchmark to understand the performance of SSL techniques in French.
SUPERB @ SLT 2022: Challenge on Generalization and Efficiency of Self-Supervised Speech Representation Learning
ZeroSpeech [38] is a challenge to build speech and language understanding with zero expert resources and is also used for analyzing the quality of SSL models [39,40]. These efforts help researchers understand SSL techniques in downstream performance. Since SUPERB has diverse speech processing tasks and uses both popular publicly available datasets and newly generated datasets in those tasks. In this challenge, we extend SUPERB and encourage the community to think further for efficient use of computation resources while building powerful techniques.
SUPERB @ SLT 2022: Challenge on Generalization and Efficiency of Self-Supervised Speech Representation Learning
This challenge benchmarks the efficacy of self-supervised learning (SSL) speech networks in various types of downstream tasks and its computation requirements. To keep the cost affordable and accelerate the development iteration, the challenge chooses Phoneme Recognition (PR), Speaker Identification (SID), Emotion Recognition (ER), Automatic Speech Recognition (ASR), Queryby-Example (QbE), Automatic Speaker Verification (ASV), Speaker Diarization (SD), Source Separation (SS), Speech Enhancement (SE), and Speech Translation (ST) from the SUPERB and extended SUPERB-SG [11,12] tasks. Researchers participate in the challenge by submitting their SSL pre-trained (i.e., upstream) model. Following the same evaluation framework introduced in them, we, the challenge organizers, extract multiple frozen hidden states from the pre-trained SSL model by default and then train a learnable weighted-sum over the hidden states along with the downstream model with predefined architecture task-by-task. Participants are allowed to apply some task-specified pre/post-processing on their upstream model for better adaptation to each dwnstream task.
SUPERB @ SLT 2022: Challenge on Generalization and Efficiency of Self-Supervised Speech Representation Learning
We use two types of metrics to measure the generalizability and computation requirements of SSL speech networks. For the former one, we proposed SUPERB score (superbs) as an overall metric to show how well each SSL technique performs in all downstream tasks. For the latter one, we choose the theoretical multiply-accumulate operation (MACs) and the number of parameters (Params) as indicators for time and space cost.
SUPERB @ SLT 2022: Challenge on Generalization and Efficiency of Self-Supervised Speech Representation Learning
Let st,i be the ith metrics for task t, st,i(u) be the corresponding score of upstream model u, T be the set of tasks, and It be the set of metrics for task t. We first convert all scores to the same scale by linear interpolation with respect to baseline and SOTA. If a task has more than one metric, we also apply an intra-task average. Then the last step is doing an inter-task average and multiplying 1000.
SUPERB @ SLT 2022: Challenge on Generalization and Efficiency of Self-Supervised Speech Representation Learning
To keep the value of SUPERB score static, we obtain the baseline and SOTA from the SUPERB leaderboard snapshot on October 15, 2021, which is the launch of leaderboard. superbs is designed to provide a comprehensive view of model capability and take the difficulty of each task into consideration. If the performances of the SOTA and baseline are close on a task, then a little improvement should be more valuable than a task has a large difference. 
SUPERB @ SLT 2022: Challenge on Generalization and Efficiency of Self-Supervised Speech Representation Learning
Taking advantage of DeepSpeed 3 , an existing deep learning optimization library including a function for profiling neural networks, we use it as the backbone and customize it for profiling networks in this challenge. We implement the estimation of MACs at the software level by wrapping each function / operator in pytorch with an approximation formula, regardless of the real implementations at the hardware level. For feasibility, we select 32 real audios from LibriSpeech [41] test-clean split, chosen equally from short to long. More implementation detail can be found at our fork repository 4 .
SUPERB @ SLT 2022: Challenge on Generalization and Efficiency of Self-Supervised Speech Representation Learning
To facilitate the development of SSL techniques and fair comparison over challenge submissions, we choose to have two datasets for each task. One is public-available, and representative in the corresponding speech processing task, e.g., LibriSpeech [41] used in ASR and PR. The other is newly created by us and held out as hidden-set. The differences between the two datasets are controlled to be only in 1) recording conditions, 2) spoken content / text scripts, 3) speakers, and 4) the amount of labeled data. To build the hidden datasets, we collected English text from selected existing corpora and split text into sentences. We worked with LxT 5 and recruited 60, gender balanced human speakers to read sentences and record the audio. All collected data is processed into subsets of audio between 1 to 5 hours for train, dev, test split of each downstream task. LxT also translated provided text to German for our en-to-de Speech Translation task.
SUPERB @ SLT 2022: Challenge on Generalization and Efficiency of Self-Supervised Speech Representation Learning
We provide two submission tracks for participants to choose from. The public-set track is mainly for development, demonstrating the task designs of our challenge. Participants can tune the hyperparameters and apply any fine-tuning techniques to reach the best performance of their models. Submissions to public track are made by uploading the prediction files. Participants cannot access data in the hidden-set track, and they have to submit their pre-trained model checkpoints and model architecture. We evaluate submissions with a fixed training procedure and sweep over several default learning rates defined by ourselves, which will be a subset of {1, 0.1, ..., 10 −6 , 10 −7 } for each task. To lower the bar of participation and encourage more diverse aspects of efforts, we also allow only evaluation on a subset of tasks, and participants can control whether to present their results on the online leaderboard or not.
SUPERB @ SLT 2022: Challenge on Generalization and Efficiency of Self-Supervised Speech Representation Learning
As an online challenge, we are capable of demonstrating results in more comprehensive ways. The classic table show the score of each task. The scatter chart (Fig. 1) compares efficiency metrics with performance metrics, and can select which one to use by users, visualizing the trade-off between them. The radar chart (Fig. 2) enables fast performance comparison of models in each task.
SUPERB @ SLT 2022: Challenge on Generalization and Efficiency of Self-Supervised Speech Representation Learning
This subsection briefly introduces the models submitted to the public-set track. The models in this subsection are not submitted to the hidden-set track, so their results are not in Table 1 and Fig. 1, except WavLM series and DistilHuBERT, which are evaluated on the hidden-set by the organizers for comparison.
SUPERB @ SLT 2022: Challenge on Generalization and Efficiency of Self-Supervised Speech Representation Learning
WavLM series (WavLM Base, WavLM Base+, WavLM Large) [33]: WavLM used the same pretext task as HuBERT [9]. The WavLM framework proposed an utterance mixing strategy where partially overlapped signals from different speakers are constructed to augment the training data. The pre-training data used for WavLM extends the 60k hours of pre-trained audio used for HuBERT and wav2vec 2.0 to reach a total of 94k hours of audio.
SUPERB @ SLT 2022: Challenge on Generalization and Efficiency of Self-Supervised Speech Representation Learning
data2vec Large [42] 6 : data2vec predicts contextualized latent input representations given the masked view of the input. The data2vec approach was shown to work well for speech representation learning and visual and text representations [42].
SUPERB @ SLT 2022: Challenge on Generalization and Efficiency of Self-Supervised Speech Representation Learning
FaST-VGS+ [43]: FaST-VGS+ is an SSL model that learns to associate raw speech waveforms with semantically related images. It is learned in a multi-task fashion with a masked language modeling objective in addition to the visual grounding objective. It uses wav2vec 2.0 base as its initialization.  DistilHuBERT [44]: DistilHuBERT is trained with a teacherstudent learning framework with knowledge distillation. The student network consists of a subnet followed by some prediction heads, where the subnet is constructed by reducing the number of transformer encoder layers of the HuBERT teacher model. Given an input speech utterance, prediction heads predict the hidden layer representations of some specific layers of the teacher model.
SUPERB @ SLT 2022: Challenge on Generalization and Efficiency of Self-Supervised Speech Representation Learning
LightHuBERT series (LightHuBERT Stage 1 and LightHu-BERT Small) [45]: LightHuBERT is a model compression framework that consists of a once-for-all Transformer, a contextualized latent representation distillation objective, and a two-stage training strategy. Stage one trains the largest architecture of the once-for-all Transformer from scratch via the loss function of the pre-training distillation. Stage two implements the once-for-all training on the supernet initialized by distilled weights. 
SUPERB @ SLT 2022: Challenge on Generalization and Efficiency of Self-Supervised Speech Representation Learning
Here we introduce the SSL models submitted to the hidden-set track. The models without reference are not yet officially published or submitted to any conference when writing this paper.
SUPERB @ SLT 2022: Challenge on Generalization and Efficiency of Self-Supervised Speech Representation Learning
SpeechCLIP [46]: SpeechCLIP is a framework bridging speech and text through images to enhance speech SSL models by leveraging Contrastive Language-Image Pre-training (CLIP), a model pretrained to align parallel image-text data [47]. SpeechCLIP has two versions: parallel and cascaded. The parallel SpeechCLIP aligns speech and CLIP image encoders and implicitly bridges speech and text representations since CLIP's image and text encoders are wellalign. The cascaded SpeechCLIP cascades CLIP's text encoder on top of the speech encoder, forcing the model to output subword embeddings. All the SpeechCLIP models add additional layers on top of HuBERT, and only the additional layers are learned with CLIP. For the SpeechCLIP submissions, the model with "small" in its name is based on HuBERT-base, while "large" means based on HuBERTlarge.
SUPERB @ SLT 2022: Challenge on Generalization and Efficiency of Self-Supervised Speech Representation Learning
Robust SSL [48,49]: Pre-training SSL models with target domain data is an intuitive way to adapt SSL models to another do-main [50]. This submission continuously pre-trained the HuBERTbase model with distorted speech data for some additional steps to enhance robustness. Distorted speech is generated by applying both additive distortions and non-additive distortions, and the details are in [49].
SUPERB @ SLT 2022: Challenge on Generalization and Efficiency of Self-Supervised Speech Representation Learning
Robust SSL with Knowledge distillation [49]: This SSL model uses the teacher-student framework as DistilHuBERT [44].
SUPERB @ SLT 2022: Challenge on Generalization and Efficiency of Self-Supervised Speech Representation Learning
To overcome the problem that DistilHuBERT is especially vulnerable to distorted speech, the teacher model performs continual training as in the last paragraph so that the student model would have a more robust target to learn with, and during knowledge distillation, the teacher and student model have different distorted inputs.
SUPERB @ SLT 2022: Challenge on Generalization and Efficiency of Self-Supervised Speech Representation Learning
ccc-wav2vec 2.0 [51] 7 : ccc-wav2vec 2.0 uses clustering and an augmentation-based cross-contrastive loss as its self-supervised objective. The Cross-Contrastive loss is computed between the encoder output of the original sample and the quantizer output of its augmentation, and vice-versa, bringing robustness to the pre-training strategy. ccc-wav2vec 2.0 achieves up to 15.6% and 12.7% relative WER improvement over the baseline wav2vec 2.0 on the test-clean and test-other sets, respectively, of LibriSpeech, without the use of any language model. It also achieves up to 14.9% relative WER improvement over the baseline wav2vec 2.0 when fine-tuned on Switchboard data.
SUPERB @ SLT 2022: Challenge on Generalization and Efficiency of Self-Supervised Speech Representation Learning
Adding silence [52]: It has also been shown that in the SID task (public-set track) of SUPERB, if the silence ratio is less than 5%, performance will be reduced by about 30% to 50% compared to other cases [52]. The observation inspired a straightforward way to improve the accuracy of speaker-related tasks -adding silence to the utterances without sufficient silence. This submission modifies the preprocessing process by padding silence in front of the utterances before HuBERT-base and HuBERT-large models extract the representations. The lengths of silence are 1/10 of the whole utterances. This submission only modifies the preprocessing without changing the existing SSL models. The submissions are only evaluated on ASV because this approach is only designed to improve speaker-related tasks.
SUPERB @ SLT 2022: Challenge on Generalization and Efficiency of Self-Supervised Speech Representation Learning
MelHuBERT: MelHuBERT is a replication study of HuBERT on Melspectrogram with simplified HuBERT loss. The only difference in model architecture is that MelHuBERT does not have convolutional feature extractor at the beginning. Instead, it directly takes Melspectrogram as input. There are two variants with different frame period, MelHuBERT-10ms and MelHuBERT-20ms. Despite that MelHuBERT-10ms has more MACs comparing to HuBERT due to input sequence length, MelHuBERT-20ms successfully reduce the MACs by 33% by removing the convolutional feature extractor. To further reduce the computational overhead during pre-training, MelHuBERT-10ms and MelHuBERT-20ms only use a smaller batch size of 32.
SUPERB @ SLT 2022: Challenge on Generalization and Efficiency of Self-Supervised Speech Representation Learning
Sequence reduction [53]: The paper [53] investigates variablelength subsampling to reduce the sequence length along the time axis to reduce the computational cost. In variable-length subsampling, a sequence of boundaries is first detected by Continuous Integrateand-Fire (CIF) [54]; vectors within each segment are pooled and passed to the subsequent layers. The paper also investigates using different approaches to guide the learning of CIF. "w2v2u" and "l25" in the submissions refer to different guidance approaches. Please refer to [53] for details. "last layer" means only the last layer output is used in downstream tasks, while "all layers" means the represen-tations from all layers are weighted sum. As the sequence reduction approaches change the sequence lengths, they cannot be evaluated on some tasks.
SUPERB @ SLT 2022: Challenge on Generalization and Efficiency of Self-Supervised Speech Representation Learning
Chimera MelHuBERT v1: This model combines Robust SSL (DistilHuBERT) and MelHuBERT. MelHuBERT has demonstrated the computation requirements of taking Melspectrogram as input while reducing the CNN feature extractor. Chimera MelHuBERT v1 further incorporates this technique in Robust SSL. Specifically, in the robust distillation framework, the teacher model is the robust Hu-BERT, and the student model is the DistilHuBERT with Melspectrogram as input. Therefore, during distillation, only two transformer layers are trained on top of the Melspectrogram with the robust distill loss as the objective. As a result, Chimera MelHuBERT v1 can achieve low MACs and few trainable parameters.
SUPERB @ SLT 2022: Challenge on Generalization and Efficiency of Self-Supervised Speech Representation Learning
Unsupervised ASR (wav2vec-u 2.0): Hidden representations from self-supervised models have been used in the unsupervised ASR task [55][56][57]. The submission considers an adversarial-trained unsupervised ASR model (i.e., wav2vec-u 2.0) as an SSL model. The phoneme posteriorgram is an additional feature for downstream tasks. As phoneme posteriorgram and wav2vec2's representations do not have the same time resolution, we upsample the posteriorgram, which is done just by repeating the vectors, by a factor of three to match the 20ms frame-shift for wav2vec2. The predicted phoneme posteriorgram is concatenated with the wav2vec2 feature as the downstream models' input. The unsupervised ASR model is trained on Librispeech-960 hours with unpaired text from the Librispeech language modeling corpus. Note that the unsupervised ASR model is trained without the auxiliary K-means loss as its original version to improve stability [56].
SUPERB @ SLT 2022: Challenge on Generalization and Efficiency of Self-Supervised Speech Representation Learning
Unsupervised ASR + Phoneme T5: Unsupervised ASR transcribes speech signals into phoneme sequences. This submission leverages the encoder of phoneme-based T5 8 , a variant of T5 that takes phonemicized text as input. Phoneme-based T5 encoder takes the output of unsupervised ASR as input and generates a sequence of representations. The representations generated from the phonemebased T5 encoder are concatenated with the wav2vec2 feature and phone posteriorgram as an additional feature. The model's target is to generate semantic representation from the phoneme-based T5 encoder to improve the spoken language understanding tasks. Only the last hidden representation from the phoneme-based T5 encoder is applied for the additional feature.
SUPERB @ SLT 2022: Challenge on Generalization and Efficiency of Self-Supervised Speech Representation Learning
In the following, we summarize the work that modified the upstream models used in the SUPERB benchmark. The results in these papers cannot be compared with the submission in Table 1 because they have different downstream models.
SUPERB @ SLT 2022: Challenge on Generalization and Efficiency of Self-Supervised Speech Representation Learning
Adapter [58]: This study aims to explore efficient tuning methods for speech self-supervised learning. Adapters are lightweight modules inserted into SSL models. In downstream tasks, the parameters of SSL models are frozen, and only the adapters are trained. The study shows that with an adapter, the performance parity can be achieved with over 90% parameter reduction. The study further finds that the Houlsby adapter [59] is the most efficient in the trade-off between performance and the number of parameters, and the Weighted sum strategy used in the SUPERB challenge is a very suitable efficient method to use in SSL speech tasks.
SUPERB @ SLT 2022: Challenge on Generalization and Efficiency of Self-Supervised Speech Representation Learning
Correlation Pooling [60]: For utterance-level classification tasks in SUPERB (e.g., SID, ER, etc.), the default downstream model in the challenge is to aggregate the speech representations across time by mean pooling. Correlation pooling [60], which extracts correlations between the coefficients of the representations, shows improvements over mean pooling.
SUPERB @ SLT 2022: Challenge on Generalization and Efficiency of Self-Supervised Speech Representation Learning
Here we summarize the general observations in the challenge, including both the public-set and hidden-set tracks. The results of hidden-set track are shown in Table 1 and Fig. 1.
SUPERB @ SLT 2022: Challenge on Generalization and Efficiency of Self-Supervised Speech Representation Learning
The designs of the submitted models vary a lot. WavLM series uses more pre-training data and new augmentation methods. data2vec is a new self-supervised objective. FaST-VGS+ and SpeechCLIP improve SSL with visual information associated with audio. DistilHuBERT, LightHuBERT, MelHuBERT compress the SSL models to reduce the number of parameters, while Sequence reduction shortens the sequence to reduce the computation. Robust SSL (with knowledge distillation) and ccc-wav2vec 2.0 improve the robustness of SSL models. "Adding silence" modifies the audio preprocessing pipeline. Unsupervised ASR (+ Phoneme T5) leverages text unpaired with audio. Chimera MelHuBERT is the integration of different approaches.
SUPERB @ SLT 2022: Challenge on Generalization and Efficiency of Self-Supervised Speech Representation Learning
The submissions have a vast range of MACs and network parameters. Among all the submissions, Chimera MelHuBERT v1 is the model with the least MACs (271G) and network parameters (22.50M). Unsupervised ASR + T5 has the most MACs (5149G) and network parameters (619.82M).
SUPERB @ SLT 2022: Challenge on Generalization and Efficiency of Self-Supervised Speech Representation Learning
On the SUPERB public leaderboard, WavLM Large achieved the SOTA on all the tasks, except ASR (ranked 2nd place), while data2vec achieved the SOTA on ASR. On the hidden-set, considering all the SSL models in terms of superbs and their computation requirements in Fig. 1, for the group with larger MACs, WavLM is still the best; for the small models, "Robust SSL (DistilHuBERT)" is the best. Comparing SSL models with FBANK, all the submissions outperformed FBANK, except in some cases in SS and SE. The observation shows that SSL models are beneficial for most speech processing tasks. However, we need more study on leveraging the SSL models effectively in tasks involving speech generation like SS and SE.
SUPERB @ SLT 2022: Challenge on Generalization and Efficiency of Self-Supervised Speech Representation Learning
The results of incorporating visual information are mixed. The comparison of FaST-VGS+ and its initialization wav2vec 2.0 on the public-set track is in Fig. 2. FaST-VGS+ is especially strong on KS and IC. On KS and IC, it outperforms other models, except WavLM Base+ and WavLM Large, which use much more pretraining data than FaST-VGS+ (1.7k hours v.s. 94k hours). But FaST-VGS+ especially degrades the performance of SID. On the hidden-set track, for the SpeechCLIP models, when the models have the same size, the parallel model and the cascade model have comparable superbs ("cascade small" v.s. "parallel small"). The SpeechCLIP models labeled with "small" adds extra layers on top of HuBERT-base, but their performance is inferior to HuBERT-base for almost all tasks, especially the speaker-related tasks, SID, ASV, and SD, with the same trend as FaST-VGS+. Increasing the model size of SpeechCLIP improves its performance remarkably ("parallel small" v.s. "parallel large"). We found that SpeechCLIP (parallel large) outperformed HuBERT-large on SID, PR, and ASR, achieved a very low PR error rate, only worse than WavLM-base+. We see the benefit of incorporating visual information during pre-training on some downstream tasks, but how to leverage visual information to improve all downstream tasks requires further study 9 .
SUPERB @ SLT 2022: Challenge on Generalization and Efficiency of Self-Supervised Speech Representation Learning
On the public-set track, DistilHuBERT reduces HuBERTbased's size by 75% while retaining the performance on some 9 Since FaST-VGS+ does not submit to the hidden-set track, and Speech-CLIP does not submit to the public-set, we cannot compare SpeechCLIP and FaST-VGS+ side-by-side. downstream tasks, but compared with HuBERT-based, the performance of DistilHuBERT remarkably degrades on ASR and ASV. LightHuBERT small achieves comparable performance to the teacher model (HuBERT-base) in most tasks with a parameter reduction of 29%. The comparison of HuBERT, DistilHuBERT, and LightHuBERT are in Fig. 2. On the hidden-set track, the three submissions regarding sequence reduction approaches have different performances on ER and QbE, but it is hard to conclude which one is the best. The sequence reduction submissions have approximately the same parameter numbers as DistilHuBERT while decreasing the MACs, but their performances on most of the tasks also decrease compared with DistilHuBERT. The results show that we still need more study to reduce representation sequence effectively. Sequence reduction is a promising new research direction to reduce the computational cost of the speech SSL models because its reduction is orthogonal to the existing compression models like DistilHuBERT and LightHuBERT and can integrate with them.
SUPERB @ SLT 2022: Challenge on Generalization and Efficiency of Self-Supervised Speech Representation Learning
The comparison of the robust SSL models and their counterparts on the hidden-set track are shown in Fig. 3. In summary, robust SSL models improve the superbs in the challenge (Robust SSL (HuBERT-base) v.s. HuBERT-base, Robust SSL (Distil-HuBERT) v.s. DistilHuBERT), and they are especially good at SE and SID. Although the testing set in the challenge does not have speech distortion like the original papers proposing the robustness approaches [48,49], making the DistilHuBERT and HuBERT-based more robust by training with distorted speech still increases their superbs in the challenge.
SUPERB @ SLT 2022: Challenge on Generalization and Efficiency of Self-Supervised Speech Representation Learning
On the hidden-set track, adding silence to the front of Hu-BERT improves the performance of ASV. The results support the hypothesis that HuBERT uses the representations corresponding to silence to store speaker information [52].
SUPERB @ SLT 2022: Challenge on Generalization and Efficiency of Self-Supervised Speech Representation Learning
Then, we compare all the submissions regarding unsupervised ASR on the hidden-set track. Unsupervised ASR is based on wav2vec 2.0-large, so it is reasonable to compare the two models. Unsupervised ASR improves PR, SID, and ER, only slightly degrading in ASR, SD, and ST. Unsupervised ASR has outstanding SID performance ranking at the 2nd in the Table of 1), only worse than WavLM-large. Moreover, in terms of the performance of ER, unsupervised ASR is only worse than WavLM-large and Unsupervised ASR plus T5. It is reasonable that by adding text information, unsupervised ASR can improve PR, but why it achieved good results on SID and ER is unclear and still under investigation. Adding T5 does not further improve any tasks except ER.
SUPERB @ SLT 2022: Challenge on Generalization and Efficiency of Self-Supervised Speech Representation Learning
SUPERB challenge at SLT 2022 challenges the speech community to build performant SSL techniques, and meanwhile achieve strong task generalizability and computation efficiency. The challenge aims to encourage researchers to consider more design criteria beyond model performance while building SSL techniques, such that SSL can expand its impact in more practical use cases. We review 14 models participating in this challenge, present our discoveries and highlight potential research directions in SSL.
SUPERB @ SLT 2022: Challenge on Generalization and Efficiency of Self-Supervised Speech Representation Learning
We thank National Center for High-performance Computing (NCHC) of National Applied Research Laboratories (NARLabs) in Taiwan and Taiwan Web Service (TWS) for providing computational and storage resources used in this challenge.
Towards End-to-end Unsupervised Speech Recognition
unsupervised speech segmentation is usually far from perfect [10]. This results in cascading errors from segmentation and other feature engineering steps. To address these issues, we propose an endto-end system, which, through a better architecture, is able to replace all audio-side pre-processing steps with a single speech-to-phoneme neural network.
Towards End-to-end Unsupervised Speech Recognition
In addition to relaxing the aforementioned limitation, we also improve the unsupervised ASR training step. The core idea of this step is to train a model which maps features extracted from a model pre-trained on unlabeled speech data into phoneme sequences by restricting the output vocabulary distribution and n-gram structures to be similar to real phoneme sequences. Existing solutions for imposing such restrictions include adversarial training [4] and empirical output distribution matching [5]. However, these methods do not ensure that the output is closely related to the spoken content. To make up for this deficiency, we propose a self-supervised auxiliary objective function that provides content-based guidance to the model by utilizing pseudo labels derived from the raw audio signal.
Towards End-to-end Unsupervised Speech Recognition
To summarize our work, we propose wav2vec-U 2.0 -an unsupervised ASR framework with an end-to-end style training and an improved objective function. In our experiments, we validate the effectiveness of the pipeline simplification and the proposed modification of the objective function. We show that wav2vec-U 2.0 performs better than the existing unsupervised frameworks across different languages from high-resource to low-resource. We also combine wav2vec-U 2.0 with selftraining strategies [11,6] to compare against prior works and supervised systems. The implementation is based on fairseq [12] and the code will be released soon.
Towards End-to-end Unsupervised Speech Recognition
Liu et al. [4] first achieved unsupervised phoneme recognition on TIMIT [13], a small and clean English dataset for proof-of-concept experiments. Their system takes MFCC representations of speech as the input and performs the pre-processing step combining speech segmentation with (a separately trained) segmental audio word2vec [7] and feature discretization with k-means clustering. Using the segmented discrete features, they train a phone-based ASR system through adversarial training inspired by Generative Adversarial Network (GAN) [14]. In the adversarial training framework, the ASR model serves as the generator which transcribes the discrete index sequence into phone sequence with a learnable embedding table followed by softmax activation. A 2-layer convolution-based discriminator is trained to distinguish the output of the generator and real phone sequences. The goal of the generator is to output phone sequences that cannot be distinguished by the discriminator. By iteratively training the two modules, the generator can learn to map the pre-processed feature sequences into phone sequences in an unsupervised manner.
Towards End-to-end Unsupervised Speech Recognition
Different from the prior work, Baevski et al. [6] proposed wav2vec-U ( Fig. 1) that first achieved unsupervised speech recognition on benchmark speech recognition datasets in many different languages. Wav2vec-U takes speech representations extracted from the wav2vec 2.0 model [15] as the input instead of human-defined features. Such representations can be obtained by self-supervised training on audio and have been proven to be much more effective than human-defined features for ASR.
Towards End-to-end Unsupervised Speech Recognition
For the pre-processing step, wav2vec-U applies a sequence of feature engineering on the speech representations: k-means clustering input frames, reducing the representation dimension with PCA, merging adjacent PCA features that share the same k-means cluster, and finally, mean pooling features from adjacent pairs of timesteps to better match the length of the phone sequences. To train the ASR model taking the pre-processed features as input, wav2vec-U followed the adversarial training approach with a single layer CNN generator and a two-layer discriminator.
Towards End-to-end Unsupervised Speech Recognition
Since the proposed wav2vec-U 2.0 model is closely related to wav2vec-U, we detail the prior work in Section 3.1 and highlight our contributions as follows: 1) we remove the need for the audio pre-processing step, and 2) we introduce an auxiliary objective that improves the segment transcriber.  Figure 1: Wav2vec-U [6]. The input wav2vec2.0 feature is pre-processed before feeding into the generator as described in Section 2.2. The generator is optimized through adversarial training against the discriminator as described in Section 3.1.  Figure 2: Proposed wav2vec-U 2.0. The generator takes raw wav2vec 2.0 feature as input without pre-processing step as described in Section 3.2. In addition to adversarial training, an auxiliary selfsupervised objective is introduced with pseudo label derived from the raw waveform as described in Section 3.3.
Towards End-to-end Unsupervised Speech Recognition
Though training unsupervised ASR through adversarial learning is feasible, results have shown that the performance of such models is usually far from supervised models. To close the gap, Chen et al. [11] discovered that GAN training can be harmonized with Hidden Markov Models (HMMs), iteratively training GAN and HMM with pseudo labels provided by the previous step and this can improve the system performance by over 30% after 3 iterations. Following this method, wav2vec-U proposed a 3-stage pipeline, training GAN followed by HMM, followed by fine-tuning pre-trained wav2vec2.0 models. For the HMM training, the unlabeled speech used for adversarial training is pseudo-labeled by the generator along with a phoneme-to-phoneme Weighted Finite-State Transducers (WFST; Mohri 16,Mohri et al. 17) to incorporate a phone-based language model for decoding. Next, the unlabeled speech is re-labeled with the HMM using a phoneme-to-word WFST to incorporate a word-based language model for decoding. Finally, the HMM-labeled data is used to fine-tune a pre-trained wav2vec2.0 model with the target being letters, decoded with a letter-to-word WFST. The self-training strategy significantly boosted the performance of the unsupervised system, making it comparable to the state-of-the-art fully-supervised system from 2019 [18]. In this work, we followed the same self-training approach to compare wav2vec-U 2.0 against existing ASR systems.
Towards End-to-end Unsupervised Speech Recognition
To introduce the proposed wav2vec-U 2.0, we begin with the adversarial training defined in wav2vec-U ( Fig. 1; Baevski et al. 6). Given X the sequence of pre-processed features (described in Section 2.2 and Fig. 1) and Y u the phonemized unpaired text sequence, the goal of adversarial training is to train a Generator G which transcribes X into a phone sequence that cannot be distinguished from Y u by a discriminator C. The full objective function is defined as
Towards End-to-end Unsupervised Speech Recognition
where the first two terms are the standard GAN objective [14]; the third term
Towards End-to-end Unsupervised Speech Recognition
is the gradient penalty on discriminator calculated with respect to a sample mixing real and fake input for better convergence [19] where α is the mixing weight sampled from uniform distribution; the fourth term
Towards End-to-end Unsupervised Speech Recognition
is the smoothness penalty to penalize locally inconsistent phone prediction from the generator where p t is the unnormalized generator output at time t; the fifth term
Towards End-to-end Unsupervised Speech Recognition
is the phoneme diversity loss encouraging a diversed output over the phone set by maximizing entropy of the generator output distribution averaged over every frame in a mini-batch B; and λ, γ, η are the weights for each term.
Towards End-to-end Unsupervised Speech Recognition
As mentioned in Section 1, existing unsupervised ASR systems including wav2vec-U rely on handcrafted pre-processing. Interestingly, we discovered that the pre-processing step in wav2vec-U is not necessary.
Towards End-to-end Unsupervised Speech Recognition
Without the need of speech segmentation, a generator taking raw speech representation sequence as the input itself can be sufficient for unsupervised ASR with proper output frequency (i.e. number of phones predicted per second). While there are various ways to control the output frequency of the generator, we found that simply changing the stride of the convolution neural network works well. With the input wav2vec 2.0 features sampled at 50 Hz, we found that outputting a phone candidate at around 16 Hz by using a stride of 3 worked consistently well across all 9 languages tested in this work. To further downsample the 16 Hz generator outputs to results closer to phone sequence, consecutive outputs sharing the identical most possible phone can be treated as a single output by random sampling as done in wav2vec-U. In addition, we found that PCA-based dimensionality reduction can be replaced by a simple batch normalization [20] along the time axis over the wav2vec 2.0 features.
Towards End-to-end Unsupervised Speech Recognition
As illustrated in Fig. 2, wav2vec-U 2.0 simply takes the raw representations extracted from wav2vec 2.0 model as input X instead of pre-processed features as is done in wav2vec-U (Fig. 1). These modifications allow wav2vec-U 2.0 to be trained without the need of a complicated pre-processing pipeline. More importantly, learning the mapping from speech representations to phone sequences in an end-to-end style avoids the defects brought by hand-crafted pre-processing and performs better as demonstrated in Section 4.2.
Towards End-to-end Unsupervised Speech Recognition
Since speech transcriptions are not available in the unsupervised setting, the generator relies only on adversarial training to learn the mapping between speech representations and phone units. The generator can possibly learn to satisfy the adversarial criterion without learning the correct mappingfor example, by consistently producing the most common n-grams regardless of the input speech.
Towards End-to-end Unsupervised Speech Recognition
To address this issue, we introduce an auxiliary objective function that aids self-supervised learning by reconstructing a pseudo label sequence Z = (z 1 , z 2 , ..., z T ) derived from the input audio. We use pseudo-labels generated by k-means clustering on Mel-frequency cepstral coefficient (MFCC) features with 64 clusters. We found that this choice of pseudo labels works consistently well across different languages, and we show the effect of different choices of pseudo labels in Section 4.3. The auxiliary self-supervised objective for the generator is defined as
Towards End-to-end Unsupervised Speech Recognition
where P G (z t |X), the probability of predicting the pseudo label z t at time t, is obtained by an additional linear transformation with the softmax function from the intermediate layer of the generator as illustrated in Fig. 2. The complete objective function of wav2vec-U 2.0 can be obtained by adding the auxiliary objective to Eq. 1 with a weight δ:
Towards End-to-end Unsupervised Speech Recognition
The benefit of introducing the auxiliary loss is two-fold: a) it provides a content-based regularization to ensure the output of the generator is closely related to the input speech and b) it guides the generator with a more explicit signal by approximating the underlying phone sequence with pseudo labels derived from speech.
Towards End-to-end Unsupervised Speech Recognition
We follow Baevski et al. [6] and perform experiments on 9 different languages from LibriSpeech [21], Multi-Lingual LibriSpeech [22], and CommonVoice [23]. Unpaired text data is phonemized with an off-the-shelf phonemizer for English [24] and other languages ‡ with silence token padded on both sides of each sentence and inserted at word boundaries with a probability of 0.5. LJSpeech [25] is also considered for preliminary experiments on end-to-end models outputting characters. We summarize each dataset below and refer readers to prior work for more details on the dataset pre-processing.
Towards End-to-end Unsupervised Speech Recognition
LibriSpeech & Libri-Light. 960 hours of English speech from LibriSpeech, discarding the transcriptions, is used for adversarial training. In addition, 53.2k hours of speech from Libri-Light is used for self-training. For the unpaired text used to train wav2vec-U 2.0 and language model, we used the official text corpus provided by LibriSpeech excluding the transcriptions of the speech data [26]. Evaluation is done on the standard dev/test sets.
Towards End-to-end Unsupervised Speech Recognition
Multi-Lingual LibriSpeech. For each of the 6 languages (Dutch, French, German, Italian, Portuguese, and Spanish) selected from Multilingual LibriSpeech, 100 hours of speech is used for adversarial training and the official text corpus is used as unpaired text. Evaluation is done on the standard test set.
Towards End-to-end Unsupervised Speech Recognition
CommonVoice. We follow Baevski et al. [6] to select 2 (Kyrgyz and Tatar) out of 38 languages from CommonVoice using 1.8 and 4.6 hours of unpaired speech respectively. For the unpaired text of both languages, we use a mixed text corpus following Baevski et al. [6]. Evaluation is done on the dev/test set following Rivière et al. [27].
Towards End-to-end Unsupervised Speech Recognition
LJSpeech. This is a single-speaker corpus containing 24 hours of English speech. Following prior works [28,29] on weakly-supervised speech recognition, we randomly select 300 utterances for validation and testing each and used the rest as unpaired speech for training. For the unpaired text, we used the same data as the LibriSpeech setup.
Towards End-to-end Unsupervised Speech Recognition
Implementation Details For English speech from LibriSpeech, the input features are extracted from the 15th layer of a pre-trained wav2vec2.0 Large model [15] encoded from the audio of which silences have been removed using an unsupervised voice activity detection tool [30]. For non-English speech, features are extracted from the XLSR-53 [31] model, a multi-lingual version of wav2vec 2.0.
Towards End-to-end Unsupervised Speech Recognition
As illustrated in Fig. 2, the generator is a 2-layer CNN with batch normalization over the input features. We found that initializing the scaling factor of batch normalization to 30 for wav2vec 2.0 Large and 35 for XLSR-53 features is critical for convergence. During adversarial training, when ‡ https://github.com/bootphon/phonemizer 
Towards End-to-end Unsupervised Speech Recognition
13.6 ± 0.9 input wav2vec 2.0 feature 50 ground truth phone sequence ∼10 obtaining the output probability distribution from the generator, consecutive frames that share the same most probable phone are merged into one by random selection.
Towards End-to-end Unsupervised Speech Recognition
The discriminator is a 2-layer CNN that uses the output of the generator or a one-hot representation of the real phone sequence as input. A single scalar indicating whether the probability of the input sequence is real or not is obtained by taking the average over output sequence followed by the sigmoid activation.
Towards End-to-end Unsupervised Speech Recognition
The generator and discriminator are trained for 100k steps in total interleaving the updates of each (50k updates for each) with a fixed learning rate of 5e-5 and 3e-4 respectively. Each batch contains 160 audio samples and 160 randomly selected text samples. With the choice of weights in Eq. 6 being 1.0 / 1.5, 1.5 / 2.5, 0 / 3, 0.3 / 0.5 for λ, γ, η, δ respectively, we run each model with 4 different random seeds and we observe that over 80% of the models will converge during training. Models are selected with the Unsupervised Cross-Validation Metric proposed by Baevski et al. [6].
Towards End-to-end Unsupervised Speech Recognition
Pykaldi [32] is used to decode the phone-based ASR outputs into words. While the generator must be trained with a stride of 3, we found decoding using a language model with a stride of 2 (higher output frequency) leads to a more accurate output.
Towards End-to-end Unsupervised Speech Recognition
We first compare wav2vec-U 2.0 with the closely related prior work wav2vec-U [6] to show the benefit of removing pre-processing steps and introducing the auxiliary objective. Table 1 shows our proposed changes to wav2vec-U (Fig. 1 vs. Fig.2) where changes are applied gradually from step (i) to step (vii).
Towards End-to-end Unsupervised Speech Recognition
Note, steps (i)-(iv) use pooling mechanisms to control the output frequency of the model. Removing these steps results in higher frequency and the model fails to converge to a reasonable accuracy. However, accuracy can be recovered by adjusting the output frequency by changing the stride of the generator (step (ii) & (iv)). The fact that models with higher output frequency (step (i) & (iii)) failed to converge shows that having output frequency close to ground truth (around 10 Hz) is important.
Towards End-to-end Unsupervised Speech Recognition
We also observe a similar performance to the original framework after completely removing all pre-processing (step (iv)) and conclude that pre-processing speech representations in order to segment them into phone-like units are not necessary to obtain a working unsupervised ASR system, which can benefit from learning in an end-to-end fashion.
Towards End-to-end Unsupervised Speech Recognition
Comparing steps (iv) and (v), we see that some feature engineering in form of PCA dimensionality reduction is indeed important for adversarial training to work, but it can be replaced with a simple batch normalization layer as in step (vi).
Towards End-to-end Unsupervised Speech Recognition
Finally, looking at steps (vii) and the final step (viii), we see that introducing the auxiliary selfsupervised objective improves PER significantly. Next, we investigate different options for pseudo labels that serve as an approximation of the underlying phone sequence for the auxiliary loss. We experiment on LibriSpeech but found a similar trend on Multi-Lingual LibriSpeech and CommonVoice.
Towards End-to-end Unsupervised Speech Recognition
To acquire the pseudo label, we need to categorize input features according to some approximation of the underlying phone sequence from speech. We first consider the wav2vec 2.0 vector quantization (VQ) indices § which come together with the input feature extraction. We also consider k-means clustering on wav2vec 2.0 features which are used to detect phone boundary in wav2vec-U. Finally, we consider performing k-mean clustering on the Mel-frequency cepstral coefficients (MFCC) audio features, which have been shown effective for self-supervised speech representation learning [33].
Towards End-to-end Unsupervised Speech Recognition
Note that the choice of the pseudo labels is important for the proposed objective. Table 2 shows the results. VQ indices perform less well than the baseline method where no pseudo labels are used. MFCC clustering yields the best results, leading to a reduction of up to 2.3% PER, which is a 14% relative improvement over the baseline. The number of pseudo label classes can significantly affect results: 64 k-means clusters perform best on both wav2vec2.0 PCA features and the MFCC audio features. Since the pseudo labels are extracted from audio, we believe a smaller label set might not be diverse enough to cover the phoneme inventory of the language and a larger label set will encode redundant information in the raw audio.
Towards End-to-end Unsupervised Speech Recognition
In Table 3, we compare wav2vec-U 2.0 with supervised, semi-supervised, and unsupervised prior work on the well-studied English benchmark dataset LibriSpeech. Without self-training, wav2vec-U 2.0 performs reducing word error rate (WER) by 2% to 4% compared to wav2vec-U. This shows that removing the pre-processing steps and introducing the self-supervised auxiliary objective can lead to improved performance on speech recognition. Next, we combine our method with the 3-stage self-training pipeline described in Section 2.3. Results show that WER can be significantly reduced with self-training, reaching below 3.7% WER on the clean data and 6.3% on the noisy data.
Towards End-to-end Unsupervised Speech Recognition
On the other hand, we observe the gap between wav2vec-U and wav2vec-U 2.0 is much smaller after applying the self-training strategy. This suggested that unsupervised ASR on LibriSpeech might be bottle-necked by the self-training pipeline. Improving the self-training approach can likely further improve the performance of unsupervised ASR. Unsupervised learning + Self-Training wav2vec-U 54k 4-gram 3.4 6.0 3.8 6.5 wav2vec-U 2.0 54k 4-gram 3.5 6.0 3.7 6.3 
Towards End-to-end Unsupervised Speech Recognition
We have also tested our approach on a number of European languages from Multi-lingual LibriSpeech and show the results in Table 4.
Towards End-to-end Unsupervised Speech Recognition
We observe that wav2vec-U 2.0 also outperforms wav2vec-U in this setup, especially on the languages with higher error rate. Overall, the proposed framework is able to improve the average WER on MLS by 8% absolute without self-training. This further demonstrates the benefit of having an end-to-end learning framework with self-supervision.
Towards End-to-end Unsupervised Speech Recognition
The results show that wav2vec-U 2.0 consistently improves over wav2vec-U with self-training. Average WER improves to 15.3% compared to 18.6% for wav2vec-U. This is a significant improvement  
Towards End-to-end Unsupervised Speech Recognition
Other than English and European languages, we experiment on two low-resource Turkic languages in CommonVoice and the results are shown in Table 5. Since we only measure the phone error rate, we only use the HMM stage of the self-training pipeline.
Towards End-to-end Unsupervised Speech Recognition
Compared to the supervised topline using 17 hours of labeled speech, wav2vec-U 2.0 is able to achieve 10.1% / 15.2% PER on Tatar / Kyrgyz using just 4.6 / 1.8 hours of speech. This once again shows the value of developing unsupervised ASR for low-resource languages.
Towards End-to-end Unsupervised Speech Recognition
While wav2vec-U 2.0 removed the need for human-engineered pre-processing steps and pipeline design, it is still not fully end-to-end. Like conventional supervised ASRs and existing unsupervised ASRs, wav2vec-U 2.0 relied on a lexicon which is an expert-defined word-to-phone mapping. Since the phoneme inventory is defined by a set of distinctive sounds, learning a speech-to-phoneme transformation is naturally easier for the model than a speech-to-grapheme mapping or a speech-toword mapping. However, collecting a lexicon for every language can be difficult in practice as it requires expertise in both linguistics and phonetics.
Towards End-to-end Unsupervised Speech Recognition
We, therefore, evaluate wav2vec-U 2.0 without relying on lexicon by replacing all phones (for both adversarial training and language modeling) with letters that can be directly inferred from the text data. Results on LJSpeech with the end-to-end version of wav2vec-U 2.0 are presented in Table 6. We first show that wav2vec-U 2.0, using no labeled data at all, is already outperforming previous works on semi-supervised ASR [28,29] which use small amounts of human-labeled data.
Towards End-to-end Unsupervised Speech Recognition
When switching to an entirely letter-based system without a lexicon, the unit error rate increases substantially. Such degradation is reasonable as the former needs to learn to spell alongside the mapping between acoustic and writing units. Unfortunately, we also discover that the word error rate of the letter-based model is significantly higher than the phone-based model. These results suggest that end-to-end unsupervised ASR could be possible but further research and development are necessary.
Towards End-to-end Unsupervised Speech Recognition
An end-to-end approach for unsupervised ASR is key to increasing applicability to low-resource languages. In this work, we move towards this goal by removing the need for human-engineered pre-processing and by improving accuracy. While wav2vec-U 2.0 is simpler and more accurate, additional work is required to enable a fully end-to-end approach. Future work includes simplifying the self-training pipeline and removing the need for a phonemizer. Possible solutions for the latter are incorporating unsupervised lexicon discovery [41] or training unsupervised ASR with graphemic text units such as letters directly as for supervised systems.