Extracting HuBERT hidden units
For analyses similar to the one in the supplemental proceedings of the original wav2vec 2.0 paper about what kinds of phonetic/phonemic information are learned by self-supervised pre-trained models (screenshot below fromAppendix D), I’m hoping to extract information related to the quantization process of the pre-trained wav2vec 2.0 and HuBERT models.Screen Shot 2022-07-10 at 1.56.44 PM1906×1092 125 KBFor wav2vec 2.0, I’ve been able to modify theWav2Vec2GumbelVectorQuantizerclass so that the forward function returns codebook-related information (see implementationhere). Basically, for a given audio file withNframes, I can get theN x 2matrix where the value in each column is the corresponding index in one of the two codebooks for a given speech frame:get_codebook_indices("test.wav")
# [[ 54 269]
#  [146 284]
#  [ 28  18]
#   ... ...
#  [118 111]
#  [146 252]]Looking at themodelling_hubert.pycode, it’s less clear where I might intercept information about which of the ‘hidden units’ each of the speech frames as assigned to. Any pointers would be appreciated! (@patrickvonplaten?)Thanks!
2022-07-26T13:49:59Z
Hey@fauxneticien,This is currently indeed difficult because we haven’t implemented and verified HUBERT’s pretraining algorithm. Note that HUBERT doesn’t use a codebook like Wav2Vec2 does, but instead clusters hidden states for pretraining. Those hidden states don’t necessary have to correspond to acoustic units however.