- Markus Müller, Sebastian Stüker, A. Waibel. 2018. E NHANCING M ULTILINGUAL G RAPHEMIC RNN B ASED ASR S YSTEMS U SING P HONE I NFORMATION. Abstract: : In the past, we proposed the use of Language Feature Vectors (LFVs) to better adapt multilingual speech recognition systems to languages. Recently, we applied this method to RNN/CTC based systems. The recognition accuracy could be improved by modulating the network using LFVs. In this work, we propose an improvement to this approach by reﬁning the network architecture as well as the training strategy. We ﬁrst evaluated multiple methods for applying the modulation. As we are using bi-directional layers, each unit outputs two values, one per direction. Optimizing the combination of the outputs for each direction did improve the performance. In addition, we propose a method for including phonetic information into the training process of a graphemic system. By pre-training layers using phones as targets, the network did learn features to discriminate phones. Adding more layers and a two stage ﬁne-tuning process using graphemes, we ﬁrst forced the network map phonetic features to graphemes. In the second stage, we allowed the network to update the phonetic feature detectors as well. Both methods improved the performance of our setup. We evaluated our setup using a combination of 4 languages (English, French, German, Turkish), with a joint set of acoustic units.
