- Chaitanya Narisetty, Tomoki Hayashi, Ryunosuke Ishizaki, Shinji Watanabe, K. Takeda. 2021. Leveraging State-of-the-art ASR Techniques to Audio Captioning. Abstract: This paper details our work towards leveraging state-of-the-art ASR techniques for the task of automated audio captioning. Our model architecture comprises of a convolution-augmented Transformer (Conformer) encoder and a Transformer decoder to generate nat-ural language descriptions of acoustic signals in an end-to-end manner. To overcome the limited availability of captioned audio samples for model training, we incorporate the Audioset-tags and audio-embeddings obtained from pretrained audio neural networks (PANNs) as an auxiliary input to our model. We train our model over audio samples from Clotho & AudioCaps datasets, and test over Clotho dataset’s validation and evaluation splits. Experimental results indicate that our trained models signiﬁcantly outperform the baseline system from DCASE 2021 challenge task 6.
