- D. Ellis, Alexander Hauptmann. 2016. DRAFT On Machine Perception of Sound Ph. Abstract: One of the desiderata in machine intelligence is that computers must be able to comprehend sounds as humans do. They must know about various sounds, associate them with physical objects, entities or events, be able to recognize and categorize them, know or discover relationships between them, etc. Successful solutions to these tasks is critical to and can have immediate effect on a variety of applications including content based indexing and retrieval of multimedia data on web which has grown exponentially in past few years. Automated machine understanding of sounds in specific forms such as speech, language and music has become fairly advanced, and has successfully been deployed into systems which are now part of daily life. However, the same cannot be said about natural occurring sounds in our environment. The problem is exacerbated by the sheer vastness of number of sound types, the diversity and variability of sound types, the variations in their structure, and even their interpretation. This dissertation aims to expand the scale and scope of machine hearing capabilities by addressing challenges in recognition of sound events, cataloging large number of potential word phrases that identify sounds and using them to categorize and learn relationships for sounds and by finding ways to efficiently evaluate trained models on large scale. On the sound event recognition front we address the major hindrance of lack of labeled data by describing ways to effectively use the vast amount of data on web. We describe methods for audio event detection which uses only weak labels in the learning process, combines weakly supervised learning with fully supervised learning to leverage labeled data in both forms and finally semi-supervised learning approaches which exploits the vast amount of available unlabeled data on web. Further, we describe methods to automatically mine sound related knowledge and relationships from vast amount of information stored in textual data. The third part once again addresses labeling challenges but now during evaluation phase. Evaluation of trained models on large scale once again requires data labeling. We describe ways to precisely estimate the performance of a trained model under restricted labeling budget. In this proposal we describe the completed works in the above directions. Empirical evaluation shows the effectiveness of the proposed methods. For each component framework described, we discuss expected research directions for successful completion of this dissertation. November 20, 2016 DRAFT
