- Cees G. M. Snoek, M. Worring, O. de Rooij, K. V. D. Van De Sande, Rong Yan, Alexander Hauptmann, John R Standards, Smith. 2008. VideOlympics : Real-Time Evaluation of Multimedia Retrieval Systems. Abstract: Alexander G. Hauptmann Carnegie Mellon University Interactive prototypes are often the best way to convince an audience of a new multimedia technology’s possible impact. Because of its dynamic audiovisual nature, a multimedia application demonstration communicates applied science more effectively than a static description in a journal publication would. Ideally, a multimedia demonstrator grasps the audience’s attention by presenting effective results, advanced multimodal interfaces, novel means of user interaction, or combinations thereof. An interactive demo offers researchers a means to engage their audience in a way that they could never achieve with a written manuscript alone. All major multimedia conferences have adopted demo sessions where researchers, equipped with their laptop-installed system, show their demo to the conference attendees on a one-on-one basis. While effective, these demo sessions lack a common denominator. Individual systems provide solutions for different tasks on different data sets, and the conference attendees have to become familiar with the peculiarities of each individual demo. Moreover, a one-on-one demo session makes it infeasible to present a system to the entire conference audience. In addition, due to their lack of focus, the impact of demo sessions on the audience is suboptimal. Establishing focus A good way to establish focus is letting several demo systems solve the same problem on a similar data set. A notable example of such a focused effort takes place at the demo session at the National Institute for Standards and Technology’s (NIST’s) Trecvid workshop (see the ‘‘Trecvid Interactive Video Retrieval Task’’ sidebar). Trecvid promotes progress in video retrieval by providing a large video collection, common retrieval tasks, uniform evaluation procedures, and a forum for researchers interested in comparing their results. In the months preceding the workshop, researchers work on a common retrieval problem. Results are submitted offline and then presented to the benchmark participants during the workshop. In the Trecvid demo session, participants showcase their video-retrieval systems. Because the audience is knowledgeable in the problem area and the task at hand, the result is a lively demo session where the audience gains deep insight in various video-retrieval systems. Unfortunately, the audience at this event is limited to researchers who participate in the evaluation campaign. Of course, the videoretrieval systems are shown at regular demo sessions, but they’re never exposed simultaneously to an audience. A common data set and a common task aid in establishing a focused demo session, but these aren’t sufficient to engage an uninformed audience.
