- M. Walker, Alexander I. Rudnicky, J. Aberdeen, Elizabeth Owen Bratt, J. Garofolo, H. Hastie, Audrey N. Le, B. Pellom, A. Potamianos, R. Passonneau, R. Prasad, S. Roukos, G. Sanders, S. Seneff, D. Stallard. 2002. DARPA communicator evaluation: progress from 2000 to 2001. Abstract: ABSTRACTThis paper describes the evaluation methodology and resultsof the DARPA Communicator spoken dialog system evaluationexperiments in 2000 and 2001. Nine spoken dialog systems in thetravel planning domain participated in the experiments resulting ina total corpus of 1904 dialogs. We describe and compare the ex-perimental design of the 2000 and 2001 DARPA evaluations. Wedescribe how we established a performance baseline in 2001 forcomplex tasks. We present our overall approach to data collection,the metrics collected, and the application of PARADISE to thesedata sets. We compare the results we achieved in 2000 for a num-ber of core metrics with those for 2001. These results demonstratelarge performance improvements from 2000 to 2001 and show thatthe Communicator program goal of conversational interaction forcomplex tasks has been achieved.1. INTRODUCTIONThe objective of the DARPA Communicator project is to supportrapid development of multi-modal speech-enabled dialog systemswith advanced conversational capabilities. Figure 1 illustrates theCommunicator challenge problem; asystem must support complexconversational interaction to complete this task within 10 minutes.You are in Denver, Friday night at 8pm on the road to the air-port after a great meeting. As a result of the meeting, youneed to attend a group meeting in San Diego on Point Lomaon Monday at 8:30, a meeting Tuesday morning at Miramar at7:30, then one from 3-5 pm in Monterey; you need reservations(car, hotel, air).You pull over to the side of the road and whip out your Com-municator. Through spoken dialog (augmented with a displayand pointing), you make the appropriate reservations, discovera conﬂict, and send an e-mail message (dictated) to inform thegroup of the changed schedule. Do this in 10 minutes.Fig. 1. Darpa Communicator Challenge ProblemDuring the course of the Communicator program, we havebeen involved in developing methods for measuring progress to-wards the program goals and assessing advances in the componenttechnologies required to achieve such goals. In previous work, wereport on an exploratory data collection experiment with nine par-ticipating Communicator systems in the travel planning domain
