- Zhou Yu, Ziyu Xu, A. Black, Alexander I. Rudnicky. 2016. Chatbot Evaluation and Database Expansion via Crowdsourcing. Abstract: Chatbots use a database of responses often culled from a corpus of text generated for a different purpose, for example ﬁlm scripts or interviews. One consequence of this approach is a mismatch between the data and the inputs generated by participants. We describe an approach that while starting from an existing corpus (of interviews) makes use of crowdsourced data to augment the response database, focusing on responses that people judge as inappropriate. The long term goal is to create a data set of more appropriate chat responses; the short term consequence appears to be the identiﬁcation and replacement of particularly inappropriate responses. We found the version with the expanded database was rated signiﬁcantly better in terms of the response level appropriateness and the overall ability to engage users. We also describe strategies we developed that target certain breakdowns discovered during data collection. Both the source code of the chatbot, TickTock, and the data collected are publicly available.
