How much is enough?: Data requirements for statistical NLP
Abstract Feeding training data to statistical representa-tions of language has become a popular past-time for computational linguists, but our un-derstanding of what constitutes a suﬃcientvolume of data remains shadowy. For ex-ample, Brown et al. (1992) used over 500million words of text to train their languagemodel. Is this enough? Could devouring evenmore data further improve the accuracy of themodel learnt? In this paperI explorea numberof issues in the analysis of data requirementsfor statistical NLP systems. 1 A framework forviewing such systems is proposed and a sam-ple of existing works are compared within thisframework. Finally, the ﬁrst steps toward atheory of data requirements are made by es-tablishing an upper bound on the expected er-ror rate of a class of statistical language learn-ers as a function of the volume of trainingdata. 1 Introduction Statistical approaches to natural languageprocessing are becoming increasingly popular,being applied to a wide variety of tasks. Forexample, Weischedel