I've been asking myself this question all the time. Let's say you are given a large set of time series data. Your task is to find out patterns that are meaningful or that you can use for future trend prediction.


The issue now is, how do you know for sure that the patterns you extract are valid, in the sense that they don't suffer from data snooping bias or a case of "torture-the-data-until-it-confesses"?


I can always test my hypothesis as new data comes in, but even if it can predict all the trends in the past, that doesn't mean that it will continue to do so in the future. No?



Data Mining-- how do you know whether the pattern you extract is valid?
A good way to do this is called 'cross-validation'. The data can be divided into three disjoint sets: the training set, the test set and the validation set.


Different models are developed using the training set. A reasonable way to do this is to take different subsets of points from the training set uniform randomly (for instance by generating 100 subsets of size 90 from a training set of 100 points) and to fit your model as you normally would. This will give you a set of models with varying ability to predict. Pick the model that gives the best prediction for the test set. (Taking random subsets has the benefit of eliminating outliers.) Now, having done all this, the training set and the test set have been 'tainted' by the fact that you used them to build your model.


Therefore, finally, the model should be evaluated on the validation set. This set gives a more honest estimate of how well the model generalizes to a new set of data.


There is some sophistication involved in picking the right relative sizes of the sets and in getting more out of less data. For instance, you can look up k-fold cross-validation.


I also support looking at the Elements of Statistical Learning. The second edition is available free online! <http://www-stat.stanford.edu/~tibs/ElemStatLearn/>


