- Yiming Yang, Tom Ault, Thomas Pierce. 2000. Combining Multiple Learning Strategies for Effective Cross Validation. Abstract: Parameter tuning through cross-validation becomes very diicult when the validation set contains no or only a few examples of the classes in the evaluation set. We address this open challenge by using a combination of classiiers with diierent performance characteristics to eeectively reduce the performance variance on average of the overall system across all classes, including those not seen before. This approach allows us to tune the combination system on available but less-representative validation data and obtain smaller performance degradation of this system on the evaluation data than using a single-method classiier alone. We tested this approach by applying k-Nearest Neighbor, Rocchio and Language Modeling classiiers and their combination to the event tracking problem in the Topic Detection and Tracking (TDT) domain, where new classes (events) are created constantly over time, and representative validation sets for new classes are often diicult to obtain on time. When parameters tuned on an early benchmark TDT corpus were evaluated on a later TDT benchmark corpus with no overlapping events, we observed a 38-65% reduction in tracking cost (a weighted combination of errors) by the combined system over the individual methods evaluated under the same conditions, strongly suggesting the robustness of this approach as a solution for improving cross-class performance consistency of statistical classi-ers when standard cross-validation fails due to the lack of representative validation sets.
