Applying meta-analytical procedures to software engineering experiments
Abstract Deriving reliable empirical results from a single experiment is an unlikely event. Hence to progress multiple experiments must be undertaken per hypothesis and the subsequent results effectively combined to produce a single reliable conclusion. Since results are quantitative in nature, a quantitative conclusion would be the optimal solution. Other disciplines use meta-analytic techniques to achieve this result. The treatise of this paper is: can meta-analysis be successfully applied to current software engineering experiments? The question is investigated by examining a series of experiments, which themselves to investigate — which defect-detection technique is best? Applying meta-analysis techniques to the software engineering data is relatively straightforward, but unfortunately the results are highly unstable, as the meta-analysis shows that the results are highly disparate and do not lead to a single reliable conclusion. The reason for this deficiency is the excessive variation within various components of the experiments. The paper outlines various ideas from other disciplines for controlling this variation and describes a number of recommendations for controlling and reporting empirical work to advance the discipline towards a position, where meta-analysis can be profitably employed.