4.4 Summary, One- and Two-Sample Tests
Table IV, adapted from Siegel & Castellan (1988), attempts a summary, demonstrating an apparent wide world of non-parametric tests available for sample comparison; but is this really so? In deciding which test(s) are relevant, the following points should be noted; the decision may be made for you.
The two-sample and k-sample cases each contain columns of tests for related samples, i.e. matched-pair samples or samples of paired replicates. This is common experimental practice in biological and behavioural sciences, where the concept of the control sample is highly developed. It is not so common in astronomy for obvious reasons, but has been exploited on occasion. The powerful tests available to treat such experiments are listed in Table IV, and are described by Siegel & Castellan.
The table runs downward in order of increasing sophistication of measurement level, from ``nominal'' (in which the test objects are simply dumped into classes or bins) through ``ordinal'' (by which objects are ranked or ordered) to ``interval'' (for which objects are placed on a scale, not necessarily numerical, in which distance along the scale matters). None of the tests requires measurement on a ``ratio'' scale, the strongest scale of measurement in which to the properties of the ``interval'' scale a true zero point is added. (Degrees Celsius for temperature measurement represents an ``interval'' scale, and Kelvins a ``ratio'' scale.) An important feature of test selection lies in the level of measurement required by the test: the table is cumulative downward in the sense that at any level of measurement, all tests above this level are applicable.
The efficiency of a particular test depends very much on the individual application. Is the search for goodness-of-fit and general difference, i.e. is this sample from a given population? Are these samples from the same population? Or is it a particular property of the distribution that is of interest, such as the location, e.g. central tendency, mean or median; or the dispersion e.g. extremes, variance, rms. For instance in the two-sample case, the 2 and the Kolmogorov-Smirnov (two-tailed) tests are both sensitive to any type of difference in the two distributions, location, dispersion, skewness, while the U test is reasonably sensitive to most properties, but is particularly powerful for location discrimination. To aid the process of choice, Tables V (single samples) and VI (two samples) summarize the attributes of the one- and two-sample tests.
The choice of test may thus come down to Hobson's. If it does not, however, and two (or more) alternatives remain - beware of this plot of the devil - it might be possible to ``test the tests'' in search of support of a point of view. If such a procedure is followed, quantification of the amount by which significance is reduced must be considered: for a chosen significance level p in a total of N tests, the chance that one test will (randomly) come up significant is Np (1 - p)N-1 Np for small p. The application of efficient statistical procedure has power, but the application of common sense has more.