(Sorry if this is a noob question. I'm a mathematician learning statistics.)


I would like to know if it's sound (or advisable) to test many p-values against the continuous uniform distribution using Kolmogorov-Smirnov or Anderson-Darling.


For example, suppose I want to demonstrate that Rock-Paper-Scissors is a fair game. My H0 is that all players win 1/3 of their games. I get 100 people to play 100 others (varying number of games for each pairing, varying number of total games for each player).


For each player, I can compare their wins against their expected wins (1/3) with chi-squared. This would give me 100 p-values. With so many players, it's probably that a few will not meet my significance level. So I figure I can test the p-values themselves using Anderson-Darling. I would only reject H0 if this last test failed.


Any thoughts are appreciated, thanks.



Can I test many p-values with KS or AD
I think what you're looking for is the higher criticism statistic, which was suggested originally by Tukey: he compares the number of $\alpha$-level significant detections to the expected number of such detections you should expect under the joint null. Higher criticism has some good mathematical properties, as shown by Donoho and Jin (<http://www.jstor.org/stable/3448581>).


In practice, however, I'd probably follow Kjetil's advice and just do FDR control.


