Given $s$ successes in $n$ trials, where $p=\frac{s}{n}$, is there a standard way to determine if I have enough data to compute a meaningful statistic? For example, given $s=1, n=10, p=0.1$, the 95% confidence interval ranges from $0.002 < p < 0.445$.


It seems like I could just use the gap between the 95% confidence interval, but it falls apart with rare events (ie: $\frac{s=1}{n=10}$ vs $\frac{s=10}{n=100}$), the 95% confidence of the one on the left is +/- 0.345 while the one on the right is +/- 0.045, yet relative to the estimated probability they are the about the same.


Since I am using the estimated probability to tell if a process is in control in the context of it's historical trend, I don't want to raise an alert on an outlier that's caused by too little data.


Am I trying to solve this the wrong way?


I am very grateful for any guidance that points me in the right direction. Thanks for reading!



Standard way of determining if you have enough data to reliably compute success probability
It doesn't seem clear how you got your alleged 95% confidence interval. Whether you're going about it the wrong way would be easier to assess if you told us how you're going about it.


The most frequently taught method of finding a confidence interval for $p$ works reasonably well when the observed number of successes and failures are reasonably large, but in your case you've observed only one success. That method says that the expected value and variance of the number of successes in $n$ independent trials with probability $p$ of success on each trial are respectively $np$ and $npq=np(1-p)$, and then approximates the distribution by a normal distribution with that mean and that variance, using $s/n$ as the value of $p$ estimated from the sample. The mean and variance of the distribution of the **proportion** of successes are $p$ and $pq/n$. Thus the endpoints of a 95% confidence interval would be
$$
\frac{s}{n} \pm 1.96 \sqrt{\frac{(s/n)(1-s/n)}{n}}.
$$
In your case the interval given by this formula is
$$
-0.086 < p < 0.286
$$
and of course seeing a negative number there should tell you that this isn't very reliable, which you expect since you've observed only one success. But this leaves the question of how you got $0.002$ and $0.445$? Also, have you got something like a null hypothesis based on that historical trend? Conceivably you'd also want to use other prior information.


(Also, stats.stackexchange.com would probably give you more information than this forum would.)


