It is known that the first order error term in the Shannon entropy formula for a binomial distribution is $1/n$ (for example, see the Wikipedia page [Binomial distribution](https://en.wikipedia.org/wiki/Binomial_distribution)), where in the limit $n \to \infty$ the entropy of a binomial approaches the Gaussian one. There are quite technical papers which have calculated the entropy of a binomial distribution exactly; however, I would like to see this error correction $1/n$ to the Gaussian in a simpler way. So, I do the following.


[Here](http://scipp.ucsc.edu/%7Ehaber/ph116C/NormalApprox.pdf), in Equation (4), the binomial distribution in the limit $n \to \infty$ has been written as $$f(x) = \frac{1}{\sqrt{2\pi npq}} e^{- (x - np)^2/2npq} \bigg[1+\mathcal{O}\Big(\frac{1}{\sqrt{n}}\Big)\bigg].$$.


In order the limit $n \to \infty$ to be well-defined and $f(x)$ doesn't vanish, we standardize $f(x)$ by defining $\mu \equiv np$, $\sigma \equiv \sqrt{npq}$, and $g(x) \equiv \sigma f(\sigma x+\mu)$, then we have: $$g(x) = \frac{1}{\sqrt{2\pi}} e^{- x^2/2} \bigg[1+\mathcal{O}\Big(\frac{1}{\sqrt{n}}\Big)\bigg],$$ where the term outside the bracket is a Gaussian with mean $0$ and standard deviation $1$.


Now, we can calculate the differential Shannon entropy of $g(x)$ as $S = -\int dx \, g(x) \ln g(x)$, which reads $$S = \frac{1}{2} \ln 2 \pi e + \mathcal{O}\Big(\frac{1}{\sqrt{n}}\Big).$$


The first term is correctly the entropy of the standard Gaussian but the error is of the order $1 / \sqrt{n}$. If we assume the result quoted in Wikipedia is correct, what is wrong with my approach?


**Edit 1 (Comment on answer 1)**


By the method of moment-generating functions, one can show that the standardized binomial distribution, $B(n, p)$, approaches the moment-generating function of the standard Gaussian, $\mathcal{N}(0,1)$, when $n \to \infty$. Thus, one expects the entropy of the former approaches $(1/2) \ln 2 \pi e$, as $n \to \infty$, I'm looking for the error terms of this expansion.



Obtaining the error term of binomial distribution's entropy from the differential entropy of a Gaussian distribution
The confusion about $1/\sqrt n$ versus $1/n$ corrections is addressed below. First, for reference, let me quote the relevant results, all following from this [source](https://core.ac.uk/download/pdf/82776425.pdf).


The Gaussian entropy is $S\_0=\tfrac{1}{2}\ln(2\pi \sigma^2)+\tfrac{1}{2}$, the leading correction $\delta S$ is of order $1/n$ and depends on the distribution.


• For the binomial distribution, with variance $\sigma^2=npq$ one has
$\delta S=\frac{1}{3n}-\frac{1}{12\sigma^2}.$ This vanishes if $p=q=1/2$, the leading order correction is then of order $1/n^2$.  

• For the Poisson distribution, with variance $\sigma^2=n\lambda$ one has
$\delta S=-\frac{1}{12\sigma^2}.$  

• For the negative binomial distribution, with variance $\sigma^2=nqp^{-2}$ one has
$\delta S =\frac{1}{6n}-\frac{1}{12\sigma^2}.$


More generally, for a discrete distribution with variance $\sigma^2=n\kappa\_2$ and third cumulant $n\kappa\_3$ one has
$$\delta S=-\frac{\kappa\_3^2}{12\kappa\_2^3 n}.$$
If the third cumulant vanishes the correction becomes of higher order in $1/n$.




---


Let me try to make contact with the calculation in the OP, to see where the confusion arises. I rescale $z=(x-np)/\sqrt{npq}$ to write the expansion
$$f(x) = \frac{1}{\sqrt{2\pi npq}} e^{- (x - np)^2/2npq} \bigg[1+\mathcal{O}\Big(\frac{1}{\sqrt{n}}\Big)\bigg]$$
of the binomial distribution around the Gaussian in the form
$$\tilde{f}(z)=\frac{1}{\sqrt{2\pi npq}} e^{-z^2/2}\bigg[1+h(z)n^{-1/2}\bigg].$$
In the large-$n$ limit, the sum over the discrete variable $x$ may be replaced by $\sqrt{npq}\int\_{-\infty}^\infty dz$. We thus find the entropy
$$S=-\sqrt{npq}\int\_{-\infty}^\infty \tilde{f}(z)\ln \tilde{f}(z)\,dz.$$


Now comes the key point: the function $h(z)$ is odd in $z$ (see eq. 4.3 in the cited reference). The order $1/\sqrt n$ correction then vanishes since it is an integral over an odd function, and the next term of order $1/n$ is the first nonzero correction.
