I'd like to know if there is a concentration inequality for the sample covariance matrix that don't assume the knowledge of the true mean.




---


**Background.**


Given a probability distribution $\mu$ on $\mathbb R^d$, the covariance matrix of $\mu$ is defined as follows:
$$\Sigma := \mathbb E [(x - \bar \mu)(x -\bar \mu)^\top] $$
where $x \sim \mu$ and $\bar \mu = \mathbb E [x]$.


If $X = (x\_1, \cdots x\_m)$ is an i.i.d. sample drawn from $\mu$, then we can define two estimators:
\begin{align\*}
& \hat \Sigma\_1 := \frac1m \sum\_{i=1}^m (x\_i - \bar \mu)(x\_i - \bar \mu)^\top, \text{ where } \bar \mu = \mathbb E\_{x \sim \mu} [x] \\
& \hat \Sigma\_2 := \frac1{m-1} \sum\_{i=1}^m (x\_i - \bar x)(x\_i - \bar x)^\top, \text{ where } \bar x = \frac1m (x\_1 + \cdots x\_m)
\end{align\*}
They both satisfy $\mathbb E\_X \hat \Sigma\_1 = \mathbb E\_X \hat \Sigma\_2 = \Sigma$.


The second estimator $\hat \Sigma\_2$ is of interest because $\bar \mu$ is often not known in practice.




---


**Question.**


I'm interested in the concentration of $\hat \Sigma\_2$ to $\Sigma$ as $m \rightarrow \infty$. More precisely, given a number $t > 0$, I'd like to know whether there exists a constant $A>0$ and a term $\alpha \in (0,1)$ that depend on $\mu$ and $t$ such that
$$\text{Prob}(\| \Sigma - \hat \Sigma\_2 \| \ge t) \le A \cdot \alpha^m$$
where $\|\cdot \|$ is the spectral norm, also known as the 2-norm. (The Frobenius norm is also fine, since for any $d \times d$ matrix $A$, $\|A\| \le \|A\|\_F \le \sqrt{d} \|A\|$)


In the case of the difference $\|\Sigma - \hat \Sigma\_1\|$, such an answer can be obtained using the [matrix Bernstein inequality](https://arxiv.org/abs/1501.01571). However, I'm less sure about $\|\Sigma - \hat \Sigma\_2\|$. I have an idea, which is to use the fact that:
$$\hat \Sigma\_1 - \hat \Sigma\_2 = \frac1{m(m-1)} \sum\_{i\neq j} (x\_i-\bar\mu) (x\_j-\bar\mu)^\top$$
which follows from:
\begin{align\*}
 \hat \Sigma\_2 =& \frac1m \sum\_i x\_i x\_i^\top - \frac1{m(m-1)} \sum\_{i\neq j} x\_i x\_j^\top \\
 =& \frac1m \sum\_i (x\_i-\bar\mu) (x\_i-\bar\mu)^\top - \frac1{m(m-1)} \sum\_{i\neq j} (x\_i-\bar\mu) (x\_j-\bar\mu)^\top \\
 =& \hat \Sigma\_1 - \frac1{m(m-1)} \sum\_{i\neq j} (x\_i-\bar\mu) (x\_j-\bar\mu)^\top
\end{align\*}
But now I'm not sure how to control the sum of the quantities $(x\_i-\bar\mu) (x\_j-\bar\mu)^\top$, which are *not independent*.


This should be a fairly standard question with a standard answer, but I couldn't find an answer to this. A [similar question](https://math.stackexchange.com/questions/2374810/concentration-inequality-for-covariance)'s only answer wasn't addressing my question; it was addressing the case for $\hat \Sigma\_1$.



Concentration inequality for the sample covariance matrix
A variant of this (taking $\frac1m$ instead of $\frac1{m-1}$ for the empirical covariance) is answered on Proposition 2.6, page 10 of <https://arxiv.org/pdf/2110.06357.pdf> .


