Suppose $X \in \mathbb{R}^{n \times d}$ is a random matrix where $n > d$. Given a matrix $A \in \mathbb{R}^{n \times n}$ such that $AX$ is a zero matrix in expectation, i.e., $\mathbb{E}\_{X}[AX] = 0$. Let $\sigma^2$ be the variance of the norm of $AX$, i.e., $\sigma^2:=\mathbb{V}[\lVert AX \rVert^2\_F]$.


Now I would like to study the property of the random matrix $B=X(X^\top X)^{-1}X^\top AX$. Do we also have $\mathbb{E}[B] = 0$ and can we bound $\mathbb{V}[\lVert B \rVert^2\_F]$ by $\sigma^2$?



expectation and variance of the norm of a random matrix
$\newcommand\si\sigma\newcommand\bm[1]{\begin{bmatrix}#1\end{bmatrix}}$No and no: In general, (i) $EB\ne0$ and (ii) we cannot bound $Var\,\|B\|\_F^2$ by $\si^2$.


E.g., let $n=3$, $d=1$,
$$y\_1:=\bm{1\\ -1\\ -1},\quad y\_2:=\bm{-1\\ 1\\ -1},\quad 
y\_3:=\bm{-1\\ -1\\ 1},\quad y\_4:=\bm{1\\ 1\\ 1},
$$
$$A:=\bm{1&-1&0\\0&1&0\\0&0&1}.$$
Let $Y$ be a random matrix such that $P(Y=y\_j)=1/4$ for $j=1,2,3,4$. Let
$$X:=A^{-1}Y.$$


Then $EAX=EY=0$ and $\|AX\|\_F^2=\|Y\|\_F^2=3$ almost surely (a.s.), so that
$$\si^2=Var\,\|AX\|\_F^2=0.$$


However,
$$EB=\bm{0\\ 0\\ -1/6}\ne0.$$
Also, the values of $\|B\|\_F^2$ at $Y=y\_1$ and at $Y=y\_4$ are the non-equal numbers $2$ and $8/3$, respectively, so that $Var\,\|B\|\_F^2$ is strictly greater than $0=\si^2$. (In fact, $Var\,\|B\|\_F^2=1/9$.)




---


One might feel some affinity for the OP's conjectures. Indeed,


(i) we have $B=P\_X AX$, where $P\_X$ is the matrix of the orthoprojector onto the column space of the matrix $X$. So, if $P\_X$ did not depend on $X$, the conjectured conclusion $EB=0$ would follow from $EAX=0$ and the linearity of any orthoprojector. Also,


(ii) since $P\_X$ is the matrix of an orthoprojector, the equality $B=P\_X AX$ does imply $\|B\|\_F^2\le\|AX\|\_F^2$. However, as shown above, this will not in general imply that $Var\,\|B\|\_F^2\le Var\,\|AX\|\_F^2$.


