$\DeclareMathOperator\Cov{Cov}$**Backround of my Question**


Let $Y$ be the response variable, $\mathbb{X}$ be the explanatory variables. The ultimate goal of prediction is finding a function $f^{\*}$ that minimize $\mathbb{E}[(Y - f^{\*}(\mathbb{X})^2)]$, we know that the solution is $f^{\*}(\mathbb{X}) = \mathbb{E}[Y | \mathbb{X}]$. Let $\epsilon = Y - \mathbb{E}[Y | \mathbb{X}]$, then we have
$$
Y = f^{\*}(\mathbb{X}) + \epsilon
$$
where $\mathbb{E}[\mathbb{\epsilon}] = 0$, and $\Cov[\mathbb{X}, \epsilon] = 0$.


So, I think the goal of regression somehow become finding a function $g \approx f^\*$ from some hypothesis space (e.g. $g(\mathbb{X}) = \mathbb{X}\beta$ for linear regression).


One way of defining $g \approx f^\*$ (which can persuade myself that it is a good approximation) is
$$
\mathbb{P}[|f^{\*}(\mathbb{X}) - g(\mathbb{X})| > \eta] < \delta
$$
for small $\eta$ and $\delta$.


**Question**


Given that $g \approx f^{\*}$ (in the sense defined above), if I can show that $\Cov[\widehat{Y}, \widehat{\epsilon}] \approx 0$ ($\widehat{Y} = g(\mathbb{X})$, and $\widehat{\epsilon} = Y - g(\mathbb{X})$), then when I see a residual plot like this
[Residual Plot](https://i.stack.imgur.com/7cAYM.png)


I can persuade myself this might be a signal suggesting that $g(\mathbb{X})$ probably a good approximation.


So, my question is how to show that $g \approx f^\* \Longrightarrow \Cov[\widehat{Y}, \widehat{\epsilon}] \approx 0$?


**My Attemp**


If further assume that $\mathbb{X}$ and $\epsilon$ are independent (is $\Cov[\mathbb{X}, \epsilon] = 0$ sufficient?), then $\Cov[f^{\*}\mathbb(X), \epsilon] = 0$, hence


\begin{equation}
\begin{aligned}
|\Cov[\widehat{Y}, \widehat{\epsilon}]| &= |\Cov[f^\*(\mathbb{X}) + (g(\mathbb{X}) - f^\*\mathbb(X)), (f^\*(\mathbb{X}) - g(\mathbb{X})) + \epsilon]| \\
&\leq |\Cov[f^{\*}(\mathbb{X}), f^{\*}(\mathbb{X}) - g(\mathbb{X})]| + |\Cov[g(\mathbb{X}) - f^{\*}(\mathbb{X}), f^{\*}(\mathbb{X}) - g(\mathbb{X})]| + |\Cov[g(\mathbb{X}) - f^{\*}(\mathbb{X}), \epsilon]| \\
& \leq |\Cov[f^{\*}(\mathbb{X}), f^{\*}(\mathbb{X}) - g(\mathbb{X})]| + |\Cov[g(\mathbb{X}) - f^{\*}(\mathbb{X}), \epsilon]| + |Var[f^\*(\mathbb{X}) - g(\mathbb{X})]|
\end{aligned}
\end{equation}
Intuitively, since $f^{\*}(\mathbb{X}) - g(\mathbb{X}) \approx 0$ with high probability, the last three terms may be small (depend on $\epsilon$ and $\delta$). But I don't know how to do it formally.


**Edit**


By @losifpinelis 's construction, $\exists (Y, \mathbb{X})$, for any given $\eta, \delta, M > 0$, there exists a function $g\_N$ such that
$$
\mathbb{P}[|f^{\*}(\mathbb{X}) - g\_N(\mathbb{X})| > \eta] < \delta
$$
but $|\Cov[g\_N(\mathbb{X}), \epsilon\_N]| > M$. Therefore, this is not a proper definition for this problem.


**My next Question** is "Is $\mathbb{E}[(f^{\*}(\mathbb{X}) - g(\mathbb{X}))^2] < \eta$ a proper definition of goodness of approximation?". That is, does
$$
\mathbb{E}[(f^{\*}(\mathbb{X}) - g(\mathbb{X}))^2] < \eta \Longrightarrow \Cov[\widehat{Y}, \widehat{\epsilon}] < SomeFunction(\eta)
$$
hold?



Justification of the use of residual plot
$\newcommand\ep\epsilon\newcommand{\de}{\delta}$Getting rid of the instances of $\approx$, one can state the question as follows:



> 
> Let $f(X):=E(Y|X)$, where $X$ and $Y$ are random variables (r.v.'s) ($X$ possibly a multivariate one) such that $EY^2<\infty$. Note that $E\ep=0$ and $Cov(f(X),\ep)=0$, where $\ep:=Y-f(X)$.
> 
> 
> Suppose that for a sequence $(g\_n)$ of Borel-measurable functions one has $g\_n(X)\to f(X)$ in probability (as $n\to\infty$). Does it then follow that
> \begin{equation\*}
> Cov(g\_n(X),\ep\_n)\to0, \tag{1}\label{1} 
> \end{equation\*}
> where $\ep\_n:=Y-g\_n(X)$?
> 
> 
> 


The answer to this question is: Of course, not.


Indeed, let e.g. $X$ be a r.v. uniformly distributed on the interval $[-1,1]$, and let $Y:=X$, so that $f(X)=X$ and $\ep=0$.


Let
\begin{equation\*}
 g\_n(X):=X\,1(|X|\ge1/n)+n^2 X\,1(|X|<1/n). 
\end{equation\*}
Then $g\_n(X)\to X=f(X)$ in probability.


However, $Eg\_n(X)=0$, $\ep\_n=Y-g\_n(X)=X-g\_n(X)=(1-n^2)X\,1(|X|<1/n)$, and hence
\begin{equation\*}
 Cov(g\_n(X),\ep\_n)=Eg\_n(X)\ep\_n=n^2(1-n^2)\,EX^2\,1(|X|<1/n) \\ 
 =(1-n^2)/(3n)\to-\infty\ne0. 
\end{equation\*}
So, \eqref{1} fails to hold. $\quad\Box$




---


On the other hand, if $g\_n(X)\to f(X)$ in $L^2$, then it is easy to see that \eqref{1} will hold.


Details on this: We have $\|g\_n(X)-f(X)\|\_2\to0$, where $\|Z\|\_2:=\sqrt{EZ^2}$.
We also have the identity
\begin{equation\*}
 g\_n(X)\ep\_n=g\_n(X)(Y-g\_n(X))=f(X)(Y-f(X))+Y(g\_n(X)-f(X))+2f(X)(f(X)-g\_n(X))-(g\_n(X)-f(X))^2. 
\end{equation\*}
Taking here the expectations and recalling that $Ef(X)(Y-f(X))=Ef(X)\ep=Cov(f(X),\ep)=0$, we get
\begin{equation\*}
 Eg\_n(X)\ep\_n=EY(g\_n(X)-f(X))+2Ef(X)(f(X)-g\_n(X))-E(g\_n(X)-f(X))^2. \tag{2}\label{2}
\end{equation\*}
By the Cauchy--Schwarz inequality and the condition $\|g\_n(X)-f(X)\|\_2\to0$,
\begin{equation\*}
 |EY(g\_n(X)-f(X))|\le\|Y\|\_2\,\|g\_n(X)-f(X)\|\_2\to0. \tag{3}\label{3}
\end{equation\*}
Note all that
\begin{equation}
\|f(X)\|\_2\le\|Y\|\_2<\infty \tag{3.5}\label{3.5} 
\end{equation}
since $f(X)$ is an orthogonal projection of $Y$ in $L^2$. So,
\begin{equation\*}
 |Ef(X)(f(X)-g\_n(X))|\le\|f(X)\|\_2\,\|g\_n(X)-f(X)\|\_2\to0. \tag{4}\label{4}
\end{equation\*}
Also,
\begin{equation\*}
 E(g\_n(X)-f(X))^2=\|g\_n(X)-f(X)\|\_2^2\to0. \tag{5}\label{5}
\end{equation\*}
Collecting \eqref{2}--\eqref{5}, we get
\begin{equation\*}
 Eg\_n(X)\ep\_n\to0. \tag{6}\label{6}
\end{equation\*}
Next, $\ep\_n=Y-g\_n(X)=Y-f(X)+f(X)-g\_n(X)=\ep+f(X)-g\_n(X)$. Taking here the expectations and recalling that $E\ep=0$, we get
\begin{equation}
 |E\ep\_n|=|E(f(X)-g\_n(X))|\le\|g\_n(X)-f(X)\|\_2\to0. \tag{7}\label{7}
\end{equation}
Further, $|Eg\_n(X)-Ef(X)|\le E|g\_n(X)-Ef(X)|\le\|g\_n(X)-Ef(X)\|\_2\to0$, again by the Cauchy--Schwarz inequality and the condition $\|g\_n(X)-f(X)\|\_2\to0$. So,
\begin{equation}
 Eg\_n(X)\to Ef(X), \tag{8}\label{8}
\end{equation}
and $|Ef(X)|\le\|f(X)\|\_2<\infty$ by the Cauchy--Schwarz inequality and \eqref{3.5}.
By \eqref{6}--\eqref{8},
\begin{equation}
 Cov(g\_n(X),\ep\_n)=Eg\_n(X)\ep\_n-Eg\_n(X)\,E\ep\_n\to0-Ef(X)\times 0=0,
\end{equation}




---


In view of \eqref{3}, \eqref{3.5}, \eqref{4}, \eqref{5}, and \eqref{7}, one can also get an explicit bound on $|Cov(g\_n(X),\ep\_n)|$:
\begin{equation}
 |Cov(g\_n(X),\ep\_n)|\le3y\de\_n+2\de\_n^2,
\end{equation}
where $y:=\|Y\|\_2$ and $\de\_n:=\|g\_n(X)-f(X)\|\_2$.


