In my recent work I've become interested in working with the minimizer of
$$
\mathbb{E}[(Y-Z)^2] + \lambda P(Z),
$$
$Y$ is an observed random variable, $P$ is a positive-convex penalty function, $Z$ is a measurable random variable with respect to the $\sigma$-algebra generated by $Y$ and $\lambda\geq 0$. 


If $\lambda=0$ this is the MSE and is minimized by the conditional expectation. My question is, is there a well developed theory of "penalized conditional expectation"?  

That is a theory studying the above equation's minimizer? So far I have found nothing really. All help is greatly appreciated. 



Does there exist a Penalized Conditional Expectation?
This is a bit different and doesn't address the question, but hopefully close enough to be useful: we know some things about $\mathbb{E} L(Z,Y)$ for other loss functions $L$.


If and only if $L$ is a Bregman divergence, the mean is the minimizer. This is attributed to Banerjee et al. 2005; try [Mark Reid's blog post](http://mark.reid.name/blog/meet-the-bregman-divergences.html) and its references.


By phrasing things differently and asking to maximize an expected score rather than minimize an expected loss (though there is no formal difference), the setting starts to look like **proper scoring rules** (e.g. Savage 1971, Gneiting and Raftery 2007). These are loss functions for eliciting distributions and they are well-understood; again, essentially just Bregman divergences.


A generalization of this question which is just receiving study recently is *property elicitation*: other loss functions and how they connect to the "property" of the distribution that solves the minimization problem. For example, if the loss is $|Y-Z|$, the minimizer is the median; if it is $\mathbf{1}\_{Y=Z}$ then it is the mode, etc.


For references there I would just point to the [Information Elicitation page](https://www.sites.google.com/site/informationelicitation/) created by Rafael Frongillo and I for this purpose, which has some slides as well. I think the paper to look at first there would be Lambert et al. 2008.


...


(1971) Leonard J. Savage. "Elicitation of personal probabilities and expectations".


(2005) Banerjee et al. "On the Optimality of Conditional Expectation as a Bregman Predictor".


(2007) Tilman Gneiting and Adrian E. Raftery. "Strictly proper scoring rules, prediction, and estimation".


(2008) Nicolas S. Lambert, David M. Pennock, and Yoav Shoham. "Eliciting properties of probability distributions".


