Optimality of the conditional expectation

This post supplements the supervised learning slides. Please see the slides for the setup.

We wish to show that the conditional expectation $\mathbf{E}[Y\mid X=x]$ is the minimum mean squared error (MSE) prediction function of $Y$ from $X$; i.e.

$$\mathbf{E}[(Y-f_{\ast }(X){)}^2]\le \mathbf{E}[(Y-f(X){)}^2]\text{ for any (other) function }f.$$

First, we note that the problem of finding the minimum MSE prediction function of $Y$ from $X$ is equivalent to the problem of finding the minimum MSE constant prediction of $Y_x\triangleq Y\mid X=x$; i.e. finding the constant ${\mu }_x\in \mathbf{R}$ such that

$$\mathbf{E}[(Y_x-{\mu }_x{)}^2]\le \mathbf{E}[(Y_x-c{)}^2]\text{ for any (other) constant }c\in \mathbf{R}.$$

This is because the minimum MSE prediction function $f_{\ast }$ must equal ${\mu }_x$ at $x$; i.e. $f_{\ast }(x)={\mu }_x$. Otherwise, it is possible to reduce the MSE of $f_{\ast }$ by replacing its value at $x$ with ${\mu }_x$:

$$f(x^{\prime })=\{\begin{array}{ll}{\mu }_x& \text{if }{x}^{\prime }=x,\\ f_{\ast }(x^{\prime })& \text{otherwise}.\end{array}$$

Second, we show that ${\mu }_x=\mathbf{E}[Y_x]$ by solving the optimization problem: $\underset{c\in \mathbf{R}}{min}\mathbf{E}[(Y_x-c{)}^2]$. The cost function seems complicated, but it is actually a quadratic function of $c$:

$$\mathbf{E}[(Y_x-c{)}^2]=\mathbf{E}[Y_x^2]-2c\mathbf{E}[Y_x]+c^2.$$

We differentiate the cost function and find its root to deduce ${\mu }_x=\mathbf{E}[Y_x]$. Recalling $f_{\ast }(x)={\mu }_x$ from the first part, we conclude

$$f_{\ast }(x)=\mathbf{E}[Y_x]=\mathbf{E}[Y\mid X=x].$$

Posted on August 30, 2021 from Ann Arbor, MI