DATASCI 415

Least squares as maximum Gaussian likelihood

This post supplements the linear regression slides. Please see the slides for the setup.

The Gaussian linear regression model is

YX=xN(β0+βx,σ2),

where β0R,βRd are parameters (we assume $\sigma^2$ is known for now). It is a collection of probability distributions: one for each parameter setting/value. The model (implicitly) encodes distributional/probabilistic restrictions on the data because it rules out distributions not in the collection from generating the data. For example, the Gaussian linear regression model rules out non-linear dependencies between the inputs and the output because all distributions in the model posit a linear relation between the inputs and the output. Note that the restrictions imposed by the model may be incorrect; i.e. the data may actually come from a distribution that is not in the model. When the model does not include the data generating distribution, the model is said to be misspecified.

We fit the model to training data by estimating the parameters with maximum likelihood. To motivate maximum likelihood, consider two possible sets of model parameters. One way to decide which set of parameters fits the training data better is to compare the probability of observing the training data from (the distributions associated with) the two sets of parameters: the higher the probability of observing the training data, the better the parameters fit. For the Gaussian linear regression model, the probability of observing the training data {(X1,Y1),,(Xn,Yn)} from parameters (β0,β) is

L(β0,β)P{{(Xi,Yi)}i=1n;β0,β}=i=1nP{YiXi;β0,β}(samples are independent)=i=1n1(2πσ2)12exp{12σ2(Yiβ0βXi)2}(Gaussian lin reg model).

We call L the likelihood; it is a function of (i.e. its inputs are) the model parameters, and it outputs the probability of observing the data from the input parameters. We emphasize that the likelihood is not the same as the pdf of the data; the likelihood is a function of parameter values while the pdf is a function of data values. Although their functional forms are similar, their inputs are different.

The maximum likelihood estimator (MLE) is the parameter value that maximizes the likelihood; i.e. the parameter values that best fit the training data in the sense that the probability of observing the training data from the MLE is higher than the probability of observing the training data from any other parameter values. In practice, we usually maximize the log of the likelihood (called the log-likelihood) or minimize the negative of the log-likelihood (called the negative log-likelihood). For the Gaussian linear regression model, the negative log-likelihood is

logL(β0,β)=i=1n12σ2(Yiβ0βXi)2+12log(2πσ2).

We drop the second term that does not depend on the parameters to obtain the least squares cost function. Thus least squares is the maximum likelihood estimator for the Gaussian linear regression model.

Posted on September 15, 2024 from San Francisco, CA.