Weighted least squares

Weighted least squares (WLS), also known as weighted linear regression, is a generalization of ordinary least squares and linear regression in which knowledge of the unequal variance of observations (heteroscedasticity) is incorporated into the regression. WLS is also a specialization of generalized least squares, when all the off-diagonal entries of the covariance matrix of the errors, are null. The fit of a model to a data point is measured by its residual, , defined as the difference between a measured value of the dependent variable, and the value predicted by the model, : If the errors are uncorrelated and have equal variance, then the function is minimised at , such that . The Gauss–Markov theorem shows that, when this is so, is a best linear unbiased estimator (BLUE). If, however, the measurements are uncorrelated but have different uncertainties, a modified approach might be adopted. Aitken showed that when a weighted sum of squared residuals is minimized, is the BLUE if each weight is equal to the reciprocal of the variance of the measurement The gradient equations for this sum of squares are which, in a linear least squares system give the modified normal equations, When the observational errors are uncorrelated and the weight matrix, W=Ω−1, is diagonal, these may be written as If the errors are correlated, the resulting estimator is the BLUE if the weight matrix is equal to the inverse of the variance-covariance matrix of the observations. When the errors are uncorrelated, it is convenient to simplify the calculations to factor the weight matrix as . The normal equations can then be written in the same form as ordinary least squares: where we define the following scaled matrix and vector: This is a type of whitening transformation; the last expression involves an entrywise division. For non-linear least squares systems a similar argument shows that the normal equations should be modified as follows. Note that for empirical tests, the appropriate W is not known for sure and must be estimated.