 ##### Least Squares  # Ordinary Least Squares for Multiple Regression

#### II.II.1 OLS for Multiple Regression

The general linear statistical model can be described in matrix notation as (II.II.1-1)

where y is a stochastic T*1 vector, X is a deterministic (exogenous) T*K matrix, b is a K*1 vector of invariant parameters to be estimated by OLS, e is a T*1 disturbance vector, T is the number of observations in the sample, and K is the number of exogenous variables used in the right hand side of the econometric equation.

It is furthermore assumed that (II.II.1-2)

which is the equivalent matrix expression of the weak set of assumptions under section II.I.3.

The least squares estimator minimizes e'e (the sum of squared residuals).

Solving the normal equations X'Xb = X'y with respect to b yields (II.II.1-3)

where X'X must be a non singular symmetric K*K matrix!

Obviously, the OLS estimator is unbiased (II.II.1-4)

since E(X'e) = 0 by assumption (X is exogenous). This result can be proved quite easily. Note that if X is not exogenously given (thus stochastic) the small sample property of unbiasedness only holds if E(X'e) = 0.

Under the assumption of OLS it can be proved that the covariance matrix of the parameters is (II.II.1-5)

The Gauss-Markov theorem states that if (II.II.1-6)

then any other estimator (II.II.1-7)

has a parameter covariance matrix which is at least as large as the covariance matrix of the OLS parameters (II.II.1-8)

This important theorem therefore proves that the OLS estimator is a best linear unbiased estimator (BLUE).

If D* is a K by T matrix which is independent from y and if (II.II.1-9)

the parameter vector is by definition a linear estimator, and if (II.II.1-10)

then it follows that (II.II.1-11)

Evidently, it follows from (II.II.1-11) that the parameter vector can only be unbiased if DX = 0 and if E(D*e) = 0.

Now what happens to the covariance matrix of this estimator? Obviously, we find (II.II.1-12)

which proves the theorem (on comparing (II.II.1-12) with (II.II.1-5); Q.E.D.).

It can be proved that (II.II.1-13)

which states that the OLS estimator of the variance is unbiased.

The operational formula for calculating the variance is (II.II.1-14)

The prediction of y values outside the sample range is (II.II.1-15)

which is an unbiased prediction function (II.II.1-16)

###### Example of extrapolation forecast The point forecast error can be found as (II.II.1-17)

whereas the average forecast error is equal to (II.II.1-18)

The degree of explanation can be measured by the determination coefficient (R-squared) or by the F-statistic which is defined as (II.II.1-19)

and (II.II.1-20)

and (II.II.1-21)

where the F statistic is valid for all ß coefficients except for the constant term.

To test the significance of a subset of m parameters (out of a total number of K) the following F test is used (II.II.1-22)

which is in fact a generalization of (II.II.1-21).

The parameter estimation of a multiple and a simple regression are related to each other. It is also possible to prove that if all explanatory variables are independent (orthogonal), there is no difference between multiple and simple regression coefficients. Assume (II.II.1-23) then it is easily deduced from (II.II.1-23) that any multiple regression parameter can be computed by (II.II.1-24)

Since it is assumed that the explanatory variables are orthogonal it follows that (II.II.1-25)

and due to the OLS assumptions we know that (II.II.1-26)

On substituting (II.II.I-25) and (II.II.1-26) into (II.II.1-24) we obtain (II.II.1-27)

which proves the theorem (Q.E.D.).  No news at the moment... © 2000-2018 All rights reserved. All Photographs (jpg files) are the property of Corel Corporation, Microsoft and their licensors. We acquired a non-transferable license to use these pictures in this website.
The free use of the scientific content in this website is granted for non commercial use only. In any case, the source (url) should always be clearly displayed. Under no circumstances are you allowed to reproduce, copy or redistribute the design, layout, or any content of this website (for commercial use) including any materials contained herein without the express written permission.

Information provided on this web site is provided "AS IS" without warranty of any kind, either express or implied, including, without limitation, warranties of merchantability, fitness for a particular purpose, and noninfringement. We use reasonable efforts to include accurate and timely information and periodically updates the information without notice. However, we make no warranties or representations as to the accuracy or completeness of such information, and it assumes no liability or responsibility for errors or omissions in the content of this web site. Your use of this web site is AT YOUR OWN RISK. Under no circumstances and under no legal theory shall we be liable to you or any other person for any direct, indirect, special, incidental, exemplary, or consequential damages arising from your access to, or use of, this web site.

Contributions and Scientific Research: Prof. Dr. E. Borghers, Prof. Dr. P. Wessa
Please, cite this website when used in publications: Xycoon (or Authors), Statistics - Econometrics - Forecasting (Title), Office for Research Development and Education (Publisher), http://www.xycoon.com/ (URL), (access or printout date).