Xycoon logo
Least Squares
Home    Site Map    Site Search    Free Online Software    
horizontal divider
vertical whitespace

Ordinary Least Squares for Multiple Regression

[Home] [Up] [Introduction] [Criterion] [OLS] [Assumptions] [Inference] [Multiple Regression] [MLE] [OLS]

II.II.1 OLS for Multiple Regression

The general linear statistical model can be described in matrix notation as


where y is a stochastic T*1 vector, X is a deterministic (exogenous) T*K matrix, b is a K*1 vector of invariant parameters to be estimated by OLS, e is a T*1 disturbance vector, T is the number of observations in the sample, and K is the number of exogenous variables used in the right hand side of the econometric equation.

It is furthermore assumed that


which is the equivalent matrix expression of the weak set of assumptions under section II.I.3.

The least squares estimator minimizes e'e (the sum of squared residuals).

Solving the normal equations X'Xb = X'y with respect to b yields


where X'X must be a non singular symmetric K*K matrix!

Obviously, the OLS estimator is unbiased


since E(X'e) = 0 by assumption (X is exogenous). This result can be proved quite easily. Note that if X is not exogenously given (thus stochastic) the small sample property of unbiasedness only holds if E(X'e) = 0.

Under the assumption of OLS it can be proved that the covariance matrix of the parameters is


The Gauss-Markov theorem states that if


then any other estimator


has a parameter covariance matrix which is at least as large as the covariance matrix of the OLS parameters


This important theorem therefore proves that the OLS estimator is a best linear unbiased estimator (BLUE).

If D* is a K by T matrix which is independent from y and if


the parameter vector is by definition a linear estimator, and if


then it follows that


Evidently, it follows from (II.II.1-11) that the parameter vector can only be unbiased if DX = 0 and if E(D*e) = 0.

Now what happens to the covariance matrix of this estimator? Obviously, we find


which proves the theorem (on comparing (II.II.1-12) with (II.II.1-5); Q.E.D.).

It can be proved that


which states that the OLS estimator of the variance is unbiased.

The operational formula for calculating the variance is


The prediction of y values outside the sample range is


which is an unbiased prediction function


Example of extrapolation forecast

The point forecast error can be found as


whereas the average forecast error is equal to


The degree of explanation can be measured by the determination coefficient (R-squared) or by the F-statistic which is defined as






where the F statistic is valid for all coefficients except for the constant term.

To test the significance of a subset of m parameters (out of a total number of K) the following F test is used


which is in fact a generalization of (II.II.1-21).

The parameter estimation of a multiple and a simple regression are related to each other. It is also possible to prove that if all explanatory variables are independent (orthogonal), there is no difference between multiple and simple regression coefficients. Assume


then it is easily deduced from (II.II.1-23) that any multiple regression parameter can be computed by


Since it is assumed that the explanatory variables are orthogonal it follows that


and due to the OLS assumptions we know that


On substituting (II.II.I-25) and (II.II.1-26) into (II.II.1-24) we obtain


which proves the theorem (Q.E.D.).

vertical whitespace

Multiple Regression
horizontal divider
horizontal divider

© 2000-2022 All rights reserved. All Photographs (jpg files) are the property of Corel Corporation, Microsoft and their licensors. We acquired a non-transferable license to use these pictures in this website.
The free use of the scientific content in this website is granted for non commercial use only. In any case, the source (url) should always be clearly displayed. Under no circumstances are you allowed to reproduce, copy or redistribute the design, layout, or any content of this website (for commercial use) including any materials contained herein without the express written permission.

Information provided on this web site is provided "AS IS" without warranty of any kind, either express or implied, including, without limitation, warranties of merchantability, fitness for a particular purpose, and noninfringement. We use reasonable efforts to include accurate and timely information and periodically updates the information without notice. However, we make no warranties or representations as to the accuracy or completeness of such information, and it assumes no liability or responsibility for errors or omissions in the content of this web site. Your use of this web site is AT YOUR OWN RISK. Under no circumstances and under no legal theory shall we be liable to you or any other person for any direct, indirect, special, incidental, exemplary, or consequential damages arising from your access to, or use of, this web site.

Contributions and Scientific Research: Prof. Dr. E. Borghers, Prof. Dr. P. Wessa
Please, cite this website when used in publications: Xycoon (or Authors), Statistics - Econometrics - Forecasting (Title), Office for Research Development and Education (Publisher), http://www.xycoon.com/ (URL), (access or printout date).

Comments, Feedback, Bugs, Errors | Privacy Policy