II.II.1 OLS for Multiple Regression
general linear statistical
model can be described in matrix notation as
y is a stochastic T*1
vector, X is a
deterministic (exogenous) T*K matrix, b
is a K*1 vector of invariant parameters to be estimated by OLS, e
is a T*1 disturbance vector, T is the number of observations in the
sample, and K is the number of exogenous variables used in the right
hand side of the econometric equation.
is furthermore assumed
is the equivalent matrix expression of the weak set of assumptions
under section II.I.3.
least squares estimator minimizes e'e
(the sum of squared residuals).
the normal equations X'Xb
= X'y with respect to b
X'X must be a non
singular symmetric K*K matrix!
the OLS estimator is unbiased
E(X'e) = 0
by assumption (X is
exogenous). This result can be proved quite easily.
Note that if X is not
exogenously given (thus stochastic) the small sample property of
unbiasedness only holds if E(X'e)
the assumption of OLS it can be proved that the covariance matrix of the parameters is
states that if
any other estimator
a parameter covariance matrix which is at least as large as the
covariance matrix of the OLS parameters
important theorem therefore proves that the OLS estimator is a best
linear unbiased estimator (BLUE).
D* is a K by T
matrix which is independent from y and if
parameter vector is by definition a linear estimator, and if
it follows that
it follows from (II.II.1-11) that the parameter vector can only be
unbiased if DX = 0
and if E(D*e) = 0.
what happens to the covariance matrix of this estimator? Obviously,
which proves the theorem (on comparing
(II.II.1-12) with (II.II.1-5); Q.E.D.).
It can be proved that
states that the OLS estimator of the variance is unbiased.
operational formula for calculating the variance
prediction of y
values outside the sample range is
is an unbiased
point forecast error can
be found as
the average forecast error
is equal to
degree of explanation can
be measured by the determination coefficient (R-squared) or by the
F-statistic which is defined as
the F statistic is valid
for all ß coefficients except for the constant term.
test the significance of a subset of m parameters (out of a total number of K) the following F
test is used
is in fact a generalization of (II.II.1-21).
parameter estimation of a multiple and a simple
regression are related to each other. It is also possible to
prove that if all explanatory variables are independent
(orthogonal), there is no difference between multiple and simple
regression coefficients. Assume
it is easily deduced from (II.II.1-23) that any multiple regression
parameter can be computed by
it is assumed that the explanatory variables are orthogonal it
due to the OLS assumptions we know that
substituting (II.II.I-25) and (II.II.1-26) into (II.II.1-24) we
proves the theorem (Q.E.D.).