- Email: [email protected]
Independent or uncorrelated disturbances in linear regression
Economics Letters 19 (19X5) 35-38 North-Holland
INDEPENDENT OR UNCORRELATED An Illustration of the Difference * Harry
H. KELEJIAN
C’nr/~r.\r[,, OJ Mu~$um/,
Collqy
Received 20 February
Recently
models
independence importance
and Ingmar
DISTURBANCES
IN LINEAR REGRESSION
R. PRUCHA
Purh, MD 1074.?.
C’SA
1985
with
powblq
non-normally
and uncorrelatedness
distributed
are not equivalent.
of distinguishing between true independence
disturbances In this paper
hair
altracted
ue gne
more
an example
atwntion. that
For
illubtrater
uch
model\
the p~~tentl,~l
and only unccvrrlatedne~s.
1. Introduction A widely used assumption in econometrics is that the disturbances are normally distributed. Under this assumption. independence and uncorrelatedness are equivalent. In the recent literature. however. more attention has been given to models with non-normally distributed disturbances. ’ For such models. independence and uncorrelatedness are not equivalent. To the best of our kno\vledge. the importance of this difference as it relates to econometric estimation has not been studied. ’ The purpose of this paper is to give (as a stimulant for further research) an example which illustrates the potential Importance of distinguishing between independence and (only) uncorrelatedness of the disturbance terms. In the example, it turns out that if the disturbances are assumed to be independent when they are only uncorrelated. and the regression parameters are correspondingI> estimated, the estimator of the varianceecovariance (VC) matrix involved is inconsistent. A On the other hand. if the disturbances are independent, but they are only assumed to be uncorrelated. efficiency is lost and inferences would be based on an incorrect large sample distribution. It also turna out that the efficiency loss is substantial for certain parameter values.
016S-3765/85,‘$3.30
’ 1985, Elsevier Science Publishera B.V. (North-Holland)
In somewhat more detail, in our illustration we consider the linear regression model under two alternative sets of assumptions on the disturbance distribution. Under the first set of assumptions the joint distribution of the disturbances is taken to be multivariate Student-t with scalar diagonal covariance matrix. Under this assumption the marginal distribution of the disturbances is univariate Student-t and the disturbances are uncorrelated but not independent over time. This model has been analyzed by Zellner (1976). The second set of assumptions is identical to the first, except that the disturbance terms are taken to be independent. The marginal distributions of the disturbances terms are the same in both cases. This model has been analyzed by Prucha and Kelejian (1984). It turns out that the asymptotic VC matrix of the maximum likelihood (ML) estimator corresponding to the independent case is a scalar multiple. say cy. of that corresponding to the uncorrelated disturbance case. We take N as a measure of the importance of the difference between independence and uncorrelatedness of the disturbance terms. Somewhat expectedly. (YI 1. Less expectedly. for extreme values of the distribution parameters, (Yapproaches zero: ’ for more typical parameter values N = 0.80. We do not advocate one assumption over the other. The purpose of this note is simply to demonstrate. using the Student-r as an example. that it may be important to distinguish in non-normal situations between uncorrelated but dependent and independent disturbances. Section 2 contains the specification of the model and the comparison of the two alternative assumptions. Concluding remarks are given in section 3.
2. The model and comparison of two alternative assumptions Consider _)‘ =
the multiple
linear regression
model
xp + u,
(1)
where 1’ is the tz x 1 vector of observations on the dependent variable; X is a non-stochastic tz matrix: whose elements are bounded and has rank k; /3 and u are. respectively. the X-X 1 and n vectors of regression parameters and disturbance terms. Assunrpriotr 1. The disturbance probability density function ’ p,,(qr.
a)=c,(n,
vector u = [u,.
as a multivariate
Student-r
where c(n, (1. 0) = 1~”‘I‘[( ~1+ tr)/2]/{
X
1
with
“‘t’)‘2,
u)[r~+u’u/u’]
I’,
. , u,,]’ is distributed
x k
(2)
17”) ‘I’( c/2)(a’)“~
‘} with u, L’> 0.
It is not difficult to show that (2) implies E(u) = 0, for (1> 1. and E( uu’) = (C,/(I) - 2))a’l,,. for 11> 2. The marginal density of u, is given by (2) with n = 1. Note that p,,( u 1I’. a) z IT:_, p,( u, ) r. a) and therefore the elements of II, are uncorrelated but not independent. Consistent with this observation we find, e.g., cov( r+,‘, 112)= u’~L>‘/(( v - 2)‘( 1’- 4)) # 0 for all t # s. .a ‘I‘hi\ the
rewlt
reflects
dlsturbancr
e\tinutor
remain
’ For c‘nw of Furthermore.
the terms
are
only
of
the aymptotic
uncorrelated:
in
the
variances Independent
of
the
ML
disturbance
estimator ax.
for the
extreme asymptotic
parameter variance!,
values. of
the
when ML
bounded.
exposition. let
unboundedneas
(1. h.
we c’ and
do d
not
distinguish
be scalars:
then
in
our
&/cd
notation stands
between for
the
(uh)/( d).
random
vector
and
the
values
it
takes
on.
H. H. Ke&an,
An assumption frequently made in the econometric literature We hence consider the following alternative assumption: Assumption
densities
2. The elements as under Assumption
p;(uIu,
a)=p,(u,lu,
of u are independent 1. That is, the density
&..P,(%lu,
is that the disturbance
with the same marginal of u is given by
terms are i.i.d.
univariate
Student-r
01.”
The model under Assumption 1 has been analyzed by Zellner tion 1 by several examples and determines that the corresponding is the least squares estimator b = (X’X))‘x’y, and its VC matrix Assuming that Z, = lim, _ mn -‘( X’X) is finite and non-singular, Student-t with limiting distribution of n ‘/*(j - /I) is multivariate @, and u degrees of freedom where
@=-L&,‘,
31
I. R. Prucha / Independent or uncorrelated dmturbances
(1976). Zellner motivates Assumpmaximum likelihood estimator of /? is (u/( u - 2))a*( XX))‘. for u > 2. it is not difficult to show that the a mean vector of zeroes, VC matrix
0 > 2.’
u-2
The model under Assumption 2 has been analyzed by Prucha and_Kelejian (1984). ’ They show that under Assumption 2 the maximum likelihood estimator for p, say /? * is determined by
)* = (xw-‘x’kjJ, iu, = [ 1 + (y, - xJ*)?/u62]
L&‘= diagy=,(@,), -‘,
u~‘=(u+1)(y-X~*)‘(L’-X~*)/n,
where y, and x,. are the t th element and t th row of y and X, respectively. of n”2( s * - p> 1s normal, with zero mean vector and variance-covariance @* =
a@,
~=(u-2)(u+3)/u(u+l),
2
9 The limiting matrix
distribution
(6)
Observe that (Y< 1 for u < co. Therefore, (6) implies that if the disturbances are independent as compared to only uncorrelated we can estimate the structural parameter vector /? more precisely. “’ Let K = (1 - a)/a be the gain factor in precision (in terms of variance-covariances). Then note that as 6 Clearly the first two moments of an arbitrary non-trivial linear combination of the disturbances is the same under both Assumption 1 and 2. Therefore the difference in the two assumptions should manifest itself in the tail thickness of the density of such linear combinations. It turns out that this tail thickness is less if independence is assumed since E,(ufu,?)=04(~/(u-2))*
(*+ 2 the gain factor K approaches infinity: the reason for this is that the element of @* = (( (’ + 3)/( 1: + l))o’z’,’ remain bounded as I: + 2. unlike those of @ = (P/( r - 2))~“‘; ‘. On the other hand. as (7+ 00 the gain factor K approaches zero; this result reflects convergence of the Student-f distribution to the normal as 11--j ~13.Some intermediate values of K (in percent) are given in (7): Degrees of freedom
(1%)
Gain in precision in percent (100~)
3
5
100
25
10 5.1
30 0.6
(7)
Consider now the case in which the disturbance terms are assumed to be independent, but they are only uncorrelated. For this case. ,L?, in (5) would be taken as the estimator of j3, and @* in (6) as the corresponding VC matrix. The Rao-Cramer theorem implies that, for this case, @ is the lower bound VC matrix of any estimator of p. Since @* = 4, with (Y< 1 for 11< W. use of @* would be incorrect and. more specifically, would lead to underestimates of the variance involved. Since (Y+ 0 as 1’+ 2 this underestimate may be serious. Consider, on the other hand. the case in which the disturbance terms are assumed to be only uncorrelated, but in fact they are independent. Then. in this case, the inefficient least squares estimator b would be used, and its large sample distribution would be taken to be the multivariate-t with cl degrees of freedom. However, if the disturbance terms are independent. the correct limiting distribution of the least squares estimator is the multivariate normal.
3. Concluding
remarks
The above analysis demonstrates that the distinction between independence and uncorrelatedness may be important even for distributions that are rather similar to the normal, such as the Student-r. This suggests that it may be important to make this distinction in a wide range of cases. This in turn suggests a need to develop specification tests for uncorrelatedness vs. independence as we move away from the normality assumption. Such tests should. preferably, not be specific to particular distributions.
References Amemiya. T., 1982. Two stage least absolute deviations estimators, Econometrica 50. 6X9-71 1, Gilstein, C.Z. and E.E. Learner. 1983. Robust sets of regression estimates. Econometrica 51. 321-334. Goldfeld. SM. and R.E. Quandt, 1981. Econometric modelling wth non-normal disturhancea, Journal of Econornetrlcs 17. 141-155. Huber, P.J., 1981, Robust statistics (Wiley, New York). Judge. G.G., W.E. Gri,ffitha, R.C. Hill and T.C. Lee. 1980, The theory and practice of econometrics (Wiley, Neu York). Koenkrr, R., 1982, Robust methods in econometrics. Econometric Reviews 1. 213-255. Mansky. C.F.. 1983. Closest empirical distribution estimation, Econometrica 51. 305-320. Mansky, C.F., n.d., Adaptive estimation of nonlinear regression models, Econometric Reviews 3. forthcoming. Prucha. I.R. and H.H. Kelejian, 1984, The structure of simultaneous equation estimators: A generalization towards nonnormal disturbances, Econometrica 52, 721-736. Zellner. A.. 1976, Bayesian and non-Bay&an analysis of the regression model with multivariate Student-t error terms, Journal of the American Statistical Association 71. 400-405.
INDEPENDENT OR UNCORRELATED An Illustration of the Difference * Harry
H. KELEJIAN
C’nr/~r.\r[,, OJ Mu~$um/,
Collqy
Received 20 February
Recently
models
independence importance
and Ingmar
DISTURBANCES
IN LINEAR REGRESSION
R. PRUCHA
Purh, MD 1074.?.
C’SA
1985
with
powblq
non-normally
and uncorrelatedness
distributed
are not equivalent.
of distinguishing between true independence
disturbances In this paper
hair
altracted
ue gne
more
an example
atwntion. that
For
illubtrater
uch
model\
the p~~tentl,~l
and only unccvrrlatedne~s.
1. Introduction A widely used assumption in econometrics is that the disturbances are normally distributed. Under this assumption. independence and uncorrelatedness are equivalent. In the recent literature. however. more attention has been given to models with non-normally distributed disturbances. ’ For such models. independence and uncorrelatedness are not equivalent. To the best of our kno\vledge. the importance of this difference as it relates to econometric estimation has not been studied. ’ The purpose of this paper is to give (as a stimulant for further research) an example which illustrates the potential Importance of distinguishing between independence and (only) uncorrelatedness of the disturbance terms. In the example, it turns out that if the disturbances are assumed to be independent when they are only uncorrelated. and the regression parameters are correspondingI> estimated, the estimator of the varianceecovariance (VC) matrix involved is inconsistent. A On the other hand. if the disturbances are independent, but they are only assumed to be uncorrelated. efficiency is lost and inferences would be based on an incorrect large sample distribution. It also turna out that the efficiency loss is substantial for certain parameter values.
016S-3765/85,‘$3.30
’ 1985, Elsevier Science Publishera B.V. (North-Holland)
In somewhat more detail, in our illustration we consider the linear regression model under two alternative sets of assumptions on the disturbance distribution. Under the first set of assumptions the joint distribution of the disturbances is taken to be multivariate Student-t with scalar diagonal covariance matrix. Under this assumption the marginal distribution of the disturbances is univariate Student-t and the disturbances are uncorrelated but not independent over time. This model has been analyzed by Zellner (1976). The second set of assumptions is identical to the first, except that the disturbance terms are taken to be independent. The marginal distributions of the disturbances terms are the same in both cases. This model has been analyzed by Prucha and Kelejian (1984). It turns out that the asymptotic VC matrix of the maximum likelihood (ML) estimator corresponding to the independent case is a scalar multiple. say cy. of that corresponding to the uncorrelated disturbance case. We take N as a measure of the importance of the difference between independence and uncorrelatedness of the disturbance terms. Somewhat expectedly. (YI 1. Less expectedly. for extreme values of the distribution parameters, (Yapproaches zero: ’ for more typical parameter values N = 0.80. We do not advocate one assumption over the other. The purpose of this note is simply to demonstrate. using the Student-r as an example. that it may be important to distinguish in non-normal situations between uncorrelated but dependent and independent disturbances. Section 2 contains the specification of the model and the comparison of the two alternative assumptions. Concluding remarks are given in section 3.
2. The model and comparison of two alternative assumptions Consider _)‘ =
the multiple
linear regression
model
xp + u,
(1)
where 1’ is the tz x 1 vector of observations on the dependent variable; X is a non-stochastic tz matrix: whose elements are bounded and has rank k; /3 and u are. respectively. the X-X 1 and n vectors of regression parameters and disturbance terms. Assunrpriotr 1. The disturbance probability density function ’ p,,(qr.
a)=c,(n,
vector u = [u,.
as a multivariate
Student-r
where c(n, (1. 0) = 1~”‘I‘[( ~1+ tr)/2]/{
X
1
with
“‘t’)‘2,
u)[r~+u’u/u’]
I’,
. , u,,]’ is distributed
x k
(2)
17”) ‘I’( c/2)(a’)“~
‘} with u, L’> 0.
It is not difficult to show that (2) implies E(u) = 0, for (1> 1. and E( uu’) = (C,/(I) - 2))a’l,,. for 11> 2. The marginal density of u, is given by (2) with n = 1. Note that p,,( u 1I’. a) z IT:_, p,( u, ) r. a) and therefore the elements of II, are uncorrelated but not independent. Consistent with this observation we find, e.g., cov( r+,‘, 112)= u’~L>‘/(( v - 2)‘( 1’- 4)) # 0 for all t # s. .a ‘I‘hi\ the
rewlt
reflects
dlsturbancr
e\tinutor
remain
’ For c‘nw of Furthermore.
the terms
are
only
of
the aymptotic
uncorrelated:
in
the
variances Independent
of
the
ML
disturbance
estimator ax.
for the
extreme asymptotic
parameter variance!,
values. of
the
when ML
bounded.
exposition. let
unboundedneas
(1. h.
we c’ and
do d
not
distinguish
be scalars:
then
in
our
&/cd
notation stands
between for
the
(uh)/( d).
random
vector
and
the
values
it
takes
on.
H. H. Ke&an,
An assumption frequently made in the econometric literature We hence consider the following alternative assumption: Assumption
densities
2. The elements as under Assumption
p;(uIu,
a)=p,(u,lu,
of u are independent 1. That is, the density
&..P,(%lu,
is that the disturbance
with the same marginal of u is given by
terms are i.i.d.
univariate
Student-r
01.”
The model under Assumption 1 has been analyzed by Zellner tion 1 by several examples and determines that the corresponding is the least squares estimator b = (X’X))‘x’y, and its VC matrix Assuming that Z, = lim, _ mn -‘( X’X) is finite and non-singular, Student-t with limiting distribution of n ‘/*(j - /I) is multivariate @, and u degrees of freedom where
@=-L&,‘,
31
I. R. Prucha / Independent or uncorrelated dmturbances
(1976). Zellner motivates Assumpmaximum likelihood estimator of /? is (u/( u - 2))a*( XX))‘. for u > 2. it is not difficult to show that the a mean vector of zeroes, VC matrix
0 > 2.’
u-2
The model under Assumption 2 has been analyzed by Prucha and_Kelejian (1984). ’ They show that under Assumption 2 the maximum likelihood estimator for p, say /? * is determined by
)* = (xw-‘x’kjJ, iu, = [ 1 + (y, - xJ*)?/u62]
L&‘= diagy=,(@,), -‘,
u~‘=(u+1)(y-X~*)‘(L’-X~*)/n,
where y, and x,. are the t th element and t th row of y and X, respectively. of n”2( s * - p> 1s normal, with zero mean vector and variance-covariance @* =
a@,
~=(u-2)(u+3)/u(u+l),
2
9 The limiting matrix
distribution
(6)
Observe that (Y< 1 for u < co. Therefore, (6) implies that if the disturbances are independent as compared to only uncorrelated we can estimate the structural parameter vector /? more precisely. “’ Let K = (1 - a)/a be the gain factor in precision (in terms of variance-covariances). Then note that as 6 Clearly the first two moments of an arbitrary non-trivial linear combination of the disturbances is the same under both Assumption 1 and 2. Therefore the difference in the two assumptions should manifest itself in the tail thickness of the density of such linear combinations. It turns out that this tail thickness is less if independence is assumed since E,(ufu,?)=04(~/(u-2))*
(*+ 2 the gain factor K approaches infinity: the reason for this is that the element of @* = (( (’ + 3)/( 1: + l))o’z’,’ remain bounded as I: + 2. unlike those of @ = (P/( r - 2))~“‘; ‘. On the other hand. as (7+ 00 the gain factor K approaches zero; this result reflects convergence of the Student-f distribution to the normal as 11--j ~13.Some intermediate values of K (in percent) are given in (7): Degrees of freedom
(1%)
Gain in precision in percent (100~)
3
5
100
25
10 5.1
30 0.6
(7)
Consider now the case in which the disturbance terms are assumed to be independent, but they are only uncorrelated. For this case. ,L?, in (5) would be taken as the estimator of j3, and @* in (6) as the corresponding VC matrix. The Rao-Cramer theorem implies that, for this case, @ is the lower bound VC matrix of any estimator of p. Since @* = 4, with (Y< 1 for 11< W. use of @* would be incorrect and. more specifically, would lead to underestimates of the variance involved. Since (Y+ 0 as 1’+ 2 this underestimate may be serious. Consider, on the other hand. the case in which the disturbance terms are assumed to be only uncorrelated, but in fact they are independent. Then. in this case, the inefficient least squares estimator b would be used, and its large sample distribution would be taken to be the multivariate-t with cl degrees of freedom. However, if the disturbance terms are independent. the correct limiting distribution of the least squares estimator is the multivariate normal.
3. Concluding
remarks
The above analysis demonstrates that the distinction between independence and uncorrelatedness may be important even for distributions that are rather similar to the normal, such as the Student-r. This suggests that it may be important to make this distinction in a wide range of cases. This in turn suggests a need to develop specification tests for uncorrelatedness vs. independence as we move away from the normality assumption. Such tests should. preferably, not be specific to particular distributions.
References Amemiya. T., 1982. Two stage least absolute deviations estimators, Econometrica 50. 6X9-71 1, Gilstein, C.Z. and E.E. Learner. 1983. Robust sets of regression estimates. Econometrica 51. 321-334. Goldfeld. SM. and R.E. Quandt, 1981. Econometric modelling wth non-normal disturhancea, Journal of Econornetrlcs 17. 141-155. Huber, P.J., 1981, Robust statistics (Wiley, New York). Judge. G.G., W.E. Gri,ffitha, R.C. Hill and T.C. Lee. 1980, The theory and practice of econometrics (Wiley, Neu York). Koenkrr, R., 1982, Robust methods in econometrics. Econometric Reviews 1. 213-255. Mansky. C.F.. 1983. Closest empirical distribution estimation, Econometrica 51. 305-320. Mansky, C.F., n.d., Adaptive estimation of nonlinear regression models, Econometric Reviews 3. forthcoming. Prucha. I.R. and H.H. Kelejian, 1984, The structure of simultaneous equation estimators: A generalization towards nonnormal disturbances, Econometrica 52, 721-736. Zellner. A.. 1976, Bayesian and non-Bay&an analysis of the regression model with multivariate Student-t error terms, Journal of the American Statistical Association 71. 400-405.