- Email: [email protected]
A simulation study of the maximum entropy moment matrix
Economics Letters 4 (1979) 0 North-Holland Publishing
A SIMULATION MATRIX *
333-336 Company
STUDY OF THE MAXIMUM ENTROPY MOMENT
James F. MEISNER University of Chicago, IL 60637, USA Received
28 April 1980
Whereas the sample moment matrix may not yield an estimate of the inverse of the population moment matrix because of singularity, the maximum entropy (ME) moment matrix is always positive definite. An experiment indicates superiority of the ME moment matrix over the sample moment matrix even when the tatter is not singular.
1. Introduction Many econometric techniques require the inverse of the moment matrix of certain variables, but the application of the technique is hampered by the singularity of the sample moment matrix caused by an insufficient number of observations relative to the number of variables. Theil and Laitinen (1980) proposed an alternative estimator of the population moment matrix of continuous random variables. Subject to mass- and mean-preserving constraints, their technique fits a distribution to an observed distribution by maximizing the entropy. The cumulative distribution function of the maximum entropy (ME) distribution is continuous everywhere and piecewise linear except for the tails (which are exponential), and the ME covariance matrix is symmetric positive definite; for further details see Theil and Laitinen (1980). Given the importance of estimating the inverse of the population covariance matrix, the objective of this paper is a simulation experiment to investigate the degree of accuracy to which this inverse is approximated by the inverse of the ME covariance matrix and by the inverse of the sample covariance matrix.
2. A simulation
experiment
Let X be an n X p matrix consisting of n stochastically independent observations on p variates. These variates are specified as independent pseudo-normal variates * Research supported Theil and Kenneth
in part by NSF Grant SOC 76-82718. 1 am indebted Laitinen for their many helpful suggestions.
333
to Professor
Henri
334
J.F. Meisner /A simulation study of the ME moment matrix
with zero mean and covariance matrix C = I of order p X p so that C-’ =I. The sample moment matrix is then (l/n) X’X. I used 100 trials (t = 1, .. .. 100) for each specification of y1and p. Let [#I be the inverse of the sample moment matrix of trial t, where i, j = 1, .... p, and let [8/i:‘]be the inverse of the ME moment matrix. The empirical mean of the diagonal elements of the latter matrices for all trials is
(1) I used this mean for bias correction, which yields 100 corrected inverses [$/MP,]. The root mean squared error (RMSE) of the diagonal elements is then
[&
zg($- g2]“*> Pn
and the RMSE of the off-diagonal
(2)
elements is
(3) These three expressions are shown in the first three columns of table 1, followed in the next three columns by the analogous results for the inverse of the sample moment matrix, Sy being replaced by stij . The results show that for n = 20 the inverse of the ME covariance matrix is a distinct improvement over the inverse of the sample covariance matrix, and that the improvement increases with increasing values of p. For n = 30 there is little difference at p = 10, but here too the improvement becomes noticeable and increases when p increases. It is also interesting to note that the RMSEs tend to decline when p increases beyond n. In the case of the inverse of the sample moment matrix the true bias correction is known to be n/(n - p - 1); see, e.g., Johnson and Kotz (1972, p. 164). The RMSEs (2) and (3) with Mpn replaced by n/(n - p - 1) and 6p by $ are shown in the last two columns of table 1. As could be expected, the entries in these columns are close to the corresponding values in the two previous columns, particularly for n = 30. Although the results are obviously conditional on the special form of C which is selected here, it is possible to conclude that there are situations in which the ME covariance matrix is superior to the sample covariance matrix for the estimation of the inverse of the population covariance matrix. The true bias correction for the inverse of the ME covariance matrix is unknown at this stage, but since the correction used is multiplicative, it is irrelevant when we need the inverse of the population covariance matrix up to a multiplicative scalar.
M nP
2.30 2.77 3.14 5.36 8.74 14.28 39.31 63.89 82.37 91.66
1.61 1.11 2.02 2.31 2.18 3.22 6.23 18.49 41.56 19.44
0.353 0.360 0.387 0.401 0.459 0.517 0.669 0.760 0.565 0.412
0.490 0.530 0.630 0.781 0.870 0.831 0.607 0.512 0.468 0.476
RMSE diagonal
0.231 0.246 0.270 0.278 0.315 0.340 0.440 0.504 0.372 0.272
0.345 0.365 0.432 0.502 0.559 0.541 0.370 0.263 0.204 0.168
RMSE offdia
1.58 1.76 2.00 2.31 2.80 3.33 7.19 b b b
2.28 2.80 3.96 6.42 17.12 b b b b b
M nP
0.352 0.363 0.393 0.408 0.471 0.590 0.855 b b b
0.506 0.558 0.726 1.028 1.653 b b b b b
RMSE diagonal
0.232 0.251 0.275 0.284 0.326 0.378 0.542 b b b
0.355 0.392 0.491 0.652 1.025 b b b b b
RMSE off-dia
a Based on the true bias correction n/(n - p - 1). b Not applicable. ITor p > n, S is singular; for p = n and p = n - 1, there is no true bias correction.
p= 10 p= 12 p=14 p=16 p=18 p = 20 p = 25 p = 30 p = 35 p = 40
n = 30 observations
p=lO p=l2 p=14 p=16 p=18 p = 20 p = 25 p = 30 p = 35 p = 40
n = 20 observations
Table 1
0.353 0.362 0.394 0.407 0.484 0.589 0.820 b b b
0.520 0.548 0.719 0.991 1.423 b b b b b
RMSE diagonal a
0.232 0.251 0.276 0.284 0.334 0.311 0.519 b b b
0.365 0.385 0.486 0.627 0.878 b b b b b
RMSB offdia a
336
J.F. Meisner / A simulation study of the ME moment matrix
References Johnson, N.L. and S. Katz, 1972, Distributions in statistics: Continuous multivariate tions (Wiley, New York). Theil, H. and K. Laitineri, 1980, Singular moment matrices in applied econometrics, Krishnaiah, ed., Multivariate analysis-V (North-Holland, Amsterdam) 629-649.
distribuin: P.R.
A SIMULATION MATRIX *
333-336 Company
STUDY OF THE MAXIMUM ENTROPY MOMENT
James F. MEISNER University of Chicago, IL 60637, USA Received
28 April 1980
Whereas the sample moment matrix may not yield an estimate of the inverse of the population moment matrix because of singularity, the maximum entropy (ME) moment matrix is always positive definite. An experiment indicates superiority of the ME moment matrix over the sample moment matrix even when the tatter is not singular.
1. Introduction Many econometric techniques require the inverse of the moment matrix of certain variables, but the application of the technique is hampered by the singularity of the sample moment matrix caused by an insufficient number of observations relative to the number of variables. Theil and Laitinen (1980) proposed an alternative estimator of the population moment matrix of continuous random variables. Subject to mass- and mean-preserving constraints, their technique fits a distribution to an observed distribution by maximizing the entropy. The cumulative distribution function of the maximum entropy (ME) distribution is continuous everywhere and piecewise linear except for the tails (which are exponential), and the ME covariance matrix is symmetric positive definite; for further details see Theil and Laitinen (1980). Given the importance of estimating the inverse of the population covariance matrix, the objective of this paper is a simulation experiment to investigate the degree of accuracy to which this inverse is approximated by the inverse of the ME covariance matrix and by the inverse of the sample covariance matrix.
2. A simulation
experiment
Let X be an n X p matrix consisting of n stochastically independent observations on p variates. These variates are specified as independent pseudo-normal variates * Research supported Theil and Kenneth
in part by NSF Grant SOC 76-82718. 1 am indebted Laitinen for their many helpful suggestions.
333
to Professor
Henri
334
J.F. Meisner /A simulation study of the ME moment matrix
with zero mean and covariance matrix C = I of order p X p so that C-’ =I. The sample moment matrix is then (l/n) X’X. I used 100 trials (t = 1, .. .. 100) for each specification of y1and p. Let [#I be the inverse of the sample moment matrix of trial t, where i, j = 1, .... p, and let [8/i:‘]be the inverse of the ME moment matrix. The empirical mean of the diagonal elements of the latter matrices for all trials is
(1) I used this mean for bias correction, which yields 100 corrected inverses [$/MP,]. The root mean squared error (RMSE) of the diagonal elements is then
[&
zg($- g2]“*> Pn
and the RMSE of the off-diagonal
(2)
elements is
(3) These three expressions are shown in the first three columns of table 1, followed in the next three columns by the analogous results for the inverse of the sample moment matrix, Sy being replaced by stij . The results show that for n = 20 the inverse of the ME covariance matrix is a distinct improvement over the inverse of the sample covariance matrix, and that the improvement increases with increasing values of p. For n = 30 there is little difference at p = 10, but here too the improvement becomes noticeable and increases when p increases. It is also interesting to note that the RMSEs tend to decline when p increases beyond n. In the case of the inverse of the sample moment matrix the true bias correction is known to be n/(n - p - 1); see, e.g., Johnson and Kotz (1972, p. 164). The RMSEs (2) and (3) with Mpn replaced by n/(n - p - 1) and 6p by $ are shown in the last two columns of table 1. As could be expected, the entries in these columns are close to the corresponding values in the two previous columns, particularly for n = 30. Although the results are obviously conditional on the special form of C which is selected here, it is possible to conclude that there are situations in which the ME covariance matrix is superior to the sample covariance matrix for the estimation of the inverse of the population covariance matrix. The true bias correction for the inverse of the ME covariance matrix is unknown at this stage, but since the correction used is multiplicative, it is irrelevant when we need the inverse of the population covariance matrix up to a multiplicative scalar.
M nP
2.30 2.77 3.14 5.36 8.74 14.28 39.31 63.89 82.37 91.66
1.61 1.11 2.02 2.31 2.18 3.22 6.23 18.49 41.56 19.44
0.353 0.360 0.387 0.401 0.459 0.517 0.669 0.760 0.565 0.412
0.490 0.530 0.630 0.781 0.870 0.831 0.607 0.512 0.468 0.476
RMSE diagonal
0.231 0.246 0.270 0.278 0.315 0.340 0.440 0.504 0.372 0.272
0.345 0.365 0.432 0.502 0.559 0.541 0.370 0.263 0.204 0.168
RMSE offdia
1.58 1.76 2.00 2.31 2.80 3.33 7.19 b b b
2.28 2.80 3.96 6.42 17.12 b b b b b
M nP
0.352 0.363 0.393 0.408 0.471 0.590 0.855 b b b
0.506 0.558 0.726 1.028 1.653 b b b b b
RMSE diagonal
0.232 0.251 0.275 0.284 0.326 0.378 0.542 b b b
0.355 0.392 0.491 0.652 1.025 b b b b b
RMSE off-dia
a Based on the true bias correction n/(n - p - 1). b Not applicable. ITor p > n, S is singular; for p = n and p = n - 1, there is no true bias correction.
p= 10 p= 12 p=14 p=16 p=18 p = 20 p = 25 p = 30 p = 35 p = 40
n = 30 observations
p=lO p=l2 p=14 p=16 p=18 p = 20 p = 25 p = 30 p = 35 p = 40
n = 20 observations
Table 1
0.353 0.362 0.394 0.407 0.484 0.589 0.820 b b b
0.520 0.548 0.719 0.991 1.423 b b b b b
RMSE diagonal a
0.232 0.251 0.276 0.284 0.334 0.311 0.519 b b b
0.365 0.385 0.486 0.627 0.878 b b b b b
RMSB offdia a
336
J.F. Meisner / A simulation study of the ME moment matrix
References Johnson, N.L. and S. Katz, 1972, Distributions in statistics: Continuous multivariate tions (Wiley, New York). Theil, H. and K. Laitineri, 1980, Singular moment matrices in applied econometrics, Krishnaiah, ed., Multivariate analysis-V (North-Holland, Amsterdam) 629-649.
distribuin: P.R.