Estimating the variability of the Stein estimator by bootstrap

Economics Letters North-Holland

293

37 (1991) 293-298

Estimating the variability of the Stein estimator by bootstrap Gang Yi * Indiana

Uniwrsity, Indianapolis, IN 36202, USA

Received Accepted

Theoretical parameters

5 March 1991 2 August 1991

and empirical results indicate of Stein-like estimators.

that the bootstrap

method

provides

a viable

alternative

in estimating

the scale

1. Introduction well known

of the linear statistical and under the maximum likelihood estimator Yet practical applications of are limited and no of precision exists. Conseand hypothesis testing are precluded and its use in many practical situations. In order to in this paper we investigate the performance of estimating the variance of Stein estimators by bootstrap method [Efron and Freedman (198411.

2.

statistical For expository

that

for variants

model and Stein estimators purposes,

consider

orthonormal

linear

statistical

y=xp+e,

or its k-mean

(1)

counterpart

X’y=p+tX’~

or

x=/3+(1,

where y is a (N x 1) vector, X is a (N X k) matrix, X’X= I,, E - N(0, a2Z,,), L’- N(0, u2Zk> and the conditional distribution of x given p is (X 1p> - N(/3, ~‘1,). Let us, since it does not greatly affect the results, assume (T2 is known and without loss of generality is set equal to one. Our * I am grateful 01651765/91/$03.50

to George

Judge

for his helpful

0 1991 - Elsevier

Science

comments. Publishers

Of course,

any remaining

B.V. All rights reserved

errors

are mine.

294

Gang Yi / Estimating

the rariahility

of

the Stein-estimator

objective is to consider alternative estimation rules p^ to estimate the vector p, using the following normalized squared error loss function as a measure of performance:

L(P>If>=(l/k,(p^P)‘(p^-P) =(l/k)

i= I

(PidJ2.

The maximum likelihood-least squares estimator b = X’y - N(/3, Ik) is a best linear estimator, under the squared error loss is minimax and has risk R(P,

b) = E[L(P,

b)] = 1

(2) unbiased

(3)

for all values of p, where the risk is defined as the expected value of the quadratic loss, Stein (1955) showed that this best invariant MLE estimator of the k-dimensional (k 2 3) location vector was inadmissible under the squared error loss and Brown (1966) extended this result and proved inadmissibility for a wide class of distributions and loss functions. James and Stein (1961) demonstrated an estimator

(4) with risk (k - 2)2

R(fh

b”‘) = (l,‘k)E[(b”‘-P)‘(b”‘-p)]

= 1-

k

E[l/X;&],

That is uniformly risk superior to the MLE. In eq. (5), (l/b’b) is distributed as [l/& A\,], where k is the degree of freedom and A = p’/3 is the noncentral parameter of the X2 distribution and the second term on the right-hand side is always positive. The variance-covariance matrix of the Stein estimator (4) is c,,(

1) =

E[ (b”’ - E[ b”‘])(b’”

- E[b’“I)‘]

1

The evaluation of eq. (6) is somewhat cumbersome because it involves the unknown parameter p and expectations of the reciprocal of a noncentral X2 distribution. The Stein estimator (4) shrinks the MLE toward a zero vector. In general, the Stein estimator can be written as (k-2)u2

b”’ = PO +

1I

(b-&)‘(b-p,,)

1

(b-p0)7

(7)

Gang Yi / Estimating

the variability of the Stein-estimator

where PO is the prior information of the location parameter p. use a sample statistic as the plausible shrink point. For instance, want the estimator to shrink toward the grand mean calculated discussion of different variations of Stein-type estimators and non-orthogonal design matrices, see Judge and Bock (1978, Ch.

3. The bootstrap

295

If we do not have a prior, we can in the k-mean problem, we might from the sampling data. For a their more general forms under 10).

approach

Given the complexity of the covariance matrix of the Stein estimator, we propose to use the bootstrap method to obtain a measure of precision. The bootstrap procedure [Efron (1979, 1982)] is a nonparametric method of estimating standard errors by resampling the data. A discussion of the use of this procedure as it relates to the regression model has been provided by Freedman (1981, 1984). Suppose we want to estimate k independent means from k samples. The MLE is simply equal to the mean of each sample respectively. The Stein estimator takes advantage of the so called ‘Stein effect’ by pooling the samples together and shrinks the MLE toward a plausible prior. The scale parameter of the Stein estimator can be estimated by bootstrap procedure as follows. (1) In model (l), the error term is assumed to be normally distributed with mean 0 and variance (T’. Under this assumption, the covariance matrix of the Stein estimator is given by equation (6). As far as bootstrap is concerned, we need only assume that the error terms are independent and have a common unknown distribution F with mean 0 and finite variance. Although F is unknown, we can use an empirical distribution P to approximate F, where P is a discrete uniform distribution with mass (l/N). (2) We are interested in the distribution @ of (b (I) - p) . This distribution depends on both the location parameter /3 and the distribution of the error term F. The bootstrap approach is to approximate @(p, F) by @(b, F). This is done by a resampling process. Consider the statistical model cl), the fitted error term is calculated by e=y-X7,

where b is the MLE. The bootstrap error term e * is obtained by random drawing from components of e N times with replacement. Then e * is used to generate the ‘pseudo-data’ Y * =Xb

The bootstrap

+e*.

(9)

MLE location

parameter

is estimated

by

b* =X'y*. This pseudo

(10)

MLE will be used to calculate

the bootstrap

Stein estimator,

(3) Repeating step 2 many times, say m times, we will have many Stein estimator is estimated by

6; = [l/cm

the

- 1)] : i=l

(b;“*

-_,(‘)*)‘,

b(l)*, by using eq. (7).

b(l)*. Then

the variance

of the

(11)

Gung Yi / Estimuting the ruriuhility

2%

qf the Stein-cstirnutor

where

b(l)*

=

(l/m)

E by*.

(12)

1=l

The theoretical (1981). In terms error should be (8) or the Stein

foundation of bootstrap in a linear regression setting was laid out in Freedman of using the bootstrap method for Stein estimators, there is a question as to which used for resampling. The two possibilities are the MLE error term defined by eq. error term

($1) =y

_Xb”‘.

(13)

In this case using the MLE error is appropriate, because the Stein estimator following nonzero systematic component in its error term:

is biased

W (I)-p) = +-2)E[l/x,Z+2,h]P.

and has the

( 14)

The key idea of the bootstrap method is that the fitted error term used for resampling must preserve the stochastic structure of the original unobservable error term.

purposes

4. Two examples In this section, we report the results of two experiments in which the variances of Stein estimators are estimated by bootstrap. In order to investigate the effectiveness of the bootstrap method, we design the experiment in such a way that the theoretical variances can be calculated and then compared with the bootstrap estimates. From eq. (6), we see the covariance matrix of the Stein estimator depends on the true parameter /3, and the expected value of the reciprocal of the X2 distribution. The latter can be evaluated by the following theorem in Bock, Judge and Yancey ( 1984): Let k and n be positil,e integers such that k > In. !f k is an el,en integer, then

h/Z-.c-2

xf(k/2-s-

4.1. Example

1) ePA”i

I: A linear model with orthonormal

c I=0

( -W)’ t!

(13 :

design

Consider the linear statistical model as specified in equation (l), where y is a (30 X 1) vector, X is a (30 x 6) orthonormal matrix and X’X = I, p = (15, 10, 8, 6, 4, 2)‘, E - NO, c2 = 1). The bootstrap resampling procedure discussed in Section 3 has been followed. Using a bootstrap repetition equal to 10000, the results are summarized in Table 1. The bootstrap estimates of the variances of the Stein estimator (Column 3 of Table 1) are compared to the theoretical variances (Column 2) which are calculated by using equation (6), equation (15) and the true p with the grand mean ?J

Gang Yi / Estimating

Table 1 Estimating

the variance

of Stein estimator

the lwiubility

in a regression

i

True Vadbj”)

Bootstrap

1

0.9217 0.8655 0.8605 0.8655 0.8805 0.9055

0.8768 0.8363 0.8306 0.839X 0.8319 0.8738

2 3 4 5 6

setting

of the Stein-estimator

297

(true CT’ = 1)

Vadb,“‘)

Percentage

of error (o/o)

4.9 3.4 3.5 3.0 5.5 3.5

equal to 8. It is clear from Table 1 that the bootstrap estimates tend to be biased downward. The percentage of errors, which is equal to (true variance-bootstrap variance)/(true variance), range from 3.0% to 5.5%. The bootstrap estimates of the variances in this case seem quite reliable and have an average downward bias of only 4%. 4.2. Example 2: A k-mean problem Next we examine the performance of the bootstrap method for a k-mean problem which involves estimating the true means {p,} of k normal distributions from k independent sample means. We use the data set from Morris (1977) which is concerned with the batting average of baseball players. Ten players are picked up from the original data set and their batting average data are summarized in Table 2. We assume that the common true standard deviation for all players is known and equal to 0.0659. The second column of the table is the true batting avarage, which is equal to the overall batting average of each player for the entire season. The third column is the MLE, which is equal to the sample mean of the first 45 appearances of each player respectively. The fourth column is the Stein estimator defined in equation (7). If we use the quadratic loss function defined in equation (2), the average loss of the Stein estimator is 57.9% smaller than the MLE. Column 5 of Table 2 is the true standard error (SE) of the Stein estimator calculated by equation (6) with the grand mean b equal to 0.2641. Column 6 is the bootstrap estimates of the standard error with repetition also equal to 10000. The percentage of error, (true SE-bootstrap SE)/(true

Table 2 The batting

averages

of ten baseball

players

Player

True

MLE

i

P,

b,

9 IO

0.346 0.279 0.276 0.266 0.211 0.271 0.258 0.267 0.318 0.200

Average

0.264

8

(true v = 0.0659)

b’?’

Stein

True SE(bj”)

Btrap SE(bj2’)

Error (Y0)

0.395 0.355 0.313 0.291 0.269 0.247 0.224 0.224 0.175 0.145

0.33 I 0.310 0.289 0.278 0.266 0.255 0.243 0.243 0.218 0.204

0.0434 0.0331 0.0330 0.0328 0.0376 0.0328 0.0328 0.0328 0.0377 0.0396

0.0412 0.0324 0.03 15 0.0301 0.0296 0.0296 0.0304 0.0304 0.0346 0.0380

5.1 2.1 4.5 x.2 21.3 9.x 7.3 7.3 8.2 4.0

0.274

0.264

0.0357

0.0330

7.8

1

SE), is reported in the last column. Two observations arc worth mentioning from Table 2. First, the bootstrap estimates are always biased downward, ranging from 2.1% to 21.3%. On average the bias is 7.85% downward. Second, since the Stein estimator shrinks the MLE toward the grand mean, the sample estimates that have values close to the grand mean tend to have smaller variances, whereas those components that are far away from the grand mean tend to have larger variances. Overall. the bootstrap estimation is reasonably good. Recall that here we are comparing the bootstrap cstimatcs with the theoretical values of the variances. Although WC assume that the design matrix, X, is orthonormal throughout the paper, the bootstrap procedure discussed above can bc easily generalized to linear models with more general design matrices under mild regularity conditions on ( X’X) matrix. The bootstrap estimations of the above two cxamplcs are computed by the VAX computer at Indiana University. 5. Summary The bootstrap simulates the original unobservable disturbance terms by a resampling process. It provides a fcasiblc and convenient alternative to assess the variability of the Stein estimator. Howcvcr, readers should bear in mind that the bootstrap estimates of the scale parameters tend to be biased downward, because the fitted error terms tend to bc a bit smaller than the real disturbance terms [Eaton (19X3)]. This bias could be significant when k is large rclativc to the sample size N. Scaling the fitted residuals up by [N/( N - k ,]‘I’ will help, but this is only a partial fix of the problem [Bickel and Freedman (1983)]. Despite this drawback, the bootstrap method offers a reliable way to estimate the precision of the Stein estimator which would certainly cnhancc its potential for practical applications. Further research is needed to investigate to what degree the bootstrap precision cstimatcs depend on /3 and the distance between the MLE h and @. Our Monte Carlo simulation results suggest that replacing /3 by h will not affect the standard errors much as long as @(B. F) is a smooth function of both its arguments. Of course, the analytical proof remains to bc seen. References Bickel.

P.J. and D.A.

Freedman

lY8.1, Bootstrapping

and .I. Hodges. ctls. A frstschrift Bock. M.E..

G.G.

wriahlca Ijrown.

Judge and T.A.

and certain

L. 1%).

rrgrcssion

for Erich Lehmann. Yanccy,

confluent

California

Wadsworth.

in: I’. Bickcl.

funtions.

of invariant

Journal

estimators

of Econometrics

K. Doksum.

3X-48.

lYX3. A Gmple form for the inverse momcnth 01‘ non-central

hypcrgrometric

On the admissihillty

models with many p;iramcter\.

Belmont.

k’ and F random

75. 217-221.

01 one or more location

pat-amctrrs.

Annala

oC Mathematical

statistics 37. 10x7- Il30. Eaton.

IOH3.The

M.I...

Minnesota.

Gauss-Markov

Minneapolis.

Efron.

B.. 1070. Bootstrap

Et’ron,

B.. lYt(2. The jackknife,

Applied

Mathematics.

Freedman.

D.A..

Frrrdman.

D.A..

Aswciation

look at the Jackknife.

the hootstrap

Monograph

and other

analy$i\.

Technical

Rrpol-t

No.

422

(University

of

Annals of Statistics 7. 1-X

resampling

3X (Society for Industrial

plans. CBMS-NSF

and Applied

;I regression

equation:

Some

Regional

Mathematics.

regression models. Annals of Statistics 0, IZIX-

Bootstrappins

1401, Estimation

Press. Berkeley.

Judg!e. G.G. and (North-Holland, Morri\.

in multivat-iate

empirical

C‘ontercnce

Phlldelphia.

Scrirs

in

PA).

12X.

result\.

Journal

of Amcricnn

Statistical

7’). 07- IOh.

James, W. and c‘. Stein. California

methods: Another

IYXI. Bootstrapping 10%.

theorem

MN).

M.E. Bock. 197X. The Amsterd;lm).

C.. 1477. Interval

with quadratic

estimation

statistical

for empirical

ond confc~-once on the design of experiments Stein, C. lY5.5. Inadmisd%lity third Berkeley

los\. Procrcdings

of the fourth symposium, Vol.

I (University

of

CA) .101-379.

Baya

I (University

of pre-tat

generalizations

in army rrscarch

01‘ the usual estimator

symposium. Vol.

implications

and Stein-t-ulr

of Stein‘s c\timator,

dcvelopmcnt

Press. Berkeley.

CA)

of the twznty-szc-

Report

normal distribution. lY7-206.

in econometrics

f’rocccdings

and testing. AR0

for the mean of II multivariate

of <‘alifornia

estimators

77-2.

Procrrdinga

of the