- Email: [email protected]
A ray method of confidence band construction for multiple linear regression models
Journal of Statistical Planning and Inference 139 (2009) 329 -- 334
Contents lists available at ScienceDirect
Journal of Statistical Planning and Inference journal homepage: w w w . e l s e v i e r . c o m / l o c a t e / j s p i
A ray method of confidence band construction for multiple linear regression models A.J. Haytera,∗ , W. Liub , P. Ah-Kineb a b
Department of Statistics and Operations Technology, University of Denver, Denver, CO, USA S3RI and School of Mathematics, University of Southampton, Southampton, UK
A R T I C L E
I N F O
Article history: Received 17 November 2007 Received in revised form 21 April 2008 Accepted 29 April 2008 Available online 14 May 2008 Keywords: Multiple linear regression Confidence bands
A B S T R A C T
This paper addresses the problem of confidence band construction for a standard multiple linear regression model. A "ray” method of construction is developed which generalizes the method of Graybill and Bowden [1967. Linear segment confidence bands for simple linear regression models. J. Amer. Statist. Assoc. 62, 403--408] for a simple linear regression model to a multiple linear regression model. By choosing suitable directions for the rays this method requires only critical points from t-distributions so that the confidence bands are easy to construct. Both one-sided and two-sided confidence bands can be constructed using this method. An illustration of the new method is provided. © 2008 Elsevier B.V. All rights reserved.
1. Introduction Consider a multiple linear regression model with k input variables x = (x1 , . . . , xk ) and with normal errors. Suppose that the design matrix X has the ith row given by (1, x1i , . . . , xki ), 1 i n, and define x¯ i. = nm=1 xim /n with x¯ = (x¯ 1. , . . . , x¯ k. ). Also, let n Sij = m=1 (xim − x¯ i. )(xjm − x¯ j. ) with S being the k × k matrix with entries Sij . Finally, let ˆ 2 be the usual variance estimate
distributed as a 2 2n−k−1 /(n − k − 1) random variable, and let 0 and b = (1 , . . . , k ) be the regression parameters which are estimated by ˆ 0 and bˆ = (ˆ 1 , . . . , ˆ k ), respectively. The objective of this paper is to construct confidence bands for the multiple linear regression model which may be one-sided
0 + xb ˆ 0 + xbˆ + g1, (x) for all x ∈ Rk or two-sided
0 + xb ∈ ˆ 0 + xbˆ ± g2, (x) for all x ∈ Rk with the property that they guarantee a confidence level of at least 1 − . Moreover, the emphasis in this paper is on confidence bands that are convenient and easy to construct due to their reliance on only readily available critical points. Construction of confidence bands for a simple linear regression starts from Working and Hotelling (1929) and has been considered by Gafarian (1964), Bowden and Graybill (1966), Graybill and Bowden (1967), Wynn and Bloomfield (1971), Uusipaikka (1983), Piegorsch et al. (2000) and Pan et al. (2003), among others. See Liu et al. (2008) for a recent review. In particular, the
∗
Corresponding author. Tel.: +1 303 871 4341; fax: +1 303 871 2297. E-mail addresses: [email protected] (A.J. Hayter), [email protected] (W. Liu).
0378-3758/$ - see front matter © 2008 Elsevier B.V. All rights reserved. doi:10.1016/j.jspi.2008.04.029
330
A.J. Hayter et al. / Journal of Statistical Planning and Inference 139 (2009) 329 -- 334
frequently quoted two-sided two-segment band of Graybill and Bowden (1967) for a simple linear regression line is constructed by setting up simultaneous confidence intervals on the regression line at x1 = x¯ 1. and on its gradient. For a multiple linear regression model the most well-known simultaneous confidence band is the hyperbolic band given by Scheffé (1953) 1 + (x¯ − x)S−1 (x¯ − x) for all x ∈ Rk . 0 + xb ∈ ˆ 0 + xbˆ ± ˆ F,k+1,n−(k+1) n This band is over the whole range Rk of the k input variables x. These hyperbolic bands have some nice properties, such as that they are in agreement with the law of likelihood whereby parameter values for models within the bands have larger likelihoods than for models outside the bands. However, there are other criteria for selecting confidence bands, and other bands can be better in certain regions. Construction of confidence bands for a multiple linear regression model over a restricted region of the input variables is much harder. Notable works include Knafl et al. (1985), Naiman (1987, 1990), Sun and Loader (1994) and Liu et al. (2005) for the situation that the input variables are constrained to a hyper-rectangle, and Bohrer (1973), Casella and Strawderman (1980), Seppanen and Uusipaikka (1992) and Liu and Lin (2007) for the situation that the input variables are constrained to a hyper-ellipsoid. The purpose of this paper is to generalize the idea of Graybill and Bowden (1967) from a simple linear regression line to a multiple regression model to construct simultaneous confidence bands by setting up simultaneous confidence limits on the multiple regression model at x = (x¯ 1. , . . . , x¯ k. ) and on its "gradients” in k directions. For k = 1, the two-sided confidence band provided in this paper reduces to the two-sided two-segment band of Graybill and Bowden (1967). Bowden (1970) also proposed a two-sided confidence band which generalizes the idea of Graybill and Bowden (1967). However, that band requires the critical value of max0 i k |ˆi |, the computation of which involves (k + 1)-dimensional numerical integration in general. The method given in this paper requires only a one-dimensional numerical integration for computing the exact critical values and, moreover, the exact critical values are well approximated (conservatively) by critical values from t-distributions when n − k − 1 is moderate or large. Furthermore, the method of this paper works for both two-sided and one-sided confidence bands. The outline of this paper is as follows. Section 2 provides the method of construction and explicit formulae of the one-sided and two-sided simultaneous confidence bands. Section 3 contains a numerical example. 2. The ray method Notice that the design matrix X satisfies 1 ¯ −1 ¯ −1 x¯ −xS + xS (X X)−1 = n −S−1 x¯ S−1 so that if the k × k matrix P satisfies PP = Ik , then 1 0 1/n 0 1 x¯ (X X)−1 = . 1/2 1/2 x¯ S P 0 Ik 0 PS Let bi ∈ Rk be the ith row of PS1/2 , 1 i k. Then it follows that ⎛ˆ ⎞ 0 + x¯ bˆ ⎜ ˆ ⎟ ⎜ b1 bˆ ⎟ 0 1 x¯ 1/n 0 ⎟ = Cov Cov ⎜ = 2 . 1/2 .. ⎜ ⎟ 0 PS 0 Ik bˆ ⎝ ⎠ . b bˆ k
Hence bi b (1 i k) are independent with variances 2 , and they are also independent of ˆ 0 + x¯ bˆ which has a variance 2 /n. k+1 (1 − i ). Then the inequalities Suppose that some error rates 0 i 1 are chosen with 1 − = i=1 ˆ
√
0 + x¯ b ˆ 0 + x¯ bˆ + zk+1 / n,
bi b ∈ bi bˆ ± zi /2 , 1 i k,
(1)
and √
0 + x¯ b ∈ ˆ 0 + x¯ bˆ ± zk+1 /2 / n,
bi b ∈ bi bˆ ± zi /2 , 1 i k,
(2)
each have a confidence level of exactly 1 − . Moreover, if the standard deviation is unknown and is replaced by its estimate ˆ in these inequalities, and the standard normal critical points are replaced by critical points from a t-distribution with n − k − 1 degrees of freedom, then it follows from Kimball's (1951) inequality that the inequalities √ 0 + x¯ b ˆ 0 + x¯ bˆ + ˆ t ,n−k−1 / n, bi b ∈ bi bˆ ± ˆ ti /2,n−k−1 , 1 i k, (3) k+1
A.J. Hayter et al. / Journal of Statistical Planning and Inference 139 (2009) 329 -- 334
331
and √
0 + x¯ b ∈ ˆ 0 + x¯ bˆ ± ˆ tk+1 /2,n−k−1 / n,
bi b ∈ bi bˆ ± ˆ ti /2,n−k−1 , 1 i k,
(4)
each have a confidence level of at least 1 − . For moderate degrees of freedom n − k − 1 the amount of conservativeness will be very small because it is only due to the modification of the inequalities in (1) and (2) with the variance estimate. Pages 10 and 11 of Hsu (1996) and references in Finner and Roters (2001) contain a discussion regarding the very slight amount of conservativeness resulting from the application of Kimball's inequality in this way. In addition, Hayter et al. (2007) provides some numerical computations of the amount of conservativeness when there is only one input variable. Of course, if it is necessary then the exact confidence levels of (3) and (4) can easily be computed involving only a one-dimensional numerical integration as for the Studentized maximum modulus distribution (see, for example, Hochberg and Tamhane, 1987). The confidence sets (3) and (4) provide the basis for the following one-sided and two-sided confidence bands, respectively. 1/2 )−1 ¯ Defining (1 , . . . , k ) = (x − x)(PS ⎛
0 + xb ˆ 0 + xbˆ + ˆ ⎝t
√
k+1 ,n−k−1
/ n+
k
⎞ |i |ti /2,n−k−1 ⎠
∀x ∈ Rk
is a one-sided upper confidence band with a confidence level of at least (1 − ). Similarly ⎛ ⎞ k √ ˆ ˆ ⎝ 0 + xb ∈ 0 + xb ± ˆ t /2,n−k−1 / n + |i |ti /2,n−k−1 ⎠ ∀ x ∈ Rk k+1
(5)
i=1
(6)
i=1
is a two-sided confidence band with a confidence level of at least (1 − ). These confidence bands are easy to construct because they only require critical points from a t-distribution. Note that the upper confidence band (5) is formed by 2k adjacent V-shaped plane segments in Rk+1 . These segments meet at one common √ point, the tip of all the V-shapes, at x = x¯ with a height ˆ 0 + xbˆ + ˆ t ,n−k−1 / n. This height is the upper confidence limit on k+1 the regression model at x = x¯ stipulated in 3. The edges of the 2k adjacent V-shapes are given by 2k half lines, the projections of which onto the x space form k straight lines that pass through x = x¯ and are in the directions of the "rays” b1 , . . . , bk . The upper part of the two-sided band (6) is of a similar shape to the upper confidence band (5). The lower and upper parts of the two-sided band are symmetric about the fitted regression model ˆ 0 + xbˆ . Moreover, the confidence band is narrowest along the directions of the rays b1 , . . . , bk . There is a certain amount of flexibility in applying these confidence band methods due to the choices of the individual error rates i together with the directions of the rays bi resulting from the orthogonal matrix P. As a default, the method can be applied with i = 1 − (1 − )1/(k+1) , 1 i k + 1, and with P equal to the identity matrix so that the rays bi are the rows of S1/2 . In fact, Liu and Hayter (2007) propose the use of confidence set volume as a criterion in the comparison of confidence bands, and it can be shown that the default choice of 1 = · · · = k+1 makes the two-sided confidence set (2) for the unknown coefficients k+1 (1 − i ), irrespective of the choice of P. As noted above, (0 , 1 , . . . , k ) have the smallest volume under the constraint 1 − = i=1 the confidence bands are narrowest along the directions of the rays. Hence if there is one particular direction for which narrow confidence bands are particularly desired, then the matrix P can be chosen so that this direction is one of the rays. Furthermore, in designed experiments the matrix S may also be formed to allow rays to be chosen in more than one specified direction. With only a single input variable, k = 1, this method of constructing confidence bands simplifies to the method shown in Graybill and Bowden (1967) for a simple linear regression model where exact critical points from the bivariate maximum modulus distribution are considered. The graph in that paper shows how the choice 1 = 2 produces confidence bands which are narrower than the hyperbolic confidence bands for values of x1 close to x¯ 1. and also for values of x1 far away from x¯ 1. . The example in the next section illustrates how this phenomenon is replicated by this new method for multiple linear regression. 3. An illustration of the ray method In this section, a portion of the acetylene data in Snee (1977) is used to illustrate the new methodology discussed in this paper. The two predictor variables are reactor temperature (x1 ) and ratio of H2 to n-heptane (x2 ). The response variable (y) is conversion of n-heptane to acetylene. There are 16 data points so k = 2, n = 16 and = n − k − 1 = 13. The fitted linear regression model is given by y = −130.69 + 0.134x1 + 0.351x2 with ˆ = 3.624 and R2 = 0.92. Also, if b1 = (b11 , b21 ) and b2 = (b12 , b22 ) then
1 =
b22 (x1 − x¯ 1. ) − b12 (x2 − x¯ 2. ) , b11 b22 − b12 b21
2 =
−b21 (x1 − x¯ 1. ) + b11 (x2 − x¯ 2. ) . b11 b22 − b12 b21
332
A.J. Hayter et al. / Journal of Statistical Planning and Inference 139 (2009) 329 -- 334
Fig. 1. The two-sided confidence band.
Fig. 2. The two confidence bands.
For = 0.05, 1 = 2 = 3 and P = I2 , the two-sided confidence band (6) is plotted in Fig. 1. Note that the conservative t-critical value is given by 2.737, which is only marginally larger than the exact critical value 2.72 from Table 7 of Hochberg and Tamhane (1987). The upper part of the band consists of 2 × 2 = 4 adjacent V-shaped plane segments. The tips of these V-shapes √ meet at x = x¯ = (1212.5, 12.4) and y = ˆ 0 + xbˆ + ˆ t /2,n−k−1 / n = 38.6. These four V-shapes are determined by four half k+1
straight lines, and their projections onto the x = (x1 , x2 ) plane are depicted in the picture. These projections form two straight lines that go through x = x¯ = (1212.5, 12.4) and are in the directions of b1 = (312.2162, 4.5893) and b2 = (4.5893, 21.4434), with the confidence bands being narrowest along these two straight lines. One may vary the directions of these two lines by changing the orthogonal matrix P. The middle of the confidence band is given by the fitted regression plane y = −130.69 + 0.134x1 + 0.351x2 . Any plane that is contained within the confidence band is deemed by the confidence bands to be a plausible candidate for the true but unknown regression plane.
A.J. Hayter et al. / Journal of Statistical Planning and Inference 139 (2009) 329 -- 334
333
Fig. 3. Plots of slices of confidence bands.
In Fig. 2, both the hyperbolic band and the confidence band in Fig. 1 are plotted. From Fig. 2 it can be seen that the confidence ¯ especially band proposed in this paper is narrower than the hyperbolic band when x is near x¯ and when x is far away from x, along the directions of the rays. This generalizes the phenomenon observed by Graybill and Bowden (1967) for a simple linear regression line. Of course, one can show a slice of the confidence bands to get a better picture of the bands in one particular direction. For example, one can plot the two bands in Fig. 2 in the direction of x1 while fixing the value of x2 , or vice versa. Fig. 3 shows the slices of the bands when x2 is fixed at x¯ 2. and when x1 is fixed at x¯ 1. . Acknowledgment This research is partially supported by the British Engineering and Physics Sciences Research Council. References Bohrer, R., 1973. A multivariate t probability integral. Biometrika 60, 647--654. Bowden, D.C., 1970. Simultaneous confidence bands for linear regression models. J. Amer. Statist. Assoc. 65, 413--421. Bowden, D.C., Graybill, F.A., 1966. Confidence bands of uniform and proportional width for linear models. J. Amer. Statist. Assoc. 61, 182--198. Casella, G., Strawderman, W.E., 1980. Confidence bands for linear regression with restricted predictor variables. J. Amer. Statist. Assoc. 75, 862--868. Finner, H., Roters, M., 2001. Asymptotic sharpness of product-type inequalities for maxima of random variables with applications in multiple comparisons. J. Statist. Plann. Inference 98, 39--56. Gafarian, A.V., 1964. Confidence bands in straight line regression. J. Amer. Statist. Assoc. 59, 182--213. Graybill, F.A., Bowden, D.C., 1967. Linear segment confidence bands for simple linear regression models. J. Amer. Statist. Assoc. 62, 403--408. Hayter, A.J., Liu, W., Wynn, H.P., 2007. Easy-to-construct confidence bands for comparing two simple linear regression lines. J. Statist. Plann. Inference 137 (4), 1213--1225. Hochberg, Y., Tamhane, A.C., 1987. Multiple Comparison Procedures. Wiley, New York. Hsu, J.C., 1996. Multiple Comparisons---Theory and Methods. Chapman & Hall, London. Kimball, A.W., 1951. On dependent tests of significance in analysis of variance. Ann. Math. Statist. 22, 600--602. Knafl, G., Sacks, J., Ylvisaker, D., 1985. Confidence bands for regression-functions. J. Amer. Statist. Assoc. 80, 683--691. Liu, W., Hayter, A.J., 2007. Minimum area confidence set optimality for confidence bands in simple linear regression. J. Amer. Statist. Assoc. 102 (477), 181--190. Liu, W., Lin, S., 2007. Construction of exact simultaneous confidence bands in multiple linear regression with predictor variables constrained in an ellipsoidal region. Statistica Sinica, to appear. Liu, W., Jamshidian, M., Zhang, Y., Donnelly, J., 2005. Simulation-based simultaneous confidence bands in multiple linear regression with predictor variables constrained in intervals. J. Comput. Graph. Statist. 14 (2), 459--484. Liu, W., Lin, S., Piegorsch, W.W., 2008. Construction of exact simultaneous confidence bands for a simple linear regression model. Internat. Statist. Rev. 76, 39--57. Naiman, D.Q., 1987. Simultaneous confidence-bounds in multiple-regression using predictor variable constraints. J. Amer. Statist. Assoc. 82, 214--219. Naiman, D.Q., 1990. On volumes of tubular neighborhoods of spherical polyhedra and statistical inference. Ann. Statist. 18, 685--716. Pan, W., Piegorsch, W.W., West, R.W., 2003. Exact one-sided simultaneous confidence bands via Uusipaikka's method. Ann. Inst. Statist. Math. 55 (2), 243--250. Piegorsch, W.W., West, R.W., Al-Saidy, O.M., Bradley, K.D., 2000. Asymmetric confidence bands for simple linear regression over bounded intervals. Comput. Statist. Data Anal. 34 (2), 193--217.
334
A.J. Hayter et al. / Journal of Statistical Planning and Inference 139 (2009) 329 -- 334
Scheffé, H., 1953. A method for judging all contrasts in analysis of variance. Biometrika 40, 87--104. Seppanen, E., Uusipaikka, E., 1992. Confidence bands for linear-regression over restricted regions. Scand. J. Statist. 19, 73--81. Snee, R.D., 1977. Validation of regression models: methods and examples. Technometrics 19, 415--428. Sun, J., Loader, C.R., 1994. Simultaneous confidence bands for linear regression and smoothing. Ann. Statist. 22, 1328--1346. Uusipaikka, E., 1983. Exact confidence bands for linear-regression over intervals. J. Amer. Statist. Assoc. 78, 638--644. Working, H., Hotelling, H., 1929. Applications of the theory of error to the interpretation of trends. J. Amer. Statist. Assoc. 24, 73--85. Wynn, H.P., Bloomfield, P., 1971. Simultaneous confidence bands in regression analysis. J. Roy. Statist. Soc. B 33, 202--217.
Contents lists available at ScienceDirect
Journal of Statistical Planning and Inference journal homepage: w w w . e l s e v i e r . c o m / l o c a t e / j s p i
A ray method of confidence band construction for multiple linear regression models A.J. Haytera,∗ , W. Liub , P. Ah-Kineb a b
Department of Statistics and Operations Technology, University of Denver, Denver, CO, USA S3RI and School of Mathematics, University of Southampton, Southampton, UK
A R T I C L E
I N F O
Article history: Received 17 November 2007 Received in revised form 21 April 2008 Accepted 29 April 2008 Available online 14 May 2008 Keywords: Multiple linear regression Confidence bands
A B S T R A C T
This paper addresses the problem of confidence band construction for a standard multiple linear regression model. A "ray” method of construction is developed which generalizes the method of Graybill and Bowden [1967. Linear segment confidence bands for simple linear regression models. J. Amer. Statist. Assoc. 62, 403--408] for a simple linear regression model to a multiple linear regression model. By choosing suitable directions for the rays this method requires only critical points from t-distributions so that the confidence bands are easy to construct. Both one-sided and two-sided confidence bands can be constructed using this method. An illustration of the new method is provided. © 2008 Elsevier B.V. All rights reserved.
1. Introduction Consider a multiple linear regression model with k input variables x = (x1 , . . . , xk ) and with normal errors. Suppose that the design matrix X has the ith row given by (1, x1i , . . . , xki ), 1 i n, and define x¯ i. = nm=1 xim /n with x¯ = (x¯ 1. , . . . , x¯ k. ). Also, let n Sij = m=1 (xim − x¯ i. )(xjm − x¯ j. ) with S being the k × k matrix with entries Sij . Finally, let ˆ 2 be the usual variance estimate
distributed as a 2 2n−k−1 /(n − k − 1) random variable, and let 0 and b = (1 , . . . , k ) be the regression parameters which are estimated by ˆ 0 and bˆ = (ˆ 1 , . . . , ˆ k ), respectively. The objective of this paper is to construct confidence bands for the multiple linear regression model which may be one-sided
0 + xb ˆ 0 + xbˆ + g1, (x) for all x ∈ Rk or two-sided
0 + xb ∈ ˆ 0 + xbˆ ± g2, (x) for all x ∈ Rk with the property that they guarantee a confidence level of at least 1 − . Moreover, the emphasis in this paper is on confidence bands that are convenient and easy to construct due to their reliance on only readily available critical points. Construction of confidence bands for a simple linear regression starts from Working and Hotelling (1929) and has been considered by Gafarian (1964), Bowden and Graybill (1966), Graybill and Bowden (1967), Wynn and Bloomfield (1971), Uusipaikka (1983), Piegorsch et al. (2000) and Pan et al. (2003), among others. See Liu et al. (2008) for a recent review. In particular, the
∗
Corresponding author. Tel.: +1 303 871 4341; fax: +1 303 871 2297. E-mail addresses: [email protected] (A.J. Hayter), [email protected] (W. Liu).
0378-3758/$ - see front matter © 2008 Elsevier B.V. All rights reserved. doi:10.1016/j.jspi.2008.04.029
330
A.J. Hayter et al. / Journal of Statistical Planning and Inference 139 (2009) 329 -- 334
frequently quoted two-sided two-segment band of Graybill and Bowden (1967) for a simple linear regression line is constructed by setting up simultaneous confidence intervals on the regression line at x1 = x¯ 1. and on its gradient. For a multiple linear regression model the most well-known simultaneous confidence band is the hyperbolic band given by Scheffé (1953) 1 + (x¯ − x)S−1 (x¯ − x) for all x ∈ Rk . 0 + xb ∈ ˆ 0 + xbˆ ± ˆ F,k+1,n−(k+1) n This band is over the whole range Rk of the k input variables x. These hyperbolic bands have some nice properties, such as that they are in agreement with the law of likelihood whereby parameter values for models within the bands have larger likelihoods than for models outside the bands. However, there are other criteria for selecting confidence bands, and other bands can be better in certain regions. Construction of confidence bands for a multiple linear regression model over a restricted region of the input variables is much harder. Notable works include Knafl et al. (1985), Naiman (1987, 1990), Sun and Loader (1994) and Liu et al. (2005) for the situation that the input variables are constrained to a hyper-rectangle, and Bohrer (1973), Casella and Strawderman (1980), Seppanen and Uusipaikka (1992) and Liu and Lin (2007) for the situation that the input variables are constrained to a hyper-ellipsoid. The purpose of this paper is to generalize the idea of Graybill and Bowden (1967) from a simple linear regression line to a multiple regression model to construct simultaneous confidence bands by setting up simultaneous confidence limits on the multiple regression model at x = (x¯ 1. , . . . , x¯ k. ) and on its "gradients” in k directions. For k = 1, the two-sided confidence band provided in this paper reduces to the two-sided two-segment band of Graybill and Bowden (1967). Bowden (1970) also proposed a two-sided confidence band which generalizes the idea of Graybill and Bowden (1967). However, that band requires the critical value of max0 i k |ˆi |, the computation of which involves (k + 1)-dimensional numerical integration in general. The method given in this paper requires only a one-dimensional numerical integration for computing the exact critical values and, moreover, the exact critical values are well approximated (conservatively) by critical values from t-distributions when n − k − 1 is moderate or large. Furthermore, the method of this paper works for both two-sided and one-sided confidence bands. The outline of this paper is as follows. Section 2 provides the method of construction and explicit formulae of the one-sided and two-sided simultaneous confidence bands. Section 3 contains a numerical example. 2. The ray method Notice that the design matrix X satisfies 1 ¯ −1 ¯ −1 x¯ −xS + xS (X X)−1 = n −S−1 x¯ S−1 so that if the k × k matrix P satisfies PP = Ik , then 1 0 1/n 0 1 x¯ (X X)−1 = . 1/2 1/2 x¯ S P 0 Ik 0 PS Let bi ∈ Rk be the ith row of PS1/2 , 1 i k. Then it follows that ⎛ˆ ⎞ 0 + x¯ bˆ ⎜ ˆ ⎟ ⎜ b1 bˆ ⎟ 0 1 x¯ 1/n 0 ⎟ = Cov Cov ⎜ = 2 . 1/2 .. ⎜ ⎟ 0 PS 0 Ik bˆ ⎝ ⎠ . b bˆ k
Hence bi b (1 i k) are independent with variances 2 , and they are also independent of ˆ 0 + x¯ bˆ which has a variance 2 /n. k+1 (1 − i ). Then the inequalities Suppose that some error rates 0 i 1 are chosen with 1 − = i=1 ˆ
√
0 + x¯ b ˆ 0 + x¯ bˆ + zk+1 / n,
bi b ∈ bi bˆ ± zi /2 , 1 i k,
(1)
and √
0 + x¯ b ∈ ˆ 0 + x¯ bˆ ± zk+1 /2 / n,
bi b ∈ bi bˆ ± zi /2 , 1 i k,
(2)
each have a confidence level of exactly 1 − . Moreover, if the standard deviation is unknown and is replaced by its estimate ˆ in these inequalities, and the standard normal critical points are replaced by critical points from a t-distribution with n − k − 1 degrees of freedom, then it follows from Kimball's (1951) inequality that the inequalities √ 0 + x¯ b ˆ 0 + x¯ bˆ + ˆ t ,n−k−1 / n, bi b ∈ bi bˆ ± ˆ ti /2,n−k−1 , 1 i k, (3) k+1
A.J. Hayter et al. / Journal of Statistical Planning and Inference 139 (2009) 329 -- 334
331
and √
0 + x¯ b ∈ ˆ 0 + x¯ bˆ ± ˆ tk+1 /2,n−k−1 / n,
bi b ∈ bi bˆ ± ˆ ti /2,n−k−1 , 1 i k,
(4)
each have a confidence level of at least 1 − . For moderate degrees of freedom n − k − 1 the amount of conservativeness will be very small because it is only due to the modification of the inequalities in (1) and (2) with the variance estimate. Pages 10 and 11 of Hsu (1996) and references in Finner and Roters (2001) contain a discussion regarding the very slight amount of conservativeness resulting from the application of Kimball's inequality in this way. In addition, Hayter et al. (2007) provides some numerical computations of the amount of conservativeness when there is only one input variable. Of course, if it is necessary then the exact confidence levels of (3) and (4) can easily be computed involving only a one-dimensional numerical integration as for the Studentized maximum modulus distribution (see, for example, Hochberg and Tamhane, 1987). The confidence sets (3) and (4) provide the basis for the following one-sided and two-sided confidence bands, respectively. 1/2 )−1 ¯ Defining (1 , . . . , k ) = (x − x)(PS ⎛
0 + xb ˆ 0 + xbˆ + ˆ ⎝t
√
k+1 ,n−k−1
/ n+
k
⎞ |i |ti /2,n−k−1 ⎠
∀x ∈ Rk
is a one-sided upper confidence band with a confidence level of at least (1 − ). Similarly ⎛ ⎞ k √ ˆ ˆ ⎝ 0 + xb ∈ 0 + xb ± ˆ t /2,n−k−1 / n + |i |ti /2,n−k−1 ⎠ ∀ x ∈ Rk k+1
(5)
i=1
(6)
i=1
is a two-sided confidence band with a confidence level of at least (1 − ). These confidence bands are easy to construct because they only require critical points from a t-distribution. Note that the upper confidence band (5) is formed by 2k adjacent V-shaped plane segments in Rk+1 . These segments meet at one common √ point, the tip of all the V-shapes, at x = x¯ with a height ˆ 0 + xbˆ + ˆ t ,n−k−1 / n. This height is the upper confidence limit on k+1 the regression model at x = x¯ stipulated in 3. The edges of the 2k adjacent V-shapes are given by 2k half lines, the projections of which onto the x space form k straight lines that pass through x = x¯ and are in the directions of the "rays” b1 , . . . , bk . The upper part of the two-sided band (6) is of a similar shape to the upper confidence band (5). The lower and upper parts of the two-sided band are symmetric about the fitted regression model ˆ 0 + xbˆ . Moreover, the confidence band is narrowest along the directions of the rays b1 , . . . , bk . There is a certain amount of flexibility in applying these confidence band methods due to the choices of the individual error rates i together with the directions of the rays bi resulting from the orthogonal matrix P. As a default, the method can be applied with i = 1 − (1 − )1/(k+1) , 1 i k + 1, and with P equal to the identity matrix so that the rays bi are the rows of S1/2 . In fact, Liu and Hayter (2007) propose the use of confidence set volume as a criterion in the comparison of confidence bands, and it can be shown that the default choice of 1 = · · · = k+1 makes the two-sided confidence set (2) for the unknown coefficients k+1 (1 − i ), irrespective of the choice of P. As noted above, (0 , 1 , . . . , k ) have the smallest volume under the constraint 1 − = i=1 the confidence bands are narrowest along the directions of the rays. Hence if there is one particular direction for which narrow confidence bands are particularly desired, then the matrix P can be chosen so that this direction is one of the rays. Furthermore, in designed experiments the matrix S may also be formed to allow rays to be chosen in more than one specified direction. With only a single input variable, k = 1, this method of constructing confidence bands simplifies to the method shown in Graybill and Bowden (1967) for a simple linear regression model where exact critical points from the bivariate maximum modulus distribution are considered. The graph in that paper shows how the choice 1 = 2 produces confidence bands which are narrower than the hyperbolic confidence bands for values of x1 close to x¯ 1. and also for values of x1 far away from x¯ 1. . The example in the next section illustrates how this phenomenon is replicated by this new method for multiple linear regression. 3. An illustration of the ray method In this section, a portion of the acetylene data in Snee (1977) is used to illustrate the new methodology discussed in this paper. The two predictor variables are reactor temperature (x1 ) and ratio of H2 to n-heptane (x2 ). The response variable (y) is conversion of n-heptane to acetylene. There are 16 data points so k = 2, n = 16 and = n − k − 1 = 13. The fitted linear regression model is given by y = −130.69 + 0.134x1 + 0.351x2 with ˆ = 3.624 and R2 = 0.92. Also, if b1 = (b11 , b21 ) and b2 = (b12 , b22 ) then
1 =
b22 (x1 − x¯ 1. ) − b12 (x2 − x¯ 2. ) , b11 b22 − b12 b21
2 =
−b21 (x1 − x¯ 1. ) + b11 (x2 − x¯ 2. ) . b11 b22 − b12 b21
332
A.J. Hayter et al. / Journal of Statistical Planning and Inference 139 (2009) 329 -- 334
Fig. 1. The two-sided confidence band.
Fig. 2. The two confidence bands.
For = 0.05, 1 = 2 = 3 and P = I2 , the two-sided confidence band (6) is plotted in Fig. 1. Note that the conservative t-critical value is given by 2.737, which is only marginally larger than the exact critical value 2.72 from Table 7 of Hochberg and Tamhane (1987). The upper part of the band consists of 2 × 2 = 4 adjacent V-shaped plane segments. The tips of these V-shapes √ meet at x = x¯ = (1212.5, 12.4) and y = ˆ 0 + xbˆ + ˆ t /2,n−k−1 / n = 38.6. These four V-shapes are determined by four half k+1
straight lines, and their projections onto the x = (x1 , x2 ) plane are depicted in the picture. These projections form two straight lines that go through x = x¯ = (1212.5, 12.4) and are in the directions of b1 = (312.2162, 4.5893) and b2 = (4.5893, 21.4434), with the confidence bands being narrowest along these two straight lines. One may vary the directions of these two lines by changing the orthogonal matrix P. The middle of the confidence band is given by the fitted regression plane y = −130.69 + 0.134x1 + 0.351x2 . Any plane that is contained within the confidence band is deemed by the confidence bands to be a plausible candidate for the true but unknown regression plane.
A.J. Hayter et al. / Journal of Statistical Planning and Inference 139 (2009) 329 -- 334
333
Fig. 3. Plots of slices of confidence bands.
In Fig. 2, both the hyperbolic band and the confidence band in Fig. 1 are plotted. From Fig. 2 it can be seen that the confidence ¯ especially band proposed in this paper is narrower than the hyperbolic band when x is near x¯ and when x is far away from x, along the directions of the rays. This generalizes the phenomenon observed by Graybill and Bowden (1967) for a simple linear regression line. Of course, one can show a slice of the confidence bands to get a better picture of the bands in one particular direction. For example, one can plot the two bands in Fig. 2 in the direction of x1 while fixing the value of x2 , or vice versa. Fig. 3 shows the slices of the bands when x2 is fixed at x¯ 2. and when x1 is fixed at x¯ 1. . Acknowledgment This research is partially supported by the British Engineering and Physics Sciences Research Council. References Bohrer, R., 1973. A multivariate t probability integral. Biometrika 60, 647--654. Bowden, D.C., 1970. Simultaneous confidence bands for linear regression models. J. Amer. Statist. Assoc. 65, 413--421. Bowden, D.C., Graybill, F.A., 1966. Confidence bands of uniform and proportional width for linear models. J. Amer. Statist. Assoc. 61, 182--198. Casella, G., Strawderman, W.E., 1980. Confidence bands for linear regression with restricted predictor variables. J. Amer. Statist. Assoc. 75, 862--868. Finner, H., Roters, M., 2001. Asymptotic sharpness of product-type inequalities for maxima of random variables with applications in multiple comparisons. J. Statist. Plann. Inference 98, 39--56. Gafarian, A.V., 1964. Confidence bands in straight line regression. J. Amer. Statist. Assoc. 59, 182--213. Graybill, F.A., Bowden, D.C., 1967. Linear segment confidence bands for simple linear regression models. J. Amer. Statist. Assoc. 62, 403--408. Hayter, A.J., Liu, W., Wynn, H.P., 2007. Easy-to-construct confidence bands for comparing two simple linear regression lines. J. Statist. Plann. Inference 137 (4), 1213--1225. Hochberg, Y., Tamhane, A.C., 1987. Multiple Comparison Procedures. Wiley, New York. Hsu, J.C., 1996. Multiple Comparisons---Theory and Methods. Chapman & Hall, London. Kimball, A.W., 1951. On dependent tests of significance in analysis of variance. Ann. Math. Statist. 22, 600--602. Knafl, G., Sacks, J., Ylvisaker, D., 1985. Confidence bands for regression-functions. J. Amer. Statist. Assoc. 80, 683--691. Liu, W., Hayter, A.J., 2007. Minimum area confidence set optimality for confidence bands in simple linear regression. J. Amer. Statist. Assoc. 102 (477), 181--190. Liu, W., Lin, S., 2007. Construction of exact simultaneous confidence bands in multiple linear regression with predictor variables constrained in an ellipsoidal region. Statistica Sinica, to appear. Liu, W., Jamshidian, M., Zhang, Y., Donnelly, J., 2005. Simulation-based simultaneous confidence bands in multiple linear regression with predictor variables constrained in intervals. J. Comput. Graph. Statist. 14 (2), 459--484. Liu, W., Lin, S., Piegorsch, W.W., 2008. Construction of exact simultaneous confidence bands for a simple linear regression model. Internat. Statist. Rev. 76, 39--57. Naiman, D.Q., 1987. Simultaneous confidence-bounds in multiple-regression using predictor variable constraints. J. Amer. Statist. Assoc. 82, 214--219. Naiman, D.Q., 1990. On volumes of tubular neighborhoods of spherical polyhedra and statistical inference. Ann. Statist. 18, 685--716. Pan, W., Piegorsch, W.W., West, R.W., 2003. Exact one-sided simultaneous confidence bands via Uusipaikka's method. Ann. Inst. Statist. Math. 55 (2), 243--250. Piegorsch, W.W., West, R.W., Al-Saidy, O.M., Bradley, K.D., 2000. Asymmetric confidence bands for simple linear regression over bounded intervals. Comput. Statist. Data Anal. 34 (2), 193--217.
334
A.J. Hayter et al. / Journal of Statistical Planning and Inference 139 (2009) 329 -- 334
Scheffé, H., 1953. A method for judging all contrasts in analysis of variance. Biometrika 40, 87--104. Seppanen, E., Uusipaikka, E., 1992. Confidence bands for linear-regression over restricted regions. Scand. J. Statist. 19, 73--81. Snee, R.D., 1977. Validation of regression models: methods and examples. Technometrics 19, 415--428. Sun, J., Loader, C.R., 1994. Simultaneous confidence bands for linear regression and smoothing. Ann. Statist. 22, 1328--1346. Uusipaikka, E., 1983. Exact confidence bands for linear-regression over intervals. J. Amer. Statist. Assoc. 78, 638--644. Working, H., Hotelling, H., 1929. Applications of the theory of error to the interpretation of trends. J. Amer. Statist. Assoc. 24, 73--85. Wynn, H.P., Bloomfield, P., 1971. Simultaneous confidence bands in regression analysis. J. Roy. Statist. Soc. B 33, 202--217.