- Email: [email protected]
Linear trend analysis: a comparison of methods
Atmospheric Environment 36 (2002) 4420–4421
Discussion
Linear trend analysis: a comparison of methods$ P. Steven Portera,*, S.T. Raob, C. Hogrefeb b
a University of Idaho, 1776 Science Center Drive, Idaho Falls, ID 83402, USA University at Albany—SUNY, Atmospheric Sciences Research Center, 251 Fuller Road, Albany, NY 12203, USA
Hess et al. (2001. Atmospheric Environment 35, 5211– 5222) compare several tests for linear trends in air quality time series. They found that low-pass (i.e., KZ filter) filtered data produced actual type I error rates much smaller than nominal when based on the use of the following equation adapted from Eskridge et al. (1997): P # 2 1=2 ½ ðY YÞ ME ¼ t0:975;f ; P ðn 2Þ1=2 ½ ðt t%Þ2 1=2 f# ¼
1 þ 2=n
n Pn1
i¼1 ðn
iÞr# i
;
ð1Þ
where ME is the 95% margin of error, t0:975;f is the 0.975th percentile of Student’s t distribution with f degrees-of-freedom, Y is the filtered time series, Y# is the least squares linear approximation to Y ; n is sample size and r# is the sample autocorrelation. We believe that the type I error rates for the filtered data in Hess et al. (2001) are much too small because the effective sample size used for the slope standard deviation calculation is not n 2; as in Eq. (1), but f:# The expression for ME should therefore be P # 2 1=2 ½ ðY YÞ ME ¼ t0:975; f# : ð2Þ P f#1=2 ½ ðt t%Þ2 1=2 Use of Eq. (2) will result in approximately nominal type I error rates for filtered data. The use of autocorrelations to compute degrees-offreedom for filtered time series is awkward in any case, more so for irregularly sampled time series, because sample autocorrelations for filtered data may be significant at very long lag times, and as such are difficult to estimate (Law and Kelton, 1991; Kendall et al., 1983). Another way to estimate f is to notice that the reduction in degrees-of-freedom due to filtering is related to the amount of variance reduction. For the $
PII of original article S1352-2310(02)00189-9 *Corresponding author. E-mail address: [email protected] (P.S. Porter).
KZ filter, f ((n 2)/variance reduction) is given approximately by f# ¼
n2 pffiffiffi; k
ð2q þ 1Þ
ð3Þ
where 2q þ 1 is the window size for the moving average and k is the number of iterations. Perhaps a better method for estimating slopes and confidence intervals is to use a filter designed to remove only seasonality. This can be accomplished using both a low-pass and a high-pass filter (i.e., a notch filter). For example, define an air quality time series as the sum of 3 components: Rt ¼ Lt þ St þ Wt ;
ð4Þ
where Lt represents low-frequency variation, S seasonality, and W short-term (synoptic) variation. Estimate the slope and its standard deviation by applying classical least squares formulas to Yt : # t; Yt ¼ L# t þ W
ð5Þ
where L# t is a time series smoothed by a low-pass filter # t is the with a cut-off period of about 2 years, and W residual of a low-pass filter with a cut-off period of about 50 days. Use of a notch filter reduces the slope variance (by reducing seasonality) while retaining the high-frequency component simplifies computations, as for many air quality time series, Yt will more closely match the trend+white noise model assumed by classical least squares than will L# t : In other words, estimating the slope from Yt (the notch filtered time series) follows the same paradigm as the other methods examined by Hess et al. (2001) in that it only removes seasonality while retaining the effects of both long-term and short-term fluctuations on the slope estimates and, thus, the confidence intervals. This procedure was applied (although sometimes this was not explicitly stated) to estimate the trend confidence intervals in previous studies using the KZ filter (Milanchus et al., 1998; Hogrefe et al., 1998; Chan et al., 1999).
1352-2310/02/$ - see front matter r 2002 Elsevier Science Ltd. All rights reserved. PII: S 1 3 5 2 - 2 3 1 0 ( 0 2 ) 0 0 5 4 6 - 0
P.S. Porter et al. / Atmospheric Environment 36 (2002) 4420–4421
That filtering appears to greatly reduce the slope variance is an artifact of least-squares estimation, which assumes independent residuals. The effect of filtering on the slope variance is perhaps best illustrated by expressing the slope variance as a product of the leastsquares weight function, wðf Þ; and the process spectral density, sðf Þ (Bloomfield and Nychka, 1992): s2 ðA# 1 Þ ¼ 2
Z
1=2
wðf Þsðf Þ df ;
ð6Þ
0
Where s2 ðA# 1 Þ is the true slope variance and f is frequency. The least-squares weight function, a transfer function for the least squares slope estimate A# 1 ; indicates the extent to which spectral bands in the time series of interest are transferred to the slope estimate. The transfer function wðf Þ is given by Pn ðt t%Þexpð2piftÞ wðf Þ ¼ t¼1 Pn ; ð7Þ 2 t¼1 ðt t%Þ where t is the time index. When data are filtered, the slope variance is given by Porter et al. (2001): s2 ðA# 1 Þ ¼ 2
Z
1=2
B2 ðf Þwðf Þsðf Þ df ;
ð8Þ
0
where B is the transfer function of the filter. Eq. (8) can also be used to estimate the slope variance given estimates of sðf Þ: This and similar approaches are discussed in Trenberth (1984), and Porter et al. (2001).
4421
References Bloomfield, P.R., Nychka, D., 1992. Climate Change 21, 275–287. Chan, D., Rao, S.T., Zurbenko, I.G., Porter, P.S., 1999. Linking changes in ozone to changes in emissions and meteorology. In: Brebbia, C.A., Jacobson, M., Power, H. (Eds.), Air Pollution VII. WIT Press, Southampton, UK, pp. 663–675. Eskridge, R.E., Ku, J.Y., Rao, S.T., Porter, P.S., Zurbenko, I.G., 1997. Separating different scales of motion in time series of meteorological variables. Bulletin of American Meteorological Society 78 (7), 1473–1483. Hess, A., Iyer, H., Malm, W., 2001. Linear trend analysis: a comparison of methods. Atmospheric Environment 35, 5211–5222. Hogrefe, C., Rao, S.T., Zurbenko, I.G., 1998. Detecting trends and biases in time series of ozonesonde data. Atmospheric Environment 32, 2569–2586. Kendall, M., Stuart, A., Ord, J.K., 1983. The Advanced Theory of Statistics. Vol. 3. Macmillan, New York. Law, A.M., Kelton, W.D., 1991. Simulation Modeling and Analysis. McGraw-Hill, New York. Milanchus, M.L., Rao, S.T., Zurbenko, I.G., 1998. Evaluating the effectiveness of ozone management efforts in the presence of meteorological variability. Journal of Air and Waste Management Society 48, 201–215. Porter, P.S., Rao, S.T., Zurbenko, I.G., Dunker, A.M., Wolff, G.T., 2001. Ozone air quality over North America: part II— an analysis of trend detection and attribution techniques. JAWMA 51, 283–306. Trenberth, K.E., 1984. Some effects of finite sample size and persistence on meteorological statistics. Part I: autocorelations. Mon. Weather Rev. 112, 2359–2368.
Discussion
Linear trend analysis: a comparison of methods$ P. Steven Portera,*, S.T. Raob, C. Hogrefeb b
a University of Idaho, 1776 Science Center Drive, Idaho Falls, ID 83402, USA University at Albany—SUNY, Atmospheric Sciences Research Center, 251 Fuller Road, Albany, NY 12203, USA
Hess et al. (2001. Atmospheric Environment 35, 5211– 5222) compare several tests for linear trends in air quality time series. They found that low-pass (i.e., KZ filter) filtered data produced actual type I error rates much smaller than nominal when based on the use of the following equation adapted from Eskridge et al. (1997): P # 2 1=2 ½ ðY YÞ ME ¼ t0:975;f ; P ðn 2Þ1=2 ½ ðt t%Þ2 1=2 f# ¼
1 þ 2=n
n Pn1
i¼1 ðn
iÞr# i
;
ð1Þ
where ME is the 95% margin of error, t0:975;f is the 0.975th percentile of Student’s t distribution with f degrees-of-freedom, Y is the filtered time series, Y# is the least squares linear approximation to Y ; n is sample size and r# is the sample autocorrelation. We believe that the type I error rates for the filtered data in Hess et al. (2001) are much too small because the effective sample size used for the slope standard deviation calculation is not n 2; as in Eq. (1), but f:# The expression for ME should therefore be P # 2 1=2 ½ ðY YÞ ME ¼ t0:975; f# : ð2Þ P f#1=2 ½ ðt t%Þ2 1=2 Use of Eq. (2) will result in approximately nominal type I error rates for filtered data. The use of autocorrelations to compute degrees-offreedom for filtered time series is awkward in any case, more so for irregularly sampled time series, because sample autocorrelations for filtered data may be significant at very long lag times, and as such are difficult to estimate (Law and Kelton, 1991; Kendall et al., 1983). Another way to estimate f is to notice that the reduction in degrees-of-freedom due to filtering is related to the amount of variance reduction. For the $
PII of original article S1352-2310(02)00189-9 *Corresponding author. E-mail address: [email protected] (P.S. Porter).
KZ filter, f ((n 2)/variance reduction) is given approximately by f# ¼
n2 pffiffiffi; k
ð2q þ 1Þ
ð3Þ
where 2q þ 1 is the window size for the moving average and k is the number of iterations. Perhaps a better method for estimating slopes and confidence intervals is to use a filter designed to remove only seasonality. This can be accomplished using both a low-pass and a high-pass filter (i.e., a notch filter). For example, define an air quality time series as the sum of 3 components: Rt ¼ Lt þ St þ Wt ;
ð4Þ
where Lt represents low-frequency variation, S seasonality, and W short-term (synoptic) variation. Estimate the slope and its standard deviation by applying classical least squares formulas to Yt : # t; Yt ¼ L# t þ W
ð5Þ
where L# t is a time series smoothed by a low-pass filter # t is the with a cut-off period of about 2 years, and W residual of a low-pass filter with a cut-off period of about 50 days. Use of a notch filter reduces the slope variance (by reducing seasonality) while retaining the high-frequency component simplifies computations, as for many air quality time series, Yt will more closely match the trend+white noise model assumed by classical least squares than will L# t : In other words, estimating the slope from Yt (the notch filtered time series) follows the same paradigm as the other methods examined by Hess et al. (2001) in that it only removes seasonality while retaining the effects of both long-term and short-term fluctuations on the slope estimates and, thus, the confidence intervals. This procedure was applied (although sometimes this was not explicitly stated) to estimate the trend confidence intervals in previous studies using the KZ filter (Milanchus et al., 1998; Hogrefe et al., 1998; Chan et al., 1999).
1352-2310/02/$ - see front matter r 2002 Elsevier Science Ltd. All rights reserved. PII: S 1 3 5 2 - 2 3 1 0 ( 0 2 ) 0 0 5 4 6 - 0
P.S. Porter et al. / Atmospheric Environment 36 (2002) 4420–4421
That filtering appears to greatly reduce the slope variance is an artifact of least-squares estimation, which assumes independent residuals. The effect of filtering on the slope variance is perhaps best illustrated by expressing the slope variance as a product of the leastsquares weight function, wðf Þ; and the process spectral density, sðf Þ (Bloomfield and Nychka, 1992): s2 ðA# 1 Þ ¼ 2
Z
1=2
wðf Þsðf Þ df ;
ð6Þ
0
Where s2 ðA# 1 Þ is the true slope variance and f is frequency. The least-squares weight function, a transfer function for the least squares slope estimate A# 1 ; indicates the extent to which spectral bands in the time series of interest are transferred to the slope estimate. The transfer function wðf Þ is given by Pn ðt t%Þexpð2piftÞ wðf Þ ¼ t¼1 Pn ; ð7Þ 2 t¼1 ðt t%Þ where t is the time index. When data are filtered, the slope variance is given by Porter et al. (2001): s2 ðA# 1 Þ ¼ 2
Z
1=2
B2 ðf Þwðf Þsðf Þ df ;
ð8Þ
0
where B is the transfer function of the filter. Eq. (8) can also be used to estimate the slope variance given estimates of sðf Þ: This and similar approaches are discussed in Trenberth (1984), and Porter et al. (2001).
4421
References Bloomfield, P.R., Nychka, D., 1992. Climate Change 21, 275–287. Chan, D., Rao, S.T., Zurbenko, I.G., Porter, P.S., 1999. Linking changes in ozone to changes in emissions and meteorology. In: Brebbia, C.A., Jacobson, M., Power, H. (Eds.), Air Pollution VII. WIT Press, Southampton, UK, pp. 663–675. Eskridge, R.E., Ku, J.Y., Rao, S.T., Porter, P.S., Zurbenko, I.G., 1997. Separating different scales of motion in time series of meteorological variables. Bulletin of American Meteorological Society 78 (7), 1473–1483. Hess, A., Iyer, H., Malm, W., 2001. Linear trend analysis: a comparison of methods. Atmospheric Environment 35, 5211–5222. Hogrefe, C., Rao, S.T., Zurbenko, I.G., 1998. Detecting trends and biases in time series of ozonesonde data. Atmospheric Environment 32, 2569–2586. Kendall, M., Stuart, A., Ord, J.K., 1983. The Advanced Theory of Statistics. Vol. 3. Macmillan, New York. Law, A.M., Kelton, W.D., 1991. Simulation Modeling and Analysis. McGraw-Hill, New York. Milanchus, M.L., Rao, S.T., Zurbenko, I.G., 1998. Evaluating the effectiveness of ozone management efforts in the presence of meteorological variability. Journal of Air and Waste Management Society 48, 201–215. Porter, P.S., Rao, S.T., Zurbenko, I.G., Dunker, A.M., Wolff, G.T., 2001. Ozone air quality over North America: part II— an analysis of trend detection and attribution techniques. JAWMA 51, 283–306. Trenberth, K.E., 1984. Some effects of finite sample size and persistence on meteorological statistics. Part I: autocorelations. Mon. Weather Rev. 112, 2359–2368.