- Email: [email protected]
Are the GARCH models best in out-of-sample performance?
Economics Letters North-Holland
305
37 (1991) 305-308
Are the GARCH models best in out-of-sample performance? Keun Yeong Lee Unilwsityof Wisconsin - Madison, Madison, Received Accepted
WI 53706, USA
24 June 1991 22 July 1991
Out-of-sample performance of exchange rate volatility model depends on criteria models cannot generally outperform the nonlinear models in the RMSE criterion. model is best in the MAE criterion.
used to measure it. The linear GARCH Furthermore, the nonparametric kernel
1. Introduction This paper basically investigates out-of-samples forecasting accuracy of the nonparametric kernel model, the GARCH models and their modified models for exchange rate volatility. Several authors [see, e.g., Engle, Hong and Kane (199O)l in the stock and foreign exchange market compared out-of-sample performance. They discovered that the GARCH models tend to do best. Since the GARCH models seem to dominate the other models, it is meaningful to develop any better model than the GARCH model and its extensions. I study five spot exchange rates Wednesdays, Noon, from the ninth week of 1973 (March 7) through the 40th week of 1989 (October 4) as published in The Federal Reserve Bulletin. Sample size is therefore equal to 866. The exchange rates are Japanese Yen, German Mark, British Pound, French Franc and Canadian Dollar. All are measured in US$/foreign currency. My interest centers on percent exchange rate changes, A In S,, on which we have 865 observations (10th week of 1973 through 40th week of 1989).
2. Models I choose
the following (In S,-In
GARCH(l,l)
model.
S,_,).lOO=U,,
(1)
WI, - N(O,u,*),
(2)
(3) * I would like to thank 0165-1765/91/$03.50
Wen Ling Lin for helpful 0 1991 - Elsevier
Science
comments. Publishers
B.V. All rights reserved
where
S is a spot
h is set to c ri,Tp’ where c is constant and &, is the sample standard deviation of I,. I conduct experiments with I, = U, ,. Bierens (1990) proposed that the best k-steps-ahead forecast of one of the variables can be consistently estimated by the nonparametric regression on an ARMA memory index. The one-step-ahead variance forecast of the ARMA model uses least squares. while the one-step-ahead variance of the GARCH model uses maximum likelihood. An ARMAtGARCH-I) memory index should be a one step ahead forecast with a linear ARMAtGARCH) model.
3. Empirical
results
Observations 452 (October 21. 1981) through 865 (October 11, 1989) arc reserved for out-ofsample forecast comparision. I use two statistics to measure out-of-sample accuracy - the root mean square error (RMSEI and the mean absolute error (MAE). Comparision of R’ has the same result as the RMSE in out-of-sample performance. I first add the sample by keeping the first observation and then roll the sample by fixing T = 450. Table 1 shows out-of-sample performance. Rankings of the models are different between the RMSE and MAE criterion. The RMSE is improved by rolling the sample. But the kernel model has a better MAE by adding the sample. I use 6 grid points tc = 1 - 5 and ml for Gaussian kernel. while 18 grid points (c = 0.5, 1. 1.5, 2, 2.5, 3 and E = 0.05, 0.1, 0.151 for truncated kernel. I do cross-validation over the sample size from 350 to 449 for the ARMA(GARCH) index model. It is applied with 6 grid points (c = 1 - 5 and ~1 for Gaussian kernel. The ARMACGARCH) index models generally outperform the GARCH models for three countries except U.K. and Japan. The smallest RMSE and MAE for the kernel model are only reported to save space in table 1. The MAEs of the nonparametric kernel model arc smaller than those of the other models. When bandwidth h goes to infinite, one-step-ahead out-of-sample performance of the Gaussian kernel model is the same as that of the homoscedastic model because the forecast value of as + x,7 is mean of Cl,‘(I77l,U/T). As the RMSE criterion gives large values more weights than the MAE criterion, the GARCH(l,l) model may explain high volatility better than the nonparametric kernel model. The nonparametric kernel model or homoscadastic model is relatively better than the GARCH and IGARCH models in the multi-steps-ahead out-of-sample performance. When 1actual value - forecast value 1 is greater than constant . (r we can improve the forecasting ability of the GARCH model by replacing the actual value with the other value (e.g., constant. (T) from those of the for Canada, France and Germany. These results are not different ARMA(GARCH) index models.
K.-Y. Lee / GARCH Table 1 One-step-ahead
RMSE L NL
L NL
MAE L NL
L NL
out-of-sample
forecast.
models and our-of-sample performance
307
“
Canada
Japan
UK
France
Germany
GARCHfA) IGARTtAI EGARCHtAl Index(A) Kerl(A) Ker2(A) GARCHtRI ICART EGARCHtRl Index(R) KerltRl Ker2(R)
0.687 0.685 0.682 0.683 0.688 0.685 0.683 0.686 0.679 ’ 0.681 ’ 0.682 ’ 0.682
4.421 4.417 4.416 4.397 j 4.4% 4.411 4.391 ’ 4.3Y7 7.072 4.731 4.427 4.395 z
5.565 3 5.538 ’ 5.610 7.131 5.737 5.714 5.651 5.549 z 6.345 6.8Y3 5.708 5.678
5.307 5.066 5.123 5.030 ’ 5.051 -q 5.059 5.288 5.063 6.192 5.040 z 5.062 5.058
4.601 4.49Y 4.494 4.477 2 4.541 4.482 7 4.592 4.4Y2 4504 4.447 ’ 4.512 4.494
GARCH(A1 IGARTtA) EGARCHtAl Index(A) KerltA) Ker2fA) GARCH(R1 IGARTtR) EGARCHtR) Index(R) Kerl(R1 Ker2tRl
0.372 0.391 0.364 0.350 z 0.340 z 0.338 ’ 0.367 0.398 0.368 0.366 0.358 0.357
2.525 2.529 2.527 2.294 3 2.271 ’ 2.288 z 2.397 2.481 3.272 2.439 2.394 2.372
2.826 2.865 2.712 2.804 2.561 ’ 2.537 ’ 2.Y49 2.990 3.416 2.x59 2.705 2.682 ’
2.8X4 2.678 2.855 2.362 ’ 2.344 ’ 2.295 ’ 2.660 2.729 3.324 2.450 2.414 2.3YY
2.610 2.549 2.629 2.364 2.201 ’ 2.254 ’ 2.524 2.55 1 2.55 I 2.384 2.287 a 2.308
” LfNLI: linear (nonlinear) model. IGART: IGARCH model with trend. A adds the sample with the first sample keeping. R rolls the sample by fixing T = 450. Index: ARMACGARCH) index model using Gaussian kernel. Kerlt2): nonparametric Gaussian (truncated) kernel model. Ifulic in Table 1 means that bandwidth h =F= thomoscadastic model). I only report the smallest RMSE and MAE for the kernel models.
4. Conclusion The linear GARCH models cannot generally outperform the nonlinear models in the RMSE criterion. But the nonparametric kernel model outperforms the GARCH model and its applied models in the MAE criterion. The MAE criterion is less sensitive to outlier observations. The homoscadastic model is considered to be the special case of the Gaussian kernel model in out-of-sample forecast. The kernal model or the homoscedastic model is relatively better than the GARCH(l,l) models in the multi-steps-ahead out-of-sample performance. The ARMACGARCH) index model may give us information for the extent of misspecification of the GARCH model.
References Bierens, H.J., 1990, Model-free asymptotically best forecasting of stationary economic time series, Econometric Theory 6. 348-383. Engle, R.F.. T. Hong and A. Kane, 1990, Valuation of variance forecasts with simulated options markets, Unpublished manuscript (Department of Economics. University of California, San Diego, CA).
Hsieh, D.A..
1989, Modelling
heteroskedasticity
in daily foreign exchange rates. Journal of Business and Economic
Statistics
7.307-317. Pagan.
A.R.
267-290.
and G.W.
Schwert,
1990. Alternative
models
for conditional
stock volatility,
Journal
of Econometrics
45.
305
37 (1991) 305-308
Are the GARCH models best in out-of-sample performance? Keun Yeong Lee Unilwsityof Wisconsin - Madison, Madison, Received Accepted
WI 53706, USA
24 June 1991 22 July 1991
Out-of-sample performance of exchange rate volatility model depends on criteria models cannot generally outperform the nonlinear models in the RMSE criterion. model is best in the MAE criterion.
used to measure it. The linear GARCH Furthermore, the nonparametric kernel
1. Introduction This paper basically investigates out-of-samples forecasting accuracy of the nonparametric kernel model, the GARCH models and their modified models for exchange rate volatility. Several authors [see, e.g., Engle, Hong and Kane (199O)l in the stock and foreign exchange market compared out-of-sample performance. They discovered that the GARCH models tend to do best. Since the GARCH models seem to dominate the other models, it is meaningful to develop any better model than the GARCH model and its extensions. I study five spot exchange rates Wednesdays, Noon, from the ninth week of 1973 (March 7) through the 40th week of 1989 (October 4) as published in The Federal Reserve Bulletin. Sample size is therefore equal to 866. The exchange rates are Japanese Yen, German Mark, British Pound, French Franc and Canadian Dollar. All are measured in US$/foreign currency. My interest centers on percent exchange rate changes, A In S,, on which we have 865 observations (10th week of 1973 through 40th week of 1989).
2. Models I choose
the following (In S,-In
GARCH(l,l)
model.
S,_,).lOO=U,,
(1)
WI, - N(O,u,*),
(2)
(3) * I would like to thank 0165-1765/91/$03.50
Wen Ling Lin for helpful 0 1991 - Elsevier
Science
comments. Publishers
B.V. All rights reserved
where
S is a spot
h is set to c ri,Tp’ where c is constant and &, is the sample standard deviation of I,. I conduct experiments with I, = U, ,. Bierens (1990) proposed that the best k-steps-ahead forecast of one of the variables can be consistently estimated by the nonparametric regression on an ARMA memory index. The one-step-ahead variance forecast of the ARMA model uses least squares. while the one-step-ahead variance of the GARCH model uses maximum likelihood. An ARMAtGARCH-I) memory index should be a one step ahead forecast with a linear ARMAtGARCH) model.
3. Empirical
results
Observations 452 (October 21. 1981) through 865 (October 11, 1989) arc reserved for out-ofsample forecast comparision. I use two statistics to measure out-of-sample accuracy - the root mean square error (RMSEI and the mean absolute error (MAE). Comparision of R’ has the same result as the RMSE in out-of-sample performance. I first add the sample by keeping the first observation and then roll the sample by fixing T = 450. Table 1 shows out-of-sample performance. Rankings of the models are different between the RMSE and MAE criterion. The RMSE is improved by rolling the sample. But the kernel model has a better MAE by adding the sample. I use 6 grid points tc = 1 - 5 and ml for Gaussian kernel. while 18 grid points (c = 0.5, 1. 1.5, 2, 2.5, 3 and E = 0.05, 0.1, 0.151 for truncated kernel. I do cross-validation over the sample size from 350 to 449 for the ARMA(GARCH) index model. It is applied with 6 grid points (c = 1 - 5 and ~1 for Gaussian kernel. The ARMACGARCH) index models generally outperform the GARCH models for three countries except U.K. and Japan. The smallest RMSE and MAE for the kernel model are only reported to save space in table 1. The MAEs of the nonparametric kernel model arc smaller than those of the other models. When bandwidth h goes to infinite, one-step-ahead out-of-sample performance of the Gaussian kernel model is the same as that of the homoscedastic model because the forecast value of as + x,7 is mean of Cl,‘(I77l,U/T). As the RMSE criterion gives large values more weights than the MAE criterion, the GARCH(l,l) model may explain high volatility better than the nonparametric kernel model. The nonparametric kernel model or homoscadastic model is relatively better than the GARCH and IGARCH models in the multi-steps-ahead out-of-sample performance. When 1actual value - forecast value 1 is greater than constant . (r we can improve the forecasting ability of the GARCH model by replacing the actual value with the other value (e.g., constant. (T) from those of the for Canada, France and Germany. These results are not different ARMA(GARCH) index models.
K.-Y. Lee / GARCH Table 1 One-step-ahead
RMSE L NL
L NL
MAE L NL
L NL
out-of-sample
forecast.
models and our-of-sample performance
307
“
Canada
Japan
UK
France
Germany
GARCHfA) IGARTtAI EGARCHtAl Index(A) Kerl(A) Ker2(A) GARCHtRI ICART EGARCHtRl Index(R) KerltRl Ker2(R)
0.687 0.685 0.682 0.683 0.688 0.685 0.683 0.686 0.679 ’ 0.681 ’ 0.682 ’ 0.682
4.421 4.417 4.416 4.397 j 4.4% 4.411 4.391 ’ 4.3Y7 7.072 4.731 4.427 4.395 z
5.565 3 5.538 ’ 5.610 7.131 5.737 5.714 5.651 5.549 z 6.345 6.8Y3 5.708 5.678
5.307 5.066 5.123 5.030 ’ 5.051 -q 5.059 5.288 5.063 6.192 5.040 z 5.062 5.058
4.601 4.49Y 4.494 4.477 2 4.541 4.482 7 4.592 4.4Y2 4504 4.447 ’ 4.512 4.494
GARCH(A1 IGARTtA) EGARCHtAl Index(A) KerltA) Ker2fA) GARCH(R1 IGARTtR) EGARCHtR) Index(R) Kerl(R1 Ker2tRl
0.372 0.391 0.364 0.350 z 0.340 z 0.338 ’ 0.367 0.398 0.368 0.366 0.358 0.357
2.525 2.529 2.527 2.294 3 2.271 ’ 2.288 z 2.397 2.481 3.272 2.439 2.394 2.372
2.826 2.865 2.712 2.804 2.561 ’ 2.537 ’ 2.Y49 2.990 3.416 2.x59 2.705 2.682 ’
2.8X4 2.678 2.855 2.362 ’ 2.344 ’ 2.295 ’ 2.660 2.729 3.324 2.450 2.414 2.3YY
2.610 2.549 2.629 2.364 2.201 ’ 2.254 ’ 2.524 2.55 1 2.55 I 2.384 2.287 a 2.308
” LfNLI: linear (nonlinear) model. IGART: IGARCH model with trend. A adds the sample with the first sample keeping. R rolls the sample by fixing T = 450. Index: ARMACGARCH) index model using Gaussian kernel. Kerlt2): nonparametric Gaussian (truncated) kernel model. Ifulic in Table 1 means that bandwidth h =F= thomoscadastic model). I only report the smallest RMSE and MAE for the kernel models.
4. Conclusion The linear GARCH models cannot generally outperform the nonlinear models in the RMSE criterion. But the nonparametric kernel model outperforms the GARCH model and its applied models in the MAE criterion. The MAE criterion is less sensitive to outlier observations. The homoscadastic model is considered to be the special case of the Gaussian kernel model in out-of-sample forecast. The kernal model or the homoscedastic model is relatively better than the GARCH(l,l) models in the multi-steps-ahead out-of-sample performance. The ARMACGARCH) index model may give us information for the extent of misspecification of the GARCH model.
References Bierens, H.J., 1990, Model-free asymptotically best forecasting of stationary economic time series, Econometric Theory 6. 348-383. Engle, R.F.. T. Hong and A. Kane, 1990, Valuation of variance forecasts with simulated options markets, Unpublished manuscript (Department of Economics. University of California, San Diego, CA).
Hsieh, D.A..
1989, Modelling
heteroskedasticity
in daily foreign exchange rates. Journal of Business and Economic
Statistics
7.307-317. Pagan.
A.R.
267-290.
and G.W.
Schwert,
1990. Alternative
models
for conditional
stock volatility,
Journal
of Econometrics
45.