- Email: [email protected]
The forecasting accuracy of major time series methods
International North-Holland
Journal
of Forecasting
2 (1986)
119-127
119
BOOK REVIEWS
The International
Journal of Forecasting Policy
The purposes of this section are to help our readers become aware of all books that are relevant to forecasting, and to emphasize those books making highly significant contributions. The Book Reviews section will include a listing of Books Received and Reviews. Readers, authors and publishers are invited to contribute to the listings or to the reviews. If you would like to do a review, please contact us first to determine whether a review is already being done. (For exceptional books, we may publish more than one review.) In preparing a review, please follow reference format given in the ‘Instructions to Authors’. The book reviews are being handled as follows: North and South America: Steven P. Schnaars Baruch College The City University of New York 17 Lexington Avenue NEW YORK, NY 10010 USA
Rest of the world: Nigel Meade Imperial College of Science and Technology Department of Management Science Exhibition Road LONDON SW7 2BX UK
Spyros Makridakis, Allen Andersen, Robert Carbone, Robert Fildes, Michele Hibon, Rudolf Lewandowski, Joseph Newton, Emmanuel Parzen and Robert Winkler, The Forecasting Accuracy of Major Time Series Methods (Wiley, New York and Chichester, 1984) $34.50/&43.50, pp. 301. * Often a reviewer is confronted with a book that delivers its message in an all too familiar way. Most definitely, such a criticism cannot be levelled at this book. The purpose of the text is to describe in detail a major empirical investigation into the strengths and weaknesses of different forecasting procedures. This study has become widely known as ‘ the M-competition’, and we shall use that label in this review. The opening chapter written by Makridakis provides an overview and also summarizes the conclusions. Chapter two is a reprint of a stimulating paper by Armstrong that first appeared in 1978 in the Journal of Business. Armstrong argues that econometric models are no better than simple extrapolative methods when used for short-term forecasting. In particular, no benefits are derived from increased model complexity. The original paper was followed by a lively discussion and a response by Armstrong. One might argue that Armstrong’s case is sustained, but that, from the viewpoint of many econometricians, the wrong question was asked. One explanation ‘for these findings is that, with some notable exceptions such as Hendry and Tremayne (1976), most * 0169-2070/86/$3.50
0 1986,
Elsevier
Science
Publishers
B.V. (North-Holland)
120
econometricians have been rather lax about the specification of the error structure of their models. @anger (1980) points out how developments in transfer function modeling are drawing time series analysis and econometrics back together, to the benefit of both. Chapter three, reprinted from the Journal of the Royal Statistical Society, series A, 1979, is a paper written by Makridakis and Hibon; it may be described as the preliminary round of the forecasting competition. In the best traditions of the Royal Statistical Society, a brisk discussion followed the reading of the paper, and the authors are to be commended for reproducing it in full. Several of the issues raised in that discussion will be considered in this review. The fourth chapter presents the results of the M-competition proper as reprinted from the Journal of Forecasting, 1982. It is perhaps unfortunate that the authors were unable to include the later discussion [Armstrong and Lusk (1983)] that appeared in the journal. The remaining chapters are all new and represent descriptions of different approaches used in the M-competition with, in some cases, an evaluation of the method’s performance. The reader may find it useful to read these before chapter four if some of the techniques are unfamiliar. The level of presentation in these later chapters is generally good. For example, Andersen and Weiss provide an excellent introduction to the Box-Jenkins approach, providing a detailed outline together with several worked examples. Other topics covered are AEP filtering (Carbone, Bilongo, Piat-Carson, and Nadeau), Bayesian forecasting (Fildes), simple extrapolative methods (Hibon), the FORSYS method (Lewandowski), ARARMA models (Newton and Parzen), and combinations of forecasts (Winkler). The material on Lewandowski’s FORSYS method is disappointing. The same description appears on pages 162-163 and on pages 264-265; it contains several misprints and numerous unspecified quantities. Since the original publications are somewhat inaccessible, a more detailed and self-contained summary would have been valuable. The results of the M-competition are now quite widely known, and so a brief summary will be sufficient, The key features were the relatively disappointing performance of the Box-Jenkins approach, the strong showing of simple methods such as exponential smoothing, and the strong performance of the FORSYS and simple combination methods. The simple combination was an unweighted average of several simple methods. Like a psychiatrist writing a whodunit, we dispose quickly of both victim and murderer and spend the rest of the time asking why. Many criticisms of the M-competition were first voiced in the Royal Statistical Society discussion. Some have been corrected, such as the use of a wider range of error measures; others remain. One problem relates to the nature of the data series used. As noted by Hill and Fildes (1984) 90% of the series are either macroeconomic or at least above company divisional level, and such series would not normally be forecast by single series extrapolation methods. Indeed, investigations of individual series reveal marked structural changes over time, thus casting doubt upon the validity of the assumptions that underly the forecasting procedures used. Herein lies a significant difference of opinion. To some, life is like that, and a forecasting method should be robust enough to withstand such shocks; to others, the methods are being misused, and the forecasts are unacceptable. This reviewer, and probably many readers also, would like to cling to the pragmatic middle approach whereby we continue to use single series methods while recognizing that the assumptions may be violated. Unfortunately, this position is not always tenable, as the following example illustrates. Suppose that a series generated by a random walk contains some additive outliers. After differencing, these outliers will induce a negative first-order autocorrelation, leading to the identification of an ARIMA (0, I, I) model. The resulting forecasting function is simple exponential smoothing. If the pattern of outliers is non-random, as is true for many macroeconomic series, the final forecasting function is inefficient. Such effects are particularly likely to occur when the series are short, and thus a single extreme value can distort the autocorrelation function. These comments suggest that robust estimation procedures [cf. Denby and Martin (1979)] need to be applied more routinely in business applications. Of course,
Book
reviews
121
the simple ad hoc procedures do not require parameter estimates, although outliers may still distort start-up values. In making the comparisons among methods, the M-competition routinely divided series into two parts: one for fitting and one for forecasting. Several discussants criticized the earlier study for the paucity of different forecasts and forecast origins that were used. Although this is somewhat rectified in the M-competition, the problem is still present. This difficulty is compounded by the series selected, many of which are macroeconomic series covering the same time periods, so that forecast errors are correlated across series. A few technical points are also apparent. The ARIMA models give optimal forecasts one step ahead, given the correct model. Chatfield (1978) found that the linear Holt-Winters (HW) schemes often do as well as a model selected by the Box-Jenkins approach. Although nominally the linear HW schemes are a subset of the ARIMA class, their coefficients are such that it is most unlikely that the usual identification schemes would lead to a linear HW model. Therefore, the de facto identification for the ARIMA model may be less than optimal. Alternatively, the model may be non-linear, as in the multiplicative Holt-Winters scheme. These factors may account for the strong performance of the simple combination of forecasts. Secondly, the ARIMA class of models is linear, yet multiplicative models of the ‘trend x seasonal + error’ form may be better. Such models include the multiplicative HW model and extensions such as, apparently, the FORSYS method. The third factor is the short-term nature of most, at least the non-seasonal, ARIMA models used for forecasting. For example, an ARIMA (0, 1, 2) scheme is clearly no better than a random walk scheme when we forecast more than two periods ahead. Since many of the identified models had a fairly simple form, we would not expect ARIMA forecasts for longer time horizons to show any improvement. Methods such as the ARARMA and FORSYS techniques, which endeavor to build in longer term components, are seen to render some improvements here. What then are we to conclude? The M-competition has its flaws, but these are not fatal. As software improves, the question to be answered will not be ‘quick and dirty’ or ‘expensive and resource-consuming’ but increasingly one of automated choices between classes of alternatives [cf. Hill and Fildes (1984)]. The M-competition has already stimulated research and will continue to do so, even if the research agenda is not always agreed; see Armstrong (1984) and the ensuing discussion. In short, everyone interested in short-term forecasting should read this book, but not uncritically. References Armstrong, J.S., 1984, Forecasting by extrapolation: Conclusions from twenty-five years of research, Interfaces 14, 52-66. Armstrong, J.S. and E. Lusk, 1983, Commentary on the Makridakis time series competition (M-competition), Journal of Forecasting 2, 259-311. Chatfield, C., 1978, The Holt-Winters forecasting approach, Applied Statistics 28, 264-279. Denby, L. and R.D. Martin, 1979, Robust estimation of the first order autoregressive parameters, Journal of the American Statistical Society 74, 140-146. Granger, C.W.J., 1980, On the synthesis of time-series and econometric models, in: D.R. Brillinger and G.C. Tiao, Directions of time series (Institute of Mathematical Statistics, Hayward, CA). Hendry, D.F. and A.R. Tremayne, 1976, Estimating systems of dynamic reduced form equations with vector autoregressive errors, International Economic Review 17, 463-471. Hill, G. and R. Fildes, 1984, The accuracy of extrapolation methods; An automatic Box-Jenkins package SIFT, Journal of Forecasting 3, 319-323.
Keith Ord Pennsylvania State University University Park, PA
Journal
of Forecasting
2 (1986)
119-127
119
BOOK REVIEWS
The International
Journal of Forecasting Policy
The purposes of this section are to help our readers become aware of all books that are relevant to forecasting, and to emphasize those books making highly significant contributions. The Book Reviews section will include a listing of Books Received and Reviews. Readers, authors and publishers are invited to contribute to the listings or to the reviews. If you would like to do a review, please contact us first to determine whether a review is already being done. (For exceptional books, we may publish more than one review.) In preparing a review, please follow reference format given in the ‘Instructions to Authors’. The book reviews are being handled as follows: North and South America: Steven P. Schnaars Baruch College The City University of New York 17 Lexington Avenue NEW YORK, NY 10010 USA
Rest of the world: Nigel Meade Imperial College of Science and Technology Department of Management Science Exhibition Road LONDON SW7 2BX UK
Spyros Makridakis, Allen Andersen, Robert Carbone, Robert Fildes, Michele Hibon, Rudolf Lewandowski, Joseph Newton, Emmanuel Parzen and Robert Winkler, The Forecasting Accuracy of Major Time Series Methods (Wiley, New York and Chichester, 1984) $34.50/&43.50, pp. 301. * Often a reviewer is confronted with a book that delivers its message in an all too familiar way. Most definitely, such a criticism cannot be levelled at this book. The purpose of the text is to describe in detail a major empirical investigation into the strengths and weaknesses of different forecasting procedures. This study has become widely known as ‘ the M-competition’, and we shall use that label in this review. The opening chapter written by Makridakis provides an overview and also summarizes the conclusions. Chapter two is a reprint of a stimulating paper by Armstrong that first appeared in 1978 in the Journal of Business. Armstrong argues that econometric models are no better than simple extrapolative methods when used for short-term forecasting. In particular, no benefits are derived from increased model complexity. The original paper was followed by a lively discussion and a response by Armstrong. One might argue that Armstrong’s case is sustained, but that, from the viewpoint of many econometricians, the wrong question was asked. One explanation ‘for these findings is that, with some notable exceptions such as Hendry and Tremayne (1976), most * 0169-2070/86/$3.50
0 1986,
Elsevier
Science
Publishers
B.V. (North-Holland)
120
econometricians have been rather lax about the specification of the error structure of their models. @anger (1980) points out how developments in transfer function modeling are drawing time series analysis and econometrics back together, to the benefit of both. Chapter three, reprinted from the Journal of the Royal Statistical Society, series A, 1979, is a paper written by Makridakis and Hibon; it may be described as the preliminary round of the forecasting competition. In the best traditions of the Royal Statistical Society, a brisk discussion followed the reading of the paper, and the authors are to be commended for reproducing it in full. Several of the issues raised in that discussion will be considered in this review. The fourth chapter presents the results of the M-competition proper as reprinted from the Journal of Forecasting, 1982. It is perhaps unfortunate that the authors were unable to include the later discussion [Armstrong and Lusk (1983)] that appeared in the journal. The remaining chapters are all new and represent descriptions of different approaches used in the M-competition with, in some cases, an evaluation of the method’s performance. The reader may find it useful to read these before chapter four if some of the techniques are unfamiliar. The level of presentation in these later chapters is generally good. For example, Andersen and Weiss provide an excellent introduction to the Box-Jenkins approach, providing a detailed outline together with several worked examples. Other topics covered are AEP filtering (Carbone, Bilongo, Piat-Carson, and Nadeau), Bayesian forecasting (Fildes), simple extrapolative methods (Hibon), the FORSYS method (Lewandowski), ARARMA models (Newton and Parzen), and combinations of forecasts (Winkler). The material on Lewandowski’s FORSYS method is disappointing. The same description appears on pages 162-163 and on pages 264-265; it contains several misprints and numerous unspecified quantities. Since the original publications are somewhat inaccessible, a more detailed and self-contained summary would have been valuable. The results of the M-competition are now quite widely known, and so a brief summary will be sufficient, The key features were the relatively disappointing performance of the Box-Jenkins approach, the strong showing of simple methods such as exponential smoothing, and the strong performance of the FORSYS and simple combination methods. The simple combination was an unweighted average of several simple methods. Like a psychiatrist writing a whodunit, we dispose quickly of both victim and murderer and spend the rest of the time asking why. Many criticisms of the M-competition were first voiced in the Royal Statistical Society discussion. Some have been corrected, such as the use of a wider range of error measures; others remain. One problem relates to the nature of the data series used. As noted by Hill and Fildes (1984) 90% of the series are either macroeconomic or at least above company divisional level, and such series would not normally be forecast by single series extrapolation methods. Indeed, investigations of individual series reveal marked structural changes over time, thus casting doubt upon the validity of the assumptions that underly the forecasting procedures used. Herein lies a significant difference of opinion. To some, life is like that, and a forecasting method should be robust enough to withstand such shocks; to others, the methods are being misused, and the forecasts are unacceptable. This reviewer, and probably many readers also, would like to cling to the pragmatic middle approach whereby we continue to use single series methods while recognizing that the assumptions may be violated. Unfortunately, this position is not always tenable, as the following example illustrates. Suppose that a series generated by a random walk contains some additive outliers. After differencing, these outliers will induce a negative first-order autocorrelation, leading to the identification of an ARIMA (0, I, I) model. The resulting forecasting function is simple exponential smoothing. If the pattern of outliers is non-random, as is true for many macroeconomic series, the final forecasting function is inefficient. Such effects are particularly likely to occur when the series are short, and thus a single extreme value can distort the autocorrelation function. These comments suggest that robust estimation procedures [cf. Denby and Martin (1979)] need to be applied more routinely in business applications. Of course,
Book
reviews
121
the simple ad hoc procedures do not require parameter estimates, although outliers may still distort start-up values. In making the comparisons among methods, the M-competition routinely divided series into two parts: one for fitting and one for forecasting. Several discussants criticized the earlier study for the paucity of different forecasts and forecast origins that were used. Although this is somewhat rectified in the M-competition, the problem is still present. This difficulty is compounded by the series selected, many of which are macroeconomic series covering the same time periods, so that forecast errors are correlated across series. A few technical points are also apparent. The ARIMA models give optimal forecasts one step ahead, given the correct model. Chatfield (1978) found that the linear Holt-Winters (HW) schemes often do as well as a model selected by the Box-Jenkins approach. Although nominally the linear HW schemes are a subset of the ARIMA class, their coefficients are such that it is most unlikely that the usual identification schemes would lead to a linear HW model. Therefore, the de facto identification for the ARIMA model may be less than optimal. Alternatively, the model may be non-linear, as in the multiplicative Holt-Winters scheme. These factors may account for the strong performance of the simple combination of forecasts. Secondly, the ARIMA class of models is linear, yet multiplicative models of the ‘trend x seasonal + error’ form may be better. Such models include the multiplicative HW model and extensions such as, apparently, the FORSYS method. The third factor is the short-term nature of most, at least the non-seasonal, ARIMA models used for forecasting. For example, an ARIMA (0, 1, 2) scheme is clearly no better than a random walk scheme when we forecast more than two periods ahead. Since many of the identified models had a fairly simple form, we would not expect ARIMA forecasts for longer time horizons to show any improvement. Methods such as the ARARMA and FORSYS techniques, which endeavor to build in longer term components, are seen to render some improvements here. What then are we to conclude? The M-competition has its flaws, but these are not fatal. As software improves, the question to be answered will not be ‘quick and dirty’ or ‘expensive and resource-consuming’ but increasingly one of automated choices between classes of alternatives [cf. Hill and Fildes (1984)]. The M-competition has already stimulated research and will continue to do so, even if the research agenda is not always agreed; see Armstrong (1984) and the ensuing discussion. In short, everyone interested in short-term forecasting should read this book, but not uncritically. References Armstrong, J.S., 1984, Forecasting by extrapolation: Conclusions from twenty-five years of research, Interfaces 14, 52-66. Armstrong, J.S. and E. Lusk, 1983, Commentary on the Makridakis time series competition (M-competition), Journal of Forecasting 2, 259-311. Chatfield, C., 1978, The Holt-Winters forecasting approach, Applied Statistics 28, 264-279. Denby, L. and R.D. Martin, 1979, Robust estimation of the first order autoregressive parameters, Journal of the American Statistical Society 74, 140-146. Granger, C.W.J., 1980, On the synthesis of time-series and econometric models, in: D.R. Brillinger and G.C. Tiao, Directions of time series (Institute of Mathematical Statistics, Hayward, CA). Hendry, D.F. and A.R. Tremayne, 1976, Estimating systems of dynamic reduced form equations with vector autoregressive errors, International Economic Review 17, 463-471. Hill, G. and R. Fildes, 1984, The accuracy of extrapolation methods; An automatic Box-Jenkins package SIFT, Journal of Forecasting 3, 319-323.
Keith Ord Pennsylvania State University University Park, PA