International Journal of Forecasting 3 (1987) 239-243 North-Holland
239
A NOTE ON THE VALUE OF COMBINING SHORT-TERM EARNINGS FORECASTS * A Test of Grauger and Ramauathan Jay S. HOEMEN University of Wisconsin, Eau Claire, WI 54701, USA
Abstract:
This paper conducts an empirical analysis of the approaches to obtaining linear combinations of forecasts. Simulated quarterly earnings were modeled using three ARIMA models. One-quarter ahead forecasts were then developed. These forecasts were combined using alternative approaches. The most accurate forecasts were obtained by adding a constant term and not constraining the weights to add up to one. The differences in the accuracy rankings were found to be statistically significant. The results are similar to those obtained by Granger and Ramanathan (1984).
Keywords: Combining forecasts, ARIMA model;.
1. introduction Granger and Ramanathan (1984) presented three methods for the linear combination of forecasts, and they illustrated an application using quarterly hog prices from an econometric model, an ARIMA model, and expert opinions. The purpose of this paper is to examine the authors’ methods on a simulated data set using quarterly earnings data from three ARIMA models. The accounting literature suggests that there are three ARIMA models that capture the variability in the quarterly series of corporate earnings: an ARIMA (100) x (011) known as the ‘Brown-Rozeff’ model; an ARIMA (100) X (OlO), known as the ‘Foster’ model; and an ARIMA (011) X (011). known as the ‘Griffin-Watts’ model. [SC Brown and Rozef (1979) Foster (1979), Griffin (1977) and Watts (1975) for details of each specific model.] Support may be found for each of these models vis a vis the other two models in terms of their one-quarter ahead forecast superiority [see Bao et al. (1983) for a review of this literature]. Of interest here is whether a combination of these ARIMA forecasts would produce more accurate predictions than any single model.
ination of forecasts Granger and Ramanathan presented three methods that could be used in the linear combination of forecasts. Method A is a linear combination with no constraints put on the coefficients and no constant value used. Method B is a linear combination with a constraint that the coefficients sum to * This research was financed in part by a University Wisconsin-Eau Claire Graduate School.
Research
and Creative Activities
Grant
by the University
of
240
J.S. Holmen / The value of combining short-term earnings forecasts
I. Method C is a linear combination with no constraints ou the coefficients, but a constant value is used. Method B is the averaging or weighted averaging approach. Newbold and Granger (1974), Makridakis et al. (1982), Winkler and Makridakis (1983), and Clemen and Winkler (1986) all have concluded that simple or weighted average linear combinations of forecasts outperform individual methods of forecasting. BUM (1985) reported on a number of application studies and provided guidance on how the weights may be determined. Granger and Ramanathan maintain that Method C is best because it yields the smallest mean squared error of the three models and it yields an unbiased combined forecast even if the individual forecasts are biased. In a simple application of the combination of ARIMA, econometric, and expert opinion forecasts, Granger and Ramanathm found Method C to yield empirically the lowest mean squared error.
3. Data Tile data used in this study are the result of a simulation study. Simulation was chosen in order to obtain large sample sizes. The drawback of the simulation procedure, however, is that other desirable forecasts, such as econometric forecasts or forecasts by security analysts, cannot be used. Since the basic thrust of this paper is to test GrangLi, and Ramanathan’s proposition that an unconstrained combination with a constant term performs better than individual forecasts, the use of simulation should not cause concern. The three proposed premier ARIMA models were used as the basic generating models. The factors that were varied were the level of the parameters, the initial seasonal pattern, and the variability level for the random number generator. The parameter levels are based on reported values found in the accounting literature [see Lorek, Icerman and Abdulkader (1983) and Griffin (1977)]. Two input patterns (one level, the other seasonal) were used to model the seasonal&y. Since the first 40 data values were discarded, the effects of these patterns were mitigated. Three levels of variability cl%,lo%, and 25%) were used to induce various levels of random fluctuations. The literature does not provide sufficient guidance in the choice of input patterns and variability levels, hence they were chosen on an ad hoc basis. For the Brown-Rozeff [ARIMA (100) X (011)] generating model, nine different levels of parameters (AR = 0.4, 0.5 or 0.6; seasonal MA = 0.7, 0.8 or 0.9) were used, yielding 54 different input combinations. For the Griffin-Watts [ARIMA (011) x (Oil)] generating model, nine different levels of parameters (MA = 0.3, 0.4 or 0.5; seasonal MA = 0.4, 0.5 or 0.6) were used, yielding 54 input combinations. For the Foster [ARIMA (100) x (OlO)] generating model, seven different levels of parameters (AR = 0.4, 0.5, 0.6, 0.7, 0.8, 0.9 or 1.0) were used, yielding 42 input combinations. Ten replications were performed, yielding a total of 1500 series. Each of the 1500 series was Imodeled using each of the proposed ‘premier’ ARIMA models (abbreviated here by BR, F, and GW). One-quarter ahead forecasts were then generated.
The results are presented in exhibits 1 and 2. The 1500 series were split into two equal groups: one for estimating the coefficients; the other for evaluating the results. Each group contained five replicates for each of the input combinations. Exhibit 1 presents the mean error (IWE), the mean
J.S. Holmen / The value of combining short-term earnings forecasts
241
absolute error (MAE), and mean squared error (MSE) ’ for the sample used to estimate the coefficients. Of the three original series the GW premier model has the lowest MSE. Of the three combination models using all three individual forecasting series, Method B reduces MSE by 3.6%, Method A by 5.596,and Method C by 5.7%. In addition, combining all three original premier models yields a lower error rate than combining only two. Notice that no matter which series are being combined, combination Method C has the lowest MSE. Finally, Methods A and B yield biased results while Method C yields unbiased results. Exhibit 2 presents the results for the holdout sample. Once again, the GW premier model has the lowest MSE of the original series. Again, all three methods which combine the three models yield a lower MSE than the best single forecast. The order of preference is the same for the holdout sample as it was for the estimating sample. Method B yields a 3.5% reduction in MSE, Method A yields an 8.0% reduction, and Method C yields an 8.5% reduction. Also, the combination of all three original models is superior to the combination of only two. Again, Method C is ‘best’ no matter which series are being combined. Exhibit 1 Weights and forecast error rates - estimate sample (n = 750). a Forecast
Mean error
Mean abs. error
Mean squared error
Constant
- 1.03 - 0.71 0.25
22.6 23.0 11.6
2020 2060 375
0.00 0.00 0.00
1.oo 0.00 0.00
Model A (unconstraineri no constant term) All - 0.69 11.4
354
0.00
0.37
- 0.25
0.86
BR&F BR&GW F&GW
890 359 365
0.00 0.00 0.00
0.79 0.14 0.00
0.15 0.00 0.11
0.00 0.85 0.88
Combination weights BR
GW
F
Original BR F GW
- 6.32 - 0.69 - 0.45
19.0 11.5 11.5
O.O!I 1.00 0.00
0.00 0.00 1.00
Model B (no constant, weights sum to I) All BR&F BR&GW
F&GW
0.09 - 0.96 0.17
11.4 22.6 11.5
361 2015 367
0.00 0.00 0.00
0.34 0.76 0.06
- 0.28 0.24 0.00
0.94 0.00 0.94
0.20
11.6
370
0.00
0.00
0.05
0.95
353 849 358 364
0.76 6.49 0.75 0.49
0.38 0.82 0.14 0.00
- 0.25 0.12 0.00 0.11
0.86 0.00 0.85 0.88
Model C (unconstrained, with constant) All BR&F BR&GW F&GW
- 0.00 - 0.00 0.00 0.00
11.3 16.8 11.3 11.5
The expected value for any of the forecasts was designed to be equal to 100. so the values in the exhibits may be cor:sidered percentage errors. BR = Brown-Roteff: ARTMA (100) x (011). F = Foster: (100) X (010). GW = Griffin-Watts: (011) X(011).
TO be consistent with Granger and Ramanathdn. mean squared error is uhcd hcrc. The mcdran &solute error gives an indication of the number of very large errors. For the estimation sample the values would be: BR: 8.50; F: 8.59: GW: 5.02; Method A: 4.45: Method B: 4.87; Method C: 4.55. For the holdout ssmple the values would be: BR: 8.36; F: 8.42: GW: 5.55; Method A: 5.61; Method B: 5.61; Method C: 5.80. The maximum ab.&rte error values observed were between 90 and 100 for the GW model and Methods A, B. and C for both the estrmation and holdout samples. The maxrmum values for the BR and F models were roughly equivalent for each sample (395 for the estimation sample am! 428 for the holdout sample). The 90% decile values would be approximately 50 for BR and F; roughly 33 for GW and Methods A, 13,and C (with the estimation and holdout samples being roughly the same).
J.S. Holmen / The value of combining short-term earnings forecasts
242
To assess whether the observed reductions in MSE are statistically significant, the van-parametric Wilcoxon signed-rank test was performed. Only the differences between the GW prentier model and the combination methods using all three models were analyzed. For the estimation sample, the difference between the GW model and Method B is significant at a p-value of 0.018. The difference between Method B and Method A is significant at a p-value of 0.004, while the difference between Method A and Method C is significant at a p-value of less than 0.001. For the holdout sample, the difference between the GW model and Method B is significant at a p-value of 0.003, the difference between Method B and Method A is not significant, and the difference between Method A and Method C is significant at a p-value of less than 0.001. Thus, Granger and Ramanathan’s claim that Method C is preferable is confirmed, with the differences being statistically significant. 5. Conclusions The results presented here support Granger and Ramanathan’s conclusions that the combination of forecasts yields an error rate lower than that for any of the original forecasts alone. Of three potential linear combinations of forecasts, an unrestricted combination with a constant term added (Method C) yields the lowest error rate. The results are quite modest, with an 8.5% reduction in MSE. In addition, Granger and Ramanathan’s contention that the unrestricted combination with a constant term is superior to a weighted average (Method B: with coefficients constrained to sum to 1) is supported by this study. The incremental reduction in MSE (between Method B and Method C) for the holdout sample was 5.2%. This difference is statistically significant, with a p-value of less than 0.001. Exhibit 2 Weights and forecast error rates - holdout sample (n = 750). a Forecast
Original BR F GW
Mean error
Mean abs. error
Mean squared error
Constant
1.19 0.99 - 0.22
20.9 20.8 12.1
1819 1782 376
0.00 0.00 0.00
1.00 0.00 0.00
0.00 1.00 0.00
0.00 0.00 1.00
346 758 350 359
0.00 0.00 0.00 0.00
0.37 0.79 0.14 0.00
- 0.25 0.15 0.00 0.11
0.86 0.00 0.85 0.88
363 1796 368 371
0.00 0.00 0.00 0.00
0.34 0.76 0.06 0.00
- 0.28 0.24 0.00 0.05
0.94 0.00 0.94 0.95
345 722 349 358
0.76 6.49 0.75 0.49
0.38 0.82 0.14 0.00
- 0.25 0.12 0.00 0.11
0.86 0.00 0.85 0.88
Model A (unconstrained no constant term) All - 0.94 11.7 BRSLF - 6.27 18.3 BRBrGW - 1.06 11.7 F&GW - 0.89 11.9 Model B (no constant, weights sum to I) All - 0.08 11.8 BR&F 1.14 20.7 BR&GW -0.13 11.9 F&GW - 0.16 12.0
Combination BR
weights F
GW
Model C (unconstrained, with constant) All BR&F BR&GW F&GW
- 0.25 - 0.01 -0.36 - 0.44
11.6 16.1 11.6 113
a The expected value for any of the forecasts was designed to be equal to 100, so the values in the exlribits may be considered percentage errors. BR = Brown-Rozeff: ARIMA (100)~(011), F= Foster: (100)~(010), GW = Griffin-Watts: (011)X(011).
J.S. Holmen / The value of combining short-term earnings forecasts
243
References Bao, D.-H., MT. Lewis, W.T. Lin and J.G. Manegold, 1983, Applications of time-series analysis in accounting: A review, Journal of Forecasting 2,405-423. Brown, L.D. and M.S. Rozeff, 1979, Univariate time-series models of quarterly accounting earnings per share: A proposed model, Journal of Accounting Research 17,179-189. Bunn, D.W., 1985, Statistical efficiency in the linear combination of forecasts, International Journal of Forecasting 1, 151-163. Clemen, R.T. and R.L. Winkler, 1986, Combining economic forecasts, Journal of Business and Economic Statistics 4.39-46. Foster, G., 1977, Quarterly accounting data: Time series properties and predictive ability results, The Accounting Review 52, 1-21. Granger, C.W.J. and R. Ramanathan, 1984, Improved methods of combining forecasts. Journal of Forecasting 3, 197-204. Griffin, P.A., 1977, The time-series behavior of quarterly earnings: Preliminary evidence, Journal of Accounting Research 15, 71-83. Lorek, K.S., J.D. Icerman and A.A. Abdulkader, 1983, Further descriptive and predictive evidence on alternative time-series models for quarterly earnings, Journal of Accounting Research 21, 317-328. Makridakis, S. et al., 1982, The accuracy of extrapolative (time series) methods: Results of a forecasting competition, Journal of Forecasting 1, 111-153. Newbold, P. and C.W.J. Granger, 1974, Experience with forecasting univariate time series and the combination of forecasts, Journal of the Royal Statistical Society Series A, 137, 131-164. Watts, R.L., 1975, The time series behavior of quarterly earnings, Working paper (Department of Commerce, University of Newcastle, Newcastle). WinkIer, R.L. and S. Makridakis, 1983, The combination of forecasts, Journal of the RoyaI Statistical Society Series A, 146, 150-157.
Biography:
Jay S. HOLMEN is an Associate Professor in the Department of Accountancy at the University of Wisconsin-Eau Claire. He received his PhD in Business Administration (Accounting emphasis) at the University of Minnesota. He has taught at universities in Minnesota, North Dakota and Wisconsin. His current research interests include the forecasting of earnings, the prediction of financial distress, and the use of forecasting by auditors.