- Email: [email protected]
Non-traditional methods of forecasting
EUROPEAN JOURNAL OF OPERATIONAL RESEARCH ELSEVIER
European Journal of Operational Research 92 (1996) 528-536
Non-traditional methods of forecasting Derek W. Bunn Department of Decision Sciences, London Business School, Sussex Place, Regents Park, London NW14SA, UK
Abstract
The traditional, statistical approach to forecasting has been based upon the identification, specification and estimation of a single model. Recently, we have seen the rise of computationally-intensive methods which depart from this protocol, such as multiple model switching, combinations and neural network methods. At the same time, we are also seeing an increased awareness of the judgemental role in forecasting, also to deal with the inadequacies of model specification. This paper seeks to address the issue of achieving a balance between data and judgement and the need to develop formal methods for it to be effective.
Keywords: Forecasting;Judgement; Combinations;Neural networks
1. Introduction
In proposing to examine the theme of recent non-traditional methods in forecasting, it might reasonably be suggested that there has never really been a central discipline in this subject to give it a methodological tradition in the first place. It is clearly the case that a variety of professions have dealt with methods of forecasting and made important contributions. Amongst many, statisticians have looked at time series diagnostics, estimation techniques and multivariate methods, economists have developed econometric model specification tests as well as business cycle and large scale macromodelling concepts, engineers have developed filtering, state-space and spectral approaches to adaptive time-series forecasting, computer scientists are beginning to introduce pattern recognition and machine learning tech-
niques into data-intensive applications, psychologists have always had an active research agenda in judgement, whilst operational researchers at large have implemented many innovative approaches such as exponential smoothing and market-response models. Furthermore, it has often taken a surprising amount of time for ideas to cross from one profession to another; witness almost a decade for Kalman filtering to be taken from control engineering into timevarying econometric models (and conversely the slow acceptance of maximum likelihood estimation in engineering). But, despite this professional disparity, it is possible to venture a characterisation of some general trends in the evolution of forecasting technology over the past 30 years, and the way these disciplines did appear to converge to core methodology in the 1970s only to start to diverge again in the 1990s. This characterisation on one aspect of the
0377-2217/96/$15.00 © 1996Elsevier Science B.V. All fights reserved PII S0377-2217(96)00006-9
D.W. Bunn/ EuropeanJournalof OperationalResearch92 (1996)528-536 evolution of forecasting methodology, upon which I wish to focus in this presentation, falls into four stages: (a) From very specific, theory-rich models (1960s): Model-building for forecasting in the 1960s remained very much in the orthodox scientific method of formulating a particular theory (eg linear or exponential growth trends, a product life-cycle, or a specific macroeconomic relationship), estimating its parameters from the available data and then validating its applicability. Thus there was a strong judgemental input (belief in the theory) and the empirical emphasis was upon estimation. The main role of data analysis was to estimate the parameters of a pre-specified model and a major thrust of research was upon the development of efficient methods of estimation (e.g. maximum likelihood, least squares or Bayesian). (b) Through broader model classes (1970s): As a consequence of a greater variety of models becoming appropriate, the theoretical work that emerged over the following decade or so was aimed at dealing with the model selection problem by creating general classes of models, such that most models could be seen as a special case of the overall class. The ARIMA class of Box and Jenkins (1976), the Bayesian multiprocess model of Harrison and Stevens (1976) and the unified view of state-space representations, e.g. Harvey (1984) are examples of such classes. What is important to notice is the extra emphasis they place upon empirical procedures (rather than judgement) f o r identi~ing the model structure from data diagnostics, rather than from a priori judgement. Thus, the data analysis is required to perform two roles: identification and estimation. (c) To very general, data.intensive, theory,sparse techniques (1980s): This trend f o r less and less judgemental input, with progressively more emphasis upon the data for model specification has accelerated in the past ten years. Firstly, the rise in popularity o f combining forecasts was a significant step o n this empirical spiral (for a review see Bunn, 1989)~ Rather than propose one model, or select one model out of several, the robustness o f pursuing ai combination, perhaps with the data determining the relative weights, has become well established in practice. More recently with the rise o f neural networks, we have machine learning techniques whereby a model
529
becomes specified out of a very general network topology, purely on the basis of the data in a way that is relatively opaque to judgemental insight. In that such an atheoretical approach to forecasting has become acceptable reflects the climate of pragmatic empiricism which dominated forecasting research during that period. The popularity of forecasting competitions placed a emphasis upon finding methods which worked best, in a generalisable sense, as established by out-of-sample testing. Thus, data now has to perform three roles, identification, estimation and validation (out-of-sample testing). (d) And to data-sparse, theory-sparse judgemental approaches (1980s): With such a move to empirically-driven model specification in the data-intensive arenas, it is not surprising to have seen a complementary move in the opposite direction (not least because of the extra data requirements), with several practitioners rejecting the prevailing climate of empirical model-based approaches to forecasting and concentrating instead upon a set of coherent scenarios instead (Waek, 1985a,b). The formulation of these scenarios may require protocols of group facilitation and soft systems thinking (Rosenhead, 1989; Senge, 1990), with a major emphasis upon consistency and the generation of insight. The objective in these approaches is to understand the impact of uncertainty, rather than to predict it. Thus, by the early 1970s, a mainstream forecasting tradition in research had emerged as one of model identification, estimation and diagnostic testing, but some 15 or so years later, methodology is appearing to diverge into the extremes of data-intensive and judgementally-oriented fields of research. We would always expect to see a divergence of methodology relating to the extremes of short and long term forecasting which differ in terms of data avalability and managerial needs, but the methodological thrusts identified above have polarised this and appear to be leaving a vacuum in those areas of forecasting where there is data (but not a lot) as well important aspects of managerial judgement to consider. These are the areas of application in which most business forecasting has to be undertaken. Thus, in looking at business practice, Huss (1987) reports on a large survey of managers, analysts and state regulators in the US electric utility industry, where they were asked to rate the most desirable
530
D.W. Bunn / European Journal of Operational Research 92 (1996) 528-536
features of a forecasting process. The first seven rankings of attributes were: 1. Does the forecast make sense? 2. Does the data support it? 3. How well has it performed? 4. Can it be explained? 5. Does it identify structural changes? 6. Is it internally consistent? 7. How sensitive is it? What is interesting here is to notice that accuracy, the criterion almost always used in forecasting research appears third, and that most of the issues are concerned with intuition, validation, explainability, etc. This again raises the issue of achieving the right balance of data and judgement in forecasting. Indeed, if credibility is one of the most desirable features of a forecast, the way in which this balance has been achieved will be a crucial aspect of the forecasting process. The structure of the remainder of this paper is therefore to discuss further this balancing issue, the research on judgemental adjustments to forecasts and what this means in the context of the rise in popularity of data-intensive methods, This review draws extensively upon previous work, partucularly the surveys on judgemental methods in Bunn and Wright (1991) and Bunn (1992), as well as work on the combination of forecasts (e.g. Bunn, 1989).
2. Balancing data and judgement By way of a motivating illustration, it is interesting to look at two series of industry forecasts. Fig. 1 is taken from Diefenbacher and Johnson (1987) and shows the history of energy forecasting in W. Germany during the 1970s. Here we see a persistent belief in proportional growth, despite the data indicating the contrary. Most countries at that time forecast energy as a factor of GDP growth, and to the extent that GDP was projected proportionally, so would energy. However, after the energy shock of 1974, that structural relationship changed, yet the simple theory of exponential growth persisted despite several years of new data. Clearly, one criticism of this history could be that inadequate weight was being given to the data, compared to judgemental theory.
;/ /
600 ~D
E 500
f " 11 t'
,,a~sX). .~ s
~,I w
400
J 300, I 70
I
I
I
75
80
85
Year Fig. 1. Energy forecasting in W. Germany (from Diefenbacher and Johnson, 1987).
On the other hand, if we look at Fig. 2, we see the history of airline forecasting in the 1960s, where simple extrapolations of the data never caught up with the rapid proportional growth. Perhaps we can say that here there is evidence of too much weight on extrapolating recent data, and not enough on theory formation and judgemental input. Together, therefore, these two examples illustrate how the issue of balancing data and judgement is quite delicate and represents a serious modelling challenge. Recent business surveys of the practical use of models in forecasting have invariably shown a managerial inclination to use relatively simple models and subject them to considerable judgemental adjustment. For example, Edmundson et al. (1988) and Huss (1985) have provided practical case-studies of the effectiveness of this approach. In addition, Mathews and Diamantopoulos (1990) have suggested from their sales forecasting data that in exercising discretion upon which model-based forecasts to adjust and which to leave, managers have generally chosen well. A similar conclusion has been reached by Willemain (1989) from experimental studies showing that judgement generally improved 'bad' quanti-
D.W. Bunn / European Journal of Operational Research 92 (1996) 528-536 85
1968 ©
75
1974
/ ©
65
C
~. 55
/
0 0
~
45
,~
°
35
25 1959 1960 1961 1962 1963 1964 1965 1966 1967 1968
Fiscal year Fig. 2. Airline industry forecasts.
tative forecasts, but was not detrimental to 'good', i.e. near optimal, ones. This use of judgemental adjustment is not just a compensation for model simplicity. With more complicated forecasting models, such as the multiple econometric equations used for economic forecasting, judgemental adjustment by the producers is equally prevalent in practice. McNees and Perna (1891), Corker et al. (1986), Wallis et al. (19841988) and Turner (1990) all discuss this process in detail. Turner (1990), in particular, provides an extensive analysis of this process on two major UK econometric models. Some of this was summarised in Bunn (1992), on which the following section is based. Unlike the more common business practice of casually adjusting the final forecasts themselves, adjustments by economic model-builders often take the form of subjective projections of the residuals of some of the variables within the model. For a wellspecified model, the residuals should just reflect random variation and would not therefore have any systematic effect on the forecasts. If, however, the
531
forecaster believes that the model is inadequate in some way, then residual adjustments will be made to one or more of the variables by projecting forward a systematic pattern of residuals. Fig. 3 shows the residual adjustments made to the forecasts in 1988 by the London Business School (LBS) and National Institute of Economic and Social Research (NIESR) for manufacturing exports. The data series up to the second quarter of 1988 shows an average (but periodic) pattern of underestimating this variable. Looking beyond the second quarter of 1988, the LBS forecasters clearly decided to project out a constant residual adjustment of about 6% to compensate for this, whereas the NIESR evidently saw the misspecification getting progressively worse and included and increasing adjustment. Both LBS and NIESR noted these adjustments as being caused by recently unreliable data and a probable misspecification of the UK elasticity with respect to world trade. Similarly Fig. 4 shows the LBS and NIESR residual adjustments to consumer expenditure. In this case, both were noted as being caused by a recent change in the savings ratio: LBS saw this as a misspecification due to financial de-regulation whereas NIESR saw it as an omitted variable in terms of mortgage equity withdrawals. Again, LBS projected the residual error at a fairly constant value, but NIESR took the view that its effect would attenuate. Of course, the added difficulty with adjusting one variable in a system of equations is caused by the interrelationships with the variables. Thus, for example, the LBS consumer expenditure adjustment
?>
1Z-
10"
i Forecast
/.---
8
6.
(%)
Fig. 3. Residual adjustments on manufacturing exports (from Turner, 1990).
D.W. Bunn /European Journal of Operational Research 92 (1996) 528-536
532
1000 1200 1000
(£m 1980
800"
Forecast
~
LB$
prices) GO0-
000200-
NIESR 0 , 1985
• '19~9'' ','90' ' '1i91' " '110i ' '
--200.
Fig. 4. Residual adjustments on consumer expenditures (from Turner, 1990).
takes the 1989 GDP growth from 3.4% to 1.4%. As Turner (1990) observes, generally less than half of the endogenous variables are adjusted, but their effects are extremely significant. Furthermore, from the perspective of a user of these forecasts, very little background to the adjustments is normally given and no sensitivity analysis performed on the effect of other possible adjustment assumptions. This is a real concern for users of these models. However, in a series of retrospective studies, Wallis (1984-1988) has shown that these adjustments have; overall, produced more accurate forecasts than if they had not been undertaken. It seems that, whilst well-specified time-series models can be most effective in filtering out noise and projecting past patterns in the data, expert intervention will pay off in practice when there is extra information about new untypical circumstances. To be more specific, judgement is needed to integrate with models where, for example: Data is inadequate. There may not be sufficient data to estimate the number of variables, or the complexity of the relationship, and so the model is underspecified. Theories are inadequate. For example, exchange rate forecasts in econometric models are essentially judgemental interpretations of government policies. Theory is currently inadequate to set up these rates as endogenous variables determined similarly to the other economic variables. There are some omitted variables. We have seen an example of this in Fig. 4 where NIESR adjusted the consumer expenditure residuals to account for inadequacy in modelling mortgage withdrawals.
There is extensive multicollinearity. Sometimes variables are omitted because of multicollinearity in the historical data set. If this is not expected to persist in the future, adjustments are necessary (Bunn and Salo, 1995). Data quality is suspect. We have seen examples of this in Fig. 3 where both NIESR and LBS chose to adjust the manufacturing export residuals because of lack of quality in the recent data. Continuity across multiple models is necessary. Period by period forecasting is often required to show consistency. Short, medium or long-term models need to show a consistent profile for reasons of credibility. Misspecification of dynamic relationship, e.g. government incentives on unemployment, new market behaviour dynamics. Change in coefficients, e.g. new market share, or savings ratio. Uncertain outcomes are partially controllable. In business situations where the outcomes or forecasts may be treated more as targets, or part of the budgetary process, then subjective intervention may be performing the function of introducing a decision-rule a n d / o r an asymmetric loss function (e.g. Willemain et al., 1991) or anticipating a propensity to achieve outcomes (e.g. Brown, 1988). Political background. This is the major source of judgemental input to economic forecasts, as such models are essentially conditional upon a particular policy framework. In practice, most judgemental adjustments appear to be undertaken in an informal way. Armstrong (1985) quotes the example of Glantz (1977) in which the US Government was sued by a group of farmers for forecasting and issuing directives for a drought which did not occur. The forecast had been based upon a judgemental adjustment to a quantitative model, which the suit contended represented unprofessional practice. Clearly an explicit process of adjustment needs to be undertaken, not just to be defensible to adversarial challenge, but also to better communicate the reasoning in the forecast. To point simply to the availability and incorporation of some 'extra information', is not enough. In fact a much more systematic approach is really required. However, in practice, we generally see, at best, some recorded commentary as to the main reasons for the
D.W. Bunn /European Journal of Operational Research 92 (1996) 528-536
adjustments. To the extent that this is carefully undertaken, this may be all that is required to provide some discipline to the forecasting process and facilitate subsequent review. Attempts to go beyond such a documented 'audit trail' and provide a formal adjustment decomposition structure have not yet matured into established practice. Various methods have been developed to help structure purely judgmental forecasts, and provide a process of integrating quantitative and qualitative data, but there have been very few applications concerned specifically with adjustment of statistical forecasts. The formal structures that have been published to date have been very limited. A few authors have sought to apply the AHP approach to decompose the judgemental adjustments (Wolfe and F lores, 1990; Saxtty and Vargas, 1991; Flores et al., 1992; Badiru et al., 1993), but for reasons discussed elsewhere (Salo and Bunn, 1995), this approach makes very restrictive assumptions for the mutual exclusivity of the factors being decomposed. Elsewhere, Lee et al. (1990) have proposed a n expert system for encoding frequently u s e d adjustment rules, but this does not address the specific structuring process to support an individual with new forecasting problem. M o r e recently, Bunn and Salo (1994) have identified a potential double-counting bias in the casual, unstructured adjustment of statistical forecasts. They observed that, just because a variable is not explicitly included in a model, does not mean that its effect is not being implicitly modelled through other variables in the model with which it might be correlated. Thus, they suggested the extensive use of 'model-consistent expectations' for all variables omitted from the model, to see whether judgemental expectations are different, and therefore imply a need for model adjustment. This is approaching a more systemic view of forecasting, taking account of variable interrelationships beyond those explicitly represented in the model.
3. Issues raised by data-intensive methods It seems reasonable to believe a prerequisite for the common style of informal judgemental adjustments to time series forecasts is a relatively intuitive
533
structure for these statistical models in the first place. Yet, one of the contemporary themes which was identified in the introduction to this paper, is the current popularity of data-intensive methods, the underlying structure of which can be quite opaque. We return now to considering this theme, and its implications for judgemental interaction. In looking at data-intensive methods, it is useful to review the development of combinations of forecasts, first of all. As an example which has been around for longer than the current generation of AI-inspired techniques such as genetic algorithms and neural networks, the topic did experience earlier much of the same criticism that is now being aimed these newer techniques. Although the idea of forming a linear combination of several forecasts may appear quite intuitive, when Newbold and Granger (1974) proposed the approach as a systematic method of forecasting, it was heavily criticised for being ad hoc and unscientific in not following the traditional scientific time series method of identifying the singularly most appropriate model. Since then, however, the weight of empirical evidence in its favour (Bunn, 1989) has caused it to be much more readily accepted, and indeed Holden and Peel (1990) suggest that consensus forecasts may now be the basis of rational expectations in the economic context. Although a large number of combination procedures have appeared in the literature, to get a sense of the methodology five well-tried methods are worth mentioning in outline as representatives of increasing degrees of sophistication: (a) Simple average: This has the virtues of impartiality,: robustness and a good 'track-record' in economic and business forecasting. It has been consistently the choice of many researchers (see Clemen, 1989), although there is some evidence that the median might be more robust. (b) Outperformance probabilities (Bunn, 1975): This is a linear combination where each individual weight is interpreted as the probability that its respective forecast will perform best (in the smallest absolute error sense) on the next occasion. Each probability is estimated as the fraction of occurrences that its respective forecasting model has performed best in the past. As a robust, nonparametric method of achieving differential weights with intu-
534
D.W. Bunn / European Journal of Operational Research 92 (1996) 528-536
itive meaning, this method performs well when there is relatively little past data a n d / o r expert judgement can be incorporated into the combining weights. (c) Optimal: Here the linear weights are calculated to minimise the error variance of the combined forecast (assuming unbiasedness for each individual forecast). Specifically, the linear (n × 1) vector of combining weights, w, is determined according to the formula
S-le w w= e,S_le
(1)
where e is the ( n × l ) unit vector and S is the (n × n) covariance matrix of forecast errors. The problem with this optimising approach is that it requires S to be properly estimated. If the error covariance matrix is not stationary or it has to be estimated on the basis of a short past history, then more robust alternatives have sometimes worked better. (d) Optimal with independence assumption: In this case, S in the above formula is restricted to be diagonal, comprising just the individual error variances. (e) Optimal with restricted weights: In this case the above formula has the additional restriction that no individual weight can be outside the interval [0,1]. Of the methods, not included above, which have often been used, regression can be shown to be theoretically equivalent to the optimal approach, if the individual forecasts are unbiased (thereby allowing the constant term in the regression to be dropped). Although most of the research on this topic has been directed at finding robust procedures to deal with situations of short data sets a n d / o r badly behaved data, with the exception of the Bayesian outperformance method, the issue of incorporating judgement with combinations has been largely ignored. Yet combinations reflect much more difficult problems in this respect than single models. Their structure is not so intuitive in providing coherent input/output relationships between all of the variables. They are a portfolio of disparate methods, motivated by a hedging approach to model selection rather than a desire to create a completely internally consistent single model. Indeed, a combination actually reflects one of two types of modelling failures;
either to identify the singularly correct model, or to build a comprehensive model from all of the variables suggested. This is not to say that the analyst should have done better, it is just a reflection of the pragmatic nature of the approach in coping with the limitations of conventional model identification and estimation techniques. However, this portfolio of models does make it harder to incorporate judgement - should adjustments be made at the level of each component model (but then how does this affect the combining weights?) or should it be done on the combined forecast itself (but how do we know the implicit affect of each predictive factor on the output?). These questions are solvable, but not intuitively, and would seem to require some knowledge-based software interfacing between model and user. When we turn to neural networks, we see many of these original criticisms now being repeated in a more extreme way. Just as combinations avoid the problem of model identification by hedging, neural networks avoid the same issue by facilitating machine learning across a very broad class of specification topologies. Furthermore, this approach appears to offer the advantages of freedom from statistical assumptions, resilience to missing observations and the ability to incorporate nonlinear functional relationships. But all this is at the price of very limited statistical diagnostics, the lack of an easily understandable structure, and the consequent need to rely upon empirical performance for validation. Many researchers in the neurocomputing literature seek an a priori justification for their configurations and heuristics by biological analogy, hence the traditional positioning of this research within artificial intelligence, but from a business applications perspective, this is less convincing. It is more relevant within our context to see this as a very general procedure for specifying and estimating nonlinear relationships in data-rich applications. Sharda (1994), provides a good recent reference on O R / M S applications and their relative performance. Much of the work is comparative with more traditional techniques such as Box-Jenkins and Linear Regression. In many situations, the artificial neural networks do seem to offer performance improvements. Some financial institutions are trading in certain markets using them (Davidson, 1994), but
D.W. Bunn / European Journal of Operational Research 92 (1996) 528-536
in general most practitioners and critics are still cautious about the lack of explanation and potential over-fitting in the procedure. Thus, the problems that combinations have faced in terms of judgemental interaction seem to be magnified severalfold in this methodology. Unlike the combining forecasts experience, it may be that even a large preponderance of empirical evidence in its favour will not be sufficient to create confidence in the technology without more research into explainability and robustness diagnostics.
4. Final c o m m e n t s
Whereas, the traditional scientific time-series method of model selection, estimation and testing predominated within the research paradigm of the 1970s, we have recently seen a divergence into data-intensive methods and judgementally-oriented procedures. In their different ways, both of these extreme directions are seeking to avoid the model selection issue. But they are being fostered in separate contexts. In short-term, data-rich applications, where accuracy is important, we have seen a move from using a single model to a combination of several, i.e. model synthesis. But in the longer term, to support business strategy where understanding the risks is the main objective, we are seeing just the opposite, ie move to model divergence, by testing many different scenarios within a decision framework. In between these two extremes is a broad class of applications where an intuitive balance of judgement and data-based modelling is required, and where few formal procedures have yet emerged. The need is apparent, however, and the work of Collopy and Armstrong (1992) in developing a data-intensive forecasting system of multiple models and a knowlwdge base ('rule-based forecasting', sic) is a useful and innovative response to this. It seems to follow from this line of thinking that if data-intensive techniques continue to rely upon empirical performance for validation and stay judgementally opaque, their impact will stay in the shortterm, data-rich class of forecasting applications. But if and when explainability and robustness diagnostics become available, there could be a exciting prospect
535
of more formal, knowledge-based procedures for judgemental interaction, and to the extent that this becomes credible to practitioners, the scope of such modelling would be that much wider. Similarly in the longer term, the more that scenario based methods (Bunn, 1993) can become model-based, moving from systems thinking to systems modelling, the more that they can provide a defensible basis for a broader class of forecasting applications in strategy. Should we start to see a convergence in forecasting methodology again, between the judgemental and data-intensive, it is clear that it will be to quite a different tradition from the 1970s, and one that will be based more upon developing intelligent software platforms for data analysis, multiple-modelling and judgemental interaction.
References Armstrong, J.S. (1985), Long Range Forecasting, Wiley, New York. Badiru, A.B., Pulat, P.S., and Kang, M. (1993), "DDM: Decision support system for hierarchical dynamic decision making", Decision Support Systems 10, 1-18. Box, G.E.P., and Jenkins, G.M. (1976), Time Series Analysis" Forecasting and Control, Holden-Day, San Francisco, CA. Brown, L.D. (1988), "Comparing judgemental to extrapolative forecasts: It's time to ask why and when", International Journal of Forecasting 4/2, 171-173. Bunn, D,W. (1975), " A Bayesian approach to the linear combination of forecasts", Operational Research Quarterly 26, 325329. Bunn, D.W. (1989), "Forecasting with more than one model" Journal of Forecasting 8/3, 161-166. Bunn, D.W., and Salo, A. (1993), "Forecasting with scenarios", European Journal of Operational Research 68/3, 291-303. Burro, D.W., and Salo, A. (1995), "Adjustment of forecasts with model consistent expectations", Research Paper, London Business School. Bunn, D.W., and Wright, G. (1991), "Interaction of judgemental and statistical forecasting methods: Issues and analysis", Management Science 37/5, 501-518. Clemen, R. (1989), "Combining forecasts: A review and annotated bibliography", 5/4, 559-584. Collopy, F., and Armstrong, J.S. (1992), "Rule-based forecasting: Development and validation of an Expert Systems approach to combining time-series extrapolations", Management Science 38/10, 1394-1414. Corker, R.J., Holly, S., and Ellis, R.G. (1986), "Uncertainty and forecast precision", 2/1, 53-70. Diefenbacher, H., and Johnson, J. (1987), "Energy forecasting in
536
D.W. Bunn / European Journal of Operational Research 92 (1996) 528-536
West Germany", in: T. Baumgartner and A. Midtum (eds.), The Politics o f Energy Forecasting, Oxford University Press, Oxford. Flores, B.E., Olson, D.L., and Wolfe, C. (1992), "Judgmental adjustment of forecasts: A comparison of methods", International Journal of Forecasting 7, 421-433. Glantz, M.H. (1977), "Consequences and responsibilities in drought forecasting", Water Resources Research 18, 3-13. Harrison, P.J., and Stevens, C.F. (1976), "Bayesian forecasting", Journal of the Royal Statistical Society, Series B 38, 205-247. Harvey, A. (1984), " A unified view of forecasting", Journal of Forecasting 3/3, 245-284. Holden, K., Peel, D.A., and Thompson, J.I., (1990), Economic Forecasting: An Introduction, Cambridge University Press, Cambridge. Huss, W.R. (1985), "Comparative analysis of company forecasts and advanced time-series techniques using annual electric utility energy sales data", 1/3, 217-239. Huss, W.R. (1987), "Forecasting in the electric utility industry", in: S. Makfidakis and S. Wheelwright, The Handbook of Forecasting, Chapter 5, Wiley, Chichester. lee, J.K., Oh, S.B., and Shin, J.C. (1990), "UNIK-FCST: Knowledge-assisted adjustment of statistical forecasts", Expert Systems with Applications 1, 39-49. Mathews, B.P., and Diamantopoulos, A. (1990),"Judgemental revision of sales forecasts: Effectiveness of selection" Journal of Forecasting 9/4, 407-416. McNees, S.K., and Pema, N.S. (1981), "Forecasting macroeco-
uomic variables: An eclectic approach", New England Economic Journal, May/June, 15-30. Newbold, P., and Granger, C.W.J. (1974), "Experience with forecasting univariate time series and the combination of forecasts", Journal of the Royal Statistical Society. Series A 137/2, 131-165. Rosenhead, J., (ed.) (1989), Rational Analysis for a Problematic WorM, Wiley, Chichester. Saaty, T.L., and Vargas, L.G. (1991), Prediction, Projection and Forecasting, Kluwer Academic Publishers, Boston, MA. Salo, A., and Bunn, D.W. (1995), "Decomposition in the assessment of judgemental probability forecasts", Technological Forecasting and Social Change, in press. Senge, P.M. (1990), The Fifth Discipline, Doubleday, New York. Sharda, R. (1994), "Neural networks for the MS/OR analyst: An application bibliography", Interfaces 24, 116-130. Turner, D. (1990), "The role of judgement in macroeconomic forecasting", Journal o f Forecasting 9, 315-346. Wack, P. (1985a), "Scenarios: Uncharted waters ahead", Harvard Business Review 63/5, 73-89. Wack, P. (1985b), "Scenarios: Shooting the rapids", Harvard Business Review 63/6, 139-150. WaUis, K.F., et al. (1984-1988), Models o f the UK Economy: Reviews 1-5, Oxford University Press, Oxford. Willemain, T.R. (1989), "Graphical adjustment of statistical forecasts", 5, 179-185. Wolfe, C., and Flores, B. (1990), "'Judgemental adjustment of earnings forecasts", Journal of Forecasting 9~/4, 389-406.
European Journal of Operational Research 92 (1996) 528-536
Non-traditional methods of forecasting Derek W. Bunn Department of Decision Sciences, London Business School, Sussex Place, Regents Park, London NW14SA, UK
Abstract
The traditional, statistical approach to forecasting has been based upon the identification, specification and estimation of a single model. Recently, we have seen the rise of computationally-intensive methods which depart from this protocol, such as multiple model switching, combinations and neural network methods. At the same time, we are also seeing an increased awareness of the judgemental role in forecasting, also to deal with the inadequacies of model specification. This paper seeks to address the issue of achieving a balance between data and judgement and the need to develop formal methods for it to be effective.
Keywords: Forecasting;Judgement; Combinations;Neural networks
1. Introduction
In proposing to examine the theme of recent non-traditional methods in forecasting, it might reasonably be suggested that there has never really been a central discipline in this subject to give it a methodological tradition in the first place. It is clearly the case that a variety of professions have dealt with methods of forecasting and made important contributions. Amongst many, statisticians have looked at time series diagnostics, estimation techniques and multivariate methods, economists have developed econometric model specification tests as well as business cycle and large scale macromodelling concepts, engineers have developed filtering, state-space and spectral approaches to adaptive time-series forecasting, computer scientists are beginning to introduce pattern recognition and machine learning tech-
niques into data-intensive applications, psychologists have always had an active research agenda in judgement, whilst operational researchers at large have implemented many innovative approaches such as exponential smoothing and market-response models. Furthermore, it has often taken a surprising amount of time for ideas to cross from one profession to another; witness almost a decade for Kalman filtering to be taken from control engineering into timevarying econometric models (and conversely the slow acceptance of maximum likelihood estimation in engineering). But, despite this professional disparity, it is possible to venture a characterisation of some general trends in the evolution of forecasting technology over the past 30 years, and the way these disciplines did appear to converge to core methodology in the 1970s only to start to diverge again in the 1990s. This characterisation on one aspect of the
0377-2217/96/$15.00 © 1996Elsevier Science B.V. All fights reserved PII S0377-2217(96)00006-9
D.W. Bunn/ EuropeanJournalof OperationalResearch92 (1996)528-536 evolution of forecasting methodology, upon which I wish to focus in this presentation, falls into four stages: (a) From very specific, theory-rich models (1960s): Model-building for forecasting in the 1960s remained very much in the orthodox scientific method of formulating a particular theory (eg linear or exponential growth trends, a product life-cycle, or a specific macroeconomic relationship), estimating its parameters from the available data and then validating its applicability. Thus there was a strong judgemental input (belief in the theory) and the empirical emphasis was upon estimation. The main role of data analysis was to estimate the parameters of a pre-specified model and a major thrust of research was upon the development of efficient methods of estimation (e.g. maximum likelihood, least squares or Bayesian). (b) Through broader model classes (1970s): As a consequence of a greater variety of models becoming appropriate, the theoretical work that emerged over the following decade or so was aimed at dealing with the model selection problem by creating general classes of models, such that most models could be seen as a special case of the overall class. The ARIMA class of Box and Jenkins (1976), the Bayesian multiprocess model of Harrison and Stevens (1976) and the unified view of state-space representations, e.g. Harvey (1984) are examples of such classes. What is important to notice is the extra emphasis they place upon empirical procedures (rather than judgement) f o r identi~ing the model structure from data diagnostics, rather than from a priori judgement. Thus, the data analysis is required to perform two roles: identification and estimation. (c) To very general, data.intensive, theory,sparse techniques (1980s): This trend f o r less and less judgemental input, with progressively more emphasis upon the data for model specification has accelerated in the past ten years. Firstly, the rise in popularity o f combining forecasts was a significant step o n this empirical spiral (for a review see Bunn, 1989)~ Rather than propose one model, or select one model out of several, the robustness o f pursuing ai combination, perhaps with the data determining the relative weights, has become well established in practice. More recently with the rise o f neural networks, we have machine learning techniques whereby a model
529
becomes specified out of a very general network topology, purely on the basis of the data in a way that is relatively opaque to judgemental insight. In that such an atheoretical approach to forecasting has become acceptable reflects the climate of pragmatic empiricism which dominated forecasting research during that period. The popularity of forecasting competitions placed a emphasis upon finding methods which worked best, in a generalisable sense, as established by out-of-sample testing. Thus, data now has to perform three roles, identification, estimation and validation (out-of-sample testing). (d) And to data-sparse, theory-sparse judgemental approaches (1980s): With such a move to empirically-driven model specification in the data-intensive arenas, it is not surprising to have seen a complementary move in the opposite direction (not least because of the extra data requirements), with several practitioners rejecting the prevailing climate of empirical model-based approaches to forecasting and concentrating instead upon a set of coherent scenarios instead (Waek, 1985a,b). The formulation of these scenarios may require protocols of group facilitation and soft systems thinking (Rosenhead, 1989; Senge, 1990), with a major emphasis upon consistency and the generation of insight. The objective in these approaches is to understand the impact of uncertainty, rather than to predict it. Thus, by the early 1970s, a mainstream forecasting tradition in research had emerged as one of model identification, estimation and diagnostic testing, but some 15 or so years later, methodology is appearing to diverge into the extremes of data-intensive and judgementally-oriented fields of research. We would always expect to see a divergence of methodology relating to the extremes of short and long term forecasting which differ in terms of data avalability and managerial needs, but the methodological thrusts identified above have polarised this and appear to be leaving a vacuum in those areas of forecasting where there is data (but not a lot) as well important aspects of managerial judgement to consider. These are the areas of application in which most business forecasting has to be undertaken. Thus, in looking at business practice, Huss (1987) reports on a large survey of managers, analysts and state regulators in the US electric utility industry, where they were asked to rate the most desirable
530
D.W. Bunn / European Journal of Operational Research 92 (1996) 528-536
features of a forecasting process. The first seven rankings of attributes were: 1. Does the forecast make sense? 2. Does the data support it? 3. How well has it performed? 4. Can it be explained? 5. Does it identify structural changes? 6. Is it internally consistent? 7. How sensitive is it? What is interesting here is to notice that accuracy, the criterion almost always used in forecasting research appears third, and that most of the issues are concerned with intuition, validation, explainability, etc. This again raises the issue of achieving the right balance of data and judgement in forecasting. Indeed, if credibility is one of the most desirable features of a forecast, the way in which this balance has been achieved will be a crucial aspect of the forecasting process. The structure of the remainder of this paper is therefore to discuss further this balancing issue, the research on judgemental adjustments to forecasts and what this means in the context of the rise in popularity of data-intensive methods, This review draws extensively upon previous work, partucularly the surveys on judgemental methods in Bunn and Wright (1991) and Bunn (1992), as well as work on the combination of forecasts (e.g. Bunn, 1989).
2. Balancing data and judgement By way of a motivating illustration, it is interesting to look at two series of industry forecasts. Fig. 1 is taken from Diefenbacher and Johnson (1987) and shows the history of energy forecasting in W. Germany during the 1970s. Here we see a persistent belief in proportional growth, despite the data indicating the contrary. Most countries at that time forecast energy as a factor of GDP growth, and to the extent that GDP was projected proportionally, so would energy. However, after the energy shock of 1974, that structural relationship changed, yet the simple theory of exponential growth persisted despite several years of new data. Clearly, one criticism of this history could be that inadequate weight was being given to the data, compared to judgemental theory.
;/ /
600 ~D
E 500
f " 11 t'
,,a~sX). .~ s
~,I w
400
J 300, I 70
I
I
I
75
80
85
Year Fig. 1. Energy forecasting in W. Germany (from Diefenbacher and Johnson, 1987).
On the other hand, if we look at Fig. 2, we see the history of airline forecasting in the 1960s, where simple extrapolations of the data never caught up with the rapid proportional growth. Perhaps we can say that here there is evidence of too much weight on extrapolating recent data, and not enough on theory formation and judgemental input. Together, therefore, these two examples illustrate how the issue of balancing data and judgement is quite delicate and represents a serious modelling challenge. Recent business surveys of the practical use of models in forecasting have invariably shown a managerial inclination to use relatively simple models and subject them to considerable judgemental adjustment. For example, Edmundson et al. (1988) and Huss (1985) have provided practical case-studies of the effectiveness of this approach. In addition, Mathews and Diamantopoulos (1990) have suggested from their sales forecasting data that in exercising discretion upon which model-based forecasts to adjust and which to leave, managers have generally chosen well. A similar conclusion has been reached by Willemain (1989) from experimental studies showing that judgement generally improved 'bad' quanti-
D.W. Bunn / European Journal of Operational Research 92 (1996) 528-536 85
1968 ©
75
1974
/ ©
65
C
~. 55
/
0 0
~
45
,~
°
35
25 1959 1960 1961 1962 1963 1964 1965 1966 1967 1968
Fiscal year Fig. 2. Airline industry forecasts.
tative forecasts, but was not detrimental to 'good', i.e. near optimal, ones. This use of judgemental adjustment is not just a compensation for model simplicity. With more complicated forecasting models, such as the multiple econometric equations used for economic forecasting, judgemental adjustment by the producers is equally prevalent in practice. McNees and Perna (1891), Corker et al. (1986), Wallis et al. (19841988) and Turner (1990) all discuss this process in detail. Turner (1990), in particular, provides an extensive analysis of this process on two major UK econometric models. Some of this was summarised in Bunn (1992), on which the following section is based. Unlike the more common business practice of casually adjusting the final forecasts themselves, adjustments by economic model-builders often take the form of subjective projections of the residuals of some of the variables within the model. For a wellspecified model, the residuals should just reflect random variation and would not therefore have any systematic effect on the forecasts. If, however, the
531
forecaster believes that the model is inadequate in some way, then residual adjustments will be made to one or more of the variables by projecting forward a systematic pattern of residuals. Fig. 3 shows the residual adjustments made to the forecasts in 1988 by the London Business School (LBS) and National Institute of Economic and Social Research (NIESR) for manufacturing exports. The data series up to the second quarter of 1988 shows an average (but periodic) pattern of underestimating this variable. Looking beyond the second quarter of 1988, the LBS forecasters clearly decided to project out a constant residual adjustment of about 6% to compensate for this, whereas the NIESR evidently saw the misspecification getting progressively worse and included and increasing adjustment. Both LBS and NIESR noted these adjustments as being caused by recently unreliable data and a probable misspecification of the UK elasticity with respect to world trade. Similarly Fig. 4 shows the LBS and NIESR residual adjustments to consumer expenditure. In this case, both were noted as being caused by a recent change in the savings ratio: LBS saw this as a misspecification due to financial de-regulation whereas NIESR saw it as an omitted variable in terms of mortgage equity withdrawals. Again, LBS projected the residual error at a fairly constant value, but NIESR took the view that its effect would attenuate. Of course, the added difficulty with adjusting one variable in a system of equations is caused by the interrelationships with the variables. Thus, for example, the LBS consumer expenditure adjustment
?>
1Z-
10"
i Forecast
/.---
8
6.
(%)
Fig. 3. Residual adjustments on manufacturing exports (from Turner, 1990).
D.W. Bunn /European Journal of Operational Research 92 (1996) 528-536
532
1000 1200 1000
(£m 1980
800"
Forecast
~
LB$
prices) GO0-
000200-
NIESR 0 , 1985
• '19~9'' ','90' ' '1i91' " '110i ' '
--200.
Fig. 4. Residual adjustments on consumer expenditures (from Turner, 1990).
takes the 1989 GDP growth from 3.4% to 1.4%. As Turner (1990) observes, generally less than half of the endogenous variables are adjusted, but their effects are extremely significant. Furthermore, from the perspective of a user of these forecasts, very little background to the adjustments is normally given and no sensitivity analysis performed on the effect of other possible adjustment assumptions. This is a real concern for users of these models. However, in a series of retrospective studies, Wallis (1984-1988) has shown that these adjustments have; overall, produced more accurate forecasts than if they had not been undertaken. It seems that, whilst well-specified time-series models can be most effective in filtering out noise and projecting past patterns in the data, expert intervention will pay off in practice when there is extra information about new untypical circumstances. To be more specific, judgement is needed to integrate with models where, for example: Data is inadequate. There may not be sufficient data to estimate the number of variables, or the complexity of the relationship, and so the model is underspecified. Theories are inadequate. For example, exchange rate forecasts in econometric models are essentially judgemental interpretations of government policies. Theory is currently inadequate to set up these rates as endogenous variables determined similarly to the other economic variables. There are some omitted variables. We have seen an example of this in Fig. 4 where NIESR adjusted the consumer expenditure residuals to account for inadequacy in modelling mortgage withdrawals.
There is extensive multicollinearity. Sometimes variables are omitted because of multicollinearity in the historical data set. If this is not expected to persist in the future, adjustments are necessary (Bunn and Salo, 1995). Data quality is suspect. We have seen examples of this in Fig. 3 where both NIESR and LBS chose to adjust the manufacturing export residuals because of lack of quality in the recent data. Continuity across multiple models is necessary. Period by period forecasting is often required to show consistency. Short, medium or long-term models need to show a consistent profile for reasons of credibility. Misspecification of dynamic relationship, e.g. government incentives on unemployment, new market behaviour dynamics. Change in coefficients, e.g. new market share, or savings ratio. Uncertain outcomes are partially controllable. In business situations where the outcomes or forecasts may be treated more as targets, or part of the budgetary process, then subjective intervention may be performing the function of introducing a decision-rule a n d / o r an asymmetric loss function (e.g. Willemain et al., 1991) or anticipating a propensity to achieve outcomes (e.g. Brown, 1988). Political background. This is the major source of judgemental input to economic forecasts, as such models are essentially conditional upon a particular policy framework. In practice, most judgemental adjustments appear to be undertaken in an informal way. Armstrong (1985) quotes the example of Glantz (1977) in which the US Government was sued by a group of farmers for forecasting and issuing directives for a drought which did not occur. The forecast had been based upon a judgemental adjustment to a quantitative model, which the suit contended represented unprofessional practice. Clearly an explicit process of adjustment needs to be undertaken, not just to be defensible to adversarial challenge, but also to better communicate the reasoning in the forecast. To point simply to the availability and incorporation of some 'extra information', is not enough. In fact a much more systematic approach is really required. However, in practice, we generally see, at best, some recorded commentary as to the main reasons for the
D.W. Bunn /European Journal of Operational Research 92 (1996) 528-536
adjustments. To the extent that this is carefully undertaken, this may be all that is required to provide some discipline to the forecasting process and facilitate subsequent review. Attempts to go beyond such a documented 'audit trail' and provide a formal adjustment decomposition structure have not yet matured into established practice. Various methods have been developed to help structure purely judgmental forecasts, and provide a process of integrating quantitative and qualitative data, but there have been very few applications concerned specifically with adjustment of statistical forecasts. The formal structures that have been published to date have been very limited. A few authors have sought to apply the AHP approach to decompose the judgemental adjustments (Wolfe and F lores, 1990; Saxtty and Vargas, 1991; Flores et al., 1992; Badiru et al., 1993), but for reasons discussed elsewhere (Salo and Bunn, 1995), this approach makes very restrictive assumptions for the mutual exclusivity of the factors being decomposed. Elsewhere, Lee et al. (1990) have proposed a n expert system for encoding frequently u s e d adjustment rules, but this does not address the specific structuring process to support an individual with new forecasting problem. M o r e recently, Bunn and Salo (1994) have identified a potential double-counting bias in the casual, unstructured adjustment of statistical forecasts. They observed that, just because a variable is not explicitly included in a model, does not mean that its effect is not being implicitly modelled through other variables in the model with which it might be correlated. Thus, they suggested the extensive use of 'model-consistent expectations' for all variables omitted from the model, to see whether judgemental expectations are different, and therefore imply a need for model adjustment. This is approaching a more systemic view of forecasting, taking account of variable interrelationships beyond those explicitly represented in the model.
3. Issues raised by data-intensive methods It seems reasonable to believe a prerequisite for the common style of informal judgemental adjustments to time series forecasts is a relatively intuitive
533
structure for these statistical models in the first place. Yet, one of the contemporary themes which was identified in the introduction to this paper, is the current popularity of data-intensive methods, the underlying structure of which can be quite opaque. We return now to considering this theme, and its implications for judgemental interaction. In looking at data-intensive methods, it is useful to review the development of combinations of forecasts, first of all. As an example which has been around for longer than the current generation of AI-inspired techniques such as genetic algorithms and neural networks, the topic did experience earlier much of the same criticism that is now being aimed these newer techniques. Although the idea of forming a linear combination of several forecasts may appear quite intuitive, when Newbold and Granger (1974) proposed the approach as a systematic method of forecasting, it was heavily criticised for being ad hoc and unscientific in not following the traditional scientific time series method of identifying the singularly most appropriate model. Since then, however, the weight of empirical evidence in its favour (Bunn, 1989) has caused it to be much more readily accepted, and indeed Holden and Peel (1990) suggest that consensus forecasts may now be the basis of rational expectations in the economic context. Although a large number of combination procedures have appeared in the literature, to get a sense of the methodology five well-tried methods are worth mentioning in outline as representatives of increasing degrees of sophistication: (a) Simple average: This has the virtues of impartiality,: robustness and a good 'track-record' in economic and business forecasting. It has been consistently the choice of many researchers (see Clemen, 1989), although there is some evidence that the median might be more robust. (b) Outperformance probabilities (Bunn, 1975): This is a linear combination where each individual weight is interpreted as the probability that its respective forecast will perform best (in the smallest absolute error sense) on the next occasion. Each probability is estimated as the fraction of occurrences that its respective forecasting model has performed best in the past. As a robust, nonparametric method of achieving differential weights with intu-
534
D.W. Bunn / European Journal of Operational Research 92 (1996) 528-536
itive meaning, this method performs well when there is relatively little past data a n d / o r expert judgement can be incorporated into the combining weights. (c) Optimal: Here the linear weights are calculated to minimise the error variance of the combined forecast (assuming unbiasedness for each individual forecast). Specifically, the linear (n × 1) vector of combining weights, w, is determined according to the formula
S-le w w= e,S_le
(1)
where e is the ( n × l ) unit vector and S is the (n × n) covariance matrix of forecast errors. The problem with this optimising approach is that it requires S to be properly estimated. If the error covariance matrix is not stationary or it has to be estimated on the basis of a short past history, then more robust alternatives have sometimes worked better. (d) Optimal with independence assumption: In this case, S in the above formula is restricted to be diagonal, comprising just the individual error variances. (e) Optimal with restricted weights: In this case the above formula has the additional restriction that no individual weight can be outside the interval [0,1]. Of the methods, not included above, which have often been used, regression can be shown to be theoretically equivalent to the optimal approach, if the individual forecasts are unbiased (thereby allowing the constant term in the regression to be dropped). Although most of the research on this topic has been directed at finding robust procedures to deal with situations of short data sets a n d / o r badly behaved data, with the exception of the Bayesian outperformance method, the issue of incorporating judgement with combinations has been largely ignored. Yet combinations reflect much more difficult problems in this respect than single models. Their structure is not so intuitive in providing coherent input/output relationships between all of the variables. They are a portfolio of disparate methods, motivated by a hedging approach to model selection rather than a desire to create a completely internally consistent single model. Indeed, a combination actually reflects one of two types of modelling failures;
either to identify the singularly correct model, or to build a comprehensive model from all of the variables suggested. This is not to say that the analyst should have done better, it is just a reflection of the pragmatic nature of the approach in coping with the limitations of conventional model identification and estimation techniques. However, this portfolio of models does make it harder to incorporate judgement - should adjustments be made at the level of each component model (but then how does this affect the combining weights?) or should it be done on the combined forecast itself (but how do we know the implicit affect of each predictive factor on the output?). These questions are solvable, but not intuitively, and would seem to require some knowledge-based software interfacing between model and user. When we turn to neural networks, we see many of these original criticisms now being repeated in a more extreme way. Just as combinations avoid the problem of model identification by hedging, neural networks avoid the same issue by facilitating machine learning across a very broad class of specification topologies. Furthermore, this approach appears to offer the advantages of freedom from statistical assumptions, resilience to missing observations and the ability to incorporate nonlinear functional relationships. But all this is at the price of very limited statistical diagnostics, the lack of an easily understandable structure, and the consequent need to rely upon empirical performance for validation. Many researchers in the neurocomputing literature seek an a priori justification for their configurations and heuristics by biological analogy, hence the traditional positioning of this research within artificial intelligence, but from a business applications perspective, this is less convincing. It is more relevant within our context to see this as a very general procedure for specifying and estimating nonlinear relationships in data-rich applications. Sharda (1994), provides a good recent reference on O R / M S applications and their relative performance. Much of the work is comparative with more traditional techniques such as Box-Jenkins and Linear Regression. In many situations, the artificial neural networks do seem to offer performance improvements. Some financial institutions are trading in certain markets using them (Davidson, 1994), but
D.W. Bunn / European Journal of Operational Research 92 (1996) 528-536
in general most practitioners and critics are still cautious about the lack of explanation and potential over-fitting in the procedure. Thus, the problems that combinations have faced in terms of judgemental interaction seem to be magnified severalfold in this methodology. Unlike the combining forecasts experience, it may be that even a large preponderance of empirical evidence in its favour will not be sufficient to create confidence in the technology without more research into explainability and robustness diagnostics.
4. Final c o m m e n t s
Whereas, the traditional scientific time-series method of model selection, estimation and testing predominated within the research paradigm of the 1970s, we have recently seen a divergence into data-intensive methods and judgementally-oriented procedures. In their different ways, both of these extreme directions are seeking to avoid the model selection issue. But they are being fostered in separate contexts. In short-term, data-rich applications, where accuracy is important, we have seen a move from using a single model to a combination of several, i.e. model synthesis. But in the longer term, to support business strategy where understanding the risks is the main objective, we are seeing just the opposite, ie move to model divergence, by testing many different scenarios within a decision framework. In between these two extremes is a broad class of applications where an intuitive balance of judgement and data-based modelling is required, and where few formal procedures have yet emerged. The need is apparent, however, and the work of Collopy and Armstrong (1992) in developing a data-intensive forecasting system of multiple models and a knowlwdge base ('rule-based forecasting', sic) is a useful and innovative response to this. It seems to follow from this line of thinking that if data-intensive techniques continue to rely upon empirical performance for validation and stay judgementally opaque, their impact will stay in the shortterm, data-rich class of forecasting applications. But if and when explainability and robustness diagnostics become available, there could be a exciting prospect
535
of more formal, knowledge-based procedures for judgemental interaction, and to the extent that this becomes credible to practitioners, the scope of such modelling would be that much wider. Similarly in the longer term, the more that scenario based methods (Bunn, 1993) can become model-based, moving from systems thinking to systems modelling, the more that they can provide a defensible basis for a broader class of forecasting applications in strategy. Should we start to see a convergence in forecasting methodology again, between the judgemental and data-intensive, it is clear that it will be to quite a different tradition from the 1970s, and one that will be based more upon developing intelligent software platforms for data analysis, multiple-modelling and judgemental interaction.
References Armstrong, J.S. (1985), Long Range Forecasting, Wiley, New York. Badiru, A.B., Pulat, P.S., and Kang, M. (1993), "DDM: Decision support system for hierarchical dynamic decision making", Decision Support Systems 10, 1-18. Box, G.E.P., and Jenkins, G.M. (1976), Time Series Analysis" Forecasting and Control, Holden-Day, San Francisco, CA. Brown, L.D. (1988), "Comparing judgemental to extrapolative forecasts: It's time to ask why and when", International Journal of Forecasting 4/2, 171-173. Bunn, D,W. (1975), " A Bayesian approach to the linear combination of forecasts", Operational Research Quarterly 26, 325329. Bunn, D.W. (1989), "Forecasting with more than one model" Journal of Forecasting 8/3, 161-166. Bunn, D.W., and Salo, A. (1993), "Forecasting with scenarios", European Journal of Operational Research 68/3, 291-303. Burro, D.W., and Salo, A. (1995), "Adjustment of forecasts with model consistent expectations", Research Paper, London Business School. Bunn, D.W., and Wright, G. (1991), "Interaction of judgemental and statistical forecasting methods: Issues and analysis", Management Science 37/5, 501-518. Clemen, R. (1989), "Combining forecasts: A review and annotated bibliography", 5/4, 559-584. Collopy, F., and Armstrong, J.S. (1992), "Rule-based forecasting: Development and validation of an Expert Systems approach to combining time-series extrapolations", Management Science 38/10, 1394-1414. Corker, R.J., Holly, S., and Ellis, R.G. (1986), "Uncertainty and forecast precision", 2/1, 53-70. Diefenbacher, H., and Johnson, J. (1987), "Energy forecasting in
536
D.W. Bunn / European Journal of Operational Research 92 (1996) 528-536
West Germany", in: T. Baumgartner and A. Midtum (eds.), The Politics o f Energy Forecasting, Oxford University Press, Oxford. Flores, B.E., Olson, D.L., and Wolfe, C. (1992), "Judgmental adjustment of forecasts: A comparison of methods", International Journal of Forecasting 7, 421-433. Glantz, M.H. (1977), "Consequences and responsibilities in drought forecasting", Water Resources Research 18, 3-13. Harrison, P.J., and Stevens, C.F. (1976), "Bayesian forecasting", Journal of the Royal Statistical Society, Series B 38, 205-247. Harvey, A. (1984), " A unified view of forecasting", Journal of Forecasting 3/3, 245-284. Holden, K., Peel, D.A., and Thompson, J.I., (1990), Economic Forecasting: An Introduction, Cambridge University Press, Cambridge. Huss, W.R. (1985), "Comparative analysis of company forecasts and advanced time-series techniques using annual electric utility energy sales data", 1/3, 217-239. Huss, W.R. (1987), "Forecasting in the electric utility industry", in: S. Makfidakis and S. Wheelwright, The Handbook of Forecasting, Chapter 5, Wiley, Chichester. lee, J.K., Oh, S.B., and Shin, J.C. (1990), "UNIK-FCST: Knowledge-assisted adjustment of statistical forecasts", Expert Systems with Applications 1, 39-49. Mathews, B.P., and Diamantopoulos, A. (1990),"Judgemental revision of sales forecasts: Effectiveness of selection" Journal of Forecasting 9/4, 407-416. McNees, S.K., and Pema, N.S. (1981), "Forecasting macroeco-
uomic variables: An eclectic approach", New England Economic Journal, May/June, 15-30. Newbold, P., and Granger, C.W.J. (1974), "Experience with forecasting univariate time series and the combination of forecasts", Journal of the Royal Statistical Society. Series A 137/2, 131-165. Rosenhead, J., (ed.) (1989), Rational Analysis for a Problematic WorM, Wiley, Chichester. Saaty, T.L., and Vargas, L.G. (1991), Prediction, Projection and Forecasting, Kluwer Academic Publishers, Boston, MA. Salo, A., and Bunn, D.W. (1995), "Decomposition in the assessment of judgemental probability forecasts", Technological Forecasting and Social Change, in press. Senge, P.M. (1990), The Fifth Discipline, Doubleday, New York. Sharda, R. (1994), "Neural networks for the MS/OR analyst: An application bibliography", Interfaces 24, 116-130. Turner, D. (1990), "The role of judgement in macroeconomic forecasting", Journal o f Forecasting 9, 315-346. Wack, P. (1985a), "Scenarios: Uncharted waters ahead", Harvard Business Review 63/5, 73-89. Wack, P. (1985b), "Scenarios: Shooting the rapids", Harvard Business Review 63/6, 139-150. WaUis, K.F., et al. (1984-1988), Models o f the UK Economy: Reviews 1-5, Oxford University Press, Oxford. Willemain, T.R. (1989), "Graphical adjustment of statistical forecasts", 5, 179-185. Wolfe, C., and Flores, B. (1990), "'Judgemental adjustment of earnings forecasts", Journal of Forecasting 9~/4, 389-406.