A comparison of the forecasting performance of WEFA and ARIMA time series methods

International Journal of Forecasting 4 (1988) 81- 101 North-Holland

81

A COMPARISON OF THE FORECASTING PERFORMANCE OF WEFA AND ARIMA TIME SERIES METHODS * Phoebus J. DHRYMES Columbia University, New York, NY 10027, USA

Stavros C. PERISTIANI City University of New York, New York, NY 10021, USA

This paper examines the forecasting performance of the Wharton model (MARK III) over the period 1973 through 1975 and compares it with that of ARIMA models’ performance over the same period. Despite strong intimation in the literature to the contrary, we find that this econometric model, at least, exhibits greater accuracy in every respect relative to ARIMA methods. in terms of its forecasts cum constant adjustments. When constant adjustments are disallowed then its forecasts are still more accurate than ARIMA forecasts over a 4- and &quarter forecasting horizon, but less accurate over a l-quarter horizon. The comparison was carried out over twenty three macrovariables, under a slight handicap for the Wharton Model, in that the latter’s parameters were estimated over a sample ending in 1969.3 while the ARIMA models were reidentified and reestimated as of the quarter immediately preceding the forecast. Forecasting accuracy, Wharton model, Forecasting horizon, Exogenous variables, Parametric stability, Comparative accuracy, causal, time series.

1. Introduction

The purpose of this paper is to examine the forecasting performance of WEFA, a large scale econometric model, vis-a-vis that of Box-Jenkins (BJ) time series procedures. A number of papers in the literature deal with the general problem of forecast evaluations, for example, Naylor et al. (1972), Nelson (1972) Narasimham et al. (1974) Levenbach et al. (1974) and Hirsch et al. (1974). More recently issues of forecasting with econometric models have been dealt with in Longbottom and Holly (1985) and extensive reviews of the subject have been published in Armstrong (1978) and Fildes (1985). A definitive exposition of the Mark III version of the Wharton model is found in McCarthy (1972). Nelson, Narasimham and Naylor find that time series ARIMA forecasts do better than forecasts made by using the FMP, BEA and WEFA models, respectively. In particular, Nelson examines some dozen or so macrovariables and claims [that] ’ the results of the comparison of FMP and ARIMA model prediction accuracy reported in the study indicate that the former were more accurate for most of the variables during the sample period over which both models were fitted’. * The research on which this paper was based was, in part, supported by NSF grant SOC 74-18761. We would like to thank the Editor and an anonymous referee for helpful suggestions on a previous draft. 0169.2070/88/$3.50

0 1988, Elsevier Science Publishers B.V. (North-Holland)

82

P.J. Dhrymes and S. C. Peristiani / Performance of WEFA and ARIMA methoa!s

Naylor et al. examine only four macrovariables, investment, the rate of inflation, GNP and the their conclusions are based solely on a comparison of one period ahcad forecasts. Narasimham et al., although they consider only a limited number of macrovariables, base their comparisons on forecasts of up to four quarters ahead. More recently, Longbottom and Holly returning to the same topic make the point that differences in forecasting performance may indicate misspecification errors; it should be noted, however, that this is a rather overreaching conclusion unless such differences are prolonged and systematic, since occasional (random) deviations in the accuracy of forecasts are to be expected even when one model is known to be ‘correct’ and the other ‘r&specified’. Finally, there exist in the literature a large number of papers that touch on one or another aspect of our subject; the interested reader is referred to the recent reviews by Makridakis et al. (1985), Makridakis and Wheelwright (1978) and Fildes (1985). A continuing record of the forecasting performance of commercial models is available in McNees (1981, 1985a, 1986). The latter work, however, compares mainly the ‘published’ forecasts from these models and as such it tends to commingle the performance of models qua models with such performance when combined with the astuteness of the model operator. Indeed, this difference is an important motivating factor for our work. A great deal of the published work on the forecasting performance of econometric models, beginning, say, with Cooper (1972) generates model forecasts not quite under the same conditions as the model operators do and, frequently, not using quite the same model. This is not due to any inclination on the part of the investigator to be unfair to the model, but rather to the fact that the investigator does not have access to the modules used by the model operators and is forced to reestimate some version of the model, not necessarily the version used at the time the forecasts were generated. This feature was amply discussed in the case of Cooper’s work. The unique aspect of this paper is that it uses an interesting set of data, consisting of the internal records of the Wharton model and the forecasts that it generates over the period 1972.4 through 1973.4, using the Mark III version. Thereafter the Mark III version was retired and the records of the successor version were not generally accessible. Thus, the data set on which we operate is uniquely free of the criticisms noted above. Another very attractive aspect of our data set is that the period over which we have forecasts from Mark III covers one of the more turbulent periods of the postwar era, spanning the imposition and aftermath of the oil embargo and the subsequent abrubt rise in the price of oil. Thus, it offers the opportunity for a particularly stringent test of the ‘robustness’ of various forecasting methods in the face of drastic change. Our data set contains a record of the assumptions made by the model operators in generating their forecasts, as well as a record of the controversial ‘constant adjustments’, common to large scale commercially available macromodels. Thus, it is possible for us to reproduce exactly the conditions under which the forecaster operated at the time of the forecast and consequently makes it possible to carry our various types of ‘experiments’ without doing violence to the structure of the model. In sections 2 and 3 we briefly review the Wharton model and BJ ARIMA time series models. In section 4 we present our empirical findings. Finally, most tables are relegated to the appendix. unemployment rate;

2. Issues in forecasting comparisons The typical econometric macromodel is generally a stochastic vector difference equation with forcing function. The model operator specifies and estimates the model and thereafter uses various assumptions regarding the path of exogenous variables over the forecasting horizon in order to generate the model’s forecasts. In addition to this fairly straighforward feature, a controversial practice has arisen in the case of forecasting with the large commercially available models. This involves the so called ‘constant adjustments’ or ‘add factors’ which simply add a constant to each

P.J. Dhvym:-s wd S i,. Perrstiani / Performance of WEE4 and ARIMA methods

83

equation prior to forecasting. Ihe practice has been defended by model operators as reflecting information they possess, regarding the variables ‘explained’ by the equation in question or data revisions of a particular type, see for example, Klein (1971). The forecasting operation, by ARIMA methods, on the other hand is comparatively simple, and indeed, the forecaster need not be a trained economist in order to operate such schemes. Still the choice of the appropriate order of the autoregressive or moving average part of the specification, or the degree of differencing involve a fair amount of discretion; however, more recently, certain rules which can be incorporated in the relevant software might make individual discretion the exception rather than the rule, see for example Harmon (1985). To just list the process of forecasting in the two contexts is to raise a question as to what is an appropriate comparison between the two methods. The comparison that receives most frequent attention is one involving the published forecasts generated by the various model operators. However, such comparisons do not help us understand the strengths and weaknesses of each approach. They merely answer the following, commercially very relevant, question: if I am to buy forecasting services from a vendor that uses the Wharton model or one that uses ARIMA methods would I get more accurate forecasts from the first or second vendor? Precisely because operators of large macromodels have been sensitive to this type of consideration they have set great store by the provision of ‘accurate’ forecasts. There is, however, no simply available method for producing ‘accurate’ forecasts i,n the long run, save through careful reexamination of the specification, the data base and perhaps the estimation of rnitLzomodels. In the short run, however, one could conceivably gain by other methods such as ‘add factoring’ and ‘constant adjustments’, which despite the protestations of their proponents are best viewed as procedures for incorporatmg the model operators’ ‘insights’ into the forecast. The use of ARIMA methods has also been suggested by critiques such as those of Sims (1980) which intimate that we do not know enough to justify the a priori restrictions placed on ‘structural models’. Despite our alleged ignorance the evolution from Sims’ critique seems to have led us to vector autoregressive systems which are really rather special cases of the unrestricted reduced from of a rather constrained General Linear Structural Econometric Model (GLSEM). Thus, it is quite appropriate to ask whether a large macro model like that of WEFA, over a turbulent period of the US, forecasts better or worse than ARIMA methods; to examine this comparison not merely in terms of the published WEFA forecasts, but also in terms of the forecasts the model would have generated had the (future) exogenous variables been known with certainty and without benefit of ‘add factoring’ or ‘constant adjustments’. Finally, we should note that despite the rather tenuous evidence presented by the major comparison studies of Naylor et al. (1972) and Nelson (1972) the impression is quite widespread that econometric models do rather poorly in comparison with ARIMA methods; see, for example, Pindyck and Rubinfeld (1976 pp. 534ff) or Makridakis and Wheelwright (1978, pp. 582ff).

3. Box-Jenkins, ARIMA and econometric model based forecasts Let y,. be a vector of nz elements containing a set of macro-variables whose behavior we are interested in forecasting (a dot on the second subscript means that y,. is row vector, hence, y.; (i= l,..., m) is column vector of T elements); at our disposal we have a set of observations (u,., x,.),

t = 1, L-9

(1)

T.

where x,. is a set of predetermined

variables. Using as the expositional

paradigm the GLSEM

P.J. Dhr)lmes and S. C. Peristicmi / Performance of WEFA and ARIiUA methods

84

forecasts based on econometric models involve the specification of a model .r;.B* = w&)+y,_~.c,

+ . . . +yp&

(2)

+ u,.,

where w,. is an s-element row vector containing the exogenous variables and the matrices B*, Ci, i=o, l,..., k contain the unknown structural parameters of the model. Having estimated the model by an appropriate technique we then forecast outside the sample period by means of the (restricted) reduced form

A. = W,.fi()+yJ?*

+ . . . +y*+&,

where if,, i=O, l,..., jj, =

@+-I,

k

are estimates of the corresponding

i=O,l,...,

unknown parameter matrices, i.e.,

k.

Thus, forecasting with an econometric model requires estimation of the structural parameter matrices B*, C,, i=O, 1,2 . . . . k as well as knowledge or estimation of future values of the exogenous variables. The required lagged dependent variables for the multiple period forecasting are generated pari passu. ARIMA forecasting is based on the work of Box and Jenkins (1970) and exposited more recently in Chatfield (1975); essentially it involves fitting an autoregressive and error driven model to suitably differenced time series. Thus, the only information required in order to forecast a macrovariable, say y,,, is the sequence of observations (yti: t=l,

2,... T).

This essential simplicity of the procedure has held great appeal for many applied economists. Indeed, one need not have any economic training whatsoever in order to engage in forecasting economic phenomena by ARIMA methods, see, for example, Anderson and Ring (1985). Recently an attempt has been made by Zellner and Palm (1974) to combine the two procedures; further work is also found in Wallis (1977). It should be noted, however, that the work by Zellner and Palm merely provides a convenient way of carrying out a likelihood ratio test for (some of) the a priori restrictions employed by structural (econometric) models; moreover, other works suggesting a transfer function formulation of the basic BJ scheme, do not constitute a particularly novel approach to forecasting, since they are, in effect, special cases of the autoregressive final form of the standard general linear structural econometric model, see, for example, Dhrymes (1974, Ch. 12). Thus, the transfer function formulation destroys the essential simplicity of ARIMA procedures, while at the same time does essentially what econometric models attempt to do, but without access to the structural form. That is, it relies on the unrestricted reduced form and the autoregressive final form and uses various conventions and pre-test rules for deciding the order of the lags. Thus, while useful as a vehicle for testing some of the a priori restrictions Xnthe fashion suggested by Zellner and Palm it does not offer a substantively different forecasting alternative.

4. Empirical results As indicated earlier the WEFA model (Mark III) was used to generate forecasts beginning with 1972.4; forecasts for WEFA are typically generated with an S-quarter horizon and we have followed

P.J. Dhrymes and SC. Peristiani / Performance of WEFA and A RIMA methods

85

this practice in our experiments. Thus, beginning with 1972.4 we have &quarter ahead forecasts. The last (initial) period for which we have a forecasting horizon of g-ahead quarters is 1973.4. Hence, the last quarter actually forecast by WEFA model is 19’75.3 and for the quarters “1973.1through 1975.2 we have multiple forecasts; this period iti one of the most turbulent in the recent economic history of the US and, consequently, provides a very de .landing environment in which the performance of the structural models is to be appraised. ARIMA forecasts are generated from a suitably identified and estimated model using data up to the first quarter in the forecasting horizon of the experiment. We remind the reader that the WEFA model is estimated with data up to 1969.3 and is not reestimated prior to each forecasting episode. The procedure is fairly straight forward: from each model an &quarter ahead forecasting sequence is obtained beginning, consecutively, with 1972.4, 1973.1 and so until 1973.4. There are many criteria in terrils of which the accuracy of the forecasts are to be compared; mainly for simplicity of interpretation, we have chosen the percent root mean squared error criterion. The results are given in exhaustive detail in the tables of the appendix; here we shall confine ourselves to a discussion of the salient features of these findings. A thorough and objective comparison ought to involve more than just a comparison between the published forecasts of WEFA and their ARIMA counterparts, but we begin with it. Comparison: Published WEFA vs. ARIMA For &quarter ahead ARIMA does uniformly better only for one variable, real output per manhour. It is unambigously superior, i.e., in four out of the five forecasting episodes it does better, only for one variable, real non-durable consumption. On the other hand, WEFA forecasts are clearly better, i.e., in three out of five forecasting episodes, WEFA forecasts dominate for five variables and for the remaining sixteen variables WEFA’s forecasts are overwhelmingly more accurate. For $-quarter ahead ARIMA does not do uniformly better for any variable; it is unambigously superior (i.e., it does better in four out of five forecasting episodes) in one variable, real nondurable consumption and is clearly superior (i.e., it does better in three out of five forecasting episodes) for three variables, nominal personal consumption expenditures, real output per manhour and in wage rate (Manufacturing and Mining). In the other nineteen variables WEFA’s forecasts are either clearly superior (seven variables), unambiguously superior (three variables) or uniformly superior (nine variables). For l-quarter ahead ARIMA is uniformly superior for only one variable, real nondurable consumption; it is unambiguously superior for two variables private compensation per hour and wage rate (for Manufacturing and Mining); it is clearly superior for four variables, nominal residential investment, real output per manhour, nominal nondurable consumption, deflator (Manufacturing). For the remaining sixteen variables WEFA’s forecasts were either clearly superior (four variables), unambiguously superior (two variables) or uniformly superior (ten variables). The preceding discussion makes it abundantly clear that, in terms of published records, the widespread belief that BJ ARIMA methods do better than forecasts from econometric models is certainly called into question given our empirical findings. Moreover, one thing is quite clear from this comparison: the structural WEFA model and its operators must incorporate in their forecasting procedures some (and evidently quite substantial) knowledge of the economy which is clearly lacking, by design or otherwise, in the forecasting procedures that are based on ARIMA methods. The obvious response to this claim, is to ask what ‘part’ of this knowledge is to be attributed to the ‘scientific’ aspect of the discipline (economics) and what part is to be attributed to the personal

86

P.J. Dhtymes and S.C. Peristiani / Performance of WEFA and ARIMA methoa!v

insights of the operators; this latter part is profoundly subjective and hence not easily reproducible by other researchers. Although one cannot possibly expect to obtain an unambiguous separation of the two components one may attempt to gauge their magnitudes by examining the question of ‘constant adjustments’ or ‘add factors’. To the extent that the latter is objectively defined in terms of a computable rule that is based on sample observations such practices, although clumsy, may be reasonably considered as part of the model specification. To the extent, however, that such rules are not prespecified prior to forecasting they can only be considered highly subjective and be interpreted as the incorporation of the operators’ personal insights. This is evidently the case with WEFA’s ‘constant adjustments’ for the period under consideration. Owing to the fact that we have access to the internal records and data sets employed by WEFA at the time of the forecasts it is possible for us to reconstruct the forecasts that would have resulted at the relevant time had ‘constant adjustments’ not been employed. We may thus compare these forecasts with those generated by ARIMA methods. The results are summarized in table 2 in the appendix. The salient features of this comparison are given below. Comparison: WEFA without constant adjustment vs. ARIMA For &quarters ahead ARIMA is uniformly superior for five variables, unemployment, private compensation per hour, real nondurable consumption, employment and wage rate (both for Manufacturing and Mining) and clearly superior for three variables, real GNP, real gross private domestic investment and real investment in Manufacturing and Mining. For the remaining sixteen variables WEFA’s forecasts are uniformly superior for nine variables and clearly superior for the remaining six variables. In t&s connection it should be noted that ARIMA is given far more credit than is appropriate. Notice, for example, that BJ dominates real GNP while WEFA dominates nominal GNP as well as the (GNP) implicit price deflator. On the other hand the BJ model does not obey the obvious identity that real GNP times price deflator equals nominal, while WEFA’s model does. When we similarly obtain the third variable through the appropriate identity in the BJ context as well, this superiority disappears. This is a feature that has been overlooked in previous comparisons, and relates generally to the fact that provision for the obvious identities is not made when one compares ARIMA and structural model induced forecasts. For Qquarters ahead ARIMA forecasts are uniformly superior for six variables, unemployment, nominal residential investment, real nondurable consumption, manufacturing (implicit p&c) deflator, government expenditures deflator and employment in Manufacturing and Mining; unambigsously superior for four variables and clearly superior for four variables. For the remaining nine variables WEFA is uniformly superior for six variables, unambiguously superior for two variables and clearly superior for one variable. For l-quarter ahead ARIMA forecasts are uniformly superior for six variables, unambiguously superior for four variables and clearly superior for three variables. For the remaining ten variables WEFA’s forecasts are uniformly superior for one variable, unambiguously superior for four variables and clearly superior for five variables. Thus, even in the absence of correct information on the values of exogenous variables over the forecasts horizon, the WEFA based forecasts ou:perform ARIMA forecasts, in the sense that they are more accurate more frequently for more macrovariables. In particular, for 8-quarters ahead WEFA does better for sixteen macrovariables; for 4-quarters ahead it does better for nine macrovariables while for l-quarter ahead it does better for ten macrovariables. On the other hand when the correct exogenous variables are employed, then, even in the absence of constant adjustments,

F.J. LNvymes find SC. Peristiani / Performance of WEFA and ARIMA methods

87

WEFA’s forecasts are more accurate than ARIMA, for 4- and &quarter ahead and wof~e only for l-quarter ahead. To be precise (see table A.4 in the appendix) WEFA forecasts are better in 13 of the 23 macrovariables for 4- and &quarters ahead and only 8 for l-quarter ahead. It is also interahg to observe in this connection that while the use of the ‘correct’ explanatory variables does not improve WEFA’s performance for one quarter ahead, it does produse improvements for 4- and S-quarters ahead forecasts. The major conclusion one may derive from the experimentation reported in this work is that ARIMA methods have been more accurate for l-quarter ahead forecasts only when ‘constant adjustments’ have been omitted from WEFA’s model. On the other hand ARIMA methods have been accorded the ‘bent-fit’ of reidentification and reestimation prior to each forecasting episode, while WEFA’s models was estimated with data as of 1969.3 and thereafter remained intact. Despite that, it clearly dominates ARIMA induced forecasts for this group of 23 macrovariables over 4- and 8-quarter horizons, whether ‘constant adjustments’ are used or not used and whether we employ the ‘correct’ or assumed values for the exogenous variables required over the forecasting horizon. A brief exercise, not reported herein, also shows that whether we use ARIMA methods or simply a fourth order autoregressive scheme, forecast accuracy over l-, 4- and 8-quarter horizons remains basically the same, although ARIMA does slightly better. Finally, the fact that we have identified and estimated ARIMA models for a number of macrovariables offers us an opportunity to compare our findings with those Nelson’s (1972) whose work is frequently cited as having demonstrated the ‘superiority’ (complete or near) of ARIMA to structural model based forecasts. The identification and parameter estimates for our work appear in table A.5 while those from Nelson’s work have been obtained directly from Nelson (1972) (identifications appear in the appendix of the original paper). This discussion is best understood as an obiter dictum rather than as a serious examination of the stability of parameters in an ARIMA context. We remind the reader that Nelson’s results are based on the sample 1947.1-1966.4 while ours are based on the period 1955.1-1973.3 thus producing an overlap of about 12 years, in addition, Nelson’s results are based on data that, almost certainly, have since been revised. There are five common

Table 1 Comparison of structures and parameters. Model identification

ours

Nelson’s a

Nominal GNP

ARIMA(l.l.0)

Nominal non-res. invest. (Exp. on Prod. structures in Nelson)

ARIMA(2.1.0)

Nominal residential inv. (Housing exp. in Nelson)

ARIMA(l.l.

Unemployment

ARIMA(2.0.0)

Deflator cons. exp. (Cons. goods price index in Nelson)

ARIMA(l.l.

0)

0)

0.618 (6.26) 0.279 0.063 (2 3) (0.9)

ARIMA( 1.1.0)

0.615

AF IWX(O.l.1)

0.347

0.635 (5.2)

,\RIMA(X i. 0;

0.639 0.076 -0.286

ARSA(2.

1.46 0.612 0.284

1.58 - 0.08 (17.4) (-7.4) 0.750 (9.9)

ARIMA(l.l.

’ Numbers in parentheses represent t-ratios; no t-ratios are given in Nelson’s work.

0. 1)

0)

0.414

P.J. Dhtymes and S.C. Peristiani / Performance cf WEFA and ARlMA

88

metho&

variables and a comparison is made in table I. Even a casual inspection of the comparison above shows that while GNP is a very ‘stable’ process, the other variables evidently represent rather ‘unstable’ processes; which shows that parameter instability is not a problem confined exclusively to the estimation of parameters of structural models.

5. Conclusion This paper has examined the forecasting performance of WEFA’s model (MARK III) over the period 1973-1975 and has compared it to that of models obtained by ‘purely statistical’, in particular Box-Jenkins ARIMA, methods. The salient findings are Forecasts based on econometric models decisively outperform those based on ARIMA models, when the forecasting horizon is either 4- or &quarters ahead; this is true whether constant adjustments are or are not used and whether we employ the correct or the assumed exogenous variables over the forecasting horizon. (ii) For a l-quarter ahead horizon WEFA’s forecasts dominate by an overwhelming margin, if constant adjustments are employed. If constant adjustments are not employed then ARIMA forecasts dominate by a moderate margin (over a l-quarter horizon) and this is true whether the ‘true’ or the assumed exogenous variables are employed over the forecasting horizon. (iii) As an aside, we compared the results of our ARIMA indentification and estimation with those of Nelson’s, whenever there was an overlap of variables; there were five such variables. Of these two produced the same structure and roughly similar parameters; for the other three we identified an entirely different structure.

(i)

The major inference to be drawn from this study is that the dedeficiencies of structural models, real and substantial as they are, have been exaggerated in the literature and that ARIMA models do not necessarily make for a better alternative except perhaps in a very limited r~chanical and extremely short term forecasting context.

Appendix This appendix contains the comparison statistics for the various experiments reported in the body of the paper. An extensive summary of all numerical results may be found in Dhrymes and Peristiani (1983). Tables A.l-A.4 contain all the appropriate comparisons. The symbols have the following meanings: S B BB BBB W

ARIMA’s forecasts have approximately (relative difference is less than 0.01) the same PRMSE as WEFA’s forecasts. ARIMA’s forecasts have PRMSE that are smaller, but not less than one half ( l) of WEFA’s forecasts. ARIMA’s forecasts have PRMSE which are between one quarter ($) and one half ($) of W EFA’s forecasts. ARIMA’s forecasts have PRMSE which are at most one quarter ($) of those of WEFA’s forecasts. ARIMA’s forecasts have PRMSE which are larger than (at most twice) those of WEFA’s forecasts.

P.J. Dhryntes and SC. Peristiani / Performance of WEFA and ARIA

methods

89

Table A.1 Comparison of PRMSE of 1.4 and g-quarter forecasts; WEFA model, with constant adjustments, ‘old’ exogenous variables; BJ ARIMA reestimated up to quarter immediately predeeding forecasts. Variable

Periods

1972.4

1973.1

1973.2

Nominal GNP

1 4 8

W W ww

W W ww

S

W W

R~dl GNP

1 4 8

W W ww

W W B

W* B B

Implicit price deflator

1 4 8

W W W

W W W

W W !Q

S W W

W W W

Unemployment

1 4 8

W W B

ww B B

W W W

W W W

W W W

Nominal personal consumption expenditures

1 4 8

W W W

W W W

ww B B

W B W

www B B

Nominal non-auto durables consumption

1 4 8

W W ww

W W ww

www W ww

W W www

W ww ww

Nominal gross domestic investment

1 4 8 1 4 8

ww ww www ww www www

www ww W ww www www

BBB W W BB ww www

BB W W www ww ww

W W ww B ww W

Nominal resdential investment

1 4 8

W B W

BB ww W

BB ww ww

ww ww W

BB ww W

Imports

1 4 8

W W W

W ww ww

www ww ww

W ww ww

W W www

Real gross private domestic investment

1 4 8

WWW www W

BBB W BB

BBB BB BB

www B W

W W W

Real output/hour

1 4 8

B B B

B B B

S W B

B B B

W W B

Private compensation/ hour

1 4 8

BBB www ww

WV!‘W ‘WWW ww

B BB ww

BB W ww

BB W ww

Nominal nondurable consumption

1 4 8

W ww ww

ww ww W

BB B B

BBB W W

BB B B

Nominal nonresidential investment

1973.3

1973.4

W* W* W

90

P.J. Dhrymes and S.C. Peristiani / Performance of WEFA and ARIMA methods

Table A.1 Variable

Periods

1972.4

1973.1

1973.2

1973.3

1973.4

B W B

W W W

4 8

W W W

W W B

W W W

Real nondurable consumption

1 4 8

B BB www

B BB BB

B B B

B B B

B W B

Real consumer services

1 4 8

W ww www

W ww ww

W W ww

W W ww

W W ww

Real investment in Mining and Manufacturing

1 4 8

B WW WWW

W ww www

B BBB B

W B ww

www ww WWW

Deflator, consumer expenditures

1 4 8

W W W

W W W

W W W

W W W

W W W

Deflator, Manufacturing

1 4 8

B B B

B B B

W W W

B W W

W ww W

Deflator, Government expenditures

1 4 8

ww www

W W W

ww W W

www W

B

W W B

Employment, Mining and Manufacturing

1 4 8

W W ww

W ww B

W B W

B B W

www www W

Wage rate, Mining and Manufacturing

1 4 8

B ww ww

W www ww

B B ww

B B W

B B W

IUominal consumer services

1

ww

ARIMA’s forecasts have PRMSE which are between two (2) and four (4) times those of WEFA’s forecasts. www ARIMA’s forecasts have PRMSE which are more than four (4) times larger than those of WEFA’s forecasts. W*. B* Same as defined above, however, their counterparts have PRMSE which is very small, hence, both methods provide rather accurate forecasts. Table A.5 gives the identification and estimation of ARIMA processes. Parameters are given as follows: first the autoregressive parameters beginning with the lowest lag and then the moving average parameters also beginning with the lowest lag. Table A.6 contains some general quantitative information by providing the median PRMSE of all five forecasting time periods.

P.J. Dhrymes and S.C. Peristiani / Performance of WEFA and A RIMA methodr

91

Table A.2 Comparison of PRMSE of 1.4 and 8-quarter forecasts; WEFA model: without constant variables; BJ ARIMA reestimated up to quarter immediately preceeding forecasts.

adjsustments,

Variable

Periods

1972.4

1973.1

1973.2

1973.3

Nominal GNP

1

4 8

W ww WWW

W ww ww

B B ww

1 4 8

W W ww

B W W

W*

W*

B*

BB B

BBB B

BBB B

Implicit price deflator

1 4 8

W WWW

W WWW WWW

B B W

B B W

B B W

Unemployment

1 4 8

B BB BBB

BBB BBB BBB

BBB BB BB

BB

B

BB BBB

BB BB

Nominal personal consumption expenditures

1 4 8

W W ww

W ww ww

B B S

B B W

Nominal non-auto durables consumption

1 4 8

W ww ww

W ww ww

B W ww

WWW W ww

Nominal gross domestic investment

1 4 8

W W B

ww S BB

BBB W W

Nominal nonresidential investment

1 4 8

W ww W

W W B

BB WW W

ww ww W

W W B

Nominal residential investment

1 4 8

BB BB W

BBB B W

BBB B W

B B B

BB B B

Imports

1 4 8

W W W

W W ww

W W ww

W ww ww

WW WWW www

Real gross private domestic investment

1 4 8

W W B

B BB BB

B B BB

www B W

BBB BBB W

Real output/hour

1 4 8

W ww ww

www www ww

W W W

B W W

B W W

Private compensation/ hour

1 4 8

BBB B B

B W B

B BBB BBB

BEB BB BB

BB BB BB

Nominal nondurable consumption

1 4 8

B B W

W ww ww

B BB ww

BBB B BB

B B BB

Real GNP

‘old’ exogenous

1973.4 B W WW

92

P.J. Dhrymes and S.C. Peristiani / Performance of WEFA and ARIMA methods

Table A.2 Variable

Periods

1972.4

1973.1

1973.2

1973.3

1973.4

Nominal consumer services

1 4 8

B W ww

B W W

B W ww

BB B W

B B W

Real nondurable consumption

1 4 8

B BB BBB

B B BB

B B BB

B BB BB

B B BB

Real consumer services

1 4 8

B W ww

W ww

W W ww

W ww W

Real investment in Mining and Manufacturing

1 4 8

B B W

BB W W

WWW BBB BB

ww B BB

W W B

Deflator, consumer expenditures

1 4 8

W WWW WWW

W W ww

W W ww

B W W

B W W

Deflator, Manufacturulg

1 4 8

BB BBB ww

B BB B

B BB W

BBB B B

BB W W

Deflator, Government expenditures

1 4 8

B B W

ww

BBB B W

BB B W

B B S

Employment, Mining and Manufacturing

1 4 8

B B B

www BB BB

B BBB BBB

BB BBB BBB

B BBB BBB

Wage rate, Mining and Manufacturing

1 4 8

BBB

W* W B

B BS BBB

B BB BBB

B BB BBB

BB BB

B B

P.J. Dhrymes and S. C. Peristiani / Performance of WEFA and ARIMA methoa5 Table A.3 Comparison of PRMSE of 1, 4 and 8-quarter forecasts; WEFA model, with constant variables; BJ ARIMA, reestimated up to quarter immediately preceeding forecasts. Variable

Periods

Nominal GNP

1 4 8

Real GNP

Implicit price deflator

1 4 8

1972.4

1973.1

‘correct* exogenous

1973.2

1973.3

1973.4

WWW W B

W B

B BBB

W

W* W B

W* B BB

B* BBB BBB

W W W B W W

WWW W WWW WWW

adjustments,

93

1 4 8

W W

W W W

Unemployment

1 4 8

BBB BB BB

BBB BBB BB

BBB B W

BBB B W

Nominal personal consumption expenditures

1 4 8

W W ww

W ww ww

W B B

ww W B

Nominal non-auto durables consumption

1 4 8

W ww ww

W ww www

W W ww

B W

B W W

Nominal gross domestic investment

1 4 8

B ww W

BBB W B

BBB ww B

BB B B

BBB BB B

Nominal nonresidential investment

1 4 8

ww ww www

ww ww WWW

B B W

B W W

ww ww S

Nominal residential investment

1 4 8

W B S

B W W

Imports

1 4 8

W W W

ww ww ww

W W W

W W W

B W B

Real gross private domestic investment

1 4 8

B W B

BBB BB BB

BBB BB BB

B BB B

BBB BBB B

Real private output/ hour

1 4 8

B B B

B B BB

B B B

B W B

Private compensation/ hour

1 4 8

B W www

W www ww

B W W

B W W

Nominal nondurable consumption

1 4 8

W W WWW

W W ww

BBB B W

BBB W B

BB ww W

94

P.J. Dhymes and S.C. Peristiani / Performance of WEFA and ARIMA methods

Table A.3 Periods

1972.4

1973.1

1973.2

1973.3

1973.4

1 4 8

W W W

B B W

W W W

B B B

W W W

Real nondurable consumption

1 4 8

B BBB BBB

B

B B B

B B B

B W B

Real consumer services

1 4 8

W ww ww

W WW

W W ww

W ww

W W ww

Real investment in Mining and Manufacturing

1 4 8

BB W W

W W B

W

ww

www

BBB BBB

BB BBB

B BB

W W W B

BB B B B B

W W W B BB B BB B B

W W ww B B ww W ww www

W W W B B ww W ww W

W ww ww W ww www W W W

1 4 8

W www ww

www W W

W B

ww

BB

BB B BB

1 4 8

B ww ww

B WWW ww

BB B ww

B B W

B B W

Variable Nominal consumer services

Deflator, consumer expenditures Deflator, Manufacturing

Deflator, Government expenditures Employment, Mining and Manufacturing Wage rate, Mining and Manufacturing

BB BB

W B

P.J. Dhrymes and SC. Peristiani / Performance of WEFA and ARIMA methoak

95

Table A.4 Comparison of PRMSE of 1, 4 and &quarter forecasts; WEFA model, without constant adjustments, ‘correct’ exogenous variables; BJ ARIMA reestimated up to quarter immediately preceeding forecasts. Variable

Periods

Nominal GNP

1

1972.4

1973.1

1973.2

1973.3

1973.4

WWW www

W W W

W W W

W B W

4 8

WWW WWW

Real GNP

1 4 8

ww www WWW

ww W S

WWW B* S

W* W* B

B B B

Implicit price deflator

1 4 8

W WWW WWW

W ww

B W W

B W W

B B W

Unemployment

1 4 8

B BBB BBB

BB BBB BBB

BBB BB BB

BBB BB BB

B BB BB

Nominal personal consumption expenditures

I 4 8

W ww

W ww ww

B B W

B B W

ww B W

Nominal non-auto durables consumption

1 4 8

W WWW

W W ww

Nominal gross domestic investment

1 4 8

W W B

BB BB B

BBB BB W

BB B W

BBB BBB B

Nominal nonresidental investment

1 4 8

W W W

W W W

B B ww

B W W

B W W

Nominal residential investment

1 4 a

BB BB W

BBB W ww

BBB W W

B ww W

BB B B

Imports

1 4 8

W W W

W ww ww

B W W

W W W

W W W

Real gross private domestic investment

1 4 8

B B BB

BBB BB BBB

BB BB BB

B BB B

BBB BBB B

Real private output/ hour

1 4 8

B B W

B W B

B W W

B B S

B W S

Private compensation/ hour

1 4 8

BBB W B

www W BB

B B B

BBB BB BB

BBB W W

Nominal nondurable consumption

1 4 8

B B W

S B B

B BBB BBB

BBB B B

BBB W B

96

P.J. Dhrymes and SC. Peristiani 1’ Performance of WEFA and AROMA methods

Table A.4 Variable

Periods

1972.4

1973.1

1973.2

1973.3

1973.4

Nominal consumer

1 4 8

B W W

S

W W

B W W

BB B B

W W W

Real nondurable consumption

1 4 8

BB B BBB

BB BB BBB

B BB BB

B BB BB

B B B

Real consumer services

1 4 8

W W

Real investment in Mining and manufacturing

1 4 8

BBB B B

Deflator, consumer expenditures

1 4 8

W W

Deflator, Manufacturing

1 4 8

B BB BBB

B BBB B

Deflator, Government expenditures

1 4 8

B B B

Employment, Mining and Mzmdfacturing

1 4 8

Wage rate, Mining and Manufacturing

1 4 8

services

W W W

www

ww

BBB BBB

BB BB

WWW B BB

W W W

W W W

BB BB BB

BBB BB B

W ww WWW

BBB BB BB

BBB BB B

BBB B W

W W W

W B B

B BB BBB

B BB BBB

B BB BBB

ww

B B B

W B B

B B BBB

B B BBB

B B W

B B B

W B

P.J. Dhrynes and S.C. Peristiani / Performance of WEFA and ARIMA metho&

97

Table A.5 BJ identification of the economic time series (1955.1-1973.3). a Variable

Model

Nominal GNP

ARIMA(l, 1,0)

0.618 (6.26)

Real GNP

ARIMA(1, 1,O)

0.430 (3.88)

Implicit price deflator

ARIMA(4,1,0)

0.425 (3.54)

0.211 (1.58)

Unemployment

ARIMA(2, 0.0)

1.578 (17.3)

- 0.6852 ( - 7.3)

Nominal personal consumption expenditures

ARlMA(2, 1,O)

0.325 (2.95)

0.458 (3.98)

Nominal non-auto durables consumption

ARlMA(1, 1, 0)

0.248 (2.10)

Nominal gross domestic investment

ARIMA(l,l,l)

Nominal nonresidential investment

ARIMA(2,1,0)

0.279 (2.29)

Nominal residential investment

ARIMA(1,l.O)

0.635 (5.19)

Imports

ARIMA(1, 1.0)

0.286 (2.69)

Real gross private domestic investment

ARIMA(l,O,O)

0.965 (38.1)

Real private output/ hour b

ARIMA(2,1,0)

0.060 (0.51)

0.072 (0.60)

Private compensation/ hour

ARIMA(2, 1, 0)

0.396 (3.48)

0.411 (3.59)

Nominal nondurable consumption

ARIMA(2, 1,O)

0.299 (2.84)

0.573 (5.21)

Nominal consumption services

ARIMA(2, 1,O)

0.663 (5.73)

0.336 (2.78)

Real nondurable consumption

ARIMA(2,Q 0)

0.958 (75.3)

Real consumption services

ARIMA(1, 1.0)

0.441 (3.85)

Real investment in Mining and Manufacturing

ARIMA(2,l.

1)

1.483 (14.1)

Deflator, consumer expenditures

ARIMA(1, LO)

0.750 (9.85)

Coefficient estimates (t-ratio)

- 0.636 ( - 0.77)

-0.019 (-0.12)

- 1.64 ( - 2.4) 0.063 (0.91)

- 0.10 (-1.4)

- 0.533 (- 5.39)

0.984 (I 1.2)

0.029 (2.07)

98

P.J. Dhtymes and S. C. Peristiani / Performance of WEFA and A RIMA methoak

Table A.5 (continued) Variable

Model

Coefficient estimates (t-ratio)

Deflator, Manufacturing

AlUMA(1, 1.1)

0.974 (24.4)

0.676 (6.71)

Deflator, Government expenditures

ARIMA(4,1,0)

0.091 (0.85)

0.062 (0.62)

Employment, Mining and Manufacturing

ARIMA(2, 0,O)

1.609 (16.8)

- 0.606 ( - 6.42)

Wage rate, Mining and manufacturing

ARIMA(3,l.

0.320 (2.85)

0.194 (1.65)

0)

0.136 (1.28)

0.554 (5.14)

0.396 (3.19)

B These identified models were used to create forecasts for the period 1973.4-1975.3. Since identification of the other time periods gave similar processes with similar coefficients, the above table is representative of all time periods. b Although the coefficients of this model are insignificant, it was chosen because it was the better forecasting model.

P.J. Dhrymes and SC. Peristiani / Performance of WEFA and ARIMA methods

99

Table A.6 Median PRMSE for periods 1972.4-1973.4 (A = ARIMA, B = WEFA with constant adjustments “old” exogenous variables, C = WEFA without constant adjustments “old” exogenous variables, D = WEFA with constant adjustments “correct” exogenous variables, E = WEFA without constant adjustments “correct’* exogenous variables). Variables

Period

A

B

C

D

E

Nominal GNP

1 4 8

0.0308 0.0451 0.0864

0.0251 0.0282 0.0419

0.0354 0.0304 0.0282

0.0091 0.0256 0.0604

0.0133 0.0317 0.0420

Real GNP

1 4 8

0.0090 0.0062 0.0196

0.0040 0.0101 0.0154

0.0262 0.0208 0.0312

0.0048 0.0099 0.0255

0.0074 0.0072 0.0223

Implicit price deflator

1 4 8

0.0200 0.0440 0.0696

0.0183 0.0286 0.0530

0.0373 0.0503 0.0479

0.0113 0.0205 0.0383

0.0205 0.0347 0.0358

Unemployment

1 4 8

0.0298 0.0843 0.1606

0.0142 0.0594 0.1137

0.0814 0.3410 0.5768

0.2254 0.1271 0.1985

0.0998 0.2952 0.5423

Nominal personal consumption expenditures

1 4 8

0.0464 0.0262 0.0462

0.0211 0.0316 0.0452

0.0359 0.0288 0.0252

0.0221 0.0261 0.0444

0.0268 0.0259 0.0274

Nominal non-auto durables consumption

1 4 8

0.0198 0.0494 0.0883

0.0152 0.0175 0.0427

0.0233 0.0292 0.0382

0.0382 0.0263 0.0306

0.0179 0.0284 0.0372

Nominal gross domestic investment

1 4 8

0.0281 0.0365 0.0834

0.0190 0.0209 0.0407

0.0217 0.0329 0.0908

0.1148 0.0785 0.095 1

0.0958 0.0895 0.0966

N,l~nical nonresidential investment

1 4 8

0.0370 0.0723 0.1092

0.0279 0.0110 0.0237

0.0318 0.0446 0.0862

0.0188 0.0487 0.0408

0.0517 0.0454 0.0589

Nominal residential investment

1 4 8

0.0474 0.2197 0.1910

0.0491 0.0784 0.0933

0.1610 0.1652 0.1543

0.0432 0.0796 0.1319

0.1531 0.1433 0.1335

Imports

1 4 8

0.1089 0.1454 0.2056

0.0729 0.0557 0.0667

0.0783 0.0942 0.872

0.0914 0.1017 0.1433

0.0835 0.1071 0.1442

Real gross private domestic investment

1 4 8

0.0143 0.0286 0.0562

0.0107 0.0337 0.0517

0.0252 0.0677 0.0696

0.1003 0.0684 0.0901

0.0847 0.0815 0.0897

Real output/hour

1 4 8

0.0411 0.0486 0.0615

0.0446 0.0483 0.0671

0.0412 0.0346 0.0408

0.0598 0.0624 0.0853

0.0581 0.0491 0.0612

Private compensation /hour

1 4 8

0.0041 0.0205 0.0574

0.0104 0.0156 0.0159

0.0196 0.0480 0.1163

0.0104 0.0142 0.0148

0.0148 0.0245 0.1066

Nominal nondurable consumption

1 4 8

0.0105 0.0227 0.0460

0.0132 0.0190 0.0449

0.0155 0.0261 0.0736

0.0301 0.0198 0.0310

0.0302 0.0487 0.0930

loo

P.J. Dhtymes and SC. Peristiani / Performance of WEFA and ARIMA methods

Table A.6 Variables

Period

A

B

C

D

E

0.0517 0.0566

0.0622 0.0522 0.0434

Nominal consumer services

1 4 8

0.0581 0.0656 0.0811

0.0535 0.0553 0.0718

0.0619 0.0548 0.0481

Real nondurable consumption

1 4 8

0.0338 0.0458 0.0522

0.0392 0.0558 0.0754

0.0596 0.0897 0.1246

0.0536 0.0709 0.0972

0.0653 0.0847 0.1308

Real consumer services

1 4 8

0.0340 0.0447 0.0581

0.0279 0.0216 0.0159

0.0200 0 0116 0.0202

0.0264 0.0157 0.0147

0.0181 0.0132 0.0189

Real investment in Mining and Manufacturing

1 4 8

0.0239 0.0545 0.0969

0.0311 0.0256 0.0277

0.0244 0.0621 0.1453

0.0126 0.1179 0.1755

0.0100 0.1178 0.2041

Deflator, consumer expenditures

1 4 8

0.0539 0.0774 0.1134

0.0412 0.0512 0.0915

0.0508 0.0519 0.0559

0.0370 0.0399 0.0544

0.0400 0.0391 0.0410

Deflator, Manufacturing

1 4 8

0.0290 0.0387 0.1160

0.0196 0.0374 0.0851

0.0662 0.0869 0.0926

0.0361 0.0609 0.0586

0.0688 0.1072 0.1516

Deflator, Gcvernment expendhes

1 4 8

0.0084 0.0312 0.0431

0.0053 0.0130 0.0214

0.0473 0.0427 0.0327

0.0114 0.0211 0.0275

0.0493 0.0550 0.0520

Employment, Mining and Manufacturing

1 4 3

0.0193 0.0248 0.0401

0.0123 0.0272 0.0263

0.0302 0.0897 0.1881

0.0139 0.0200 0.0476

0.0311 0.0939 0.1708

K -%eRate, Mining and Manufacturing

1 4 8

0.0089 0.0192 0.0527

0.0156 0.0255 0.0247

d.0327 0.0598 0.1546

0.0193 0.0247 0.0226

0.0184 0.0330 0.1260

Average median

1 4 8

0.0312 0.0544 0.0841

0.0257 0.0335 0.0502

0.0402 0.0679 0.1012

0.0454 0.0493 0.0662

0.0487 0.0699 0.1056

0.0566

0.0365

0.0698

0.0536

0.0747

Overall average median

P.J. Dhrymes and SC. Peristiani / Performaxe

of WEFA and ARIMA methods

101

References Anderson, O.D., ed., 1985, Time series analysis: Theory and practice, Vol. 7 (North-Ii&and, Amsterdam)_ Armstrong, J., 1978, Forecasting with econometric methods: Folklore versus fact, Journal of ~us&ss, 549-s@. Box, G. and G. Jenkins, 1970, Time series analysis: Forecasting and control (Holden-Day, New York). Chatfield, C., 1975, The analysis of time series: Theory and practice (Chapman and Hall, London). Cooper, R., 1972, The predictive performance of quarterly econometric models of the United States, in: B. Hickman, ed., Econometric models of cyclical behaviour, Vol. 2 (Columbia University Press, New York). Dhrymes, P., 1974, Econometrics, statistical foundations and applications (Springer-Verlag, New York). Dhrymes, P. and S. Peristiani, 1983, The predictive performance of WEFA versus Box-Jenkins ARMA mod&, &cm.sion paper no. 514 (Department of Economics, Columbia University, New York). Fildes, R., 1985, Quantitative forecasting the state of art: Econometric models, Journal of the Operations Research Society 36, 549-580. Harmon, P. and D. King, 1985, Expert systems: Artificial intelligence in business (Wiley, New York). Hirsch, A., T. Grimm and G. Narasimham, 1974, Some multiplier and error characteristics of the BEA quarterly model, International Economic Retriew 16, 616-631. Klein, L. 1971, An essay on tkie theory of economic prediction (Markham, Chicago, IL). Levenbach, H., J. Clearly and D. Fryck, 1974. A comparison of ARIMA and econometric models for telephone demand, Ameri-Statistical Association, Proceedings of the Business and Economic Statistics Section, 448-450. Longbottom, J. and S. Holly, 1985, The role of time series analysis in the evaluation of econometric models, Journal of Forecasting 4,75-87. McCarthy, M., 1981, The Wharton quartely econometric forecasting model III (Economics Research Unit, University of Pennsylvania, Philadelphia, PA). McNees, S., 1981, The recent record of thirteen forecasters, New England Economic Review, 5-17. McNees, S., 1982, The role of macroeconometric models in forecasting and policy analysis in the United States, Journal of Forecasting 1, 37-48. McNees, S., 1986, Forecasting accuracy of alternative techniques: A comparison of U.S. macroeconomic forecasts, Journal of Business and Economic Statistics 4, 5-15. Makridakis, S. and S. Wheelwright, 1978, Forecasting methods and application (Wiley, New York). Markridakis, S., A. Andersen, R. Carbone, R. Fiides, M. Hibon, R. Lewandowski, J. Newton, E. Parzen, and R. Winkler, 1985, The forecasting accuracy of major time series methods (Wiley, Chichester). Narasimham, G., G. Castellino and N. Singpurwalla, 1974, On the predictive performance of the BEA quarterly model and Box-Jenkins ARIMA models, American Statistical Association, Proceedings of the Business and Economic Statistics Section, 501-504. Naylor, T., T. Seaks and D. Wichem, 1972, Box-Jenkins methods: An alternative to econometric models, International Statistical Review 40, 123-137. Nelson, C., 1976, The prediction performance of the FRB-MIT-PENN model of the U.S. economy, American Economic Review 62,902-917. Pindyck, R. and D. Rubinfeld, Econometric models and economic forecasts (McGraw-Hill, New York). Sims, C., 1980, Macroeconomics and reality, Econometrica 48, l-48. Wallis, K., 1977, Multiple time series analysis and final form of econometric models, Econometrica 45, 1481-1497. ZeBner, A., 1979,Statistical analysis of econometric models, Journal of the American Statistical Association 74. 631-651. Zellner, A. and F. Palm, 1974, Time series analysis and simultaneous equation econometric models, Journal of Econometrics 14,17-54.

Biogruphy: P.J. DHRYMES is a Professor in the Department of Economics at Columbia University. His research interests include theoretical and applied econometrics. He is the author of Distributed Lags: Problems of Estimation and Formulation (Holden-Day, 1971), Econometrics: Statistical Foundatrons and Applications (Springer-Verlag, 1974) and Introducto~ Econometrics (Springer-Verlag, 1978). S. PERISTIANI is an Assistant Professor in the Department cf Economics at Queens College-CUNY. His research interest include theoretical and applied econometrics.