- Email: [email protected]
On the information provided by forecasting models
TECHNOLOGICAL
FORECASTING
AND SOCIAL CHANGE
16, 351-361
(1980)
On the Information Provided by Forecasting Models PETER G;iRDENFORS
ABSTRACT The aim of the paper is to evaluate the information provided by forecasting models that include explanatory variables besides the variables to be forecasted. It is argued that the content of a forecast is a combination of historical information about the variable to be forecasted and theoretical considerations, normally manifested by a model. The historical information is assessed by a time series model for the variable. In order to assess the theoretical information about a variable, one suggests a measure. This measure is based on the improvement of fit to the actual values of the values obtained from the forecasting model in comparison to the values obtained from the time series model. The R2 measure, which frequently is used as a measure of the explanatory power of a forecasting model, is critically discussed.
Problem and Program What do we learn from forecasting models? When forecasting a quantity as a time dependent variable we combine different sources of information about the variable. First, we normally have available data for past values of the variable. Second, we have some theoretical knowledge about the variable, which is often expressed by a model in which the variable is included. These kinds of information are then combined by using past values of the variable to estimate the parameters of the model equations and to determine the initial values for these equations. The model, with the estimated parameters and the given initial values, is then used to produce a forecast of the variable of interest. This outline of a standard procedure makes it natural to ask the following question: What is the information provided by a forecast? Since there are several ways of formulating a model including a variable to be forecasted and since these models must be compared with respect to their predictive power, a consequent question is: How is the information provided by a forecast to be measured? Before these questions can be attacked, it is necessary to discuss the concept of information in this setting. When judging the information transmitted by a forecasting model in this paper, I will adopt a Bayesian approach. The general idea is that the explanatory value of a forecasting model, with respect to a given variable, is the amount of information that is provided about the behavior of the variable that was not available before the model was given. Here the a priori information about a variable will be determined by a “naive” time series model for the variable.
PETER GARDENFORS is an Assistant Professor at the Department of Philosophy, University of Lund, Sweden. He is working on a research project on the methodology of forecasting sponsored by the Planning Division of the Research Institute of Swedish National Defense (FOA P). @ Elsevier North Holland,
Inc., 1980
0040-1625/80/04035111$02.25
352
PETER GARDENFORS
It will be argued that the additional information comes mainly from the theory used to build the forecasting model. The explanatory power of a forecasting model for a variable, which includes other variables as well, will then be judged in relation to the nai’ve model for the variable. When evaluating equations in econometric forecasting models one of the most widely used measures is R2. The traditional rationale for using R2 is that it measures the proportion of the variation in a variable that is “explained” by the equation in the model. The R2 measure will be presented in the discussion and its alleged role as a measure of the information provided by forecasting models will be criticized. The measure I of the information provided by a forecasting model suggested in this paper evades this criticism. After introducing the measure I and discussing some of its properties, the effects of the measure will be illustrated by an example. Theories of Information and Information from Theories A starting point for the mathematical theory of information is that the information content i(x) of a message that a certain event has occurred is a decreasing function of the probability x of the event. When choosing the appropriate decreasing function, one is led by the idea that if two stochastically independent events have probabilities x1 and x2, respectively, then the information content of the message that both events have occurred should equal the sum of the information contents of the events taken separately. This amounts to the requirement that i(x,x*)
= i(x,) + i(xz).
(1)
If it is assumed that i(x) is a continuous decreasing function of the probability x only, then it can be shown that the only functions satisfying the stipulated requirement are of the form i(x) = -log
x,
(2)
leaving only a choice of the logarithmic base. This general approach to measuring information forms the basis for two different research traditions-on the one hand, the mathematical and statistical communication theory, and on the other, a methodological program in line with the Bayesian approach to probability theory. The first research tradition will, however, not be of interest in what follows. A basic motive for the other tradition is that the goal of scientific theory construction is to reduce our uncertainty about the world around us. A natural thought then is that the uncertainty eliminated by a theory equals the information provided by the theory. The amount of uncertainty we are free from when accepting a theory as true can also be said to be the explanatory value of the theory.’ I turn next to the problem of what kind of information we get from theories. I will confine myself to theories about variables that can be measured in a quantitative way. Theoretical considerations about a set of variables of interest often result in a model that consists of a system of equations. In forecasting models these equations normally contain lagged variables; that is, the equations form a dynamic system?
‘A general presentation of the role of the information concept within the philosophy of science is given by Hintikka 161. *Dynamic systems and their use in forecasting and planning are presented in GErdenfors [3].
INFORMATION
PROVIDED
BY FORECASTING
MODELS
353
The information provided by such a model is what the system of equations says about the world. But how can this information be measured numerically? The theory on which the system is based does in general leave a number of parameters in the equations undetermined. Furthermore, if the equations are seen as a dynamic system, one must also specify a set of initial values for the variables before the system can be used to simulate a past period or forecast a future period. The theory can therefore not by itself produce any information about the future, but only in combination with some given data for the variables. The data for the variables to be forecasted can be seen as the a priori information of these variables. These data form a time series and can be used to establish a “naive” model for the forecasted variables, without any considerations of the underlying dynamics. A forecast, in this setting, is the outcome of a system of equations in combination with the available information about the variables. The new information provided by the forecast is therefore the information transmitted by the underlying theory into the system equations. This kind of information is difficult to measure numerically and the only way to get some quantitative data is to study the outcome of the theoretical system, that is, a simulation of historical data or a forecast? A forecasting model is seldom seen in a completely deterministic way but a random component is allowed in the equations, either as representing errors of measurement or as reflecting the stochastic behavior of some of the variables in the equation. Therefore, even if the actual values of the variable of interest do not completely coincide with the forecasted or simulated values, it may be possible to assess the probability that the actual series of values will occur, given the assumption that the model is correct. The information from the forecasting model is then naturally related to the improvement in fit of the values obtained from the model to the actual values of the variable in comparison to the fit obtained from a nai’ve model. A suggestion for a precise measure of improvement of fit will be given later on. However, it will be better first to discuss the R2 measure, which is frequently used to evaluate the explanatory power of a forecasting model. In fact, my discussion in this paper can be seen as much against R2 as for the proposed new measure of explanatory power. Against R2 Iwill now present the construction of the R2 measure and after that turn to a critical analysis of the properties of the measure. This criticism serves as a motivation for the new measure to be presented in the next section. Suppose we are interested in forecasting a variable x(t) as a function of the time t and that this is done by a forecasting model that produces simulated or forecasted values x, (I) of the variable for a certain period of time. This series is to be compared with the series x,, (t) of actual values of the variable during the same period of time. The differences X, (t) - x,~ (t) are called the residuals of the forecasted series. The residuals are not unit free, but if they are in some way divided by the total variance of the variable, one obtains a unit-free measure. In order to give a precise definition of R2, one first introduces the total sum of squares (or total variation) of the actual values x, (t) about their meanx, for a time period t,, 62,. . .> tk as follows:
3A general discussion of the evaluation of simulation models in terms of the amount of information provide about the behavioral processes being simulated is given by Hanna [51.
they
354
PETER GARDENFORS
The regression sum of squares (or explained variation) is then defined as RSS = i [x3&) - <12. ,= I Finally, the error sum of squares (or residual variation) is introduced
(4) as follows:
ESS = ; [x, (ti) - x, (ti)12. (5) ,= I Now if the equation for x, (t) is linear, then it can be shown that if the mean value of the residuals is zero and if these residuals are perfectly uncorrelated with the explaining variables in the linear equation, then the following identity holds:4 TSS = RSS + ESS.
(6)
The R2 value for the equation for the variable x(t) during the time interval t, , . . , tk is then defined as 1 - ESS/TSS. If the assumptions for the identity (6) are fulfilled, R2 can also be defined as RSSITSS. Under these assumptions the R2 ranges in values between 0 and 1 and is higher the better the x, (t) values correspond with the x, (t) values. The standard interpretation of this measure, given in most textbooks of econometrics, is that R2 is the proportion of the total variation of the variable “explained” by the given forecasting model? It is said that the total variation (TSS) of the variable of interest is partitioned into two parts. The first (RSS) is the variation of the simulated or forecasted values X?(ti) about their mean. This is referred to as the sum of squares explained by the influence of the variables in the model equation for ,r, (t). The second component (ESS) is the unexplained, or residual, variation of the variable. A first critical point is that a high R2 value cannot be the only criterion to determine since if further explanatory varwhether an equation describes some data “correctly,” iables are added to the system equation, the R2 is never lowered, but is likely to be raised, as long as linear regression is used. The reason for this is that the addition of a new explanatory variable does not alter TSS but is likely to increase RSS and thereby decrease ESS. Another drawback of R2, which is more important from a methodological point of view, is that the value of the measure is dependent on the way the equations in a model are specified, in the sense that two models that are theoretically equivalent and which have identical forecasting properties may give different results when the R2 is computed. In order to illustrate this I will give an example that is adapted from Pindyck and Rubinfeld [9, pp. 65-661. Suppose we are interested in giving a model for saving, for which we use the variable S(t). The only exogenous (explanatory) variable is supposed to be disposable income, which is denoted Y(r). A third variable of interest is aggregate consumption, denoted by C(t). By definition, we have the following relation among these three variables: Y(t) = S(t) + C(t).
(7)
We next formulate two different models. The first consists of a single equation: s(t) = a, + &Y(t)
+ e,(t).
‘For a derivation of this identity, see, e.g., Johnston [7, p, 341. ‘See, e.g., Johnston [7, p. 341, and Pindyck and Rubinfeld 19, pp. 33-373.
(8)
INFORMATION
PROVIDED
BY FORECASTING
MODELS
355
Here a, and b, are equation parameters that are to be estimated and e, (t) is an error term. The second system consists of two equations: C(r) = up + b2Y(r) + e,(t), s(t) = Y(t) -
(9)
C(r).
(10)
In the first model S(r) is explained directly with the aid of Y(t); in the second it is explained via C(I) and the given identity. The parameters ul, b, , u2, and b, can now be estimated from a set of data. If an ordinary least squares regression is made, using U.S. quarterly data from 1954-1 to 1971-4 (in billions of dollars), one obtains the following equations: s(t)
= 14.51 + O.l2Y(t)
C(t) = -14.51
+ 0.88Y(t)
+ e,(t), + ez(t).
(8a) (9a)
The error terms in these regressions are identical, but with opposite signs, that is, e, (t) = - e8 (t) for all t. The standard error of the regression is 3.72 and the ESS is 966.5 for both equations. This shows that the models are identical for forecasting purposes. However, for eq. (8a) the R2 value is 0.896, while for eq. (9a) it is 0.998. The explanation of this difference is that the TSS values are different and since the ESS values are the same, the R* values will be different as well. The point of this example is that two model equations can give identical practical results and yet differ substantially in R2. I now come to the most serious drawback of the R2 as a measure of the explanatory value of a forecasting model equation. If the time series of data on which the estimation of the equation is based is “smooth” or regular, then it is easier to construct a system of equations that will give a close fit to these data when the model is simulated over the data period than when the given data are “irregular” and seemingly uncorrelated. This means that for a variable, the historical data of which are regular in some sense, one obtains, in general, even for a very unsophisticated model, an equation that gives a high R2 ; and when historical data are irregular it is necessary to have a very good model to obtain a high RZ. Therefore a high R2 may as well be a sign of regular data as of an informative equation. The reason for this unwanted feature is that R2 is “based” on TSS which does not in any way take into account the time ordering of the data. In this sense the use of R2 means that one completely disregards some important information about the variable to be forecasted, namely, that the historical data form a time series. And since there are ways of expressing this information, this temporal structure should be included in the background knowledge if the explanatory value of an equation is to be appropriately evaluated. A Measure of the Explanatory Value of an Equation From a Bayesian point of view it may be argued that the explanatory value of an equation is the amount of information that is provided about the behavior of the variable that was not available before the equation was given. This recipe for determining the explanatory value is afflicted with two fundamental difficulties: 1) Which is the information available before the equation is constructed? and 2) How is the additional information to be measured? As regards the first difficulty, there is no clear-cut answer, but it is nevertheless possible to make some demarcation. What is known about the variable to be forecasted
356
PETER GARDENFORS
before it is related to other “explanatory” variables in a model can often be summed up by the curve that the variable describes as a function of time. If this curve shows some form of regularity, for example, a linear trend or a cyclical movement, then the output of an appropriate forecasting model equation should catch this regularity. In order to be more informative, the equation should also produce a series of simulated data that fits the actual curve better than what can be expected from the rough regularity. If on the other hand, the curve of actual values of the variable to be forecasted seems to be irregular, then an equation can have a large explanatory value with respect to the behavior of the variable even if the fit between the actual values and those produced by the equation is comparatively poor. I therefore suggest that instead of using TSS as a measure of the background information, which is what the R2 amounts to under this approach, it is better to formulate some “nai’ve” model of the behavior of the variable. By the designation “naYve” I mean that only information about the variable to be forecasted is to be used, and no information about the behavior of other variables is allowed in the model. A very simple example of a nai’ve model is the rule that says that the forecasted value for the next point of time is the same as the previous value of the variable. Another similar model is that the forecasted change in value for the next point of time is the same as the change in values between the last two points of time. Now if the forecasting model in which one is interested does not give better forecasts than these two naive models, then this is a strong reason to reject the forecasting model. The two methods mentioned here have been suggested by Milton Friedman as a kind of threshold test for forecasting models! In a naive model, in general, the values of the variable to be forecasted is seen as a time series. In order to catch as much information as possible from the a priori available data series for the variable, one should therefore use the best tools of time series analysis when constructing the naive model. Apart from various methods of trend extrapolation, exponential smoothing, and so on, one should also use the techniques systematized by Box and Jenkins.7 They consider two basic forms of models-autoregressive and moving average. Autoregressive and moving average models can be combined into ARMA models. The set of ARMA models comprise the basic arsenal for the Box-Jenkins approach. The most important merit of this approach is that it has developed systematic methods for choosing among these models based on an analysis of available data. An assumption for the application of ARMA models is that the time series to be analyzed is stationary, which means that the joint probability density function for the values of the process is the same for all sets of time points. In reality, however, one often encounters series that are not stationary. They cannot be treated directly in the ARMA framework, but by transforming the series in some way one may obtain a stationary series. The choice of transformation method is not entirely mechanical but may be “theory laden” in a weak sense. For example, when choosing between considering a given time series as produced by a linearly or exponentially growing process, theoretical considerations about the process may be taken into account. The methodology of analyzing nonstationary time series is discussed by Gardenfors and Hansson [4].
6These two naive methods have been used by Liu [81 when evaluating a forecasting model for the U.S. economy. ‘The main reference is Box and Jenkins [2]. A good presentation of their methods can also be found in Anderson [I].
INFORMATION
PROVIDED
BY FORECASTING
MODELS
351
Even if the choice of transformation method and the identification of ARMA models are not entirely mechanical processes, it does not seem as strong an idealization to assume that for every time series the methods mentioned here will be able to select a naive model that catches as much information about the time series as is possible without using other explanatory variables. This model can then be used to assess the a priori information about the variable to be forecasted. Suppose a series of actual datax,, (tj), i = 1, 2, . . ., k, is given for the variable to be forecasted. If the na’ive model, when simulated for the same time period, produces a series x,, (ri) of values, we then let NSS denote the sum of the squares of the differences between the actual values of the variable and the values obtained from the na’ive model; that is,
NSS = 1
[xc, (6) -
x,,(4 I] ‘.
(11)
I suggest that NSS is inversely correlated to the information provided by the naive model. The information provided by a forecasting model containing explanatory variables, besides the variable to be forecasted, should be evaluated in comparison to the information provided by a naive model for the variable in question. Forecasting models are best seen as stochastic models, assigning different probabilities to different outcomes. We are interested in the probability of the series of actual values of the forecasted variable, given that the suggested forecasting model is correct. And, of course, the closer the actual series is to the series produced by the model when all stochastic errors are put to zero, the more probable is the actual series as an outcome of the model. By invoking Bayesian ideas, we can argue that the closer the gap between actual and simulated values, the more information the forecasting model carries about the variable to be forecasted in comparison to the naive model. The distance between the actual values and the simulated values obtained from the forecasting model can be measured by the ESS (error sum of squares), which was introduced earlier as a measure of the residual variation. In accordance with the general form of information measures, as already presented, I suggest that the new information I provided by a forecasting model for a variable x(t() during a time period t, , tz , , tk is measured as I = log NSS -
log ESS.
(12)
The value of I is positive only if ESS is smaller than NSS, zero only if ESS equals NSS, and negative only if ESS is greater than NSS. If the average error is greater when a forecasting model including explanatory variables is used to produce the values of a variable than when a naive model is used, then it cannot be said that the forecasting model helps to explain the behavior of the variable but, rather, the forecasting model is misinformative. This is true even if it happens to be the case that the R2 for the equation of the variable in the forecasting model is close to 1. It is only when the value of I is positive that one may expect that the forecasts obtained from the forecasting model, including explanatory variables, will be more accurate than the forecasts obtained directly from the naive model. This argument can be formulated as a methodological rule of thumb: A forecasting model for a given variable that does not obtain a positive I value in relation to a natural naive model (i.e., time series model) should not be used to make forecasts. A presupposition for the use of this rule of thumb is, of course, that the studied time interval is long enough to outweigh the short-term advantages of a sophisticated time series model. How long the time interval must be determined from case to case, mainly depending on what one knows about the dynamic behavior of the variable to be forecasted.
358
PETER GARDENFORS
Another criterion that has models is the root mean square and actual values, respectively, then this measure can be defined
RMS =
d
been used to evaluate the performance of forecasting error. If x,~(ti) and X, (ti) as before denote the simulated of the variable x(t) during the time interval t, , . . , t,; , as
i [x,, (t, ) - .xs(ti ,] * . (13) ,= I The RMS error is thus another measure of the deviation of the simulated or forecasted variable from its actual time path. It may now be suggested that the RMS instead of ESS be used as a basis for a measurement of the improvement of the fit between the forecasted or simulated values and the actual values. If we let RMS,, denote the deviation of the values of the variable in the naive model from its actual values and RMS the deviation of the values produced by the forecasting model including explanatory variables, then, in analogy with the preceding reasoning, the following is another way of measuring the explanatory value of the forecasting model: l/k
I’ = log RMS,, -
log RMS,
As the following to the I measure:
I’ = log -
Ilk i-i
series of identities
[x,,(ti)
(14) show, this measure is, however, closely related
- x,, (ti)]*
I
=
i [XC, (ti) - ~9 (ti)] * ,=I %(log NSS - log k) - %(log ESS - log k)
=
%I.
logdllk
(15)
I’ is thus identical to I except for the choice of logarithmic base. This comparison with a measure based on RMS shows some of the “robustness” of the logarithmic form of the information measure. An Example I will illustrate the concepts discussed in this paper by a forecast based on a simple econometric model. The model and the simulations and forecasts made by it are borrowed from Pindyck and Rubinfeld [9, pp. 320-3281. This model, although very simple, is typical for econometric forecasting. The model is a multiequation linear macroeconomic system. The endogenous variables of the model are consumption C(t), gross domestic investment Z(t), a short-term interest rate R(r), and GNP Y(t). The exogenous variables are government spending G(t), and money stock M(t). All variables are measured in billions of 1958 dollars except for R(t), which is in percent per year. The data for the variables are U. S. quarterly values. The model consists of four equations, one for each of the endogenous variables. Here I will concentrate on the consumption variable C(t). The equation for this variable is C(t) = 2.537 + O.l85Y(t)
+ 0.714C(t
-
1).
(lo)
The parameters in this equation were obtained by ordinary least squares estimation using quarterly data from 1955- 1 to 197 l-4. The t statistics for the equation parameters are good: 1.90 for the constant term, 5.95 for the parameter Y(t), and 13.8 for the parameter C(t). What should be noted is that the R* for the equation is 0.999.
INFORMATION
PROVIDED
BY FORECASTING
359
MODELS
However, the reason for the high R2 value is mainly that the time series for the consumption variable is smooth with only small disturbances. Thus a naive model of this time series will also show a good fit to the actual curve, and the explanatory value of the econometric equation may not be great despite the high R2 value. A “visual inspection” of the consumption curve from 1955-1 to 1971-4 indicates a slowly but exponentially growing series. In order to estimate the rate of growth, first the logarithm of the time series was taken and then the regression slope and the mean value of the resulting time series were computed. The result indicated a 1.O% growth in the consumption each quarter. This estimation gives a very good fit to actual data.8 In this way we have obtained a simple time series model for the consumption variable, which can be taken as the nai’ve model when evaluating the information provided by the econometric model. Pindyck and Rubinfeld [9] used their model for making an ex posf forecast. The model was simulated forward starting at the end of the estimation period (1971-4) and continuing as long as historical data for the exogenous and endogenous variables were available ( 1973-2). The actual values of the consumption variable and the values obtained by this forecast are shown in the first two columns of Table 1. In the third column the forecast error is computed. TABLE 1 Yearquarter
Actual values
Forecast
Error
Nai’ve forecast
Error
1971-4 1972-I 1972-2 1972-3 1972-4 1973-l 1973-2
504.1 512.5 523.4 531.0 540.5 552.7 553.3
500.8 505.3 509.3 511.8 514.2 516.7 519.4
-3.3 -7.2 -14.1 - 19.2 -26.3 -36.0 -33.9
507.0 512.1 517.3 522.5 527.7 533.0 538.4
2.9 -0.4 -6.1 -8.5 - 12.8 - 19.7 -14.9
RMS
23.2
11.3
In comparison to the forecast made by the econometric model, the forecast made by the time series model is presented in the fourth column. For the time period considered, the nai’ve forecast comes considerably closer to the actual values than the econometric forecasting model does. The difference in fit is reflected in the RMS values for the two the information provided by the forecasts. According to the previous discussion, econometric forecasting model for the present time period should be measured by I = log NSS -
log ESS = 2(log RMS,, -
log RMS,).
(17)
If natural logarithms are used, the value of I is -1.44. No matter which base is chosen, the value will be negative and the econometric forecasting model is thus misinformative with respect to the consumption variable when judged against the a priori information about the variable, here expressed by an exponential growth model. This discouraging outcome for the econometric forecasting model needs some mitigating comments. First, the model is very simple, containing only a few explanatory
8Since the regression errors are strongly correlated, one can in fact construct better fit by including a first-order autoregressive process.
a naive model with an even
360
PETER GARDENFORS
variables. Second, the forecasted time period is very short. Time series models typically perform best in short-term forecasts while the “pattern” of the variable to be forecasted is not broken. Models based on explanatory variables show to advantage only when some time has elapsed so that (“unexpected”) changes in the values of the explanatory variables can have an impact on the forecast. Thus even if the equation for the consumption variable in the econometric model gives comparatively poor results for the short forecasting period investigated here, it may outperform the nai’ve model in the long run. Third, the model may yield informative forecasts for the other endogenous variables, even for the present short forecasting period. The time series of actual values for investment I(t) and interest rate R(t) are not so smooth and regular as the series for consumption is, and as a result, a naive model for these two variables will not show such a good fit as for the consumption variables. For these variables the econometric forecasting model is likely to provide some information, in the sense discussed here, even for a short forecasting period. Nevertheless, this example shows that a forecast based on an equation containing explanatory variables may say very little, or even be misleading, even if the R2 value of the equation for the estimation period is close to 1. The conclusion is that this measure should be used with a great deal of caution or, rather, be avoided when judging the explanatory power of a forecasting model.
Conclusion The main task of this paper has been to evaluate the information provided by forecasting models. I have concentrated on forecasting models containing explanatory variables besides the variables to be forecasted. It has been argued that the content of a forecast can be seen as a combination of a priori information about the variable to be forecasted, in the form of historical data, and theoretical considerations, which normally are manifested in a model including the variable of interest. The a priori information about the variable can be assessed by using a time series model for the variable. The information about a variable transmitted by a theory into a model equation is more difficult to evaluate. In order to obtain a numerical measure, one must investigate the performance of the model. Here a measure I is suggested which is based on the improvement of fit between the forecasted values of the variable obtained from the forecasting model and the actual values in comparison to the fit of the forecasted values obtained from the naive model. The R2 measure, which frequently has been used as a measure of the explanatory power of a forecasting equation has here been critically discussed, the main criticism being that it disregards some important information about the variable, namely, that the values of the variable form a time series. The I measure does not have this drawback and when taking into account other properties of this measure which have been discussed in this paper, I conclude that the I measure is useful as a measure of the information provided by a forecasting model.
The research for this paper has been supported by a grant from a research project on future studies sponsored by the Planning Division of the Research Institute of Swedish National Defence (FOA P).
INFORMATION
PROVIDED
BY FORECASTING
MODELS
361
References 1. Anderson, 0. D., Time Series Analysis and Forecasting: The Box-Jenkins Approach, Butterworth, London, 1976. 2. Box, G. E. P., and Jenkins, G. M., Times Series Analyses. Forecasting and Control, Holden-Day, San Francisco 1970. 3. G’ardenfors, P., Dynamic Systems as Tools for Forecasting and Planning: A Presentation and Some Methodological Aspects, Theory Decision (to appear). 4. Gardenfors, P., and Hansson, B.: Forecasting Non-Stationary Time Series-Some Methodological Aspects, mimeographed, University of Lund, Sweden, 1979. 5. Hanna, J. E, Information-Theoretic Techniques for Evaluating Simulation Models, in Computer Simulation ofHuman Eehaviour, Dutton and Starbuck, eds., Wiley, New York, 1971. 6. Hintikka, J., The Varieties of Information and Scientific Explanation, in Logic, Methodology and Philosophy of Science III, van Rootselaar and Staal, eds., North-Holland, Amsterdam, 1968. 7. Johnston, J., Econometric Methods, McGraw-Hill, New York, 1963. 8. Liu, T-C., A Simple Forecasting Model for the U.S. Economy, in International Monetary Fund Staff Pupers, 1954. 9. Pindyck, R. S., and Rubinfeld, D. L., Econometric Models and Economic Forecasts, McGraw-Hill, New York, 1976. 10. Theil, H., Economics and Information Theory, North-Holland, Amsterdam, 1967. Received 7 November
1979; revised 29 Januq
1980
FORECASTING
AND SOCIAL CHANGE
16, 351-361
(1980)
On the Information Provided by Forecasting Models PETER G;iRDENFORS
ABSTRACT The aim of the paper is to evaluate the information provided by forecasting models that include explanatory variables besides the variables to be forecasted. It is argued that the content of a forecast is a combination of historical information about the variable to be forecasted and theoretical considerations, normally manifested by a model. The historical information is assessed by a time series model for the variable. In order to assess the theoretical information about a variable, one suggests a measure. This measure is based on the improvement of fit to the actual values of the values obtained from the forecasting model in comparison to the values obtained from the time series model. The R2 measure, which frequently is used as a measure of the explanatory power of a forecasting model, is critically discussed.
Problem and Program What do we learn from forecasting models? When forecasting a quantity as a time dependent variable we combine different sources of information about the variable. First, we normally have available data for past values of the variable. Second, we have some theoretical knowledge about the variable, which is often expressed by a model in which the variable is included. These kinds of information are then combined by using past values of the variable to estimate the parameters of the model equations and to determine the initial values for these equations. The model, with the estimated parameters and the given initial values, is then used to produce a forecast of the variable of interest. This outline of a standard procedure makes it natural to ask the following question: What is the information provided by a forecast? Since there are several ways of formulating a model including a variable to be forecasted and since these models must be compared with respect to their predictive power, a consequent question is: How is the information provided by a forecast to be measured? Before these questions can be attacked, it is necessary to discuss the concept of information in this setting. When judging the information transmitted by a forecasting model in this paper, I will adopt a Bayesian approach. The general idea is that the explanatory value of a forecasting model, with respect to a given variable, is the amount of information that is provided about the behavior of the variable that was not available before the model was given. Here the a priori information about a variable will be determined by a “naive” time series model for the variable.
PETER GARDENFORS is an Assistant Professor at the Department of Philosophy, University of Lund, Sweden. He is working on a research project on the methodology of forecasting sponsored by the Planning Division of the Research Institute of Swedish National Defense (FOA P). @ Elsevier North Holland,
Inc., 1980
0040-1625/80/04035111$02.25
352
PETER GARDENFORS
It will be argued that the additional information comes mainly from the theory used to build the forecasting model. The explanatory power of a forecasting model for a variable, which includes other variables as well, will then be judged in relation to the nai’ve model for the variable. When evaluating equations in econometric forecasting models one of the most widely used measures is R2. The traditional rationale for using R2 is that it measures the proportion of the variation in a variable that is “explained” by the equation in the model. The R2 measure will be presented in the discussion and its alleged role as a measure of the information provided by forecasting models will be criticized. The measure I of the information provided by a forecasting model suggested in this paper evades this criticism. After introducing the measure I and discussing some of its properties, the effects of the measure will be illustrated by an example. Theories of Information and Information from Theories A starting point for the mathematical theory of information is that the information content i(x) of a message that a certain event has occurred is a decreasing function of the probability x of the event. When choosing the appropriate decreasing function, one is led by the idea that if two stochastically independent events have probabilities x1 and x2, respectively, then the information content of the message that both events have occurred should equal the sum of the information contents of the events taken separately. This amounts to the requirement that i(x,x*)
= i(x,) + i(xz).
(1)
If it is assumed that i(x) is a continuous decreasing function of the probability x only, then it can be shown that the only functions satisfying the stipulated requirement are of the form i(x) = -log
x,
(2)
leaving only a choice of the logarithmic base. This general approach to measuring information forms the basis for two different research traditions-on the one hand, the mathematical and statistical communication theory, and on the other, a methodological program in line with the Bayesian approach to probability theory. The first research tradition will, however, not be of interest in what follows. A basic motive for the other tradition is that the goal of scientific theory construction is to reduce our uncertainty about the world around us. A natural thought then is that the uncertainty eliminated by a theory equals the information provided by the theory. The amount of uncertainty we are free from when accepting a theory as true can also be said to be the explanatory value of the theory.’ I turn next to the problem of what kind of information we get from theories. I will confine myself to theories about variables that can be measured in a quantitative way. Theoretical considerations about a set of variables of interest often result in a model that consists of a system of equations. In forecasting models these equations normally contain lagged variables; that is, the equations form a dynamic system?
‘A general presentation of the role of the information concept within the philosophy of science is given by Hintikka 161. *Dynamic systems and their use in forecasting and planning are presented in GErdenfors [3].
INFORMATION
PROVIDED
BY FORECASTING
MODELS
353
The information provided by such a model is what the system of equations says about the world. But how can this information be measured numerically? The theory on which the system is based does in general leave a number of parameters in the equations undetermined. Furthermore, if the equations are seen as a dynamic system, one must also specify a set of initial values for the variables before the system can be used to simulate a past period or forecast a future period. The theory can therefore not by itself produce any information about the future, but only in combination with some given data for the variables. The data for the variables to be forecasted can be seen as the a priori information of these variables. These data form a time series and can be used to establish a “naive” model for the forecasted variables, without any considerations of the underlying dynamics. A forecast, in this setting, is the outcome of a system of equations in combination with the available information about the variables. The new information provided by the forecast is therefore the information transmitted by the underlying theory into the system equations. This kind of information is difficult to measure numerically and the only way to get some quantitative data is to study the outcome of the theoretical system, that is, a simulation of historical data or a forecast? A forecasting model is seldom seen in a completely deterministic way but a random component is allowed in the equations, either as representing errors of measurement or as reflecting the stochastic behavior of some of the variables in the equation. Therefore, even if the actual values of the variable of interest do not completely coincide with the forecasted or simulated values, it may be possible to assess the probability that the actual series of values will occur, given the assumption that the model is correct. The information from the forecasting model is then naturally related to the improvement in fit of the values obtained from the model to the actual values of the variable in comparison to the fit obtained from a nai’ve model. A suggestion for a precise measure of improvement of fit will be given later on. However, it will be better first to discuss the R2 measure, which is frequently used to evaluate the explanatory power of a forecasting model. In fact, my discussion in this paper can be seen as much against R2 as for the proposed new measure of explanatory power. Against R2 Iwill now present the construction of the R2 measure and after that turn to a critical analysis of the properties of the measure. This criticism serves as a motivation for the new measure to be presented in the next section. Suppose we are interested in forecasting a variable x(t) as a function of the time t and that this is done by a forecasting model that produces simulated or forecasted values x, (I) of the variable for a certain period of time. This series is to be compared with the series x,, (t) of actual values of the variable during the same period of time. The differences X, (t) - x,~ (t) are called the residuals of the forecasted series. The residuals are not unit free, but if they are in some way divided by the total variance of the variable, one obtains a unit-free measure. In order to give a precise definition of R2, one first introduces the total sum of squares (or total variation) of the actual values x, (t) about their meanx, for a time period t,, 62,. . .> tk as follows:
3A general discussion of the evaluation of simulation models in terms of the amount of information provide about the behavioral processes being simulated is given by Hanna [51.
they
354
PETER GARDENFORS
The regression sum of squares (or explained variation) is then defined as RSS = i [x3&) - <12. ,= I Finally, the error sum of squares (or residual variation) is introduced
(4) as follows:
ESS = ; [x, (ti) - x, (ti)12. (5) ,= I Now if the equation for x, (t) is linear, then it can be shown that if the mean value of the residuals is zero and if these residuals are perfectly uncorrelated with the explaining variables in the linear equation, then the following identity holds:4 TSS = RSS + ESS.
(6)
The R2 value for the equation for the variable x(t) during the time interval t, , . . , tk is then defined as 1 - ESS/TSS. If the assumptions for the identity (6) are fulfilled, R2 can also be defined as RSSITSS. Under these assumptions the R2 ranges in values between 0 and 1 and is higher the better the x, (t) values correspond with the x, (t) values. The standard interpretation of this measure, given in most textbooks of econometrics, is that R2 is the proportion of the total variation of the variable “explained” by the given forecasting model? It is said that the total variation (TSS) of the variable of interest is partitioned into two parts. The first (RSS) is the variation of the simulated or forecasted values X?(ti) about their mean. This is referred to as the sum of squares explained by the influence of the variables in the model equation for ,r, (t). The second component (ESS) is the unexplained, or residual, variation of the variable. A first critical point is that a high R2 value cannot be the only criterion to determine since if further explanatory varwhether an equation describes some data “correctly,” iables are added to the system equation, the R2 is never lowered, but is likely to be raised, as long as linear regression is used. The reason for this is that the addition of a new explanatory variable does not alter TSS but is likely to increase RSS and thereby decrease ESS. Another drawback of R2, which is more important from a methodological point of view, is that the value of the measure is dependent on the way the equations in a model are specified, in the sense that two models that are theoretically equivalent and which have identical forecasting properties may give different results when the R2 is computed. In order to illustrate this I will give an example that is adapted from Pindyck and Rubinfeld [9, pp. 65-661. Suppose we are interested in giving a model for saving, for which we use the variable S(t). The only exogenous (explanatory) variable is supposed to be disposable income, which is denoted Y(r). A third variable of interest is aggregate consumption, denoted by C(t). By definition, we have the following relation among these three variables: Y(t) = S(t) + C(t).
(7)
We next formulate two different models. The first consists of a single equation: s(t) = a, + &Y(t)
+ e,(t).
‘For a derivation of this identity, see, e.g., Johnston [7, p, 341. ‘See, e.g., Johnston [7, p. 341, and Pindyck and Rubinfeld 19, pp. 33-373.
(8)
INFORMATION
PROVIDED
BY FORECASTING
MODELS
355
Here a, and b, are equation parameters that are to be estimated and e, (t) is an error term. The second system consists of two equations: C(r) = up + b2Y(r) + e,(t), s(t) = Y(t) -
(9)
C(r).
(10)
In the first model S(r) is explained directly with the aid of Y(t); in the second it is explained via C(I) and the given identity. The parameters ul, b, , u2, and b, can now be estimated from a set of data. If an ordinary least squares regression is made, using U.S. quarterly data from 1954-1 to 1971-4 (in billions of dollars), one obtains the following equations: s(t)
= 14.51 + O.l2Y(t)
C(t) = -14.51
+ 0.88Y(t)
+ e,(t), + ez(t).
(8a) (9a)
The error terms in these regressions are identical, but with opposite signs, that is, e, (t) = - e8 (t) for all t. The standard error of the regression is 3.72 and the ESS is 966.5 for both equations. This shows that the models are identical for forecasting purposes. However, for eq. (8a) the R2 value is 0.896, while for eq. (9a) it is 0.998. The explanation of this difference is that the TSS values are different and since the ESS values are the same, the R* values will be different as well. The point of this example is that two model equations can give identical practical results and yet differ substantially in R2. I now come to the most serious drawback of the R2 as a measure of the explanatory value of a forecasting model equation. If the time series of data on which the estimation of the equation is based is “smooth” or regular, then it is easier to construct a system of equations that will give a close fit to these data when the model is simulated over the data period than when the given data are “irregular” and seemingly uncorrelated. This means that for a variable, the historical data of which are regular in some sense, one obtains, in general, even for a very unsophisticated model, an equation that gives a high R2 ; and when historical data are irregular it is necessary to have a very good model to obtain a high RZ. Therefore a high R2 may as well be a sign of regular data as of an informative equation. The reason for this unwanted feature is that R2 is “based” on TSS which does not in any way take into account the time ordering of the data. In this sense the use of R2 means that one completely disregards some important information about the variable to be forecasted, namely, that the historical data form a time series. And since there are ways of expressing this information, this temporal structure should be included in the background knowledge if the explanatory value of an equation is to be appropriately evaluated. A Measure of the Explanatory Value of an Equation From a Bayesian point of view it may be argued that the explanatory value of an equation is the amount of information that is provided about the behavior of the variable that was not available before the equation was given. This recipe for determining the explanatory value is afflicted with two fundamental difficulties: 1) Which is the information available before the equation is constructed? and 2) How is the additional information to be measured? As regards the first difficulty, there is no clear-cut answer, but it is nevertheless possible to make some demarcation. What is known about the variable to be forecasted
356
PETER GARDENFORS
before it is related to other “explanatory” variables in a model can often be summed up by the curve that the variable describes as a function of time. If this curve shows some form of regularity, for example, a linear trend or a cyclical movement, then the output of an appropriate forecasting model equation should catch this regularity. In order to be more informative, the equation should also produce a series of simulated data that fits the actual curve better than what can be expected from the rough regularity. If on the other hand, the curve of actual values of the variable to be forecasted seems to be irregular, then an equation can have a large explanatory value with respect to the behavior of the variable even if the fit between the actual values and those produced by the equation is comparatively poor. I therefore suggest that instead of using TSS as a measure of the background information, which is what the R2 amounts to under this approach, it is better to formulate some “nai’ve” model of the behavior of the variable. By the designation “naYve” I mean that only information about the variable to be forecasted is to be used, and no information about the behavior of other variables is allowed in the model. A very simple example of a nai’ve model is the rule that says that the forecasted value for the next point of time is the same as the previous value of the variable. Another similar model is that the forecasted change in value for the next point of time is the same as the change in values between the last two points of time. Now if the forecasting model in which one is interested does not give better forecasts than these two naive models, then this is a strong reason to reject the forecasting model. The two methods mentioned here have been suggested by Milton Friedman as a kind of threshold test for forecasting models! In a naive model, in general, the values of the variable to be forecasted is seen as a time series. In order to catch as much information as possible from the a priori available data series for the variable, one should therefore use the best tools of time series analysis when constructing the naive model. Apart from various methods of trend extrapolation, exponential smoothing, and so on, one should also use the techniques systematized by Box and Jenkins.7 They consider two basic forms of models-autoregressive and moving average. Autoregressive and moving average models can be combined into ARMA models. The set of ARMA models comprise the basic arsenal for the Box-Jenkins approach. The most important merit of this approach is that it has developed systematic methods for choosing among these models based on an analysis of available data. An assumption for the application of ARMA models is that the time series to be analyzed is stationary, which means that the joint probability density function for the values of the process is the same for all sets of time points. In reality, however, one often encounters series that are not stationary. They cannot be treated directly in the ARMA framework, but by transforming the series in some way one may obtain a stationary series. The choice of transformation method is not entirely mechanical but may be “theory laden” in a weak sense. For example, when choosing between considering a given time series as produced by a linearly or exponentially growing process, theoretical considerations about the process may be taken into account. The methodology of analyzing nonstationary time series is discussed by Gardenfors and Hansson [4].
6These two naive methods have been used by Liu [81 when evaluating a forecasting model for the U.S. economy. ‘The main reference is Box and Jenkins [2]. A good presentation of their methods can also be found in Anderson [I].
INFORMATION
PROVIDED
BY FORECASTING
MODELS
351
Even if the choice of transformation method and the identification of ARMA models are not entirely mechanical processes, it does not seem as strong an idealization to assume that for every time series the methods mentioned here will be able to select a naive model that catches as much information about the time series as is possible without using other explanatory variables. This model can then be used to assess the a priori information about the variable to be forecasted. Suppose a series of actual datax,, (tj), i = 1, 2, . . ., k, is given for the variable to be forecasted. If the na’ive model, when simulated for the same time period, produces a series x,, (ri) of values, we then let NSS denote the sum of the squares of the differences between the actual values of the variable and the values obtained from the na’ive model; that is,
NSS = 1
[xc, (6) -
x,,(4 I] ‘.
(11)
I suggest that NSS is inversely correlated to the information provided by the naive model. The information provided by a forecasting model containing explanatory variables, besides the variable to be forecasted, should be evaluated in comparison to the information provided by a naive model for the variable in question. Forecasting models are best seen as stochastic models, assigning different probabilities to different outcomes. We are interested in the probability of the series of actual values of the forecasted variable, given that the suggested forecasting model is correct. And, of course, the closer the actual series is to the series produced by the model when all stochastic errors are put to zero, the more probable is the actual series as an outcome of the model. By invoking Bayesian ideas, we can argue that the closer the gap between actual and simulated values, the more information the forecasting model carries about the variable to be forecasted in comparison to the naive model. The distance between the actual values and the simulated values obtained from the forecasting model can be measured by the ESS (error sum of squares), which was introduced earlier as a measure of the residual variation. In accordance with the general form of information measures, as already presented, I suggest that the new information I provided by a forecasting model for a variable x(t() during a time period t, , tz , , tk is measured as I = log NSS -
log ESS.
(12)
The value of I is positive only if ESS is smaller than NSS, zero only if ESS equals NSS, and negative only if ESS is greater than NSS. If the average error is greater when a forecasting model including explanatory variables is used to produce the values of a variable than when a naive model is used, then it cannot be said that the forecasting model helps to explain the behavior of the variable but, rather, the forecasting model is misinformative. This is true even if it happens to be the case that the R2 for the equation of the variable in the forecasting model is close to 1. It is only when the value of I is positive that one may expect that the forecasts obtained from the forecasting model, including explanatory variables, will be more accurate than the forecasts obtained directly from the naive model. This argument can be formulated as a methodological rule of thumb: A forecasting model for a given variable that does not obtain a positive I value in relation to a natural naive model (i.e., time series model) should not be used to make forecasts. A presupposition for the use of this rule of thumb is, of course, that the studied time interval is long enough to outweigh the short-term advantages of a sophisticated time series model. How long the time interval must be determined from case to case, mainly depending on what one knows about the dynamic behavior of the variable to be forecasted.
358
PETER GARDENFORS
Another criterion that has models is the root mean square and actual values, respectively, then this measure can be defined
RMS =
d
been used to evaluate the performance of forecasting error. If x,~(ti) and X, (ti) as before denote the simulated of the variable x(t) during the time interval t, , . . , t,; , as
i [x,, (t, ) - .xs(ti ,] * . (13) ,= I The RMS error is thus another measure of the deviation of the simulated or forecasted variable from its actual time path. It may now be suggested that the RMS instead of ESS be used as a basis for a measurement of the improvement of the fit between the forecasted or simulated values and the actual values. If we let RMS,, denote the deviation of the values of the variable in the naive model from its actual values and RMS the deviation of the values produced by the forecasting model including explanatory variables, then, in analogy with the preceding reasoning, the following is another way of measuring the explanatory value of the forecasting model: l/k
I’ = log RMS,, -
log RMS,
As the following to the I measure:
I’ = log -
Ilk i-i
series of identities
[x,,(ti)
(14) show, this measure is, however, closely related
- x,, (ti)]*
I
=
i [XC, (ti) - ~9 (ti)] * ,=I %(log NSS - log k) - %(log ESS - log k)
=
%I.
logdllk
(15)
I’ is thus identical to I except for the choice of logarithmic base. This comparison with a measure based on RMS shows some of the “robustness” of the logarithmic form of the information measure. An Example I will illustrate the concepts discussed in this paper by a forecast based on a simple econometric model. The model and the simulations and forecasts made by it are borrowed from Pindyck and Rubinfeld [9, pp. 320-3281. This model, although very simple, is typical for econometric forecasting. The model is a multiequation linear macroeconomic system. The endogenous variables of the model are consumption C(t), gross domestic investment Z(t), a short-term interest rate R(r), and GNP Y(t). The exogenous variables are government spending G(t), and money stock M(t). All variables are measured in billions of 1958 dollars except for R(t), which is in percent per year. The data for the variables are U. S. quarterly values. The model consists of four equations, one for each of the endogenous variables. Here I will concentrate on the consumption variable C(t). The equation for this variable is C(t) = 2.537 + O.l85Y(t)
+ 0.714C(t
-
1).
(lo)
The parameters in this equation were obtained by ordinary least squares estimation using quarterly data from 1955- 1 to 197 l-4. The t statistics for the equation parameters are good: 1.90 for the constant term, 5.95 for the parameter Y(t), and 13.8 for the parameter C(t). What should be noted is that the R* for the equation is 0.999.
INFORMATION
PROVIDED
BY FORECASTING
359
MODELS
However, the reason for the high R2 value is mainly that the time series for the consumption variable is smooth with only small disturbances. Thus a naive model of this time series will also show a good fit to the actual curve, and the explanatory value of the econometric equation may not be great despite the high R2 value. A “visual inspection” of the consumption curve from 1955-1 to 1971-4 indicates a slowly but exponentially growing series. In order to estimate the rate of growth, first the logarithm of the time series was taken and then the regression slope and the mean value of the resulting time series were computed. The result indicated a 1.O% growth in the consumption each quarter. This estimation gives a very good fit to actual data.8 In this way we have obtained a simple time series model for the consumption variable, which can be taken as the nai’ve model when evaluating the information provided by the econometric model. Pindyck and Rubinfeld [9] used their model for making an ex posf forecast. The model was simulated forward starting at the end of the estimation period (1971-4) and continuing as long as historical data for the exogenous and endogenous variables were available ( 1973-2). The actual values of the consumption variable and the values obtained by this forecast are shown in the first two columns of Table 1. In the third column the forecast error is computed. TABLE 1 Yearquarter
Actual values
Forecast
Error
Nai’ve forecast
Error
1971-4 1972-I 1972-2 1972-3 1972-4 1973-l 1973-2
504.1 512.5 523.4 531.0 540.5 552.7 553.3
500.8 505.3 509.3 511.8 514.2 516.7 519.4
-3.3 -7.2 -14.1 - 19.2 -26.3 -36.0 -33.9
507.0 512.1 517.3 522.5 527.7 533.0 538.4
2.9 -0.4 -6.1 -8.5 - 12.8 - 19.7 -14.9
RMS
23.2
11.3
In comparison to the forecast made by the econometric model, the forecast made by the time series model is presented in the fourth column. For the time period considered, the nai’ve forecast comes considerably closer to the actual values than the econometric forecasting model does. The difference in fit is reflected in the RMS values for the two the information provided by the forecasts. According to the previous discussion, econometric forecasting model for the present time period should be measured by I = log NSS -
log ESS = 2(log RMS,, -
log RMS,).
(17)
If natural logarithms are used, the value of I is -1.44. No matter which base is chosen, the value will be negative and the econometric forecasting model is thus misinformative with respect to the consumption variable when judged against the a priori information about the variable, here expressed by an exponential growth model. This discouraging outcome for the econometric forecasting model needs some mitigating comments. First, the model is very simple, containing only a few explanatory
8Since the regression errors are strongly correlated, one can in fact construct better fit by including a first-order autoregressive process.
a naive model with an even
360
PETER GARDENFORS
variables. Second, the forecasted time period is very short. Time series models typically perform best in short-term forecasts while the “pattern” of the variable to be forecasted is not broken. Models based on explanatory variables show to advantage only when some time has elapsed so that (“unexpected”) changes in the values of the explanatory variables can have an impact on the forecast. Thus even if the equation for the consumption variable in the econometric model gives comparatively poor results for the short forecasting period investigated here, it may outperform the nai’ve model in the long run. Third, the model may yield informative forecasts for the other endogenous variables, even for the present short forecasting period. The time series of actual values for investment I(t) and interest rate R(t) are not so smooth and regular as the series for consumption is, and as a result, a naive model for these two variables will not show such a good fit as for the consumption variables. For these variables the econometric forecasting model is likely to provide some information, in the sense discussed here, even for a short forecasting period. Nevertheless, this example shows that a forecast based on an equation containing explanatory variables may say very little, or even be misleading, even if the R2 value of the equation for the estimation period is close to 1. The conclusion is that this measure should be used with a great deal of caution or, rather, be avoided when judging the explanatory power of a forecasting model.
Conclusion The main task of this paper has been to evaluate the information provided by forecasting models. I have concentrated on forecasting models containing explanatory variables besides the variables to be forecasted. It has been argued that the content of a forecast can be seen as a combination of a priori information about the variable to be forecasted, in the form of historical data, and theoretical considerations, which normally are manifested in a model including the variable of interest. The a priori information about the variable can be assessed by using a time series model for the variable. The information about a variable transmitted by a theory into a model equation is more difficult to evaluate. In order to obtain a numerical measure, one must investigate the performance of the model. Here a measure I is suggested which is based on the improvement of fit between the forecasted values of the variable obtained from the forecasting model and the actual values in comparison to the fit of the forecasted values obtained from the naive model. The R2 measure, which frequently has been used as a measure of the explanatory power of a forecasting equation has here been critically discussed, the main criticism being that it disregards some important information about the variable, namely, that the values of the variable form a time series. The I measure does not have this drawback and when taking into account other properties of this measure which have been discussed in this paper, I conclude that the I measure is useful as a measure of the information provided by a forecasting model.
The research for this paper has been supported by a grant from a research project on future studies sponsored by the Planning Division of the Research Institute of Swedish National Defence (FOA P).
INFORMATION
PROVIDED
BY FORECASTING
MODELS
361
References 1. Anderson, 0. D., Time Series Analysis and Forecasting: The Box-Jenkins Approach, Butterworth, London, 1976. 2. Box, G. E. P., and Jenkins, G. M., Times Series Analyses. Forecasting and Control, Holden-Day, San Francisco 1970. 3. G’ardenfors, P., Dynamic Systems as Tools for Forecasting and Planning: A Presentation and Some Methodological Aspects, Theory Decision (to appear). 4. Gardenfors, P., and Hansson, B.: Forecasting Non-Stationary Time Series-Some Methodological Aspects, mimeographed, University of Lund, Sweden, 1979. 5. Hanna, J. E, Information-Theoretic Techniques for Evaluating Simulation Models, in Computer Simulation ofHuman Eehaviour, Dutton and Starbuck, eds., Wiley, New York, 1971. 6. Hintikka, J., The Varieties of Information and Scientific Explanation, in Logic, Methodology and Philosophy of Science III, van Rootselaar and Staal, eds., North-Holland, Amsterdam, 1968. 7. Johnston, J., Econometric Methods, McGraw-Hill, New York, 1963. 8. Liu, T-C., A Simple Forecasting Model for the U.S. Economy, in International Monetary Fund Staff Pupers, 1954. 9. Pindyck, R. S., and Rubinfeld, D. L., Econometric Models and Economic Forecasts, McGraw-Hill, New York, 1976. 10. Theil, H., Economics and Information Theory, North-Holland, Amsterdam, 1967. Received 7 November
1979; revised 29 Januq
1980