- Email: [email protected]
Rejoinder
Book review/Rejoinder policy implied by a commitment to a democratic society, is rather treated as a convincing argument for adopting physical controls and planned trade as the solution to Britain’s economic stagnation. Free market economists are often accused of practising the economics of Dr Pangloss, of treating the discrepancies between their model and
actuality as evidence that the world has been constructed on the wrong lines. It is depressing to find that this deplorable habit has spread to the other side of the ideological divide.
William Peterson Cambridge University Cambridge, UK
Assessment is undermined without proper documentation of the model, the data used in the model and the software used to implement the model. Assessment can be either quantitative or qualitative. In addition, because of their independent perspectives, experts from fields other than energy or econometric modelling can be employed most fruitfully in assessment work, but only if adequate documentation is available.
Model and data documentation
Rejoinder Energy forecasting of the art Improvable accuracy, rather than accuracy is a hallmark of scientific prediction: it can be as accurate as in astronomy or it can consist of rough estimates like those of the behavioral sciences. The point is not that scientific prediction is accurate, but that it is founded and, for this reason, can be improved.’ In a recent issue of Energy Economics, Sergio Koreisha’ pointed out several weaknesses in energy policy models. More recently, in Technology Review, Robert Stobaugh and Sergio Koreisha3 continued the discussion and identified five specific weaknesses in many energy forecasting models. These weaknesses, which these authors called ‘red flags’, are now listed and defined: Exclusion: any factor not included in the model is assumed to be relatively unimportant in affecting the conclusion. Aggregation: data on different subprocesses are often combined, or aggregated, as if they were one process to reduce the number of variables to manageable proportions, or simply because it is not possible or convenient to measure them separately. Range: the data in a forecast equation are calculated from observations from a range of prior experience. And, if we stay within that range, such data should be useful in making forecasts. Reversibility: if the elasticities used to make forecasts are derived during a demand period in which reduced prices are accompanied by increased consumption, then it
208
- the science
l
is assumed that forecasts can also be made using these elasticities for periods when increased prices are expected to be followed by reduced consumption. Time Lag: the modeller is seldom able to estimate accurately the time required for a change in one variable, such as price, to achieve any affect.
In addition to being warned of the existence of such weaknesses, the user should also be made aware of their extent and gravity in a particular forecasting system as Koreisha has exemplified. Users can be made aware of possible model difficulties, and thus be better served, if models were accompanied by adequate documentation; assessment of assumptions and results of the model; and evidence of stability of relationships and reliability of data.
Documentation
and assessment
The primary reason for documentation is to make available information about a model so that the gravity of such problems as exclusion, aggregation, range, reversibility, and time lag can be assessed. The science is in the character of the documentation and the nature of the assessment in which attempts are made to identify first w.hat is known about the model and then to assess its structural and data dependent limitations. Whereas the character of documentation is descriptive, that of assessment is analytic.
One requirement of documentation is complete specification of the general model upon which a forecasting model is based. As part of this effort, a desired level of disaggregation and model completeness is identified. Accordingly, a framework for examining the extent of any problems of exclusion and disaggregation is made available. For example, in an econometric energy model, a particular price elasticity coefficient is dependent on what other variables are included in the equation(s). Great variability across models is not at all surprising. For example, if several price variables and income are included in a model, the results can reasonably be expected to be much different than if only a price and income variable are included. Unqualified interpretation of a forecast from the simpler model is almost certainly in error. The listing of assumptions is also important to model documentation and should be made explicit. For example, if perfect competition is assumed as a matter of analytic convenience for a market that many believe to be oligopolistic, this choice should be stated. This particular assumption is important since it ultimately affects the reliability of forecasts such as the expected increase in production from an increase in the price of a fuel such as natural gas. The data series used to estimate values for the coefficients in relationships or to obtain future values for variables in forecasting equations should be clearly listed. Since the data used in many energy modelsare obtained from a combination of administrative records, samples, expert judgment, and other sources which contain a variety of sampling, non-sampling and measurement errors, determining the quality of the data is a persistent issue. These errors affect the reliability of a forecast independent of the assumptions and
ENERGY
ECONOMICS
July 1982
Rejoinder
completeness of the model. The underlying assumptions of a model may be correct, yet the forecast may be very unreliable due to such data errors.
Forecasting strategy documentation Finally, the analyst frequently adjusts the coefficients, the modelling system, or the data to obtain a desired forecast. Rules of thumb based on knowledge of energy markets, not captured by the model, are employed in making these adjustments. This is called forecasting strategy. Such a strategy might be required to forecast the effect of gas price deregulation on residential consumption of natural gas after several years of relatively active conservation programmes that are governmentinduced. It is the most artful part of forecasting and, therefore, probably requires the most exposure, scrutiny, and discussion by disinterested third parties to establish rules of good practice; yet it is the least reported aspect of forecasting.
Present status of documentation Sadly enough, the conceptual aspects of energy models have not been afforded the same high level of documentation as some macroeconomic forecasting models, such as Fair’s? Neither have the data been reported in as complete a fashion as census data, as reported by Gonzalez et al. 5 Yet, perhaps as a result of this great need, energy modellers, as exemplified by George Lady of the Department of Energy and others at the 1979 conference at the National Bureau of Standards, have taken the lead among soft-science modellers in establishing a most comprehensive methodology of model documentation This methodology includes complete documentation of a model from the conceptual framework to the computer software implementing the model.
Qualitative assessment One form of qualitative assessment is to interpret the affect of excluded variables, as identified in the documentation, on the magnitude of the coefficients of included variables. Another is to discuss the appropriate level of spatial or temporal disaggregation for identifying like sets of pro-
instability in the coefficient(s) and fit of econometrically based models. Accordingly, these techniques can be employed to explore existing data to begin to quantitatively address inherent difficulties with range and reversibility in many econometrically based energy forecasting models. If these diagnostics reveal that the coefficients and fit of a relationship are very unstable over the range of values for which observations are available, then there is little basis for believing in the integrity of the relationship over a range of values for which few or no observations have been made. If the diagnostics indicate that the coefficient(s) and fit are relatively stable (ie not sensitive to a few, observations) then there are, at least, Quantitative assessment empirical grounds for believing in the A quantitative assessment attempts to reliability of the relationship. establish the gravity of such problems Given recent history, the assumpas reversibility and range. A quantitation of reversibility and range are tive assessment also addresses the quite similar if price changes alone are uncertainty of the data itself that is being considered. For example, in the used to estimate the model or to case of an aggregate petrol price series forecast with it. in constant dollars, al large proportion Mike McCarthy of the University of (but not all) of the earlier observaPennsylvania and John H. Herbert tions would exhibit relatively conof the US Department of Energy have stant or falling annual percentage described methods of evaluating data changes. Up to a point in time the prior to their use in models7q8 Data reversibility assumption is that hisscreening is always recommended torically observed variations in travel because primary data is frequently behaviour with respect to price from sundry sources, and recording changes would continue over the and other measurement errors are range of large annual percentage always suspect. This screening will changes in price. prevent faulty data being responsible for surprising results rather than fundamental changes in the market Reversibility being modelled. In more recent times, several Population projections are comobservations became available that monly used and important inputs in exhibited such large annual percenmany energy models that forecast either direct consumption of a fuel or tage changes. However, for a forecasting time horizon, such as 1981-85, an energy-using good such as a new car. Nathan Keyfitz of Harvard Univer- price changes may be expected to generally fall within a range of values sity,’ and others, have established that are between the many small and that these projections are subject to the few large annual percentage great error. The importance of populchanges in price. Accordingly, variable lation is such that the impact of the errors inherent in these forecasts on a values are, once more, expected to fall along a portion of the percentage final forecast can be as great as the change in price axis for which little, impact of an imprecisely measured or no, information is available as to elasticity coefficient in a forecasting consumer response. equation. An especially relevant diagnostic The assumption of range and reverfor examining reversibility is what sibility imply great stability in an Kuh et al, refer to as DFBETAI~.~~ estimated relationship. The purpose It is a scaled measure of change in a of regression diagnostic procedures coefficient j (eg a partial price elasrecently set forth by Ed Kuh’O and ticity coefficient) as a result of leavhis associates at the Massachusetts ing out the ith observation(s). This Institute of Technology (MIT) Center diagnostic can be used to compare for Computational Research in the stability of a price coefficient Economics and Management Science is to indicate the nature and degree of during the period of constant or
ducers and consumers. Whether characteristics, such as urban/rural location, household size, age of household head, and region of the country are appropriate in categorizing consumers for estimating reliable gasoline demand equations is a question for qualitative assessment. A third is to establish exactly how differences between desired measurements and available data might bias results. A fourth is the evaluation of such topics as the technical substitutability of fuels (eg coal for oil in industrial boilers) mentioned in the previous essay. Reports by David Freedman et al,6 a probabilist, are examples of such qualitative assessment.
ENERGY ECONOMICS July 1982
209
Rejoinder falling percentage changes in price in constant dollars and the more recent period during which dramatic percentage increases in price were experienced. Rather than regression diagnostics or conventional test statistics of structural stability, graphs are possibly a better and more instructive tool for assessing the character of the relationship between variables. Graphical analysis is especially apt for examining forecasted variables which can be explained by relatively few exogenous variables. For example, since both small annual percentage increases and decreases in petrol price and large annual percentage increases in petrol price in constant dollars have occurred in the past (ie from 1960 to 1980), a line graph which displayed both percentage changes in vehicle miles travelled per capita and annual percentage changes in price plotted against time could be used to assess the degree to which annual percentage increases (decreases) in price were associated with annual percentage decreases (increases) in vehicle miles travelled per capita. Mosteller and Tukeyr2 and other data analysts have suggested several additional graphical displays for examining the nature of such relationships over time. The construction of one of these graphs is now described. First, regress annual percentage changes in per capita vehicle miles travelled on another relevant variable (other than price) such as annual percentage changes in per capita disposable income in constant dollars. Next, compute the residuals from this regression. Finally, plot these residuals as well as annual percentage changes in price against time. This graph is recommended since the relationship between vehicle miles travelled and price over time is displayed only after subtracting out the affect on vehicle miles travelled of another important variable - income.
Conclusion Thus, a model is not meant to be reality and should not be interpreted in a vacuum. A model attempts to capture relationships between some driving forces in the energy market and to establish the conditions under which these relationships are expected to hold. In this rapidly changing world, a political reality can certainly render
210
a particular forecast of a model invalid, but not the model itself, if it is well founded. The science is based on constantly improving the understanding of what is known and not known about these driving forces as time proceeds. Moreover, one must be able to make some reliable and founded guesses about the world’s energy future, for without the insights of careful modelling this task would be even more awesome. John Herbert Virginia, USA
References I Mario Bunga. ‘Sergio Koreisha, ‘The limitations of anergv policy models’, Energy Economics, Vol 2, No 2, April 1980, pp 96-l 10. “Sergio Koreisha and Robert Stobaugh, ‘Modelling: selective attention institutionalized’, Technology Review, FabruaryMarch 1981, pp 64-66. 4Ray Fair, A Model of Macroeconomic Activity, Vols I and II, Ballinger, Cambridge. - . MA. 1976. ‘Maria Gonzalez; Jack L. Ogus, Gary Shapiro and Benjamin J. Tapping, ‘Standards for discussion and presentation of
errors in survey and census data’, Journal of the American Statistical Association, 1975, pp 5-23. 6David Freedman, Thomas Rothanbarg and Richard Sutch, The Demand for Energy in rhe year 1990: An Assessment of the Regional Demand Forecasting Model, Lawrence Barkelev Laboratory, Berkeley, CA 1980 ‘Lawrence Klein and Michael D. McCarthy, A Blueprint for the Validarion of an Economerric Model and its Forecasts. or How to Build an Economerric Model and Usa it to Produce a Good Forecast, Prepared for Department of Transportation under contract number DOTFA78WA-4167. Washington. DC. 1979. *John H. Herbert, Procedu& fo; Analyzing Data for Use in Models, Department of Energy, EIA, TR-0245, Washington, DC, 1980. ‘Nathan Keyfitz, The Accuracy of Popularion Forecasts, Department of Sociologv, Harvard University, Boston, MA, 1980. lo David A. Belslev, Edwin Kuh and Roy E. Walsch, Regression Diagnostics, John z;il; New York, NY, 1980. “Frederick Mostellar and John W. Tukey, Data Analysis and Regression, AddisonWaslev, Reading. MA, 1977. Saul Gass, ad, Validation and Assessment Issues of Energy Models, National Bureau of Standards, Washington, DC, 1980. Jan Kmanta and James B. Ramsey, ads, Evaluation of Econometric Models. Academic Press, New York, NY, 1980.
The author replies Herbert’s commentary on the need for more adequate documentation and proper assessment of energy models provides more fuel to the arguments put forward in my previous articles regarding the importance of testing the impact of various plausible assumptions on the models’ results. Policy makers, ie users of models, should demand that key parameters be perturbed to simulate feasible scenarios to see ‘if the results remain plausible or [if] they change so much as to become totally ludicrous’. With proper documentation, users would be in a better position to know what lies behind the assumptions in the model and to assess the limitations which they impose on the results. Although I agree with Herbert that much can be gained in having more adequate documentation of the model structure and of the database to perform independent assessments, I think that it would be difficult to convince the econometric or other
related social science professions to engage in such activities to the extent that he recommends. The reward systems of these professions, unlike those of the ‘harder’ sciences, do not remunerate duplication efforts. In physics, the ability to duplicate someone else’s research such as discovery of a new particle or element is considered not only worthy of publication, but also of great necessity in the advancement of the science. In the ‘softer’ sciences like economics replication of someone else’s results is seldom considered interesting, as evidenced by the facts that very few journals ever publish databases (‘conservation of space’) or even demand that rigorous documentation of data sources be included in the articles. Inclusion of a sector on assessment in the documentation of models would be most useful if knowledge obtained from the assessments could somehow be used to refine the structure and the results of the model.
ENERGY
ECONOMICS
July 1982
actuality as evidence that the world has been constructed on the wrong lines. It is depressing to find that this deplorable habit has spread to the other side of the ideological divide.
William Peterson Cambridge University Cambridge, UK
Assessment is undermined without proper documentation of the model, the data used in the model and the software used to implement the model. Assessment can be either quantitative or qualitative. In addition, because of their independent perspectives, experts from fields other than energy or econometric modelling can be employed most fruitfully in assessment work, but only if adequate documentation is available.
Model and data documentation
Rejoinder Energy forecasting of the art Improvable accuracy, rather than accuracy is a hallmark of scientific prediction: it can be as accurate as in astronomy or it can consist of rough estimates like those of the behavioral sciences. The point is not that scientific prediction is accurate, but that it is founded and, for this reason, can be improved.’ In a recent issue of Energy Economics, Sergio Koreisha’ pointed out several weaknesses in energy policy models. More recently, in Technology Review, Robert Stobaugh and Sergio Koreisha3 continued the discussion and identified five specific weaknesses in many energy forecasting models. These weaknesses, which these authors called ‘red flags’, are now listed and defined: Exclusion: any factor not included in the model is assumed to be relatively unimportant in affecting the conclusion. Aggregation: data on different subprocesses are often combined, or aggregated, as if they were one process to reduce the number of variables to manageable proportions, or simply because it is not possible or convenient to measure them separately. Range: the data in a forecast equation are calculated from observations from a range of prior experience. And, if we stay within that range, such data should be useful in making forecasts. Reversibility: if the elasticities used to make forecasts are derived during a demand period in which reduced prices are accompanied by increased consumption, then it
208
- the science
l
is assumed that forecasts can also be made using these elasticities for periods when increased prices are expected to be followed by reduced consumption. Time Lag: the modeller is seldom able to estimate accurately the time required for a change in one variable, such as price, to achieve any affect.
In addition to being warned of the existence of such weaknesses, the user should also be made aware of their extent and gravity in a particular forecasting system as Koreisha has exemplified. Users can be made aware of possible model difficulties, and thus be better served, if models were accompanied by adequate documentation; assessment of assumptions and results of the model; and evidence of stability of relationships and reliability of data.
Documentation
and assessment
The primary reason for documentation is to make available information about a model so that the gravity of such problems as exclusion, aggregation, range, reversibility, and time lag can be assessed. The science is in the character of the documentation and the nature of the assessment in which attempts are made to identify first w.hat is known about the model and then to assess its structural and data dependent limitations. Whereas the character of documentation is descriptive, that of assessment is analytic.
One requirement of documentation is complete specification of the general model upon which a forecasting model is based. As part of this effort, a desired level of disaggregation and model completeness is identified. Accordingly, a framework for examining the extent of any problems of exclusion and disaggregation is made available. For example, in an econometric energy model, a particular price elasticity coefficient is dependent on what other variables are included in the equation(s). Great variability across models is not at all surprising. For example, if several price variables and income are included in a model, the results can reasonably be expected to be much different than if only a price and income variable are included. Unqualified interpretation of a forecast from the simpler model is almost certainly in error. The listing of assumptions is also important to model documentation and should be made explicit. For example, if perfect competition is assumed as a matter of analytic convenience for a market that many believe to be oligopolistic, this choice should be stated. This particular assumption is important since it ultimately affects the reliability of forecasts such as the expected increase in production from an increase in the price of a fuel such as natural gas. The data series used to estimate values for the coefficients in relationships or to obtain future values for variables in forecasting equations should be clearly listed. Since the data used in many energy modelsare obtained from a combination of administrative records, samples, expert judgment, and other sources which contain a variety of sampling, non-sampling and measurement errors, determining the quality of the data is a persistent issue. These errors affect the reliability of a forecast independent of the assumptions and
ENERGY
ECONOMICS
July 1982
Rejoinder
completeness of the model. The underlying assumptions of a model may be correct, yet the forecast may be very unreliable due to such data errors.
Forecasting strategy documentation Finally, the analyst frequently adjusts the coefficients, the modelling system, or the data to obtain a desired forecast. Rules of thumb based on knowledge of energy markets, not captured by the model, are employed in making these adjustments. This is called forecasting strategy. Such a strategy might be required to forecast the effect of gas price deregulation on residential consumption of natural gas after several years of relatively active conservation programmes that are governmentinduced. It is the most artful part of forecasting and, therefore, probably requires the most exposure, scrutiny, and discussion by disinterested third parties to establish rules of good practice; yet it is the least reported aspect of forecasting.
Present status of documentation Sadly enough, the conceptual aspects of energy models have not been afforded the same high level of documentation as some macroeconomic forecasting models, such as Fair’s? Neither have the data been reported in as complete a fashion as census data, as reported by Gonzalez et al. 5 Yet, perhaps as a result of this great need, energy modellers, as exemplified by George Lady of the Department of Energy and others at the 1979 conference at the National Bureau of Standards, have taken the lead among soft-science modellers in establishing a most comprehensive methodology of model documentation This methodology includes complete documentation of a model from the conceptual framework to the computer software implementing the model.
Qualitative assessment One form of qualitative assessment is to interpret the affect of excluded variables, as identified in the documentation, on the magnitude of the coefficients of included variables. Another is to discuss the appropriate level of spatial or temporal disaggregation for identifying like sets of pro-
instability in the coefficient(s) and fit of econometrically based models. Accordingly, these techniques can be employed to explore existing data to begin to quantitatively address inherent difficulties with range and reversibility in many econometrically based energy forecasting models. If these diagnostics reveal that the coefficients and fit of a relationship are very unstable over the range of values for which observations are available, then there is little basis for believing in the integrity of the relationship over a range of values for which few or no observations have been made. If the diagnostics indicate that the coefficient(s) and fit are relatively stable (ie not sensitive to a few, observations) then there are, at least, Quantitative assessment empirical grounds for believing in the A quantitative assessment attempts to reliability of the relationship. establish the gravity of such problems Given recent history, the assumpas reversibility and range. A quantitation of reversibility and range are tive assessment also addresses the quite similar if price changes alone are uncertainty of the data itself that is being considered. For example, in the used to estimate the model or to case of an aggregate petrol price series forecast with it. in constant dollars, al large proportion Mike McCarthy of the University of (but not all) of the earlier observaPennsylvania and John H. Herbert tions would exhibit relatively conof the US Department of Energy have stant or falling annual percentage described methods of evaluating data changes. Up to a point in time the prior to their use in models7q8 Data reversibility assumption is that hisscreening is always recommended torically observed variations in travel because primary data is frequently behaviour with respect to price from sundry sources, and recording changes would continue over the and other measurement errors are range of large annual percentage always suspect. This screening will changes in price. prevent faulty data being responsible for surprising results rather than fundamental changes in the market Reversibility being modelled. In more recent times, several Population projections are comobservations became available that monly used and important inputs in exhibited such large annual percenmany energy models that forecast either direct consumption of a fuel or tage changes. However, for a forecasting time horizon, such as 1981-85, an energy-using good such as a new car. Nathan Keyfitz of Harvard Univer- price changes may be expected to generally fall within a range of values sity,’ and others, have established that are between the many small and that these projections are subject to the few large annual percentage great error. The importance of populchanges in price. Accordingly, variable lation is such that the impact of the errors inherent in these forecasts on a values are, once more, expected to fall along a portion of the percentage final forecast can be as great as the change in price axis for which little, impact of an imprecisely measured or no, information is available as to elasticity coefficient in a forecasting consumer response. equation. An especially relevant diagnostic The assumption of range and reverfor examining reversibility is what sibility imply great stability in an Kuh et al, refer to as DFBETAI~.~~ estimated relationship. The purpose It is a scaled measure of change in a of regression diagnostic procedures coefficient j (eg a partial price elasrecently set forth by Ed Kuh’O and ticity coefficient) as a result of leavhis associates at the Massachusetts ing out the ith observation(s). This Institute of Technology (MIT) Center diagnostic can be used to compare for Computational Research in the stability of a price coefficient Economics and Management Science is to indicate the nature and degree of during the period of constant or
ducers and consumers. Whether characteristics, such as urban/rural location, household size, age of household head, and region of the country are appropriate in categorizing consumers for estimating reliable gasoline demand equations is a question for qualitative assessment. A third is to establish exactly how differences between desired measurements and available data might bias results. A fourth is the evaluation of such topics as the technical substitutability of fuels (eg coal for oil in industrial boilers) mentioned in the previous essay. Reports by David Freedman et al,6 a probabilist, are examples of such qualitative assessment.
ENERGY ECONOMICS July 1982
209
Rejoinder falling percentage changes in price in constant dollars and the more recent period during which dramatic percentage increases in price were experienced. Rather than regression diagnostics or conventional test statistics of structural stability, graphs are possibly a better and more instructive tool for assessing the character of the relationship between variables. Graphical analysis is especially apt for examining forecasted variables which can be explained by relatively few exogenous variables. For example, since both small annual percentage increases and decreases in petrol price and large annual percentage increases in petrol price in constant dollars have occurred in the past (ie from 1960 to 1980), a line graph which displayed both percentage changes in vehicle miles travelled per capita and annual percentage changes in price plotted against time could be used to assess the degree to which annual percentage increases (decreases) in price were associated with annual percentage decreases (increases) in vehicle miles travelled per capita. Mosteller and Tukeyr2 and other data analysts have suggested several additional graphical displays for examining the nature of such relationships over time. The construction of one of these graphs is now described. First, regress annual percentage changes in per capita vehicle miles travelled on another relevant variable (other than price) such as annual percentage changes in per capita disposable income in constant dollars. Next, compute the residuals from this regression. Finally, plot these residuals as well as annual percentage changes in price against time. This graph is recommended since the relationship between vehicle miles travelled and price over time is displayed only after subtracting out the affect on vehicle miles travelled of another important variable - income.
Conclusion Thus, a model is not meant to be reality and should not be interpreted in a vacuum. A model attempts to capture relationships between some driving forces in the energy market and to establish the conditions under which these relationships are expected to hold. In this rapidly changing world, a political reality can certainly render
210
a particular forecast of a model invalid, but not the model itself, if it is well founded. The science is based on constantly improving the understanding of what is known and not known about these driving forces as time proceeds. Moreover, one must be able to make some reliable and founded guesses about the world’s energy future, for without the insights of careful modelling this task would be even more awesome. John Herbert Virginia, USA
References I Mario Bunga. ‘Sergio Koreisha, ‘The limitations of anergv policy models’, Energy Economics, Vol 2, No 2, April 1980, pp 96-l 10. “Sergio Koreisha and Robert Stobaugh, ‘Modelling: selective attention institutionalized’, Technology Review, FabruaryMarch 1981, pp 64-66. 4Ray Fair, A Model of Macroeconomic Activity, Vols I and II, Ballinger, Cambridge. - . MA. 1976. ‘Maria Gonzalez; Jack L. Ogus, Gary Shapiro and Benjamin J. Tapping, ‘Standards for discussion and presentation of
errors in survey and census data’, Journal of the American Statistical Association, 1975, pp 5-23. 6David Freedman, Thomas Rothanbarg and Richard Sutch, The Demand for Energy in rhe year 1990: An Assessment of the Regional Demand Forecasting Model, Lawrence Barkelev Laboratory, Berkeley, CA 1980 ‘Lawrence Klein and Michael D. McCarthy, A Blueprint for the Validarion of an Economerric Model and its Forecasts. or How to Build an Economerric Model and Usa it to Produce a Good Forecast, Prepared for Department of Transportation under contract number DOTFA78WA-4167. Washington. DC. 1979. *John H. Herbert, Procedu& fo; Analyzing Data for Use in Models, Department of Energy, EIA, TR-0245, Washington, DC, 1980. ‘Nathan Keyfitz, The Accuracy of Popularion Forecasts, Department of Sociologv, Harvard University, Boston, MA, 1980. lo David A. Belslev, Edwin Kuh and Roy E. Walsch, Regression Diagnostics, John z;il; New York, NY, 1980. “Frederick Mostellar and John W. Tukey, Data Analysis and Regression, AddisonWaslev, Reading. MA, 1977. Saul Gass, ad, Validation and Assessment Issues of Energy Models, National Bureau of Standards, Washington, DC, 1980. Jan Kmanta and James B. Ramsey, ads, Evaluation of Econometric Models. Academic Press, New York, NY, 1980.
The author replies Herbert’s commentary on the need for more adequate documentation and proper assessment of energy models provides more fuel to the arguments put forward in my previous articles regarding the importance of testing the impact of various plausible assumptions on the models’ results. Policy makers, ie users of models, should demand that key parameters be perturbed to simulate feasible scenarios to see ‘if the results remain plausible or [if] they change so much as to become totally ludicrous’. With proper documentation, users would be in a better position to know what lies behind the assumptions in the model and to assess the limitations which they impose on the results. Although I agree with Herbert that much can be gained in having more adequate documentation of the model structure and of the database to perform independent assessments, I think that it would be difficult to convince the econometric or other
related social science professions to engage in such activities to the extent that he recommends. The reward systems of these professions, unlike those of the ‘harder’ sciences, do not remunerate duplication efforts. In physics, the ability to duplicate someone else’s research such as discovery of a new particle or element is considered not only worthy of publication, but also of great necessity in the advancement of the science. In the ‘softer’ sciences like economics replication of someone else’s results is seldom considered interesting, as evidenced by the facts that very few journals ever publish databases (‘conservation of space’) or even demand that rigorous documentation of data sources be included in the articles. Inclusion of a sector on assessment in the documentation of models would be most useful if knowledge obtained from the assessments could somehow be used to refine the structure and the results of the model.
ENERGY
ECONOMICS
July 1982