- Email: [email protected]
A cautionary tale about multiple regression
Research on forecrrst~ng
252
judgmental forecaster. The answer we know from statistical theory is ‘yes’. In the end the authors come to the conclusion that the experts possessed ‘truly valid intuition’. I agree. Why the surprise? References from outside their chosen ambit such as McNees (1990) support this. The early experimental work in this area is wholly confused. The continuing analysis of managerial forecasts provides a rich vein for management scientists to explore and one which is important to the practice of forecasting for it provides a route towards improved implementation of models. -Robert
Fildes
Reference McNees. Stephen K.. 1990, “The role of judgment in macroeconomic forecasting accuracy”. Internatmnul Journal of Forecasting, 6. 287-299.
[Professor Robert Blattberg, Graduate School of Business, University of Chicago, 1108 E. 58th St., Chicago, IL, 60637, USA]
Milton Friedman, “A cautionary tale about multiple regression” (the appendix to “Alternative approaches to analyzing economic data” by Milton Friedman and Anna J. Schwartz), American Economic Review 81 (1991) 48-49. Friedman reported on his use of multiple regression in 1945. As a statistical consultant, he was asked to analyze data on alloys used in turbine blades for engines. The task was to develop an alloy that would withstand high temperatures for long periods. Friedman proposed a single equation regression that sought to explain ‘time to fracture’ as a function of stress, temperature, and variables representing the composition of the alloy. To obtain estimates for this equation, along with associated test statistics, it would take a highly skilled operator about three months to calculate the equation. Fortunately, there was one large scale computer in the country that could perform the calculations. This computer, the Mark 1, was at Harvard. It was built from a number of IBM card-sorting machines and was housed in an
enormous air-conditioned gymnasium. Not counting data input, it required 40 hours to calculate the regression. The size of the regression was such that it could be solved in a matter of seconds on today’s desktop computers. Friedman was delighted with the results. It had a high R2 and performed well on all of the relevant statistics. This regression allowed Friedman to construct two new alloys. Using the regression model, he predicted that each alloy would survive for several hundred hours at very high temperatures. Because this was metallurgy, not economics, he was able to test the predictions in short order. A MIT lab carried out the tests. Friedman was sufficiently skeptical that he did not reveal his predictions. As it turned out, rather than surviving for hundreds of hours, each alloy ruptured in less than four hours. Friedman concluded from this that the statistical measures associated with regressions are of little value for judging the ability of the model to predict with new data. This has been a common theme in much of the research over the past three decades (see Pant and Starbuck, 1990, for recent evidence on this issue). Nevertheless, the lesson does not seem to have spread widely. Journals are still filled with papers that cling to faith in their regression statistics. What saves most of these authors is that their predictions will never be put to the test. After all, if they were unable to test it themselves, why should they expect others to do so? Imagine how much space would be saved in economics journals if authors were required to test their predictions. -J. Scott Armstrong
Reference Pant, P. Narayan and William H. Starbuck, 1990, “Innocents in the forest: Forecasting and research methods”, Journal of Management. 16, 433-460.
[Milton Friedman, Hoover CA 94305-6010, USA]
Institution,
Stanford,
R.W. Hafer, Scott E. Hein, and S. Scott MacDonald, “Market and survey forecasts of the
252
judgmental forecaster. The answer we know from statistical theory is ‘yes’. In the end the authors come to the conclusion that the experts possessed ‘truly valid intuition’. I agree. Why the surprise? References from outside their chosen ambit such as McNees (1990) support this. The early experimental work in this area is wholly confused. The continuing analysis of managerial forecasts provides a rich vein for management scientists to explore and one which is important to the practice of forecasting for it provides a route towards improved implementation of models. -Robert
Fildes
Reference McNees. Stephen K.. 1990, “The role of judgment in macroeconomic forecasting accuracy”. Internatmnul Journal of Forecasting, 6. 287-299.
[Professor Robert Blattberg, Graduate School of Business, University of Chicago, 1108 E. 58th St., Chicago, IL, 60637, USA]
Milton Friedman, “A cautionary tale about multiple regression” (the appendix to “Alternative approaches to analyzing economic data” by Milton Friedman and Anna J. Schwartz), American Economic Review 81 (1991) 48-49. Friedman reported on his use of multiple regression in 1945. As a statistical consultant, he was asked to analyze data on alloys used in turbine blades for engines. The task was to develop an alloy that would withstand high temperatures for long periods. Friedman proposed a single equation regression that sought to explain ‘time to fracture’ as a function of stress, temperature, and variables representing the composition of the alloy. To obtain estimates for this equation, along with associated test statistics, it would take a highly skilled operator about three months to calculate the equation. Fortunately, there was one large scale computer in the country that could perform the calculations. This computer, the Mark 1, was at Harvard. It was built from a number of IBM card-sorting machines and was housed in an
enormous air-conditioned gymnasium. Not counting data input, it required 40 hours to calculate the regression. The size of the regression was such that it could be solved in a matter of seconds on today’s desktop computers. Friedman was delighted with the results. It had a high R2 and performed well on all of the relevant statistics. This regression allowed Friedman to construct two new alloys. Using the regression model, he predicted that each alloy would survive for several hundred hours at very high temperatures. Because this was metallurgy, not economics, he was able to test the predictions in short order. A MIT lab carried out the tests. Friedman was sufficiently skeptical that he did not reveal his predictions. As it turned out, rather than surviving for hundreds of hours, each alloy ruptured in less than four hours. Friedman concluded from this that the statistical measures associated with regressions are of little value for judging the ability of the model to predict with new data. This has been a common theme in much of the research over the past three decades (see Pant and Starbuck, 1990, for recent evidence on this issue). Nevertheless, the lesson does not seem to have spread widely. Journals are still filled with papers that cling to faith in their regression statistics. What saves most of these authors is that their predictions will never be put to the test. After all, if they were unable to test it themselves, why should they expect others to do so? Imagine how much space would be saved in economics journals if authors were required to test their predictions. -J. Scott Armstrong
Reference Pant, P. Narayan and William H. Starbuck, 1990, “Innocents in the forest: Forecasting and research methods”, Journal of Management. 16, 433-460.
[Milton Friedman, Hoover CA 94305-6010, USA]
Institution,
Stanford,
R.W. Hafer, Scott E. Hein, and S. Scott MacDonald, “Market and survey forecasts of the