- Email: [email protected]
On the possibility of accurate public prediction
On the Possibility of Accurate Public Prediction
HERBERT
A. SIMON”
Carnegie Mellon
University
ABSTRACT: Henshel’s (1995) challenge to practical applicability of the theorem demonstrating the possibility of public predictions of social behavior fails, for his examples do not show discontinuity of the public behavior to the magnitude of the prediction of behavior.
I find it somewhat embarrassing to defend a paper that I published more than 42 years ago. Surely some of the things one has said publicly in the past are wrong, and one hardly needs to react defensively when deficiencies are pointed out in them. So when Richard L. Henshel(1995) published in a recent issue of The Journal ofSocio-Economics a thoughtful paper raising questions about the empirical plausibility of the central theorem in my venerable “Bandwagon and Underdog Effects of Election Predictions,” (Simon, 1954) my first reaction was “Right on!” It appeared not unlikely that “benchmark’ numbers, whether they be $39.99 prices, 50% election predictions, or Dow-Jones signal values, might provide dividing lines that would shift a nontrivial number of responses in a discontinuous way, thereby defining situations that violate a key assumption underlying my Possibility Theorem. But I have had afterthoughts that suggest that the matter is not as clear and simple as Henshel makes it-that the kinds of benchmarks he points to may have little or nothing to do with the (approximate) empirical validity of the theorem. The purpose of this brief account is to examine further whether the Possibility Theorem is likely to have any wide application to real-world situations. The conditions under which the theorem holds are not at issue: It holds when the behavior of interest is a con-
*Direct all correspondence Pittsburgh, PA 15213.
to: Herbert
A. Simon, Department
Journal of Socio-Economics, Volume 26, No. 2, pp. 127-132 Copyright 0 1997 by JAI Press, Inc. All rights of reproduction in any form reserved. ISSN: 10534357
of Psychology,
Carnegie
Mellon University,
128
THE JOURNAL
OF SOCIO-ECONOMICS
Vol. 26/No. 2/1997
tinuous function of the published prediction. The central question is how often, if ever, this condition is satisfied to a sufficient approximation in the real world. THE POSSIBILITY
THEOREM
AND BENCHMARK NUMBERS
Henshel makes his case against the Possibility Theorem by providing four examples of critical benchmarks that, he claims, would cause discontinuities in a prediction function. He does not, in fact, offer any systematic empirical evidence to support his claims but believes that our general experience of the world will cause us to accept them. The four benchmarks are (a) values of the Dow-Jones Index that are “critical” in Dow Theory; (b) “bargain” prices of the “$3.99 < $4.00” variety; (c) inflation that reaches “double-digit” levels; and (d) polls that show a candidate reaching a 50% preference level. Let us consider them in turn. Dow Theory Critical Levels The Possibility Theorem addresses a situation where some fraction (I, for initial percentage) of a population is disposed to behavior A, and the remainder to behavior B. A prediction (P) is made of this fraction and the prediction is published. Now the actual behavior (V) takes place and is compared with the prediction. The theorem is not concerned with whether the prediction estimates the original disposition of the population, but whether it can be made in such a way as to anticipate the actual behavior that occurs after it is published, so that P = V, for given Z, at least approximately. The Dow-Jones Index is not a prediction of behavior but a summary of market behavior in the recent past-the final prices of stocks at the end of a trading day. The predictor of behavior is the Dow Theory, which alleges that prices will change in certain ways as a function of the most recent index and changes in it. So the question for the Possibility Theorem is whether publication of the Dow Theory before trading opens will cause a significantly different movement in the market that day than if the Theory weren’t published. Henshel does not address this question at all. He argues that many people already know the Dow Theory and will act on it, therefore making the fall of the Index below a critical value potentially a self-fulfilling prophecy. But we have seen that movement of the Index is not a prophecy or prediction at all; it is a fact about past events. Is there any reason to believe (or any empirical evidence that would provide such a reason) that if anyone predicted that the market would fall to, say 649 (assuming 650 to be a critical value), it would behave substantially differently-amounting to a discontinuity-from what it would do if they predicted that it would fall to, say 650? That is wholly different from the discontinuity that might appear if the Index did, in fact, fall below 650 (assuming that were the critical value). In the latter case, the discontinuity in behavior is caused by an event, not a prediction, and is therefore irrelevant to the Possibility Theorem.
Accurate Public Prediction
129
I would not claim, of course, that the conditions of continuity for the Possibility Theorem are always satisfied in the real world; what I claim is that their satisfaction is a much less rare event than Henshel would have us believe. If the President of the United States (or perhaps even the Chairman of the Fed) predicted that the market would fall below a critical value of the Index, a substantial fall might in fact result. Even this would not prove discontinuity: we would have to test whether prediction of a slightly smaller or a slightly larger fall would have caused a signifi~~tly different market reaction, or whether, instead, the market reaction would be a smooth function of the prediction. We can use an example much simpler than Dow Theory to make the point. Suppose that the weathers predicts every morning the probability of rain before 6 p.m. that evening, and we record the percentage of co~uters who carry umbrellas on that day. Now it is conceivable (but by no means obvious) that there might be a discontinuous increase in the percentage of umbrella carriers as the rain prediction rises from 49% to 50% (or from 0% to 1%). But this is not a violation of the conditions of the Possibility Theorem, for what is being predicted is the weather, not the behavior of commuters. Suppose that the weather prediction were accompanied by a public prediction of the number of umbrella carriers. Do we have any reason to suppose, or any evidence, that the actual number of umbrella carriers, even if it were affected by our prediction of that number, would change discontinuously with it? Notice that now each prediction of carriers is being made for a fixed value of the weather prediction If V is the number of carriers, P the predicted number, and I the weather prediction (to borrow my earlier notation), the question for the Possibility Theorem is whether V is a continuous or discontinuous function of P,for a given v&e of 1. My own guess is that it would not only be a continuous function, but an almost constant function as well. “Bargain Prices”
A similar ~gument dismisses the possible discontinuity in dem~d between $3.99 and $4.00 prices as irrelevant to the Possibility Theorem. What is proved (if merchants are right in thinking there is such a discontinuity-has it ever been seriously tested?) is that small changes in prices can cause discrete changes in behavior. This is quite different from claiming, for example, that there are critical values in the Consumer Price Index such that when the Index passes one such value, demand changes in a discontinuous fashion. Even if the latter occurred, it would violate the assumptions of the Possibility Theorem only if we regarded the Consumer Price Index as a prediction rather than an (approximate) statement of fact. Do we believe that there is a critical value of a prediction of next month’s Consumer Price Index that could produce a discontinuity in response? (Remember that we are varying only the ~~e~icrio~; the actual state of affairs, and its history up to the moment of prediction is unaltered.)
130
THE JOURNAL
Double-Digit
OF SOCIO-ECONOMICS
Vol. 26/No. 2/l 997
Inflation
Again, Henshel’s examples describe critical values in a situation, not in a prediction, that might produce a discontinuity in the response. In fact, he does not demonstrate that there was such a discontinuity. All that recent history tells us is that when prices began rising more rapidly in the 1970’s, more and more people began to take prospective changes in the price level into account in their decisions, and thereby probably accelerated the rate of inflation. I see nothing in the statistics in Statistical Abstracts that reveals a discontinuity in this reaction, and nothing that shows anything different in the price movements just after the inflation rate exceeded 10% than just before. All the statistics show is a slow and rather steady increase in prices from, say, 1970 to 1980 briefly accelerated by the two oil shocks (the latter being events, not predictions). Even proof of such a discontinuity would not disconfirm the assumptions of the Possibility Theorem. The relevant question for the Possibility Theorem is whether, for a given reported level of recent and current injlation, publication of a prediction of double-digit inflation (e.g., publication of a prediction of a 10.1% inflation rate versus one of a 9.9% rate) would cause a discontinuous reaction. I know of no reason to suppose that it would, and Henshel adduces no evidence of discontinuity. The Bandwagon
Effect
We come to Henshel’s final example: the possibility that publication of an election poll (especially in the neighborhood of 50% for the leading candidate) could cause a discontinuity in the outcome of the election as a function of the prediction. Again, the question, as far as the Possibility Theorem is concerned, is not whether the publication of an election prediction could have a substantial effect on the actual vote: for all the reasons that Henshel adduces, it very well might. The question for the Theorem is whether this effect could be very different if the poll showed 49.9% for a candidate than if it showed 50.1%. Reading a poll that says that your candidate (or the opponent) is going to get 50.1% of the vote is very different from knowing this as a fact. Assuming substantial variation in beliefs as to the accuracy of the poll (a not implausible assumption), one would hardly expect massive switches to take place at a particular poll level, but rather that they would be distributed over a range of poll results. Therefore, we have little reason to believe that the continuity assumption of the Possibility Theorem will be violated even in the face of a substantial bandwagon or underdog effect. There is no hard evidence with which I am familiar about the nature or extent of the effect of polls on voting. The most plausible case for a large effect concerns abstention from voting if one candidate is almost certain to win, and then only if the abstention is at different levels for supporters of the competing candidates. Even in this case, what reason is there to believe that the abstention of many voters will begin at a particular poll percentage, creating a discontinuity in the
Accurate
PublicPrediction
131
function relating P to V? Ever since 1936, we have strong reasons to disbelieve in a large bandwagon effect associated with specific poll values. CONCLUSIONS We can sum up our conclusions about the empirical plausibility of the continuity condition of the Possibility Theorem as follows: 1. There is no doubt that events can cause large changes in people’s behavior, often over a short period of time. A declaration of war is one such event, an earthquake is another. 2. There is no doubt that predictions of events can also cause large changes in behavior: for example, the prediction, a half-day before the fact, that a hurricane is going to miss Florida but hit Virginia. 3. Neither of the two previous points implies that either small changes in events or small changes in predictions (especially the latter) will cause large changes in behavior. I would expect most measures of changes of behavior in response to an earthquake to be continuous functions of its severity measured on the Richter scale. The same would be trne of rates of evacuation from a hurricane hazard area in relation to predictions of the hu~icane’s Beaufort scale meas~e of intensity. 4. The likelihood of a discontinuity in the function relating a prediction of behavior to the actual behavior can be expected to decline rapidly with doubts about the exactness of the prediction, and also with the availability to people of other info~ation relevant to their preferences for the behavior being predicted. From these conclusions, we can draw the further conclusion that the continuity condition for the Possibility Theorem holds empirically under a wide range of real-world conditions; in fact, that continuity is the rule, empirically, and discontinuity the exception. Of course, like Henshel, I lack hard facts on these matters and must appeal to your beliefs about how the world works. But I have been at some pains to show that the beliefs that Henshel cites, even if empiric~ly correct, do not support his thesis. Why is all of this worth worrying about? It is worth worrying about because, as Henshel points out, the possible effects of predictions upon behavior have been widely inte~reted as creating a severe difficulty, if not an impossibility, for the social sciences. I hope to have shown that the critical assumption of the Possibility Theorem, the continuous relation between prediction and behavior, is in fact satisfied in the real world in most cases to a sufficient degree of approximation for building theories that describe human behavior in that world. I do not dismiss the dif~culties of prediction created by the possibility of human reaction to the predictions-and, indeed, the larger part of my earlier paper was devoted, not to the Possibility Theorem, but to these dif~culties. But dif~culties are not impossibiIities. Nor do I wish to suggest that prediction- and especially, prediction of the behavior of others-does not play an important role in social systems. So-called
132
THE JOURNAL OF SOCIO-ECONOMiCS
Vol. X/No.
211997
“rational expectations” theory, right or wrong (and I think it is fundamentally wrong), represented one view of the way predictions work. Game theory, in all of its variants, represents other views. Theorizing, no matter how sophisticated the mathematics, is not going to settle these issues, for it has been shown that rationality in such out-guessing situations can be defined in innumerable ways, each definition usually predicting different outcomes from the other definitions. What we need, and need badly, is extensive empirical work to discover how human beings actually make choices in the face of uncertainty, especially uncertainty about the actions and reactions of other human beings Having shown that building valid empirical theories is not ruled out in principle, the possibility Theorem has laid that issue to rest, and it is time now to move on to the much more difficult questions, especially empirical questions, that remain. Richard Henshel is to be thanked for turning our attention to this other side of the matter. REFERENCES Henshel, R.L. (1995). The Griinberg/Modigliani and Simon possibility theorem: a social psychological critique. The .klourtwl of So~io-E~~otlomi~s, 24, 50 l-520. Simon, H.A. (1954). Bandwagon and underdog effects of election predictions. Public Opinion Quarterly, IX, 245-253.
HERBERT
A. SIMON”
Carnegie Mellon
University
ABSTRACT: Henshel’s (1995) challenge to practical applicability of the theorem demonstrating the possibility of public predictions of social behavior fails, for his examples do not show discontinuity of the public behavior to the magnitude of the prediction of behavior.
I find it somewhat embarrassing to defend a paper that I published more than 42 years ago. Surely some of the things one has said publicly in the past are wrong, and one hardly needs to react defensively when deficiencies are pointed out in them. So when Richard L. Henshel(1995) published in a recent issue of The Journal ofSocio-Economics a thoughtful paper raising questions about the empirical plausibility of the central theorem in my venerable “Bandwagon and Underdog Effects of Election Predictions,” (Simon, 1954) my first reaction was “Right on!” It appeared not unlikely that “benchmark’ numbers, whether they be $39.99 prices, 50% election predictions, or Dow-Jones signal values, might provide dividing lines that would shift a nontrivial number of responses in a discontinuous way, thereby defining situations that violate a key assumption underlying my Possibility Theorem. But I have had afterthoughts that suggest that the matter is not as clear and simple as Henshel makes it-that the kinds of benchmarks he points to may have little or nothing to do with the (approximate) empirical validity of the theorem. The purpose of this brief account is to examine further whether the Possibility Theorem is likely to have any wide application to real-world situations. The conditions under which the theorem holds are not at issue: It holds when the behavior of interest is a con-
*Direct all correspondence Pittsburgh, PA 15213.
to: Herbert
A. Simon, Department
Journal of Socio-Economics, Volume 26, No. 2, pp. 127-132 Copyright 0 1997 by JAI Press, Inc. All rights of reproduction in any form reserved. ISSN: 10534357
of Psychology,
Carnegie
Mellon University,
128
THE JOURNAL
OF SOCIO-ECONOMICS
Vol. 26/No. 2/1997
tinuous function of the published prediction. The central question is how often, if ever, this condition is satisfied to a sufficient approximation in the real world. THE POSSIBILITY
THEOREM
AND BENCHMARK NUMBERS
Henshel makes his case against the Possibility Theorem by providing four examples of critical benchmarks that, he claims, would cause discontinuities in a prediction function. He does not, in fact, offer any systematic empirical evidence to support his claims but believes that our general experience of the world will cause us to accept them. The four benchmarks are (a) values of the Dow-Jones Index that are “critical” in Dow Theory; (b) “bargain” prices of the “$3.99 < $4.00” variety; (c) inflation that reaches “double-digit” levels; and (d) polls that show a candidate reaching a 50% preference level. Let us consider them in turn. Dow Theory Critical Levels The Possibility Theorem addresses a situation where some fraction (I, for initial percentage) of a population is disposed to behavior A, and the remainder to behavior B. A prediction (P) is made of this fraction and the prediction is published. Now the actual behavior (V) takes place and is compared with the prediction. The theorem is not concerned with whether the prediction estimates the original disposition of the population, but whether it can be made in such a way as to anticipate the actual behavior that occurs after it is published, so that P = V, for given Z, at least approximately. The Dow-Jones Index is not a prediction of behavior but a summary of market behavior in the recent past-the final prices of stocks at the end of a trading day. The predictor of behavior is the Dow Theory, which alleges that prices will change in certain ways as a function of the most recent index and changes in it. So the question for the Possibility Theorem is whether publication of the Dow Theory before trading opens will cause a significantly different movement in the market that day than if the Theory weren’t published. Henshel does not address this question at all. He argues that many people already know the Dow Theory and will act on it, therefore making the fall of the Index below a critical value potentially a self-fulfilling prophecy. But we have seen that movement of the Index is not a prophecy or prediction at all; it is a fact about past events. Is there any reason to believe (or any empirical evidence that would provide such a reason) that if anyone predicted that the market would fall to, say 649 (assuming 650 to be a critical value), it would behave substantially differently-amounting to a discontinuity-from what it would do if they predicted that it would fall to, say 650? That is wholly different from the discontinuity that might appear if the Index did, in fact, fall below 650 (assuming that were the critical value). In the latter case, the discontinuity in behavior is caused by an event, not a prediction, and is therefore irrelevant to the Possibility Theorem.
Accurate Public Prediction
129
I would not claim, of course, that the conditions of continuity for the Possibility Theorem are always satisfied in the real world; what I claim is that their satisfaction is a much less rare event than Henshel would have us believe. If the President of the United States (or perhaps even the Chairman of the Fed) predicted that the market would fall below a critical value of the Index, a substantial fall might in fact result. Even this would not prove discontinuity: we would have to test whether prediction of a slightly smaller or a slightly larger fall would have caused a signifi~~tly different market reaction, or whether, instead, the market reaction would be a smooth function of the prediction. We can use an example much simpler than Dow Theory to make the point. Suppose that the weathers predicts every morning the probability of rain before 6 p.m. that evening, and we record the percentage of co~uters who carry umbrellas on that day. Now it is conceivable (but by no means obvious) that there might be a discontinuous increase in the percentage of umbrella carriers as the rain prediction rises from 49% to 50% (or from 0% to 1%). But this is not a violation of the conditions of the Possibility Theorem, for what is being predicted is the weather, not the behavior of commuters. Suppose that the weather prediction were accompanied by a public prediction of the number of umbrella carriers. Do we have any reason to suppose, or any evidence, that the actual number of umbrella carriers, even if it were affected by our prediction of that number, would change discontinuously with it? Notice that now each prediction of carriers is being made for a fixed value of the weather prediction If V is the number of carriers, P the predicted number, and I the weather prediction (to borrow my earlier notation), the question for the Possibility Theorem is whether V is a continuous or discontinuous function of P,for a given v&e of 1. My own guess is that it would not only be a continuous function, but an almost constant function as well. “Bargain Prices”
A similar ~gument dismisses the possible discontinuity in dem~d between $3.99 and $4.00 prices as irrelevant to the Possibility Theorem. What is proved (if merchants are right in thinking there is such a discontinuity-has it ever been seriously tested?) is that small changes in prices can cause discrete changes in behavior. This is quite different from claiming, for example, that there are critical values in the Consumer Price Index such that when the Index passes one such value, demand changes in a discontinuous fashion. Even if the latter occurred, it would violate the assumptions of the Possibility Theorem only if we regarded the Consumer Price Index as a prediction rather than an (approximate) statement of fact. Do we believe that there is a critical value of a prediction of next month’s Consumer Price Index that could produce a discontinuity in response? (Remember that we are varying only the ~~e~icrio~; the actual state of affairs, and its history up to the moment of prediction is unaltered.)
130
THE JOURNAL
Double-Digit
OF SOCIO-ECONOMICS
Vol. 26/No. 2/l 997
Inflation
Again, Henshel’s examples describe critical values in a situation, not in a prediction, that might produce a discontinuity in the response. In fact, he does not demonstrate that there was such a discontinuity. All that recent history tells us is that when prices began rising more rapidly in the 1970’s, more and more people began to take prospective changes in the price level into account in their decisions, and thereby probably accelerated the rate of inflation. I see nothing in the statistics in Statistical Abstracts that reveals a discontinuity in this reaction, and nothing that shows anything different in the price movements just after the inflation rate exceeded 10% than just before. All the statistics show is a slow and rather steady increase in prices from, say, 1970 to 1980 briefly accelerated by the two oil shocks (the latter being events, not predictions). Even proof of such a discontinuity would not disconfirm the assumptions of the Possibility Theorem. The relevant question for the Possibility Theorem is whether, for a given reported level of recent and current injlation, publication of a prediction of double-digit inflation (e.g., publication of a prediction of a 10.1% inflation rate versus one of a 9.9% rate) would cause a discontinuous reaction. I know of no reason to suppose that it would, and Henshel adduces no evidence of discontinuity. The Bandwagon
Effect
We come to Henshel’s final example: the possibility that publication of an election poll (especially in the neighborhood of 50% for the leading candidate) could cause a discontinuity in the outcome of the election as a function of the prediction. Again, the question, as far as the Possibility Theorem is concerned, is not whether the publication of an election prediction could have a substantial effect on the actual vote: for all the reasons that Henshel adduces, it very well might. The question for the Theorem is whether this effect could be very different if the poll showed 49.9% for a candidate than if it showed 50.1%. Reading a poll that says that your candidate (or the opponent) is going to get 50.1% of the vote is very different from knowing this as a fact. Assuming substantial variation in beliefs as to the accuracy of the poll (a not implausible assumption), one would hardly expect massive switches to take place at a particular poll level, but rather that they would be distributed over a range of poll results. Therefore, we have little reason to believe that the continuity assumption of the Possibility Theorem will be violated even in the face of a substantial bandwagon or underdog effect. There is no hard evidence with which I am familiar about the nature or extent of the effect of polls on voting. The most plausible case for a large effect concerns abstention from voting if one candidate is almost certain to win, and then only if the abstention is at different levels for supporters of the competing candidates. Even in this case, what reason is there to believe that the abstention of many voters will begin at a particular poll percentage, creating a discontinuity in the
Accurate
PublicPrediction
131
function relating P to V? Ever since 1936, we have strong reasons to disbelieve in a large bandwagon effect associated with specific poll values. CONCLUSIONS We can sum up our conclusions about the empirical plausibility of the continuity condition of the Possibility Theorem as follows: 1. There is no doubt that events can cause large changes in people’s behavior, often over a short period of time. A declaration of war is one such event, an earthquake is another. 2. There is no doubt that predictions of events can also cause large changes in behavior: for example, the prediction, a half-day before the fact, that a hurricane is going to miss Florida but hit Virginia. 3. Neither of the two previous points implies that either small changes in events or small changes in predictions (especially the latter) will cause large changes in behavior. I would expect most measures of changes of behavior in response to an earthquake to be continuous functions of its severity measured on the Richter scale. The same would be trne of rates of evacuation from a hurricane hazard area in relation to predictions of the hu~icane’s Beaufort scale meas~e of intensity. 4. The likelihood of a discontinuity in the function relating a prediction of behavior to the actual behavior can be expected to decline rapidly with doubts about the exactness of the prediction, and also with the availability to people of other info~ation relevant to their preferences for the behavior being predicted. From these conclusions, we can draw the further conclusion that the continuity condition for the Possibility Theorem holds empirically under a wide range of real-world conditions; in fact, that continuity is the rule, empirically, and discontinuity the exception. Of course, like Henshel, I lack hard facts on these matters and must appeal to your beliefs about how the world works. But I have been at some pains to show that the beliefs that Henshel cites, even if empiric~ly correct, do not support his thesis. Why is all of this worth worrying about? It is worth worrying about because, as Henshel points out, the possible effects of predictions upon behavior have been widely inte~reted as creating a severe difficulty, if not an impossibility, for the social sciences. I hope to have shown that the critical assumption of the Possibility Theorem, the continuous relation between prediction and behavior, is in fact satisfied in the real world in most cases to a sufficient degree of approximation for building theories that describe human behavior in that world. I do not dismiss the dif~culties of prediction created by the possibility of human reaction to the predictions-and, indeed, the larger part of my earlier paper was devoted, not to the Possibility Theorem, but to these dif~culties. But dif~culties are not impossibiIities. Nor do I wish to suggest that prediction- and especially, prediction of the behavior of others-does not play an important role in social systems. So-called
132
THE JOURNAL OF SOCIO-ECONOMiCS
Vol. X/No.
211997
“rational expectations” theory, right or wrong (and I think it is fundamentally wrong), represented one view of the way predictions work. Game theory, in all of its variants, represents other views. Theorizing, no matter how sophisticated the mathematics, is not going to settle these issues, for it has been shown that rationality in such out-guessing situations can be defined in innumerable ways, each definition usually predicting different outcomes from the other definitions. What we need, and need badly, is extensive empirical work to discover how human beings actually make choices in the face of uncertainty, especially uncertainty about the actions and reactions of other human beings Having shown that building valid empirical theories is not ruled out in principle, the possibility Theorem has laid that issue to rest, and it is time now to move on to the much more difficult questions, especially empirical questions, that remain. Richard Henshel is to be thanked for turning our attention to this other side of the matter. REFERENCES Henshel, R.L. (1995). The Griinberg/Modigliani and Simon possibility theorem: a social psychological critique. The .klourtwl of So~io-E~~otlomi~s, 24, 50 l-520. Simon, H.A. (1954). Bandwagon and underdog effects of election predictions. Public Opinion Quarterly, IX, 245-253.