The Status of Hidden-Variable Theories

THE STATUS OF HIDDEN-VARIABLE THEORIES

A. SHIMONY Boston University, Boston, Mass., USA

There have been several strong motivations for maintaining that the quantum mechanical description of the physical world is incomplete and in need of supplementation by some kind of hidden-variable theory. The earliest historically was a reaction to the indeterminacy which is characteristic of quantum mechanical predictions. At the Solvay Conference of 1927 and in correspondence at that time, Einstein argued that a complete specification of the initial conditions of a process should suffice in principle to determine the course of the process, and therefore the statistical character of quantum mechanical predictions implies that the quantum description is incomplete (see JAMMER, 1966, pp. 357-9). A second motivation was to provide a solution to the paradox of EINSTEIN, PODOLSKY, and ROSEN (1935). I shall not try to recapitulate their argument here, but shall only say that the paradox arises from the conjunction of three propositions: the complete describability of the physical world in quantum mechanical terms, the nonoccurrence of action-at-a-distance, and their famous ontological criterion ("If, without in any way disturbing a system, we can predict with certainty ... the value of a physical quantity, then there exists an element of physical reality corresponding to this physical quantity"). Einstein, Podolsky, and Rosen considered action-at-a-distance to be strongly disconfirmed by physical evidence, and they believed their ontological criterion to be part of the realistic Weltanschauung of working scientists. Consequently, they concluded that the only reasonable way out of the paradox is to reject the first premise and supplement the quantum mechanical description of physical systems with hidden variables. A third motivation was to provide a solution to the problem of measurement. If the physical world can be exhaustively described in quantum mechanical terms, then the superposition principle is applicable to macroscopic as well as to microscopic systems. In particular, it is possible for

594

A. SIDMONY

a measuring apparatus to be in a superposition of states, in each of which a macroscopic observable, such as the position of a pointer, has distinct values. Not only can such peculiar states of a measuring apparatus occur, but they must occur in quite ordinary circumstances. Hence, if the physical world can be exhaustively described in quantum mechanical terms, then the occurrence of a definite result of measurement is not in general an objective feature of the physical world. A hidden-variable theory, however, would permit one to say that the result of measurement in such circumstances is definite and objective prior to the registration of the result upon the consciousness of the observer (see, for example, BOHM & BUB, 1966). These three motivations make different demands upon a hidden-variable theory. The first evidently demands determinism, a particular consequence of which is that the outcome of every measurement is in principle definite: the value of an observable A being measured by a given apparatus is unequivocally determined by the states A and A' of the object and the apparatus, respectively (where the term 'state' refers to a complete specification, including both the quantum mechanical description and the hidden variables). The second motivation demands a hidden-variable theory with no action-at-a-distance, but it allows a stochastic hidden-variable theory, in which the complete specification of the states of object and apparatus determine only a statistical distribution of the outcomes of measurement. The third motivation permits either deterministic or stochastic, and either local or nonlocal theories. One can also classify hidden-variable theories as 'noncontextualistic' or 'contextualistic'. A noncontextualistic hidden-variable theory makes a kind of intrinsic ascription to each observable of the system when the state A is specified. In the case of a deterministic noncontextualistic hiddenvariable theory, the ascription is a definite value A(A) to each observable A, and therefore in any state A there is a simultaneous ascription of definite values to all observables, even though their simultaneous exhibition may be operationally impossible. In a noncontextualistic stochastic hiddenvariable theory the ascription is rather a probability distribution PAC ex) over the range of possible values ex of the observable A. The essential characteristic of noncontextualistic hidden-variable theories can be conveyed by considering several pieces of apparatus, each of which is ideal for measuring A from the standpoint of quantum mechanics, i.e., if the system can be described quantum mechanically as being in an eigenstate of A with eigenvalue ex, then each piece of apparatus will yield the result that the value of A is ex. Then, according to a noncontextualistic hidden-

HIDDEN-VARIABLE THEORIES

595

variable theory, the result which each of these pieces of apparatus yields for A is the same when the system is in any specified state A; if the theory is deterministic the same value A(A) results from each, and if the theory is stochastic the same statistical distribution Pi.(r:t.) results from each. The reason that this agreement is noteworthy is that there may be great differences of construction and of function among different pieces of ideal apparatus for the same observable. For example, one ideal apparatus for measuring the total angular momentum p may simultaneously measure i., whereas another ideal apparatus for measuring p may simultaneously measure jz. A noncontextualistic hidden-variable theory asserts that the results of measuring p with the two pieces of apparatus are independent of the 'context' supplied by the additional measurement, of i, or i., respectively. By contrast, a contextualistic hidden-variable theory is one in which the result yielded by an ideal apparatus depends upon 'context', and hence upon the state A' of the apparatus. If the contextualistic hiddenvariable theory is deterministic then the outcome of a measurement of A can be written A(A, A'); and if it is stochastic the probability distribution of outcomes can be written Pu,(r:t.). Since the functions A(A, A') and Pi, i.,(A) may depend upon A' in many different ways, there are evidently many different kinds of contextualistic hidden-variable theories. For instance, in one class of such theories the 'context' is the set cr of observables which is measured along with A, and hence the only feature of A' which is relevant is the set
596

A. SHIMONY

a Hilbert space of dimension 2 or less. The famous theorem of VON NEUMANN

(1955), on the nonexistence of hidden variables is not decisive, though it has a conclusion similar to that of the corollary, because it relies upon an additional premise which is not physically acceptable (see BELL, 1966). A new lease on life was given to the program of seeking hidden variables by BELL (1966), who argued for the physical reasonableness of contextualistic theories, by citing Bohr's admonitions about the relevance of the experimental arrangement to questions of physical reality. There is no doubt that consistent contextualistic hidden-variable theories exist; and furthermore there exist specific models, in which probability distributions are postulated for the hidden variables, which yield statistical predictions in exact agreement with those of quantum mechanics (see BELL, 1966; BOHM, 1952; SIEGEL WIENER & 1956). The extant models, however, are peculiar, and the essential question is therefore whether one can expect to find a physically reasonable contextualistic hidden-variable theory which agrees statistically with quantum mechanics. BELL'S (1965) own answer to this question is negative, if a necessary condition for being physically reasonable is a certain kind of locality. In order to explain his concept of locality we must consider a composite physical system consisting of two spatially well-separated components 1 and 2. Suppose {Aa } is a family of observables of 1 parametrized by a, and {B b } is a family of observables of 2 parametrized by b. It is the option of the experimenter to decide which A a and which B b, from the respective families are to be measured, and he prepares suitable apparatus for the purpose. According to a contextualistic hidden-variable theory, the outcome of the measurement of A a (or the distribution of outcomes) is determined by the state A of the composite system and by the state of the apparatus, and similarly for Bb • A theory is local in Bell's sense if the outcome (or distribution of outcomes) of the measurement of A a is independent of the value of the parameter b and of the state of the apparatus used to observe component 2, and similarly for the measurement of B b • (Actually, Bell defined locality only in the case of deterministic hidden-variable theories, but the definition just stated is the natural extension of this definition to include the case of stochastic hidden-variable theories.) It is certainly reasonable to believe that nature is local in Bell's sense, for the measurement of an observable of one localized component ought not be affected by the experimenter's operations upon another component localized elsewhere. Nevertheless, one cannot say that the failure of locality in Bell's sense entails action-at-a-distance, for once the pieces of apparatus

597

HIDDEN-VARIABLE THEORIES

for measuring A a and B b are set in place, there is usually plenty of time for the hidden variables (which may be extraordinarily subtle creatures) to communicate with each other using signals no faster than light. Now suppose that each A a and each Bb has a range of possible values in the interval [-1,1], and let Aa(A,).') be the average value of A a when the state A of the composite system and the state A' of the apparatus for component 1 are specified, and let BbO, A") be the average value of Bb when A and the state )." of the apparatus for component 2 are specified. Let (l(A), (la().'), and (lbO") be the probability distributions of the states A, A', and A". The condition of locality has been satisfied by making the distribution of ).' independent of b, that of A' independent of a, and that of A independent of both a and b. We may define a correlation function

Now reasoning along BELL'S lines (1971) leads (in remarkably few steps) to the inequality IP(a, b)-pea, b)I+P(a, b)+P(a, b)-2

~

O.

This inequality is independent of the details of the local hidden-variable theory, and it is independent of the choice of the distributions (l(A), (laO'), (lbO"). Nevertheless, it makes a definite prediction regarding a certain sum of correlations. Furthermore, there are situations in which the quantum mechanical analogue of the correlation function Pea, b), namely,

violates Bell's inequality. Hence, he has shown that no local hidden-variable theory, whether deterministic or stochastic, can agree with all the statistical predictions of quantum mechanics. This result seems to be in conflict with the expectations of Einstein, Podolsky, and Rosen, who argued that a local hidden-variable theory is needed to complete quantum mechanics, but not to correct it within its usual domain of applicability. By precluding the realization of the original program of Einstein, Podolsky, and Rosen, Bell's result gives rise to a new question: are the statistical predictions of quantum mechanics always empirically correct, particularly in the domain of correlation phenomena, where they conflict with the metaphysically appealing family of local hidden-variable theories? An experiment to answer this question, by examining the correlations of polarizations of pairs of photons emitted in atomic cascades, was designed by CLAUSER, HORNE, SHIMONY, and HOLT (1969). The experiment is being

598

A. SHIMONY

performed by Holt at Harvard and by Clauser, Commins, and Freedman at Berkeley. The latter group has some preliminary results, but no announcement will be made until more data are gathered. If this caution is understood, however, there is no harm in reporting what they have seen: the sum on the left-hand side of the above inequality was found to be .07 + .035, in agreement with the quantum mechanical prediction (for their arrangement) of .044, and in evident disagreement with Bell's inequality. An experiment of KASDAY, ULLMAN, and Wu (1970) on the scattering of photons produced in positronium annihilation, also implies a violation of Bell's inequality, if a suitable assumption is made about the selection of observed photon pairs from the ensemble of pairs emitted from the source. These experimental results are disappointing to advocates of local hidden-variable theories (and I must confess to sharing their disappointment). They are not entirely surprising, since some of the most spectacular successes of quantum mechanics have concerned correlation phenomena (for example, the energy levels of the helium atom and the collective behavior of conduction electrons in metals). Yet none of these successes had really been achieved in a situation where quantum mechanics conflicts with local hidden-variable theories, and as Einstein, Podolsky, and Rosen showed, quantum mechanics is extremely counterintuitive at this point of conflict. Understandably, therefore, the advocates of local hiddenvariable theories are very much concerned with the decisiveness of the experiments, and have raised questions about the proper conclusion to be drawn from them. One question has been hinted at above. Even if there is no action-at-adistance in nature, why must Bell's inequality hold, since there is always plenty of time for the hidden variables of one polarizer to communicate with those of the other polarizer, after the two have been installed and oriented? For an advocate of hidden variables to invoke such a conspiracy among the hidden variables sounds suspiciously like resorting to epicycles, and one is inclined to dismiss the conjecture on methodological grounds. However, an alternative answer is possible, for Clauser has suggested an ingenious device for changing the orientation of the polarizer axes while the photons are in flight by means of Kerr cells, thereby thwarting any conspiracy which employs signals of light velocity or slower. Another serious question is raised by the fact that in the experimental determination of the correlation Pea, b) most photon pairs are not counted. In the experiment with photons from an atomic cascade the photodetectors detect 10 to 20 percent of the incident photons, and hence only 1 to 4 percent

mDDEN-VARIABLE THEORIES

599

of the pairs. Could there not be a selection mechanism such that the number of pairs detected is proportional to the quantum mechanical prediction, even though the number of pairs passing through the pair of polarizers is not? The answer to this question is not known. Work is in progress on the mathematical possibility of a local hidden-variable theory, with a locally operating selection mechanism, which not only violates Bell's inequality but yields quantum mechanical counting rates. In the case of the experiment of Kasday, Ullman, and Wu a mathematical model of such a selection mechanism has been constructed (see KASDAY, 1971). Work is also in progress in designing variants of the cascade experiment using correlated pairs of particles which are easier to detect than photons, so that almost all produced by the source can be observed. It has also been objected that a hidden assumption, in addition to locality, has been smuggled in regarding the space of the states A: namely, that the space has sufficient structure to admit a Lebesgue measure, which is presupposed in speaking of probability distributions. This objection must be conceded. However, it is hard to see what an advocate of hidden variables would gain thereby, for without probability distributions he cannot speak at all of the statistical predictions of his hidden-variable theory, and he cannot even formulate the question of whether his theory agrees statistically with standard quantum mechanics. Although experimental evidence is not completely decisive against local hidden-variable theories, a considerable degree of stubbornness is needed to continue their advocacy, and one may profitably recall DUHEM'S (1954) maxim, "Good sense is the judge of hypotheses which ought to be abandoned". One may ask, however, what good sense judges about nonlocal hidden-variable theories, about which the correlation experiments imply nothing. The answer, I think, is that good sense does not say anything, and suspends judgment until the issue becomes more concrete. There are too many possible nonlocal hidden-variable theories, some conflicting with the statistical predictions of quantum mechanics and some not, to permit a sweeping judgment. One elegant family of nonlocal hidden-variable theories has been investigated in some detail by BOHM and BUB (1966) (their interest in this family having probably arisen more from its amenability to mathematical analysis than from physical heuristics). This family has the property of agreeing statistically with quantum mechanics after the hidden variables become 'randomized'. However, immediately after a measurement is performed the hidden variables are in a kind of nonequilibrium distribution, and

600

A. SHIMONY

hence new measurements performed before a certain relaxation time has elapsed can be expected to disagree statistically with quantum mechanics. Bohm and Bub conjecture that this relaxation time is of the order of hJkT, where T is the temperature of the apparatus. At room temperature the relaxation time would then be about 10- 1 3 seconds. PAPALIOLIOS (1967) ingeniously tested their conjecture by measuring the intensity of light passing through a stack of three extremely thin sheets of polaroid with varying orientations of their axes of polarization. The results were entirely in accordance with quantum mechanics, even though the transit time of light through each sheet of polaroid was about 7.5 x 10- 1 4 seconds. He remarks that "it is also possible to perform a more definitive test of Bohm and Bub's choice of hJkT as the relaxation time, by repeating the experiment at lower temperatures. The lack of a theoretical understanding of this choice of T, however, does not at this time justify cooling the apparatus to liquid air (or lower) temperatures". This remark of Papaliolios suggests a line of investigation which, if successful, would enhance the significance of his experimental result. It might possibly be shown by general considerations of thermodynamics and statistical mechanics (including such results as the fluctuation-dissipation theorem, which relates properties of irreversible processes to certain characteristics of fluctuations in equilibrium) that a relaxation time of the order of hJkT is to be expected in a large family of models. De Broglie's speculations about the thermodynamics of a subquantum medium (see DE BROGLIE, 1964) might be usable in such an investigation. If such a general theorem were established, then the findings of Papaliolios would be strong evidence against hidden-variable theories, nonlocal as well as local, in which thermodynamics is applicable to a subquantum medium. In the hands of Planck and Einstein thermodynamics considerations were of central importance in the earliest period of quantum mechanics, and it would be very satisfying if they could again be invoked on a fundamental question of principle. I must warn, however, that work done by Horne and myself so far along this line has yielded no success whatever. References F. J., 1971, A survey of hidden-variables theories (Physics Department, Purdue University, Lafayette, Ind., unpublished) BELL, J. S., 1965, On the Einstein Podolsky Rosen paradox, Physics, vol. 1, pp. 195-200 BELL, J. S., 1966, On the problem of hidden variables in quantum mechanics, Reviews of Modem Physics, vol. 38, pp. 447-452 BELINFANTE,

HIDDEN-VARIABLE THEORIES

60~

BELL, J. S., 1971, Introduction to the hidden variable question, in: Foundations of Quantum Mechanics: Rendiconti della Scuola Internazionale di Fisica "Enrico Fermi", ed. B. d'Espagnat (Academic Press, New York), pp. 17-1181 BOHM, D., 1952, Quantum theory in terms of "hidden" variables I, Physical Review, vol. 85, pp. 166-179 BOHM, D. and J. BUB, 1966, A proposed solution of the measurement problem in quantum mechanics by a hidden variable theory, Reviews of Modern Physics, vol. 38, pp. 453--469 CLAUSER, J. F., M. A. HORNE, A. SlDMONY and R. A. HOLT, 1969, Proposed experiment to test local hidden-variable theories, Physical Review Letters, vol. 23, pp. 880-884 DE BROGLIE, L., 1964, La thermodynamique de la particule isolee (Gauthier-Villars, Paris) DUHEM, P., 1954, The aim and structure of physical theory, translated by P. Wiener, (princeton University Press, Princeton) EINSTEIN, A., B. PODOLSKY and N. ROSEN, 1935, Can quantum-mechanical description of physical reality be considered complete?, Physical Review, vol. 47, pp, 770-780 GLEASON, A., 1957, Measures on closed subspaces of Hilbert space, Journal of Mathematics and Mechanics, vol. 6, pp. 885-893 JAMMER, M., 1966, The conceptual development of quantum mechanics (McGraw-HilI, New York) KASDAY, L., 1971, Experimental test of quantum predictions for widely separated photons, in: Foundations of Quantum Mechanics: Rendiconti della Scuola Internazionale di Fisica "Enrico Fermi," ed. B. d'Espagnat (Academic Press, New York), pp. 195-210 KASDAY, L., J. ULLMAN and C. S. Wu, 1970, The Einstein-Podolsky-Rosen argument: positron annihilation experiment, Bulletin of the American Physical Society, vol. 15. p. 586 KOCHEN, S. and E. P. SPECKER, 1967, The problem of hidden variables in quantum mechanics, Journal of Mathematics and Mechanics, vol. 17, pp. 59-88 PAPALIOLIOS, C., 1967, Experimental test of a hidden-variable quantum theory, Physical Review Letters, vol. 18, pp. 622-625 SIEGEL, A. and N. WIENER, 1956, "Theory of measurement" in differential-space quantum theory, Physical Review, vol. 101, pp. 429--432 VON NEUMAN, J., 1955, Mathematical foundations of quantum mechanics, translated by R. Beyer (princeton University Press, Princeton)

Note added in proof: Definite experimental results against local hidden-variable theories and supporting quantum mechanics have been announced by S. J. Freedman and J. F. Clauser, Physical Review Letters, vol. 28, pp. 938 (1972).