- Email: [email protected]
Application of probability limit theorems to measurements
Photogrammetria - Elsevier Publishing C o m p a n y , A m s t e r d a m - Printed in The Netherlands
A P P L I C A T I O N OF P R O B A B I L I T Y L I M I T T H E O R E M S T O MEASUREMENTS 1 S. W, HENRIKSEN
Raytheon Company, Alexandria, Va. (U.S.A.) (Received June 28, 1966)
SUMMARY
The growth of physics and mathematics as separate disciplines has at times made interchange of techniques between the two difficult. One area where interchange has been incomplete has been in statistics. There have been consequent misunderstandings of the limitations of (1) the law of large numbers and (2) the Gaussian convergence theorem, resulting in misinterpretation of physical data. This paper investigates the use of the law (1) and the theorem (2) in physics, shows that physical data cannot be made to fit exactly within the limitations of these two statements, and discusses a distribution (and its consequences) that can describe the statistics of physical data.
INTRODUCTION
Discrepancies between measurements on physical phenomena and predictions from "mathematical models" of these phenomena are common and we accept them; we do not expect our models to be exact because we know that knowledge is limited. Even if it were unlimited, we would still make do with inexact models because they are simpler and easier to handle. Not all the discrepancies arise from systematic but unknown effects; many are the result of random disturbances for which no practical model can account except statistically, and a major field of research is the finding of valid ways of statistically "averaging out' the random discrepancies. The physicist, in selecting the mathematical tools for constructing his model, does not look into the tool's theoretical strength as much as he does their utility, and this lack of fussiness does not weaken the purely mechanistic portions of his model. But because the statistical tools often depend heavily on the concepts of "infinite population" and of "confidence", neither of which appears in the physical world, the models are often shaky in their statistical parts and 1 Paper presented at the International Symposium on Spatial Aerotriangulation, February 2 8 - M a r c h 4, 1966, University of Illinois, U r b a n a , 111. (U.S.A.).
Photo~,,rammetria.
22 (1967) ~- 11
6
S . W . HENRIKSFN
predictions are made which are at variance not only with observations but witiq common sense. Since most physical experiments are made on limited population(i.e., within a short time span) so that observational discrepancies are difficult to, pick up, we must use the "'common sense" criterion in selecting statistical tools l'¢u improving the models,
THE CENTRAL I.IMIT THEOREM
In particular, a vast number of models are constructed and predictions made on the basis of uncritical application of the law of large numbers and the central limit theorem. As a result, statistical estimates are produced which givc an un-. warranted amount of confidence in the predictions themselves, i.e,, in the faithfulness of the mathematical model to the physical reality. Expected values with their standard deviations are computed, and are relied on to an extent far beyond what common sense tells us the observational equipment accuracy warrants. It is exactly as if we are given the average value of measurements made with a cheap foot rule, and the accuracy estimate is better than could be gotten using a precision interferometer! 1 place little credence in the tale of the physics instructor wht~ was in the habit of pulling a one inch brass cube from his pocket and asking ~ bystander to guess its weight; his average of several thousand such observation> agreed to a fraction of a milligram with that gotten by weighing on a chemical balance. But a great volume of published experimental results is based on assumptions identical with those justifying the one inch cube experiment, and the conditions under which such assumptions can or cannot be made should be clearly understood. To see how the law of large numbers and the normal convergence theorem (center limit theorem) are involved in the models, we look at the physical proto-. type and see that two distinct (but not necessarily independent) kinds of effect are p r e s e n t - - t h e phenomenon being measured and the instrument (with observert doing the measuring. Although the phenomenon's random variations are themselves discrete, we can for the moment assume continuous random variation. Such an assumption cannot be made about the random effects introduced by the measuring process, and we allow at least two kinds of limits on the smallness of the effect>. The first is the readout limit of the p r o c e s s - - t h e smallest division on the scale. the least significant figure written down, etc.; the second is the resolution limit of the p r o c e s s - - t h e smallest variation the observer can detect. Either one can be the larger, and the mathematical model takes account only of the larger; for the experiments, a larger readout limit means that he can increase the accuracy by further dividing his scale or by adding more figures, until he reaches the resolution limit; a larger resolution limit means that some of the digits in his measured value~ are meaningless. Existence of this limit means that the measurements variations come out in discrete block>, with the variations in each block having an (approx-
7
L I M I T THEOREM AND APPLICATIONS
imately) rectangular distribution as shown in Fig.1. The average value of x of the variations may or may not be equidistant from the upper and lower bounds; this is determined by the measuring instruments' structure. But, it will usually be invariant within each block and is independent of the average value of the phenomenon's variations within that block unless those variations are so large and infrequent that no observations exist within some blocks. Such a case leads to statistics of infinite-resultion measurements on finite samples and has little physical interest.
+f (x)
x
+x
Fig.l. Distribution of measurements within a single resolution of readout interval. Dashed curve: envelope of measurement distribution; solid curve: Gaussian distribution.
If we accept the rectangular distribution hypotheses (and it is easy to adapt the argument to other semi-discrete distributions of the same type) then role of the law of large numbers is obvious. We have, as the number N of measurements X increases:
N --->~
rN ~
n 0
- - e (X) < b
i
--~ 1
for any ,5 > 0
But X is not the weight of a cube, the length of a desk, the density or temperature of air, etc.; it is the ensemble of measurements on those variables, and E(X) of (X), as the average value of the ensemble of values of the phenomenon. If the measuring instrument enforces distribution like that of Fig.1 and if all detectable variations be within the limits of that distribution, the average value of the measurements will be (b--a)~2 regardless of the most probable or true value of X. From this it follows that no experiment can obtain a value ,Y closer to the true value X than is allowed by the largest resolution limit within the experiment. Nor can any set of measurements, however large, reduce the uncertainty in the true value to less than what is passed by the resolution "window". But, it is often argued, does not the normal convergence theorem allow us to do better, since almost any distribution for individual variates tends to give a Photogrammetria,
22 (19671 5 - l l
8
S. \ V .
tIENRIKSI:N
G a u s s i a n distribution for the average? A t first sight this seems reasonable, for if we l o o k at average values, which are m a d e up from sums of individual variate~, we find that the distribution of such averages are gotten by taking conw~lution~ of pairs of c o m p o n e n t s : iv,_
f,,(X)
/
:
[,.
,(X--_~j'~(j)
d:
so that, for an average of n rectangular distributions of width 1 and height J we have: II I1
.f, (X)
(.-.
\!
])'"(
) (A" -'
Ill
111
ftl)"
~
q] ~
~
for m < x ~ q m ~i t. If we shift the center to :, it is easy to find E ( X ) and E(X=), which turn out to ,* (.v)
5
." (x'-')
5=
" 12
bc (2'
(c.f. CRAMEI{, 1946.) As n increases, the function /,,(x) differs less and less form the Gaussian function, as the n o r m a l convergence t h e o r e m shows for the general case. Bu| ~ . must be very careful in applying these results to real measurements, in selecting we have assumed that il is the "'truc" value of the physical variable. The n u m b e r , we use in the equation, however, have been filtered through the measuring m-L strument; the resolution "'window" is not centered on 5 but merely brackct~ .: between limits characteristic of the instrument. These limits therefore act on dw distribution function ol the physical variable as if they had a Dirac ,~-funclion distribution; i.e., as if they had a G u a s s i a n distribution with zero variance:
a (x)
-
lira
I
,i=--~()
I 2,~=
~'
- - (x/,~)='
!3;
C o n s e q u e n t l y e q . l should bc m o d i f i e d to center, not on 5, but on ldl the instrumental resolution b o u n d closest to 5 (or next below or above, etc,L A n o t h e r equivalent way of looking at the p r o b l e m is to consider the actual distribution as a limiting case of the binomial. T h e distribution is defined a~: 211
.f,, (.¥)
2
""
I
P/~*l~.~,r~tmmclrid. 2 2 I ]9&71 ~ i :
9
L I M I T T H E O R E M AND A P P L I C A T I O N S
v = 1, for m < x < m + v. This gives, the m o m e n t s : (x) = [~] ~ (x") -
18n--
[~12 +
ll
12 +f C~)
I
Jtj/i //
+x
Fig.2. Measurement distribution vs. Gaussian distribution. Letting n increase, but at the same time increasing r, we arrive at a distribution f(x) = Ae [--b x--~.]" which has a shape like that in Fig.2, and whose E(X) and E(X ~) are: ~ (x) = [5] + l/, ,~ (x'-') -- ([S] + 1/2F +
1+120 12
where: 0
A~
[.z
--
5]'-'
e -~'
[nz --
~12
As can be seen, this distribution, like the modified binomial, is retained under addition. Furthermore, it shows explicitly that the resolution limit of the instrument limits the extent to which statistical analysis can locate £. The average of an infinite n u m b e r of measurements will not improve our knowledge of ~, but merely increases our confidence in the location of the interval within which 5 lies. The foregoing is easily extended to cover linear functions of the measured values, and the same conclusions are r e a c h e d - - t h e values found for the functions lie within limits that depend on the resolution limits of measuring instruments, and which do not approach the function values for the population mean of the independent variable.
EXAMPLE
A photograph is taken of the night sky in the direction of Betelgeuse. The photograph is placed in a measuring engine of 1 - - . least interval reading and the Photo~ramnwtria.
22 ( 1 9 6 7 ) 5 I1
10
S. W. HENRIKSI, N
coordinates (X~, Y,) of images of all stars between 8th and 3rd magnitude ar~' measured. Also measured are the coordinates ( X .... Y~,,) of four marks (on ~hc photograph) whose position with respect to the optical axis of the camera arc known. Then the equations: X = X ( ~ , a , l . al, u=. •.a+~) Y " }" (q, '3-.(, bl, h~. . h,,)
t4.i
can be used to find the parameters a;, hj from known values ol stellar" righ~ ascension and declination r~,, a.~ and corresponding X~, Y.,.coordinates of the image,. The direction of the optical axis is found by computing the axis's coordinatu, X,., Y,. from the coordinates X .... Y , , o f the marks and solving eq.4 for ,~,. ,~ As usually interpreted the statistics say that the variance x2=(cl of ,~,., ;~, is of the' form "2'-'U-'t == F / ( S - - n ~ where S in the number of star images measured, S - - n the. number of degrees of freedom of the system, and F is a function of the variance~ of the measurements and known directions ~.~, O.~. Hence the direction of the optical axis can be found with a variance as close to zero as we please by making 5 large enough. By the interpretation given in this article, the statistics say that 2u(~ I has the form E=(c) F~ ~ F z ( s - - n ) where F~ is a function of the variance of that part of the system which defines the smallest resolution possible. If the photograph shows a satellite against a star background, and if the photograph is measured on a measuring engine in the usual way, with digital readout, Ft is 0 . 2 9 lO v_, m, and F= can be taken to be in the neighborhood u~ 9 10 ~ t= m (ScnMm, 1965). If the number(s) of stars and measurements is taken as 100, the standard deviation of a satellite position is 0.6 l0 " m, and increasing the number of stars or measurements will not significantly diminish this. + - -
CONCLUSION
If the law of large numbers and the Gaussian convergence (central limit~ theorem are applicable without reservation to physical data, the present practice of deriving physical state values with variances several orders of magnitude smaller than the variances inherent in the measuring equipment is valid. If. on the prob.. ability limit theorems are restricted in application to those situations where convolution of rectangular distributions makes sense and by applying binomial or multinomial based statistics to the other situations, it is found that observation averages or functions of observation averages approach limiting values which arc to a certain extent independent of the average or true values of the phenomenon measured. Removal of biases from the system or use of smoothing procedures o( any kind does not change the values found except by multiplying of the observing system's resolution limit.
Photogranlmetr.t+ 22 I 1 ~ 6 7 ] "~ t!
LIMIT THEOREM AND APPLICATIONS
11
REFERENCES
CRAMER, H., 1946. Mathematical Methods of Statistics. Princeton Univ. Press, Princeton, N.J., 575 pp. EISENnARX, C., 1963. Realistic evaluation of the precision and accuracy of instrument calibration systems. J. Res. Natl. Bur. Std., C, 67(2): 161-189. FELLER, W., 1957. An Introduction to Probability Theory and its Applications, 2nd ed. Wiley, New York, N.Y., 461 pp. PARZEN. E., 1960. Modern Probability Theory and its Applications. Wiley, New York, N.Y., 474 pp. SCHMXO, H.. 1965. Accuracy aspects of a world-wide passive satellite triangulation system. Photogrammetric Eng., 31(1): 104-117.
Photogrammetria, 22 (I967) 5-11
A P P L I C A T I O N OF P R O B A B I L I T Y L I M I T T H E O R E M S T O MEASUREMENTS 1 S. W, HENRIKSEN
Raytheon Company, Alexandria, Va. (U.S.A.) (Received June 28, 1966)
SUMMARY
The growth of physics and mathematics as separate disciplines has at times made interchange of techniques between the two difficult. One area where interchange has been incomplete has been in statistics. There have been consequent misunderstandings of the limitations of (1) the law of large numbers and (2) the Gaussian convergence theorem, resulting in misinterpretation of physical data. This paper investigates the use of the law (1) and the theorem (2) in physics, shows that physical data cannot be made to fit exactly within the limitations of these two statements, and discusses a distribution (and its consequences) that can describe the statistics of physical data.
INTRODUCTION
Discrepancies between measurements on physical phenomena and predictions from "mathematical models" of these phenomena are common and we accept them; we do not expect our models to be exact because we know that knowledge is limited. Even if it were unlimited, we would still make do with inexact models because they are simpler and easier to handle. Not all the discrepancies arise from systematic but unknown effects; many are the result of random disturbances for which no practical model can account except statistically, and a major field of research is the finding of valid ways of statistically "averaging out' the random discrepancies. The physicist, in selecting the mathematical tools for constructing his model, does not look into the tool's theoretical strength as much as he does their utility, and this lack of fussiness does not weaken the purely mechanistic portions of his model. But because the statistical tools often depend heavily on the concepts of "infinite population" and of "confidence", neither of which appears in the physical world, the models are often shaky in their statistical parts and 1 Paper presented at the International Symposium on Spatial Aerotriangulation, February 2 8 - M a r c h 4, 1966, University of Illinois, U r b a n a , 111. (U.S.A.).
Photo~,,rammetria.
22 (1967) ~- 11
6
S . W . HENRIKSFN
predictions are made which are at variance not only with observations but witiq common sense. Since most physical experiments are made on limited population(i.e., within a short time span) so that observational discrepancies are difficult to, pick up, we must use the "'common sense" criterion in selecting statistical tools l'¢u improving the models,
THE CENTRAL I.IMIT THEOREM
In particular, a vast number of models are constructed and predictions made on the basis of uncritical application of the law of large numbers and the central limit theorem. As a result, statistical estimates are produced which givc an un-. warranted amount of confidence in the predictions themselves, i.e,, in the faithfulness of the mathematical model to the physical reality. Expected values with their standard deviations are computed, and are relied on to an extent far beyond what common sense tells us the observational equipment accuracy warrants. It is exactly as if we are given the average value of measurements made with a cheap foot rule, and the accuracy estimate is better than could be gotten using a precision interferometer! 1 place little credence in the tale of the physics instructor wht~ was in the habit of pulling a one inch brass cube from his pocket and asking ~ bystander to guess its weight; his average of several thousand such observation> agreed to a fraction of a milligram with that gotten by weighing on a chemical balance. But a great volume of published experimental results is based on assumptions identical with those justifying the one inch cube experiment, and the conditions under which such assumptions can or cannot be made should be clearly understood. To see how the law of large numbers and the normal convergence theorem (center limit theorem) are involved in the models, we look at the physical proto-. type and see that two distinct (but not necessarily independent) kinds of effect are p r e s e n t - - t h e phenomenon being measured and the instrument (with observert doing the measuring. Although the phenomenon's random variations are themselves discrete, we can for the moment assume continuous random variation. Such an assumption cannot be made about the random effects introduced by the measuring process, and we allow at least two kinds of limits on the smallness of the effect>. The first is the readout limit of the p r o c e s s - - t h e smallest division on the scale. the least significant figure written down, etc.; the second is the resolution limit of the p r o c e s s - - t h e smallest variation the observer can detect. Either one can be the larger, and the mathematical model takes account only of the larger; for the experiments, a larger readout limit means that he can increase the accuracy by further dividing his scale or by adding more figures, until he reaches the resolution limit; a larger resolution limit means that some of the digits in his measured value~ are meaningless. Existence of this limit means that the measurements variations come out in discrete block>, with the variations in each block having an (approx-
7
L I M I T THEOREM AND APPLICATIONS
imately) rectangular distribution as shown in Fig.1. The average value of x of the variations may or may not be equidistant from the upper and lower bounds; this is determined by the measuring instruments' structure. But, it will usually be invariant within each block and is independent of the average value of the phenomenon's variations within that block unless those variations are so large and infrequent that no observations exist within some blocks. Such a case leads to statistics of infinite-resultion measurements on finite samples and has little physical interest.
+f (x)
x
+x
Fig.l. Distribution of measurements within a single resolution of readout interval. Dashed curve: envelope of measurement distribution; solid curve: Gaussian distribution.
If we accept the rectangular distribution hypotheses (and it is easy to adapt the argument to other semi-discrete distributions of the same type) then role of the law of large numbers is obvious. We have, as the number N of measurements X increases:
N --->~
rN ~
n 0
- - e (X) < b
i
--~ 1
for any ,5 > 0
But X is not the weight of a cube, the length of a desk, the density or temperature of air, etc.; it is the ensemble of measurements on those variables, and E(X) of (X), as the average value of the ensemble of values of the phenomenon. If the measuring instrument enforces distribution like that of Fig.1 and if all detectable variations be within the limits of that distribution, the average value of the measurements will be (b--a)~2 regardless of the most probable or true value of X. From this it follows that no experiment can obtain a value ,Y closer to the true value X than is allowed by the largest resolution limit within the experiment. Nor can any set of measurements, however large, reduce the uncertainty in the true value to less than what is passed by the resolution "window". But, it is often argued, does not the normal convergence theorem allow us to do better, since almost any distribution for individual variates tends to give a Photogrammetria,
22 (19671 5 - l l
8
S. \ V .
tIENRIKSI:N
G a u s s i a n distribution for the average? A t first sight this seems reasonable, for if we l o o k at average values, which are m a d e up from sums of individual variate~, we find that the distribution of such averages are gotten by taking conw~lution~ of pairs of c o m p o n e n t s : iv,_
f,,(X)
/
:
[,.
,(X--_~j'~(j)
d:
so that, for an average of n rectangular distributions of width 1 and height J we have: II I1
.f, (X)
(.-.
\!
])'"(
) (A" -'
Ill
111
ftl)"
~
q] ~
~
for m < x ~ q m ~i t. If we shift the center to :, it is easy to find E ( X ) and E(X=), which turn out to ,* (.v)
5
." (x'-')
5=
" 12
bc (2'
(c.f. CRAMEI{, 1946.) As n increases, the function /,,(x) differs less and less form the Gaussian function, as the n o r m a l convergence t h e o r e m shows for the general case. Bu| ~ . must be very careful in applying these results to real measurements, in selecting we have assumed that il is the "'truc" value of the physical variable. The n u m b e r , we use in the equation, however, have been filtered through the measuring m-L strument; the resolution "'window" is not centered on 5 but merely brackct~ .: between limits characteristic of the instrument. These limits therefore act on dw distribution function ol the physical variable as if they had a Dirac ,~-funclion distribution; i.e., as if they had a G u a s s i a n distribution with zero variance:
a (x)
-
lira
I
,i=--~()
I 2,~=
~'
- - (x/,~)='
!3;
C o n s e q u e n t l y e q . l should bc m o d i f i e d to center, not on 5, but on ldl the instrumental resolution b o u n d closest to 5 (or next below or above, etc,L A n o t h e r equivalent way of looking at the p r o b l e m is to consider the actual distribution as a limiting case of the binomial. T h e distribution is defined a~: 211
.f,, (.¥)
2
""
I
P/~*l~.~,r~tmmclrid. 2 2 I ]9&71 ~ i :
9
L I M I T T H E O R E M AND A P P L I C A T I O N S
v = 1, for m < x < m + v. This gives, the m o m e n t s : (x) = [~] ~ (x") -
18n--
[~12 +
ll
12 +f C~)
I
Jtj/i //
+x
Fig.2. Measurement distribution vs. Gaussian distribution. Letting n increase, but at the same time increasing r, we arrive at a distribution f(x) = Ae [--b x--~.]" which has a shape like that in Fig.2, and whose E(X) and E(X ~) are: ~ (x) = [5] + l/, ,~ (x'-') -- ([S] + 1/2F +
1+120 12
where: 0
A~
[.z
--
5]'-'
e -~'
[nz --
~12
As can be seen, this distribution, like the modified binomial, is retained under addition. Furthermore, it shows explicitly that the resolution limit of the instrument limits the extent to which statistical analysis can locate £. The average of an infinite n u m b e r of measurements will not improve our knowledge of ~, but merely increases our confidence in the location of the interval within which 5 lies. The foregoing is easily extended to cover linear functions of the measured values, and the same conclusions are r e a c h e d - - t h e values found for the functions lie within limits that depend on the resolution limits of measuring instruments, and which do not approach the function values for the population mean of the independent variable.
EXAMPLE
A photograph is taken of the night sky in the direction of Betelgeuse. The photograph is placed in a measuring engine of 1 - - . least interval reading and the Photo~ramnwtria.
22 ( 1 9 6 7 ) 5 I1
10
S. W. HENRIKSI, N
coordinates (X~, Y,) of images of all stars between 8th and 3rd magnitude ar~' measured. Also measured are the coordinates ( X .... Y~,,) of four marks (on ~hc photograph) whose position with respect to the optical axis of the camera arc known. Then the equations: X = X ( ~ , a , l . al, u=. •.a+~) Y " }" (q, '3-.(, bl, h~. . h,,)
t4.i
can be used to find the parameters a;, hj from known values ol stellar" righ~ ascension and declination r~,, a.~ and corresponding X~, Y.,.coordinates of the image,. The direction of the optical axis is found by computing the axis's coordinatu, X,., Y,. from the coordinates X .... Y , , o f the marks and solving eq.4 for ,~,. ,~ As usually interpreted the statistics say that the variance x2=(cl of ,~,., ;~, is of the' form "2'-'U-'t == F / ( S - - n ~ where S in the number of star images measured, S - - n the. number of degrees of freedom of the system, and F is a function of the variance~ of the measurements and known directions ~.~, O.~. Hence the direction of the optical axis can be found with a variance as close to zero as we please by making 5 large enough. By the interpretation given in this article, the statistics say that 2u(~ I has the form E=(c) F~ ~ F z ( s - - n ) where F~ is a function of the variance of that part of the system which defines the smallest resolution possible. If the photograph shows a satellite against a star background, and if the photograph is measured on a measuring engine in the usual way, with digital readout, Ft is 0 . 2 9 lO v_, m, and F= can be taken to be in the neighborhood u~ 9 10 ~ t= m (ScnMm, 1965). If the number(s) of stars and measurements is taken as 100, the standard deviation of a satellite position is 0.6 l0 " m, and increasing the number of stars or measurements will not significantly diminish this. + - -
CONCLUSION
If the law of large numbers and the Gaussian convergence (central limit~ theorem are applicable without reservation to physical data, the present practice of deriving physical state values with variances several orders of magnitude smaller than the variances inherent in the measuring equipment is valid. If. on the prob.. ability limit theorems are restricted in application to those situations where convolution of rectangular distributions makes sense and by applying binomial or multinomial based statistics to the other situations, it is found that observation averages or functions of observation averages approach limiting values which arc to a certain extent independent of the average or true values of the phenomenon measured. Removal of biases from the system or use of smoothing procedures o( any kind does not change the values found except by multiplying of the observing system's resolution limit.
Photogranlmetr.t+ 22 I 1 ~ 6 7 ] "~ t!
LIMIT THEOREM AND APPLICATIONS
11
REFERENCES
CRAMER, H., 1946. Mathematical Methods of Statistics. Princeton Univ. Press, Princeton, N.J., 575 pp. EISENnARX, C., 1963. Realistic evaluation of the precision and accuracy of instrument calibration systems. J. Res. Natl. Bur. Std., C, 67(2): 161-189. FELLER, W., 1957. An Introduction to Probability Theory and its Applications, 2nd ed. Wiley, New York, N.Y., 461 pp. PARZEN. E., 1960. Modern Probability Theory and its Applications. Wiley, New York, N.Y., 474 pp. SCHMXO, H.. 1965. Accuracy aspects of a world-wide passive satellite triangulation system. Photogrammetric Eng., 31(1): 104-117.
Photogrammetria, 22 (I967) 5-11