- Email: [email protected]
The connection between the Barnard-Birnbaum Monte Carlo test and the two-sample Kolmogorov-Smirnov test
Mathematicsand Computersin SimulationXXVI (1984)20-22 North-Holland
20
THE CONNECTION BETWEEN THE BARNARD-BIRNBAUM THE TWO-SAMPLE KOLMOCOROV-SMIRNOV TEST
MONTE CARLO TEST AND
LE. ANGUS Hughes Aircraft Company, Fullerron, CA 92634, U.S.A.
The equivalence of the Barnard-Birnbaum
Monte Carlo test procedures to two-sample Kolmogorov-Smirnov
to simulated data is demonstrated and a generalization of the Barnard-Birnbaum
applied
test is suggesteti.
The Monte Carlo tests introduced by Barnard [l] and Birnbaum (31 may be described as follows. A statistic S is observed and it is desired to test the hypothesis H: S has cumulative distribution function (cdf) F versus the hypothesis A :S has cdf G. For now it will be assumed that for all X, F( .u) 2 G(s) and for at least one y, F(y) > G(y ) (the modifications necessary for the other one-tailed test, and for a two-tailed test will be straight-forward). In the Barnard-Birnbaum scenario, the distribution F is assumed intractable and/or lacks a simple closed expression, but it is feasible to generate an independent Monte Carlo sample of size N, say T,, T2,. ..,TV,from F.This sample, along with the original observation S which may have distribution F orG,is used to construct the test of H versus A defined by the following: reject M iff M/N < y
0)
where y is fixed, 0 < y < 1, and M is the number of observations in r’, T,, . . ., T,,, which are strictly greater than S. Of course, it is assumed that all cdf’s are continuous. This type of test has been studied by Hope [5), and quite extensively by Rutemiller and Schafer [8] in terms of its power. Marriott [6] investigated the power for various values of N and y which resulted in sizes of 4j.05 and 0.01. Marriott’s calculations were, however, conditioned on S. Birnbaum [3] has shown that the probability of rejecting H is
(1 - F(s)).‘(F(s))‘~-‘dP(S~s)
(2)
(where [ .] is the greatest integer function) which reduces to 1 +[N-y]
-~-I-N
(3)
when P(S
F(s). Of course, (3) gives the size of the test (1). Having generated T,, T,, . . . . TN.the problem may now be viewed as a two-sample problem with S, a sample of size 1, being the other sample. The Kolmogorov-Smirnov (Hodges [4]) two-sample test of H versus A is to reject H iff sup _,m,,,,(&(.~)8’:(x))>, 1 -y, 0
H iff 1 -&(S)
(4)
which is equivalent to (1). Note that an easy proof of (3) ensues since under H, 1 - &,(S) is uniformly distributed over the set { K/N; A’= 0, 1, . . , , N }. The Barnard-Birnbaum test for the other tail (F Q G) is 0378-4754/‘84/$3.00 ~3 1984. Eisevicr Science Publishers B.V. (North-Holland)
J. E. Angus / Barnard- Bimbaum Monte Carlo test and 2 -sample Kotmogorov - Smirnov test
21
shown to be equivalent to the corresponding one-sided, two-sample Kolmogorov-Smirnov test using similar arguments. The two-sided, equal tail area, Barnard-Bimbaum test (for the case where G( x ) # F(X) for some x is the only condition on G) is to
M/N(Y
reject H iff
(
or M/Ns+-y
(9
where O
max( PJS),
1 - k&(S)) 2 1 -y,
that is, iff 1 - RN(S) 2 1 - y or &(S) 2 ! - y -&ich is exactly (5). The probability of rejecting H in (5) is given by
-
(I- -Fw)J( F(s)) "-'+(F(s))'(l
F(s))~-')
dP(S,cs)
(6)
and the size of the test (5) is clearly 2(1+ [ Ny])/(l + N). Finally, the two-sided, unequal tail area Barnard-Birnbaum test, i.e.
is easily shown to be equivalent to the two sided, unequal tail area, two-s-ampleKolmogorov-Smirnov test which is
where 0 < y,, yz < 1, y, + y2 < (N - 1)/N. The probability of rejecting H in (7) is INYZI
(~-+))‘(F(s))~-‘+
c
j-0
N Li
(1
(F(s))‘{1 -F(s))~-’
(8)
and the size of the test (7) is (2+~NY,l+P92lMN+
1).
In view of these results a generalization of the Barnard-Birnbaum tests to the case where more than one original observation of S (say S, , Sz, . . . , SK; K >, 2) is available is only a matter of applying the appropriate Kolmogorov-Smirnov tH-u-sample test. It is also worth noting that when N -+ 00, the tests (1) or (4) become equivalent to the test which rejects H if S is too large which is, of course, optimal in terms of power when S possesses a monotone likelihood
22
J E. Angus / Barnard - Birttbaum Monte Carlo test and 2 -sample Kolmoclwa, -Smirnou test
ratio. This was verified in the Monte Carlo study of Rutemiller and Schafer (8). However, in the generalization discussed above, as N + 00 the test becomes equivalent to a classical one-sample Kolmogorov-Smirnov test which is suboptimal for testing two simple hypotheses. Finally, it should be emphasized that the conclusion of this note is not that the Barnard-Birnbaum test is nothing new. On the contrary, the Barnard-Birnbaum test is an ingenious blending of computer applications and statistical theory in order to solve otherwise intractable problems. Also, the Kolmogorov-Smirnov two-sample test has been singled out for study in this note because of its long starding popularity and reputation. Other two-sample tests based on ranks could serve equally well as test procedures in the generalization of the Barnard-Birnbaum scenario when there is more than one original observatron. In fact, the Cramer-von Mises two-sample test for the two-sided alternative reduces to the ;.wo-sample Kolmogorov-Smirnov when one sample is of size one, and thus also provides a natural generalization to the Barnard-Birnbaum two-sided test procedure. However, another advantage to using the Kolmogorov-Smirnov type statistics is that they cover both one-sided alternatives, and the two-sided alternative.
References 111 GA. Barnard, J. Roy. Statist. Sec. Ser. 825 (1963) 294 (In discussion). (21 J. Besag and P.S. Diggle. Simple Monte Carlo tests for spatial patterns, A@.
Statist. 26 (1977) 327-333. (3) Z.W. Birnbaum. (1974) Computers and unconventional test statistics, in: P. Proschsn and R.J. Serfling, Eds., Reliubili~r und Biometry (SIAM, Philadelphia, 1974). [4j J.L. Hodges, The significance probability of the Smirnov two-sample test, Ark&. ht. .$ (1957) 469-486. (S] A.C.A. Hope, A simplified Monte Carlo significance test procedure, J. Roy. Statist. SOL. Ser. 830 (1968) 582-598. [6] F.H.C. Marriott, Barnard’s Monte Carlo tests: how many simulations? Appl. Statist. 28 (1979) 75-77. [7) B.D. Ripley, Modelling spatial patterns, J. Rqv. Statist. Sot. Ser. B39 (1977) 172-212. [Sj H.C. Rutemiller and R.E. Schafer, Simulation studies of the power of Birnbaum’s test, J. Statist. Comput. Sim. 9 (1979) 261-281.
20
THE CONNECTION BETWEEN THE BARNARD-BIRNBAUM THE TWO-SAMPLE KOLMOCOROV-SMIRNOV TEST
MONTE CARLO TEST AND
LE. ANGUS Hughes Aircraft Company, Fullerron, CA 92634, U.S.A.
The equivalence of the Barnard-Birnbaum
Monte Carlo test procedures to two-sample Kolmogorov-Smirnov
to simulated data is demonstrated and a generalization of the Barnard-Birnbaum
applied
test is suggesteti.
The Monte Carlo tests introduced by Barnard [l] and Birnbaum (31 may be described as follows. A statistic S is observed and it is desired to test the hypothesis H: S has cumulative distribution function (cdf) F versus the hypothesis A :S has cdf G. For now it will be assumed that for all X, F( .u) 2 G(s) and for at least one y, F(y) > G(y ) (the modifications necessary for the other one-tailed test, and for a two-tailed test will be straight-forward). In the Barnard-Birnbaum scenario, the distribution F is assumed intractable and/or lacks a simple closed expression, but it is feasible to generate an independent Monte Carlo sample of size N, say T,, T2,. ..,TV,from F.This sample, along with the original observation S which may have distribution F orG,is used to construct the test of H versus A defined by the following: reject M iff M/N < y
0)
where y is fixed, 0 < y < 1, and M is the number of observations in r’, T,, . . ., T,,, which are strictly greater than S. Of course, it is assumed that all cdf’s are continuous. This type of test has been studied by Hope [5), and quite extensively by Rutemiller and Schafer [8] in terms of its power. Marriott [6] investigated the power for various values of N and y which resulted in sizes of 4j.05 and 0.01. Marriott’s calculations were, however, conditioned on S. Birnbaum [3] has shown that the probability of rejecting H is
(1 - F(s)).‘(F(s))‘~-‘dP(S~s)
(2)
(where [ .] is the greatest integer function) which reduces to 1 +[N-y]
-~-I-N
(3)
when P(S
F(s). Of course, (3) gives the size of the test (1). Having generated T,, T,, . . . . TN.the problem may now be viewed as a two-sample problem with S, a sample of size 1, being the other sample. The Kolmogorov-Smirnov (Hodges [4]) two-sample test of H versus A is to reject H iff sup _,m,,,,(&(.~)8’:(x))>, 1 -y, 0
H iff 1 -&(S)
(4)
which is equivalent to (1). Note that an easy proof of (3) ensues since under H, 1 - &,(S) is uniformly distributed over the set { K/N; A’= 0, 1, . . , , N }. The Barnard-Birnbaum test for the other tail (F Q G) is 0378-4754/‘84/$3.00 ~3 1984. Eisevicr Science Publishers B.V. (North-Holland)
J. E. Angus / Barnard- Bimbaum Monte Carlo test and 2 -sample Kotmogorov - Smirnov test
21
shown to be equivalent to the corresponding one-sided, two-sample Kolmogorov-Smirnov test using similar arguments. The two-sided, equal tail area, Barnard-Bimbaum test (for the case where G( x ) # F(X) for some x is the only condition on G) is to
M/N(Y
reject H iff
(
or M/Ns+-y
(9
where O
max( PJS),
1 - k&(S)) 2 1 -y,
that is, iff 1 - RN(S) 2 1 - y or &(S) 2 ! - y -&ich is exactly (5). The probability of rejecting H in (5) is given by
-
(I- -Fw)J( F(s)) "-'+(F(s))'(l
F(s))~-')
dP(S,cs)
(6)
and the size of the test (5) is clearly 2(1+ [ Ny])/(l + N). Finally, the two-sided, unequal tail area Barnard-Birnbaum test, i.e.
is easily shown to be equivalent to the two sided, unequal tail area, two-s-ampleKolmogorov-Smirnov test which is
where 0 < y,, yz < 1, y, + y2 < (N - 1)/N. The probability of rejecting H in (7) is INYZI
(~-+))‘(F(s))~-‘+
c
j-0
N Li
(1
(F(s))‘{1 -F(s))~-’
(8)
and the size of the test (7) is (2+~NY,l+P92lMN+
1).
In view of these results a generalization of the Barnard-Birnbaum tests to the case where more than one original observation of S (say S, , Sz, . . . , SK; K >, 2) is available is only a matter of applying the appropriate Kolmogorov-Smirnov tH-u-sample test. It is also worth noting that when N -+ 00, the tests (1) or (4) become equivalent to the test which rejects H if S is too large which is, of course, optimal in terms of power when S possesses a monotone likelihood
22
J E. Angus / Barnard - Birttbaum Monte Carlo test and 2 -sample Kolmoclwa, -Smirnou test
ratio. This was verified in the Monte Carlo study of Rutemiller and Schafer (8). However, in the generalization discussed above, as N + 00 the test becomes equivalent to a classical one-sample Kolmogorov-Smirnov test which is suboptimal for testing two simple hypotheses. Finally, it should be emphasized that the conclusion of this note is not that the Barnard-Birnbaum test is nothing new. On the contrary, the Barnard-Birnbaum test is an ingenious blending of computer applications and statistical theory in order to solve otherwise intractable problems. Also, the Kolmogorov-Smirnov two-sample test has been singled out for study in this note because of its long starding popularity and reputation. Other two-sample tests based on ranks could serve equally well as test procedures in the generalization of the Barnard-Birnbaum scenario when there is more than one original observatron. In fact, the Cramer-von Mises two-sample test for the two-sided alternative reduces to the ;.wo-sample Kolmogorov-Smirnov when one sample is of size one, and thus also provides a natural generalization to the Barnard-Birnbaum two-sided test procedure. However, another advantage to using the Kolmogorov-Smirnov type statistics is that they cover both one-sided alternatives, and the two-sided alternative.
References 111 GA. Barnard, J. Roy. Statist. Sec. Ser. 825 (1963) 294 (In discussion). (21 J. Besag and P.S. Diggle. Simple Monte Carlo tests for spatial patterns, A@.
Statist. 26 (1977) 327-333. (3) Z.W. Birnbaum. (1974) Computers and unconventional test statistics, in: P. Proschsn and R.J. Serfling, Eds., Reliubili~r und Biometry (SIAM, Philadelphia, 1974). [4j J.L. Hodges, The significance probability of the Smirnov two-sample test, Ark&. ht. .$ (1957) 469-486. (S] A.C.A. Hope, A simplified Monte Carlo significance test procedure, J. Roy. Statist. SOL. Ser. 830 (1968) 582-598. [6] F.H.C. Marriott, Barnard’s Monte Carlo tests: how many simulations? Appl. Statist. 28 (1979) 75-77. [7) B.D. Ripley, Modelling spatial patterns, J. Rqv. Statist. Sot. Ser. B39 (1977) 172-212. [Sj H.C. Rutemiller and R.E. Schafer, Simulation studies of the power of Birnbaum’s test, J. Statist. Comput. Sim. 9 (1979) 261-281.