THE LANCET
analysis will not fulfil that need. Public health workers and health-care providers must have accurate information on the reproducibility, sensitivity, specificity, and cost of tests. Estimates of test properties must be based on valid methodology. Acceptance of discrepant analysis to evaluate a test for C trachomatis could have set a precedent for using a biased methodology to evaluate screening tests for other diseases (eg, herpes simplex virus infection and tuberculosis) Hadgu has spotted an error that went unnoticed by the original investigators, by institutional review boards, and by journal peer reviewers. His work will make all of us examine the methods used to determine test characteristics much more closely in the future. Daniel L McGee, *Gladys H Reynolds Biostatistics Section, Department of Preventive Medicine, Loyola University Medical Centre, Chicago, IL; and *Statistics in Epidemiology Section, American Statistical Association, 1600 Clifton Road, Atlanta, GA 30333, USA
1
Hadgu A. The discrepancy in discrepant analyses. Lancet 1996; 348: 592–93.
SIR—Hadgu’s article on discrepant analysis1 for the evaluation of new methods for diagnosing genital infection by Chlamydia trachomatis makes use of results obtained by DNA amplification (ligase chain [LCR] or polymerase chain [PCR] reactions) in comparison with cell culture. Discordant results are usually resolved by additional methods. This may, according to Hadgu, introduce bias in favour of the new test. The article is timely. We accept his analysis and his statement about the risk of overestimating the sensitivity of the new tests (LCR, PCR) to the disadvantage of cell culture (94·4% vs 65·0% in the article Hadgu refers to2). We have more than 3 years of experience with LCR and PCR and have found that cell culture has given by and large the same yields as a combination of two or even three DNAamplification tests—ie, false negatives and false positives within the range 5–8% (and, thus, sensitivities and predictive values of positive tests in the range 92–95%). There is no denying the fact that there are strong commercial interests to promote amplification tests, by pointing out, for example, that DNA from one to ten organisms will suffice whereas 1000 or more organisms are required for cell-culture and over 10 000 for antigen-detection tests. However, there is a discrepancy between mathematical assumptions and the real sensitivity of the test in a clinical setting. Specimens are handled differently for cell culture and for PCR/LCR. Swabs taken for cell culture are transported in 1·5 mL buffer, and then further diluted with 1·0 mL of culture medium. From this suspension 1·0 mL is transferred to each one of two wells of a 24-well culture plate (ie, 40% of the clinical specimen to each well). The swab for PCR is collected in 1·0 mL buffer from which 0·1 mL is transferred to a tube containing 1·9 mL diluent/lysis buffer. 50 µL is taken for amplification. For LCR the swab is transported in 0·5 mL diluent from which, after heating of the specimen, 0·1 mL is transferred for amplification. This means that if there were 10 000 elementary bodies in a clinical specimen 2⳯4000 would be applied to the cell-culture wells, 25 to PCR, and 2000 to LCR. In an established clinical infection Number
Test pattern Culture
PCR (cervical)
PCR (urine)
19 5 3 1 1 334
+ + ⳮ ⳮ ⳮ ⳮ
+ ⳮ + ⳮ + ⳮ
+ + + + ⳮ ⳮ
Table: Patterns of results of three diagnostic tests in 363 women
1308
the concentration of organisms is usually5 at least 100 000 per mL (secretion) or per g (tissue). In a model system when we titrated equal amounts of freshly passaged clinical strains of C trachomatis we found that cell culture, PCR (Amplicor), and LCR (Abbott) were equally sensitive within one ten-fold dilution step. Good cell culture can be just as sensitive as amplification but no test is perfect and none detects 100% of infected patients. The more you test the more you will find. Table 1 gives findings from one of our trials of invasive (endocervical/urethral) sampling tested by PCR. Culture of endocervical/urethral specimens and PCR of endocervical specimens had similar sensitivities, and a few more positives were obtained if urine was also examined. In an expanded study3 we used culture and three amplification tests (LCR and two PCR) on all specimens; all four performed equally well and no more than 1% of culture-negative patients were positive by PCR or LCR and vice versa. High-quality cell culture (well-organised sampling and transport, mycoplasma-free culture, culturing at least 0·5–1·0 mL of the specimen, and detection of inclusions by fluorescent genus or species-specific antibodies) can still be used as reference method, is cost-effective in comparison with DNA amplification at present, and should always be used in legal situations. However, DNA amplification is an excellent adjunctive test, preferably for non-invasive (urine) specimens from males but also from women when genital swabs cannot be obtained. They are also good alternatives for laboratories where other methods do not seem to perform equally well. *Dan Danielsson, Lars Falk, Kenneth Persson Department of Clinical Microbiology and Immunology, STD Outpatient Clinic, Department of Dermatovenereology, Örebro Medical Centre Hospital, S-701 85 Örebro, Sweden; and Department of Clinical Microbiology, University of Lund, Malmö General Hospital
1 2
3
Hadgu A. The discrepancy in discrepant analysis. Lancet 1996; 348: 592–93. Schachter J, Stamm WE, Quinn TC, Andrews WW, Burczac JD, Lee HH. Ligase chain reaction to detect Chlamydia trachomatis infection of the cervix. J Clin Microbiol 1994; 32: 2540–43. Falk L, Fogelqvist M, Foglé L, Danielsson D. Diagnosis of C trachomatis infection in men and women by culture compared with manual and semiautomated nucleic acid amplification techniques (AmplicorTM, LCxR/LCRTM) of genitourinary specimens and first void urine. In: Proceedings of Third Meeting of the European Society for Chlamydia Research (Vienna, Austria, Sept 11–14, 1996). Bologna: Editrice Esculapio, 1996: 290.
SIR—Hadgu raises concerns about how diagnostic tests for Chlamydia trachomatis are evaluated.1 He alleges that use of discrepant analysis strongly biases these evaluations in favour of new tests, with negative connotations for standard tests. We think that these concerns are “much ado about very little”. We and others have used discrepant analysis with DNA amplification tests for C trachomatis because this approach provides the most accurate and practical means of understanding the performance of these new and highly sensitive assays.2,3 Hadgu offers no specific alternative. The choice seems to be: (1) the imperfect “gold standard” (culture), (2) discrepant analysis as we and others have applied it, or (3) use a “tie breaker” not only with the culture negative/amplification positive specimens but also with specimens that are negative in both assays. We maintain that (2) is the most reasonable approach. The variable sensitivity of cell culture is widely known and culture alone cannot be used as a “gold standard” for comparison with the more sensitive amplification tests; to do so produces sensitivity and specificity figures for the amplification assays that are meaningless. Hadgu’s preference—a tie breaker test such as
Vol 348 • November 9, 1996
THE LANCET
the MOMP gene assay for all negative/negative specimens generates a huge and expensive workload. Further, in our laboratories, testing by LCR for the MOMP gene has yielded only three uniquely positive tests out of 813 assays done in parallel with LCR for plasmid gene. In our study that Hadgu cites2 none of 387 specimens were LCR positive for MOMP and negative for plasmid gene. Direct fluorescent antibody (DFA) testing has been even less fruitful; 1% false positives (24/2421) were found on testing double negatives in one study and not one true positive was identified. Because discrepancy analysis requires two positive tests before the specimen is accepted as truly positive, the probability that a “double negative” will be positive by both LCR (MOMP) and DFA is very low and there is very little to be gained by testing large numbers of negative/negative specimens. Discrepant analysis has been used for PCR and LCR assays for other microorganisms.4,5 In the sixth paragraph of his paper Hadgu seems to create a situation based on our data,2 where an increase in culture positivity to 34% results in more LCR false negatives than false positives. This was done by simply multiplying the number of all culture-positive specimens by a number sufficient (approximately 6·5) to generate the example he seeks. He does not recognise that apparent LCR false positives will increase as the culture positivity increases (apparent is used because most can be confirmed as true positives). His conclusion that sensitivity of culture based on discrepant analysis is “always greater than the corresponding sensitivity of LCR,” is not correct. Our results show that LCR was more sensitive than culture. Hadgu states that LCR for the MOMP gene, which provides information on chlamydial genes via a molecular target different from that in plasmid LCR, should not be used because it is not Food and Drug Administration approved. Failure to use new technology to evaluate a molecular diagnostic test strikes us as counterproductive. *Julius Schachter, Walter E Stamm, Thomas C Quinn *Chlamydia Research Laboratory, Department of Laboratory Medicine, San Francisco General Hospital, San Francisco, CA 94110, USA; Department of Medicine, University of Washington, Seattle; and Department of Medicine Johns Hopkins University, Baltimore
1 2
3
4
5
Hadgu A. The discrepancy in discrepant analysis. Lancet 1996; 348: 592–93. Schachter J, Stamm WE, Quinn TC, Andrews WW, Burczak JD, Lee HH. Ligase chain reaction to detect Chlamydia trachomatis infection of the cervix. J Clin Microbiol 1994; 32: 2540–43. de Barbeyrac B, Pellet I, Dutilh B, Bebear C, Dumon B, Geniaux M. Evaluation of the amplicor Chlamydia trachomatis test versus culture in genital samples in various prevalence populations. Genitourin Med 1994; 70: 162–66. Smith KR, Ching S, Lee H, et al. Evaluation of ligase chain reaction for use with urine for identification of Neisseria gonorrhoeae in females attending a sexually transmitted disease clinic. J Clin Microbiol 1995; 33: 455–57. Orle KA, Gates CA, Martin DH, Body BA, Weiss J. Simultaneous PCR detection of Haemophilus ducreyi, Treponema pallidum, and herpes simplex virus types 1 and 2 from genital ulcers. J Clin Microbiol 1996; 34: 49–54.
Author’s reply SIR—McGee and Reynolds are correct to note that upwardly biased estimates of sensitivity and specificity could have “profound significance for medicine”. Even a 1% decrease in specificity will result in a substantial amount of unnecessary treatment and follow-up of false-positive individuals. In urine from females without symptoms the specificity of LCR is 95·2%, according to the Food and Drug Administrationapproved package insert. With discrepant analysis by the MOMP gene test specificity would have been put at over 99·5%, a difference that has serious implications for
Vol 348 • November 9, 1996
screening, so Schachter and colleagues are wrong to imply that my article is “much ado about very little”. Danielsson and colleagues are correct to note that the additional tests should be used on all patients, and in that situation statistical modelling can help to estimate test performance where the gold standard is imperfect.1,2 Thus, Schachter and colleagues are wrong in stating that I offer no specific alternative—and even without an alternative, mistakes should still be pointed out. A longer, more mathematical version of my paper has been accepted by a statistical journal.3 According to Danielsson and colleagues, less than 1% of the tissue-culture negatives were positive by PCR or LCR. This is interesting because in Birmingham, Alabama,3 more than 10% of tissue-culture negatives were positive by LCR, whereas in the Seattle laboratory the comparable figure was 0·7%. This indicates suboptimal culture or LCR testing. Schachter and colleagues’ assertion that additional sequential testing should be done only on the apparent false positives and not on samples that are negative by both LCR and culture is also incorrect. Schachter et al4 claim that, in a side-by-side comparison, the LCR MOMP gene test had a specificity of 100%, which by definition implies that all MOMP positive results are true positives. Thus, if any sample that is culture and LCR negative is MOMP positive it is a true positive by their own definition. Secondly, assuming all negative/negative samples are uninfected is equivalent to claiming the LCR test has a sensitivity of 100% in culture negative but infected individuals—yet the sensitivity of LCR in culture positive individuals is 80–90%. In discrepant analysis the additional tests must be technologically and statistically independent of both LCR and tissue culture but it will still be biased even if the resolution test is perfect, as long as the discrepant resolution is restricted to the apparent false positives.3 Medical technology assessment requires a good understanding of statistical techniques and mathematical modelling, and interdisciplinary collaboration is essential. DNA amplification may be one of the greatest scientific developments of the past 25 years. The tests based on it (LCR and PCR) deserve better evaluation than discrepant analysis can often achieve. *Alula Hadgu Department of Health and Human Services, Public Health Service, Centers for Disease Control and Prevention, Atlanta, GA 30333, USA
1
2
3 4
Qu Y, Tan M, Kutner MH. Random effects models in latent class analysis for evaluating accuracy of diagnostic tests. Biometrics 1996; 52: 797–810. Hadgu A, Qu Y, Sensitivity and specificity of chlamydia tests using latent class analysis. The International Biometric Society, March 19–20, 1996. Abs 5. Hadgu A. Bias in the evaluation of DNA-amplification tests for detecting Chlamydia trachomatis. Stats Med (in press). Schachter J, Stamm WE, Quinn TC, Andrews WW, Burczak JD, Lee HH. Ligase chain reaction to detect Chlamydia trachomatis infection of the cervix. J Clin Microbiol 1994; 32: 2540–43.
Platelet glycoprotein IIIa gene polymorphism and myocardial infarction S IR —A study by Weiss and colleagues 1 reported an approximate two-fold increase in prevalence of A2 allele of the PlA polymorphism in the platelet glycoprotein (GP) IIIa gene amongst patients with a history of myocardial infarction (MI) or unstable angina. The finding that the PlA2 allele was a significant risk factor for coronary thrombosis, which was strongest when disease onset was before 60 years (odds ratio 6·2),1 prompted us to investigate its possible
1309