- Email: [email protected]
Letter to the Editor on Cohen et al. (Letter commenting on: J Clin Epidemiol. 2015;68:299–306)
Letters to the Editor / Journal of Clinical Epidemiology 69 (2016) 248e268
Martin Boeker Edith Motschall Werner Vach* Center for Medical Biometry and Medical Informatics University Medical Centre Freiburg Stefan-Meier-Str. 26, Freiburg D-79104 Germany *Corresponding author. Tel.: þ49-761-203-6722; fax: þ49-761-203-6711. E-mail address: [email protected] (W. Vach)
References [1] Hausner E, Guddat C, Hermanns T, Lamperta U, Waffenschmidt S. Development of search strategies for systematic reviews: validation showed the noninferiority of the objective approach. J Clin Epidemiol 2015;68:191e9. [2] Booth A. Unpacking your literature search toolbox: on search styles and tactics. Health Info Libr J 2008;25(4):313e7. [3] Atkinson KM, Koenka AC, Sanchez CE, Moshontz H, Cooper H. Reporting standards for literature searches and report inclusion criteria: making research syntheses more transparent and easy to replicate. Res Synth Methods 2015;6:87e95. [4] Lefebvre C, Glanville J, Wieland LS, Coles B, Weightman AL. Methodological developments in searching for studies for systematic reviews: past, present and future? Syst Rev 2013;2:78. [5] Li L, Tian J, Tian H, Moher D, Liang F, Jiang T, et al. Network meta-analyses could be improved by searching more sources and by involving a librarian. J Clin Epidemiol 2014;67:1001e7. [6] Rethlefsen ML, Murad MH, Livingston EH. Engaging medical librarians to improve the quality of review articles. JAMA 2014;312:999e1000. [7] Rethlefsen ML, Farrell AM, Osterhaus Trzasko LC, Brigham TJ. Librarian co-authors correlated with higher quality reported search strategies in general internal medicine systematic reviews. J Clin Epidemiol 2015;68:617e26. http://dx.doi.org/10.1016/j.jclinepi.2015.05.022
Letter to the Editor on Cohen et al. (Letter commenting on: J Clin Epidemiol. 2015;68:299e306) To the Editor: In describing and applying Cochran’s Q statistic, Cohen et al. [1] make several substantial factual errors. From the description in Section 2, readers might infer that Cochran [2] discusses the use of Q with likelihood ratios (LRs) and other measures derived from 2 2 contingency tables. He does not. That article discusses combining estimates such as ‘‘a simple mean of the observations, as in a physical determination, or the difference between the means of two treatments, as in a comparative experiment, or a median lethal dose, or a regression coefficient.’’ In addition to the estimate, each experiment produces an estimate of the corresponding standard error or variance, accompanied by its degrees of freedom. For the sorts of estimates that Cochran discusses, the degrees of freedom
257
ordinarily come from a mean square (eg, the sample variance or the residual mean square of a regression). Importantly, the estimated variance is ordinarily independent of the estimate itself. Section 2 indicates that Cochran’s Q statistic follows a chi-squared distribution on k 1 degrees of freedom (ie, in the absence of heterogeneity Qwc2k1 ). In fact, as Cochran explains, the null distribution of Q only approaches c2k1 when the numbers of degrees of freedom from the individual experiments become large. Cohen et al. do not cite evidence to show that the values of N in their simulation study are large enough for that approximation to be adequate. Kulinskaya et al [3,4]. have investigated the null distribution of Q and have shown that the nature of its departure from c2k1 depends on the measure of effect. They give results for standardized mean difference [3] and risk difference [4]. To my knowledge, results for the LR are not available. The phrase ‘‘Cochran’s Q test’’ gives the impression that Cochran [2] uses Q as the basis for a test of heterogeneity. In fact, after discussing Q in that article, he uses a different statistic to test for heterogeneity. Section 5 states that ‘‘the Q test was originally proposed by Cochran to assess heterogeneity between subgroups in a study.’’ In fact, as the title indicates, the subject of Cochran’s article was combining estimates from experiments. Inverse-variance weights based on estimated variances are a frequent source of difficulty in meta-analysis; one cannot generally use estimated variances as if they were the known true variances. That assumption allowed Biggerstaff and Jackson [5] to obtain the exact distribution of a statistic that they call Q, but that statistic is not Cochran’s Q (in which the weights are based only on estimated variances). For LRs, it is also a problem that the logarithm of an LR and its estimated variance are related (Cohen et al. [1] give the formulas in Appendix A). In assessing heterogeneity of LRs, one can avoid the difficulties of inverse-variance weights by using a standard model from discrete multivariate analysis [6]. For G subgroups, the data form a 2 2 G table. Homogeneity of the positive likelihood ratio (PLR), for example, requires that (in the underlying probabilities) the result on the index test and the subgroup be conditionally independent, given the classification from the reference standard. It is easy to fit that model, calculate the usual chi-squared measure of its fit, and refer it to the chi-squared distribution on 2(G 1) degrees of freedom. The properties of such chisquared tests have been well studied. For the first two cases of PLR in Table 3 (prevalence Z 0.30, N Z 300, PLR Z 3.00 in group 1, and PLR Z 4.00 and 6.03 in group 2), the chi-squared test (on two degrees of freedom) yields P-values that differ substantially, and in opposite directions, from those of the test based on Q: 0.521 instead of 0.295 and 0.00925 instead of 0.040, respectively. It is likely that analysis of the data for Table 3 using the chi-squared
258
Letters to the Editor / Journal of Clinical Epidemiology 69 (2016) 248e268
test would give a different picture of the minimum detectable difference in LRs. David C. Hoaglin Division of Biostatistics and Health Services Research Department of Quantitative Health Sciences University of Massachusetts Medical School 368 Plantation Street Albert Sherman Center Worcester, MA 01605 USA Tel.: 508-856-3576; fax: 508-856-8993. E-mail address: [email protected]
References [1] Cohen JF, Chalumeau M, Cohen R, Korevaar DA, Khoshnood B, Bossuyt PMM. Cochran’s Q test was useful to assess heterogeneity in likelihood ratios in studies of diagnostic accuracy. J Clin Epidemiol 2015;68:299e306. [2] Cochran WG. The combination of estimates from different experiments. Biometrics 1954;10:101e29. [3] Kulinskaya E, Dollinger MB, Bjørkestøl K. Testing for homogeneity in meta-analysis I. The one-parameter case: standardized mean difference. Biometrics 2011;67:203e12. [4] Kulinskaya E, Dollinger MB, Bjørkestøl K. On the moments of Cochran’s Q statistic under the null hypothesis, with application to the meta-analysis of risk difference. Res Synth Methods 2011;2: 254e70. [5] Biggerstaff BJ, Jackson D. The exact distribution of Cochran’s heterogeneity statistic in one-way random effects meta-analysis. Stat Med 2008;27:6093e110. [6] Bishop YMM, Fienberg SE, Holland PW. Discrete multivariate analysis: theory and practice. Cambridge, MA: MIT Press; 1975. http://dx.doi.org/10.1016/j.jclinepi.2015.03.024
Response to letter by Hoaglin: Heterogeneity in likelihood ratios Dr Hoaglin raises concerns about our proposal to use Cochran’s Q to investigate heterogeneity in diagnostic likelihood ratios across subgroups [1]. We agree with him that the 1954 article to which we refer to the original description of Cochran’s Q may be confusing to the readers [2]. In another article, Cochran explicitly described the total c2 for a 2 N contingency table as ‘‘a weighted sum of squares of the deviations of the individual proportions of success pi from their mean’’ [3]. Yet Cochran did not call this statistic Q in this other article either. Our application of Cochran’s Q comes from metaanalysis, where it is very frequently used to evaluate heterogeneity of the effect of interest across studies [4]. This requires estimating a summary estimate of the effect and then calculating the Q statistic as the weighted sum of squared Conflict of interest: None. Funding: There was no specific funding for preparing this manuscript.
differences between study effects and the pooled effect. It is well known that Cochran’s Q does not exactly follow a c2 distribution, but it is quite common to assume this approximation [4e6]. Most methodologists nowadays recommend I2 as a more meaningful expression of heterogeneity in meta-analysis: I2 Z 100% (Q df)/Q, where Q is Cochran’s statistic and df is the degrees of freedom [6]. Dr Hoaglin suggested to evaluate the conditional independence of the subgroup (X ) and the index test (Y ) given the result of the reference standard (Z ), by using a test of goodness of fit for a loglinear model for contingency tables, in which cell counts are treated as independent observations of a Poisson response variable [7]. We believe the P-values obtained from such a test are not comparable with those presented in our article because they test a different hypothesis. The approach suggested by Hoaglin compares the observed and fitted counts; our approach tests for the interaction between the subgroup variable (X ) and the reference standard (Z ). The P-values that we report correspond to the test of significance of the interaction term X Z. Jeremie F. Cohen* INSERM U1153 Obstetrical, Perinatal and Pediatric Epidemiology Research Team Research Center for Epidemiology and Biostatistics Sorbonne Paris Cite (CRESS) Paris Descartes University 53, avenue de l’Observatoire 75014 Paris, France Department of Pediatrics Necker-Enfants Malades Hospital Assistance Publique-H^ opitaux de Paris Paris Descartes University 149, rue de Sevres 75015 Paris, France Department of Clinical Epidemiology Biostatistics and Bioinformatics Academic Medical Center University of Amsterdam PO Box 22700 1100 DE Amsterdam, The Netherlands
Martin Chalumeau INSERM U1153 Obstetrical, Perinatal and Pediatric Epidemiology Research Team Research Center for Epidemiology and Biostatistics Sorbonne Paris Cite (CRESS) Paris Descartes University 53, avenue de l’Observatoire 75014 Paris, France Department of Pediatrics Necker-Enfants Malades Hospital Assistance Publique-H^ opitaux de Paris Paris Descartes University 149, rue de Sevres 75015 Paris, France
Martin Boeker Edith Motschall Werner Vach* Center for Medical Biometry and Medical Informatics University Medical Centre Freiburg Stefan-Meier-Str. 26, Freiburg D-79104 Germany *Corresponding author. Tel.: þ49-761-203-6722; fax: þ49-761-203-6711. E-mail address: [email protected] (W. Vach)
References [1] Hausner E, Guddat C, Hermanns T, Lamperta U, Waffenschmidt S. Development of search strategies for systematic reviews: validation showed the noninferiority of the objective approach. J Clin Epidemiol 2015;68:191e9. [2] Booth A. Unpacking your literature search toolbox: on search styles and tactics. Health Info Libr J 2008;25(4):313e7. [3] Atkinson KM, Koenka AC, Sanchez CE, Moshontz H, Cooper H. Reporting standards for literature searches and report inclusion criteria: making research syntheses more transparent and easy to replicate. Res Synth Methods 2015;6:87e95. [4] Lefebvre C, Glanville J, Wieland LS, Coles B, Weightman AL. Methodological developments in searching for studies for systematic reviews: past, present and future? Syst Rev 2013;2:78. [5] Li L, Tian J, Tian H, Moher D, Liang F, Jiang T, et al. Network meta-analyses could be improved by searching more sources and by involving a librarian. J Clin Epidemiol 2014;67:1001e7. [6] Rethlefsen ML, Murad MH, Livingston EH. Engaging medical librarians to improve the quality of review articles. JAMA 2014;312:999e1000. [7] Rethlefsen ML, Farrell AM, Osterhaus Trzasko LC, Brigham TJ. Librarian co-authors correlated with higher quality reported search strategies in general internal medicine systematic reviews. J Clin Epidemiol 2015;68:617e26. http://dx.doi.org/10.1016/j.jclinepi.2015.05.022
Letter to the Editor on Cohen et al. (Letter commenting on: J Clin Epidemiol. 2015;68:299e306) To the Editor: In describing and applying Cochran’s Q statistic, Cohen et al. [1] make several substantial factual errors. From the description in Section 2, readers might infer that Cochran [2] discusses the use of Q with likelihood ratios (LRs) and other measures derived from 2 2 contingency tables. He does not. That article discusses combining estimates such as ‘‘a simple mean of the observations, as in a physical determination, or the difference between the means of two treatments, as in a comparative experiment, or a median lethal dose, or a regression coefficient.’’ In addition to the estimate, each experiment produces an estimate of the corresponding standard error or variance, accompanied by its degrees of freedom. For the sorts of estimates that Cochran discusses, the degrees of freedom
257
ordinarily come from a mean square (eg, the sample variance or the residual mean square of a regression). Importantly, the estimated variance is ordinarily independent of the estimate itself. Section 2 indicates that Cochran’s Q statistic follows a chi-squared distribution on k 1 degrees of freedom (ie, in the absence of heterogeneity Qwc2k1 ). In fact, as Cochran explains, the null distribution of Q only approaches c2k1 when the numbers of degrees of freedom from the individual experiments become large. Cohen et al. do not cite evidence to show that the values of N in their simulation study are large enough for that approximation to be adequate. Kulinskaya et al [3,4]. have investigated the null distribution of Q and have shown that the nature of its departure from c2k1 depends on the measure of effect. They give results for standardized mean difference [3] and risk difference [4]. To my knowledge, results for the LR are not available. The phrase ‘‘Cochran’s Q test’’ gives the impression that Cochran [2] uses Q as the basis for a test of heterogeneity. In fact, after discussing Q in that article, he uses a different statistic to test for heterogeneity. Section 5 states that ‘‘the Q test was originally proposed by Cochran to assess heterogeneity between subgroups in a study.’’ In fact, as the title indicates, the subject of Cochran’s article was combining estimates from experiments. Inverse-variance weights based on estimated variances are a frequent source of difficulty in meta-analysis; one cannot generally use estimated variances as if they were the known true variances. That assumption allowed Biggerstaff and Jackson [5] to obtain the exact distribution of a statistic that they call Q, but that statistic is not Cochran’s Q (in which the weights are based only on estimated variances). For LRs, it is also a problem that the logarithm of an LR and its estimated variance are related (Cohen et al. [1] give the formulas in Appendix A). In assessing heterogeneity of LRs, one can avoid the difficulties of inverse-variance weights by using a standard model from discrete multivariate analysis [6]. For G subgroups, the data form a 2 2 G table. Homogeneity of the positive likelihood ratio (PLR), for example, requires that (in the underlying probabilities) the result on the index test and the subgroup be conditionally independent, given the classification from the reference standard. It is easy to fit that model, calculate the usual chi-squared measure of its fit, and refer it to the chi-squared distribution on 2(G 1) degrees of freedom. The properties of such chisquared tests have been well studied. For the first two cases of PLR in Table 3 (prevalence Z 0.30, N Z 300, PLR Z 3.00 in group 1, and PLR Z 4.00 and 6.03 in group 2), the chi-squared test (on two degrees of freedom) yields P-values that differ substantially, and in opposite directions, from those of the test based on Q: 0.521 instead of 0.295 and 0.00925 instead of 0.040, respectively. It is likely that analysis of the data for Table 3 using the chi-squared
258
Letters to the Editor / Journal of Clinical Epidemiology 69 (2016) 248e268
test would give a different picture of the minimum detectable difference in LRs. David C. Hoaglin Division of Biostatistics and Health Services Research Department of Quantitative Health Sciences University of Massachusetts Medical School 368 Plantation Street Albert Sherman Center Worcester, MA 01605 USA Tel.: 508-856-3576; fax: 508-856-8993. E-mail address: [email protected]
References [1] Cohen JF, Chalumeau M, Cohen R, Korevaar DA, Khoshnood B, Bossuyt PMM. Cochran’s Q test was useful to assess heterogeneity in likelihood ratios in studies of diagnostic accuracy. J Clin Epidemiol 2015;68:299e306. [2] Cochran WG. The combination of estimates from different experiments. Biometrics 1954;10:101e29. [3] Kulinskaya E, Dollinger MB, Bjørkestøl K. Testing for homogeneity in meta-analysis I. The one-parameter case: standardized mean difference. Biometrics 2011;67:203e12. [4] Kulinskaya E, Dollinger MB, Bjørkestøl K. On the moments of Cochran’s Q statistic under the null hypothesis, with application to the meta-analysis of risk difference. Res Synth Methods 2011;2: 254e70. [5] Biggerstaff BJ, Jackson D. The exact distribution of Cochran’s heterogeneity statistic in one-way random effects meta-analysis. Stat Med 2008;27:6093e110. [6] Bishop YMM, Fienberg SE, Holland PW. Discrete multivariate analysis: theory and practice. Cambridge, MA: MIT Press; 1975. http://dx.doi.org/10.1016/j.jclinepi.2015.03.024
Response to letter by Hoaglin: Heterogeneity in likelihood ratios Dr Hoaglin raises concerns about our proposal to use Cochran’s Q to investigate heterogeneity in diagnostic likelihood ratios across subgroups [1]. We agree with him that the 1954 article to which we refer to the original description of Cochran’s Q may be confusing to the readers [2]. In another article, Cochran explicitly described the total c2 for a 2 N contingency table as ‘‘a weighted sum of squares of the deviations of the individual proportions of success pi from their mean’’ [3]. Yet Cochran did not call this statistic Q in this other article either. Our application of Cochran’s Q comes from metaanalysis, where it is very frequently used to evaluate heterogeneity of the effect of interest across studies [4]. This requires estimating a summary estimate of the effect and then calculating the Q statistic as the weighted sum of squared Conflict of interest: None. Funding: There was no specific funding for preparing this manuscript.
differences between study effects and the pooled effect. It is well known that Cochran’s Q does not exactly follow a c2 distribution, but it is quite common to assume this approximation [4e6]. Most methodologists nowadays recommend I2 as a more meaningful expression of heterogeneity in meta-analysis: I2 Z 100% (Q df)/Q, where Q is Cochran’s statistic and df is the degrees of freedom [6]. Dr Hoaglin suggested to evaluate the conditional independence of the subgroup (X ) and the index test (Y ) given the result of the reference standard (Z ), by using a test of goodness of fit for a loglinear model for contingency tables, in which cell counts are treated as independent observations of a Poisson response variable [7]. We believe the P-values obtained from such a test are not comparable with those presented in our article because they test a different hypothesis. The approach suggested by Hoaglin compares the observed and fitted counts; our approach tests for the interaction between the subgroup variable (X ) and the reference standard (Z ). The P-values that we report correspond to the test of significance of the interaction term X Z. Jeremie F. Cohen* INSERM U1153 Obstetrical, Perinatal and Pediatric Epidemiology Research Team Research Center for Epidemiology and Biostatistics Sorbonne Paris Cite (CRESS) Paris Descartes University 53, avenue de l’Observatoire 75014 Paris, France Department of Pediatrics Necker-Enfants Malades Hospital Assistance Publique-H^ opitaux de Paris Paris Descartes University 149, rue de Sevres 75015 Paris, France Department of Clinical Epidemiology Biostatistics and Bioinformatics Academic Medical Center University of Amsterdam PO Box 22700 1100 DE Amsterdam, The Netherlands
Martin Chalumeau INSERM U1153 Obstetrical, Perinatal and Pediatric Epidemiology Research Team Research Center for Epidemiology and Biostatistics Sorbonne Paris Cite (CRESS) Paris Descartes University 53, avenue de l’Observatoire 75014 Paris, France Department of Pediatrics Necker-Enfants Malades Hospital Assistance Publique-H^ opitaux de Paris Paris Descartes University 149, rue de Sevres 75015 Paris, France