- Email: [email protected]
Significant errors
Correspondence
captured by the regression terms for HIV prevalence is causally linked to HIV infection. However, this proposition is compromised if other characteristics correlated with prevalent HIV, but incompletely reflected in the model, contribute to this mortality variation. In fact, the estimate by Hogan and colleagues seems quite high compared with existing HIV epidemiology. For example, focusing specifically on high HIV prevalence populations, about 55% of all HIV-infected women in the world live in just 13 high-prevalence countries.2 Overall adult HIV prevalence in these countries varies from 3% to 26%. A proportionate share of maternal mortality for these countries (55% of 61 400) would be about 33 800. However, if we use numeric estimates for the five of these countries that contribute most to maternal mortality provided by Hogan and colleagues, and estimates we calculated from their maternal mortality ratios for the others, the total of all maternal deaths for the 13 comes to only about 41 000–42 000. It seems quite implausible that some 80% of maternal deaths in these countries would be attributable to HIV. If HIV were adding such a new dominant burden of maternal mortality, we should have seen a several-fold increased maternal mortality. By contrast, a specific 1998 review of maternal mortality in South Africa classified AIDS as an “indirect” cause in 14·5% of maternal deaths.3 Other correlated explanations that could relate to persistent high maternal mortality in Africa include poor health infrastructure in Africa or indeed conceivably the indirect effects of HIV on the health infrastructure such as loss of health personnel. It could, therefore, be misleading and distort policy to ascribe an inappropriately large burden of maternal deaths directly to HIV. We declare that we have no conflicts of interest.
*James D Shelton, Ron Gray [email protected] Bureau for Global Health, US Agency for International Development, Washington, DC 20523, USA (JDS); and Johns Hopkins University,
1390
Bloomberg School of Public Health, Baltimore, MD, USA (RG) 1
2
3
Hogan MC, Foreman KJ, Naghavi M, at al. Maternal mortality for 181 countries, 19008: a systematic analysis of progress towards Millennium Development Goal 5. Lancet 2010; 375: 1609–23. UNAIDS. 2008 report on the global AIDS epidemic. Geneva: UNAIDS, 2008. http:// www.unaids.org/en/KnowledgeCentre/ HIVData/GlobalReport/2008/default.asp (accessed May 14, 2010). National Committee on Confidential Inquiries in Maternal Deaths. A review of maternal deaths in South Africa during 1998. S Afr Med J 2000; 90: 367–73.
We welcome the study by Margaret Hogan and colleagues,1 which provides important new estimates for maternal mortality rates (MMR) in 181 countries. However, we would have liked to see additional discussion on the presentation of the individuallevel data. As a specific example, from our work in Malawi and analysis of the Demographic and Health Survey (DHS) data on maternal mortality (unpublished), the graph for Malawi seems to overestimate MMR. The last two sibling history data points (which we assume are for the DHS 2000 and 2004 surveys) are higher than the published estimates of 1120 (2000)2 and 984 (2004).3 Although these two points are correctly centred around 1997 and 2001, respectively, reflecting the retrospective nature of the sisterhood method, the rates are around 1400 and 1500, respectively, which seems high even after accounting for upward adjustment based on the analyses of Obermeyer and colleagues.4 We presume that the upward adjustments are based on GakidouKing weights, which correct for families with high mortality being under-represented, and then an additional upward adjustment which represents the fact that deaths are less likely to be reported the longer ago they occurred. Obermeyer and colleagues report that on average the Gakidou-King weights will increase adult mortality rates upwards by 27%. The upward adjustment for Malawi was of the order of 40%. In the case
of Mali, where the DHS 2006 gives an MMR of 464 but is plotted at about 800 in the paper, the upward adjustment is even more significant. Given the scale of some of these adjustments, we believe that some discussion of these changes was necessary, and that the plots should have been labelled as “adjusted sibling history” to avoid potential for confusion. We declare that we have no conflicts of interest.
Tim Colbourn, *Anthony Costello, Christina Pagel [email protected] University College London, London WC1H 0BT, UK 1.
2
3
4
Hogan MC, Foreman KJ, Naghavi M, et al. Maternal mortality for 181 countries, 1980–2008: a systematic analysis of progress towards Millennium Development Goal 5. Lancet 2010; 375: 1609–23. National Statistics Office, Macro International USA. Malawi demographic and health survey 2000. Zomba/Calverton: NSO and Macro International, 2001. National Statistics Office, Orc Macro USA. Malawi demographic and health survey 2004. Calverton: NSO and ORC Macro, 2005. Obermeyer Z, Rajaratnam JK, Park CH, et al. Measuring adult mortality using sibling survival: a new analytical method and new results for 44 countries, 1974–2006. PLoS Med 2010; 7: e1000260.
Significant errors The Ronald Fisher in “The Validus Medicus and a new gold standard” (July 31, p 324)1 is a man of straw of Stephen Ziliak’s creation. Contrast Ziliak’s “After Fisher the erroneous belief is that failing to reach statistical significance is the same as finding no important difference between the two bad outcomes” with Fisher’s “in fact no scientific worker has a fixed level of significance at which from year to year, and in all circumstances, he rejects hypotheses; he rather gives his mind to each particular case in the light of his evidence and his ideas…A test of significance contains no criterion for ‘accepting’ a hypothesis”.2 Ziliak also misunderstands the story of rofecoxib. It was already known by the year 2000 that rofecoxib had a disadvantage compared with naproxen as regards cardiovascular side-effects.3 www.thelancet.com Vol 376 October 23, 2010
Correspondence
I consult for the pharmaceutical industry, including Merck and Novartis, and have consulted for Roche. I own shares in Novartis. I am secretary of the Fisher Memorial Trust.
Stephen Senn [email protected] Department of Statistics, University of Glasgow, Glasgow G12 8QW, UK 1 2
3
4
5
Ziliak ST. The Validus Medicus and a new gold standard. Lancet 2010; 376: 324–25. Fisher RA. Statistical methods and scientific inference. In: Bennet JH, ed. Statistical methods, experimental design and scientific inference. Oxford: Oxford University, 1990. Bombardier C, Laine L, Reicin A, et al. Comparison of upper gastrointestinal toxicity of rofecoxib and naproxen in patients with rheumatoid arthritis. N Engl J Med 2000; 343: 1520–28. Lisse JR, Perlman M, Johansson G, et al. Gastrointestinal tolerability and effectiveness of rofecoxib versus naproxen in the treatment of osteoarthritis: a randomized, controlled trial. Ann Intern Med 2003; 139: 539–46. Farkouh ME, Kirshner H, Harrington RA, et al. Comparison of lumiracoxib with naproxen and ibuprofen in the Therapeutic Arthritis Research and Gastrointestinal Event Trial (TARGET), cardiovascular outcomes: randomised controlled trial. Lancet 2004; 364: 675–84.
www.thelancet.com Vol 376 October 23, 2010
Author’s reply In my Art of Medicine essay, I said that “[t]here are few scientists who would not rejoice at the breaking of our gold fetters”. By “gold fetters” I mean the illogical system of rewards that is currently spoiling medicine and other sciences—through research based on “artificial randomisation”, “statistical significance”, and “validity” descended from Ronald Fisher.1,2 Stephen Senn is apoplectic but the cause is not scientific. His remarks do not pass the tests of history and basic human values. “Student’s” priority over Fisher—and the demonstrated value of his economic, balanced, and repeated small sample approach to the design and evaluation of experiments—is undoubted.3 Yet Student is not mentioned by Senn. On the Student-Fisher debates on design, testing, and estimation—and on the gross distortion of Student’s methods by the younger Fisher4—Senn is mute. Nietzsche said that the twilight of the idols will be denied by some. Some will try to impede the inevitable decline and new dawn of science. Senn does not seek merely to save randomisation and significance; he employs several Aristotelian fallacies to try to place them and Fisher higher up in the annals. He applauds when Fisher says that “[a] test of significance contains no criterion for ‘accepting’ a hypothesis”. But the pregnant phrase here is “no criterion”; in Fisher’s method there is no criterion for assessing medical and other knowledge— the exact opposite of what Senn claims and human beings need most. “Finally”, Fisher emphasised, “in inductive inference we introduce no cost functions for faulty judgments.” Some help. “In fact”, in his view, “scientific research is not geared to maximize the profits of any particular organization... We make no attempt to evaluate these consequences, and do not assume that they are capable of evaluation in any currency”.5 Student and his students have a better gold standard. We value
balanced designs; prior knowledge; variable—including extreme—odds; inputs of personal probability; shows of minimum real error; cost functions; and explicit demonstrations of power to detect large and real treatment differences across independent and repeated trials. Fisher’s waning faithful do not. I declare that I have no conflicts of interest.
Stephen T Ziliak [email protected] Roosevelt University, Chicago, IL 60605, USA 1
2
3
4
5
Altman DG, Schulz KF, Moher D, et al. The revised CONSORT statement for reporting randomized trials: explanation and elaboration. Ann Intern Med 2001; 134: 663–94. Bruhn M, McKenzie D. In pursuit of balance: randomization in practice in development economics. Am Econ J Appl Econ 2009; 1: 200–32. Ziliak ST. Guinnessometrics: the economic foundation of ‘Student’s’ t. J Econ Persp 2008; 22: 199–216. Ziliak ST, McCloskey DN. The cult of statistical significance: how the standard error costs us jobs, justice, and lives. Ann Arbor: University of Michigan Press, 2008. Fisher RA. Statistical methods and scientific induction. J R Stat Soc 1955; 17: 69–78.
Web-surfers beware: know thy source As the managing editor at theheart.org, one of the cardiology websites cited by Christopher Cannon (Aug 14, p 505),1 I was delighted to read Cannon’s belief that “cardiology’s move online... will hopefully improve health care.” I would, however, point to some potential pitfalls. As Cannon notes, physicians now have countless websites to turn to for information, and ever-increasing formats—news, scientific papers, blogs, tweets, webcasts, podcasts—in which to get it. Indeed, I believe the rise of independent medical websites probably spurred meeting organisers, journal editors, and now textbook publishers not only to diversify the content they offer online, but also to offer it more quickly: simultaneous presentation and publication of new research being a case in point. Add to this the rise of “sharing” via social
Science Photo Library
The non-significant difference to which Ziliak alludes4 does not overturn this result and nor does Fisher’s approach to significance testing suggest that it should. The real controversy is otherwise. Did the significant difference compared with naproxen established by 2000 logically imply that rofecoxib was inferior to placebo? It can be argued that it did not. Did it make any practical difference in choosing between the two as to whether naproxen was cardioprotective or rofecoxib was cardiotoxic? Again it can be argued that it did not. None of this confusion can be laid at Fisher’s door. As for Ziliak’s attack on randomisation, I suggest that any readers in doubt as to its value look no further than the TARGET study.5 Patients were allocated not at random to one of two substudies, either lumiracoxib versus naproxen or lumiracoxib versus ibuprofen. They were, however, randomised between groups within studies. Between substudies there is substantial imbalance in prognostic factors. Within substudies there is not. Fisher was not infallible, but he was a scientist of the very first calibre whose work deserves to be taken seriously.
1391
captured by the regression terms for HIV prevalence is causally linked to HIV infection. However, this proposition is compromised if other characteristics correlated with prevalent HIV, but incompletely reflected in the model, contribute to this mortality variation. In fact, the estimate by Hogan and colleagues seems quite high compared with existing HIV epidemiology. For example, focusing specifically on high HIV prevalence populations, about 55% of all HIV-infected women in the world live in just 13 high-prevalence countries.2 Overall adult HIV prevalence in these countries varies from 3% to 26%. A proportionate share of maternal mortality for these countries (55% of 61 400) would be about 33 800. However, if we use numeric estimates for the five of these countries that contribute most to maternal mortality provided by Hogan and colleagues, and estimates we calculated from their maternal mortality ratios for the others, the total of all maternal deaths for the 13 comes to only about 41 000–42 000. It seems quite implausible that some 80% of maternal deaths in these countries would be attributable to HIV. If HIV were adding such a new dominant burden of maternal mortality, we should have seen a several-fold increased maternal mortality. By contrast, a specific 1998 review of maternal mortality in South Africa classified AIDS as an “indirect” cause in 14·5% of maternal deaths.3 Other correlated explanations that could relate to persistent high maternal mortality in Africa include poor health infrastructure in Africa or indeed conceivably the indirect effects of HIV on the health infrastructure such as loss of health personnel. It could, therefore, be misleading and distort policy to ascribe an inappropriately large burden of maternal deaths directly to HIV. We declare that we have no conflicts of interest.
*James D Shelton, Ron Gray [email protected] Bureau for Global Health, US Agency for International Development, Washington, DC 20523, USA (JDS); and Johns Hopkins University,
1390
Bloomberg School of Public Health, Baltimore, MD, USA (RG) 1
2
3
Hogan MC, Foreman KJ, Naghavi M, at al. Maternal mortality for 181 countries, 19008: a systematic analysis of progress towards Millennium Development Goal 5. Lancet 2010; 375: 1609–23. UNAIDS. 2008 report on the global AIDS epidemic. Geneva: UNAIDS, 2008. http:// www.unaids.org/en/KnowledgeCentre/ HIVData/GlobalReport/2008/default.asp (accessed May 14, 2010). National Committee on Confidential Inquiries in Maternal Deaths. A review of maternal deaths in South Africa during 1998. S Afr Med J 2000; 90: 367–73.
We welcome the study by Margaret Hogan and colleagues,1 which provides important new estimates for maternal mortality rates (MMR) in 181 countries. However, we would have liked to see additional discussion on the presentation of the individuallevel data. As a specific example, from our work in Malawi and analysis of the Demographic and Health Survey (DHS) data on maternal mortality (unpublished), the graph for Malawi seems to overestimate MMR. The last two sibling history data points (which we assume are for the DHS 2000 and 2004 surveys) are higher than the published estimates of 1120 (2000)2 and 984 (2004).3 Although these two points are correctly centred around 1997 and 2001, respectively, reflecting the retrospective nature of the sisterhood method, the rates are around 1400 and 1500, respectively, which seems high even after accounting for upward adjustment based on the analyses of Obermeyer and colleagues.4 We presume that the upward adjustments are based on GakidouKing weights, which correct for families with high mortality being under-represented, and then an additional upward adjustment which represents the fact that deaths are less likely to be reported the longer ago they occurred. Obermeyer and colleagues report that on average the Gakidou-King weights will increase adult mortality rates upwards by 27%. The upward adjustment for Malawi was of the order of 40%. In the case
of Mali, where the DHS 2006 gives an MMR of 464 but is plotted at about 800 in the paper, the upward adjustment is even more significant. Given the scale of some of these adjustments, we believe that some discussion of these changes was necessary, and that the plots should have been labelled as “adjusted sibling history” to avoid potential for confusion. We declare that we have no conflicts of interest.
Tim Colbourn, *Anthony Costello, Christina Pagel [email protected] University College London, London WC1H 0BT, UK 1.
2
3
4
Hogan MC, Foreman KJ, Naghavi M, et al. Maternal mortality for 181 countries, 1980–2008: a systematic analysis of progress towards Millennium Development Goal 5. Lancet 2010; 375: 1609–23. National Statistics Office, Macro International USA. Malawi demographic and health survey 2000. Zomba/Calverton: NSO and Macro International, 2001. National Statistics Office, Orc Macro USA. Malawi demographic and health survey 2004. Calverton: NSO and ORC Macro, 2005. Obermeyer Z, Rajaratnam JK, Park CH, et al. Measuring adult mortality using sibling survival: a new analytical method and new results for 44 countries, 1974–2006. PLoS Med 2010; 7: e1000260.
Significant errors The Ronald Fisher in “The Validus Medicus and a new gold standard” (July 31, p 324)1 is a man of straw of Stephen Ziliak’s creation. Contrast Ziliak’s “After Fisher the erroneous belief is that failing to reach statistical significance is the same as finding no important difference between the two bad outcomes” with Fisher’s “in fact no scientific worker has a fixed level of significance at which from year to year, and in all circumstances, he rejects hypotheses; he rather gives his mind to each particular case in the light of his evidence and his ideas…A test of significance contains no criterion for ‘accepting’ a hypothesis”.2 Ziliak also misunderstands the story of rofecoxib. It was already known by the year 2000 that rofecoxib had a disadvantage compared with naproxen as regards cardiovascular side-effects.3 www.thelancet.com Vol 376 October 23, 2010
Correspondence
I consult for the pharmaceutical industry, including Merck and Novartis, and have consulted for Roche. I own shares in Novartis. I am secretary of the Fisher Memorial Trust.
Stephen Senn [email protected] Department of Statistics, University of Glasgow, Glasgow G12 8QW, UK 1 2
3
4
5
Ziliak ST. The Validus Medicus and a new gold standard. Lancet 2010; 376: 324–25. Fisher RA. Statistical methods and scientific inference. In: Bennet JH, ed. Statistical methods, experimental design and scientific inference. Oxford: Oxford University, 1990. Bombardier C, Laine L, Reicin A, et al. Comparison of upper gastrointestinal toxicity of rofecoxib and naproxen in patients with rheumatoid arthritis. N Engl J Med 2000; 343: 1520–28. Lisse JR, Perlman M, Johansson G, et al. Gastrointestinal tolerability and effectiveness of rofecoxib versus naproxen in the treatment of osteoarthritis: a randomized, controlled trial. Ann Intern Med 2003; 139: 539–46. Farkouh ME, Kirshner H, Harrington RA, et al. Comparison of lumiracoxib with naproxen and ibuprofen in the Therapeutic Arthritis Research and Gastrointestinal Event Trial (TARGET), cardiovascular outcomes: randomised controlled trial. Lancet 2004; 364: 675–84.
www.thelancet.com Vol 376 October 23, 2010
Author’s reply In my Art of Medicine essay, I said that “[t]here are few scientists who would not rejoice at the breaking of our gold fetters”. By “gold fetters” I mean the illogical system of rewards that is currently spoiling medicine and other sciences—through research based on “artificial randomisation”, “statistical significance”, and “validity” descended from Ronald Fisher.1,2 Stephen Senn is apoplectic but the cause is not scientific. His remarks do not pass the tests of history and basic human values. “Student’s” priority over Fisher—and the demonstrated value of his economic, balanced, and repeated small sample approach to the design and evaluation of experiments—is undoubted.3 Yet Student is not mentioned by Senn. On the Student-Fisher debates on design, testing, and estimation—and on the gross distortion of Student’s methods by the younger Fisher4—Senn is mute. Nietzsche said that the twilight of the idols will be denied by some. Some will try to impede the inevitable decline and new dawn of science. Senn does not seek merely to save randomisation and significance; he employs several Aristotelian fallacies to try to place them and Fisher higher up in the annals. He applauds when Fisher says that “[a] test of significance contains no criterion for ‘accepting’ a hypothesis”. But the pregnant phrase here is “no criterion”; in Fisher’s method there is no criterion for assessing medical and other knowledge— the exact opposite of what Senn claims and human beings need most. “Finally”, Fisher emphasised, “in inductive inference we introduce no cost functions for faulty judgments.” Some help. “In fact”, in his view, “scientific research is not geared to maximize the profits of any particular organization... We make no attempt to evaluate these consequences, and do not assume that they are capable of evaluation in any currency”.5 Student and his students have a better gold standard. We value
balanced designs; prior knowledge; variable—including extreme—odds; inputs of personal probability; shows of minimum real error; cost functions; and explicit demonstrations of power to detect large and real treatment differences across independent and repeated trials. Fisher’s waning faithful do not. I declare that I have no conflicts of interest.
Stephen T Ziliak [email protected] Roosevelt University, Chicago, IL 60605, USA 1
2
3
4
5
Altman DG, Schulz KF, Moher D, et al. The revised CONSORT statement for reporting randomized trials: explanation and elaboration. Ann Intern Med 2001; 134: 663–94. Bruhn M, McKenzie D. In pursuit of balance: randomization in practice in development economics. Am Econ J Appl Econ 2009; 1: 200–32. Ziliak ST. Guinnessometrics: the economic foundation of ‘Student’s’ t. J Econ Persp 2008; 22: 199–216. Ziliak ST, McCloskey DN. The cult of statistical significance: how the standard error costs us jobs, justice, and lives. Ann Arbor: University of Michigan Press, 2008. Fisher RA. Statistical methods and scientific induction. J R Stat Soc 1955; 17: 69–78.
Web-surfers beware: know thy source As the managing editor at theheart.org, one of the cardiology websites cited by Christopher Cannon (Aug 14, p 505),1 I was delighted to read Cannon’s belief that “cardiology’s move online... will hopefully improve health care.” I would, however, point to some potential pitfalls. As Cannon notes, physicians now have countless websites to turn to for information, and ever-increasing formats—news, scientific papers, blogs, tweets, webcasts, podcasts—in which to get it. Indeed, I believe the rise of independent medical websites probably spurred meeting organisers, journal editors, and now textbook publishers not only to diversify the content they offer online, but also to offer it more quickly: simultaneous presentation and publication of new research being a case in point. Add to this the rise of “sharing” via social
Science Photo Library
The non-significant difference to which Ziliak alludes4 does not overturn this result and nor does Fisher’s approach to significance testing suggest that it should. The real controversy is otherwise. Did the significant difference compared with naproxen established by 2000 logically imply that rofecoxib was inferior to placebo? It can be argued that it did not. Did it make any practical difference in choosing between the two as to whether naproxen was cardioprotective or rofecoxib was cardiotoxic? Again it can be argued that it did not. None of this confusion can be laid at Fisher’s door. As for Ziliak’s attack on randomisation, I suggest that any readers in doubt as to its value look no further than the TARGET study.5 Patients were allocated not at random to one of two substudies, either lumiracoxib versus naproxen or lumiracoxib versus ibuprofen. They were, however, randomised between groups within studies. Between substudies there is substantial imbalance in prognostic factors. Within substudies there is not. Fisher was not infallible, but he was a scientist of the very first calibre whose work deserves to be taken seriously.
1391