- Email: [email protected]
Of Statistical Significance: “Trends” Toward Significance and Optimism Bias
Journal of the American College of Cardiology © 2006 by the American College of Cardiology Foundation Published by Elsevier Inc.
Vol. 48, No. 7, 2006 ISSN 0735-1097/06/$32.00
CORRESPONDENCE Letters to the Editor Of Statistical Significance: “Trends” Toward Significance and Optimism Bias In the study by McMurray et al. (1), the investigators discuss a post hoc analysis of the VALIANT (Valsartan in Acute Myocardial Infarction) trial and conclude: “These data, although not conclusive, also support the hypothesis that adding an ARB (angiotensin receptor blocker) to an ACE (angiotensin-converting enzyme) inhibitor may have a small additional anti-infarction effect.” This conclusion was based on a “trend” toward decreased myocardial infarction (MI) and stroke compared with monotherapy. One has to question whether p values of 0.65 and 0.187 should be considered a “trend” toward significance. By the same analogy, there seems to be a “trend” toward greater incidence of angina in patients on combination therapy (based on survival curves). Moreover, based on the survival curves, at 6 months there seems to be a “trend” toward a greater number of strokes in the combination therapy compared to valsartan alone. There should be some thought/discussion on what can reasonably be defined as a “trend.” This question has surfaced in all post hoc analyses of randomized controlled trials where there is no preset criterion for significance. If we consider p ⬍ 0.05 as statistically significant, perhaps a p value between 0.05 to 0.10 may be considered as a trend toward significance. The conclusion that “These data also suggest, but do not prove, that adding an ARB to an ACE inhibitor may have a small additional anti-infarction effect” should thus be interpreted with caution. Given these somewhat slippery slopes, we call for a consensus as to what level of significance should be identified as a “trend” in a post hoc analysis and whether the observation of such a trend should be an acceptable basis for conclusions. Unless these terms are clearly defined a priori, one cannot quite avoid a euphemism such as “optimism bias” (2) to characterize the conclusions made by McMurray et al. (1).
Sripal Bangalore, MD, MHA *Franz H. Messerli, MD *Hypertension Program Division of Cardiology Columbia University College of Physicians and Surgeons St. Luke’s-Roosevelt Hospital Center 1000 Tenth Avenue Suite 3B-30 New York, New York 10025 E-mail: [email protected] doi:10.1016/j.jacc.2006.07.011
REFERENCES 1. McMurray J, Solomon S, Pieper K, et al. The effect of valsartan, captopril, or both on atherosclerotic events after acute myocardial
infarction: an analysis of the Valsartan in Acute Myocardial Infarction Trial (VALIANT). J Am Coll Cardiol 2006;47:726 –33. 2. Chalmers I, Matthews R. What are the implications of optimism bias in clinical research? Lancet 2006;367:449 –50.
REPLY Unfortunately, Drs. Bangalore and Messerli did not cite our concluding sentence about myocardial infarction in full, omitting the final statement that our observation was “a possibility that needs to be prospectively tested” (1). A p value of 0.187 (for the comparison of combination treatment with captopril and valsartan versus captopril monotherapy) might, with the appropriate caution we applied in the Abstract Conclusions and elsewhere, reasonably be described as a trend. In addition, in supportive analyses using other classifications of events (e.g., investigator reported versus adjudicated) and other statistical techniques, the p value for combination versus monotherapy was nominally statistically significant. Overall, strokes were numerically fewer in the combination treatment group than in the captopril-only group, with a p value of 0.079, which, according to Drs. Bangalore and Messerli’s own criteria, might be described as a trend. It is not clear why one would try to segment the Kaplan-Meier curves into 6-month time periods. We clearly stated that our observations had been made in a post hoc analysis, were not conclusive, and required prospective testing. Fortunately, such a trial has been conducted and will be reported in the near future (2). *John J. V. McMurray, MD, FACC Robert M. Califf, MD, FACC Marc A. Pfeffer, MD, PhD, FACC *Western Infirmary Cardiology Dumbarton Road Glasgow G11 6NT Scotland United Kingdom E-mail: [email protected] doi:10.1016/j.jacc.2006.07.012
REFERENCES 1. McMurray J, Solomon S, Pieper K, et al. The effect of valsartan, captopril, or both on atherosclerotic events after acute myocardial infarction: an analysis of the Valsartan in Acute Myocardial Infarction Trial (VALIANT). J Am Coll Cardiol 2006;47:726 –33. 2. Teo K, Yusuf S, Sleight P, et al., for the ONTARGET/ TRANSCEND investigators. Rationale, design, and baseline characteristics of 2 large, simple, randomized trials evaluating telmisartan, ramipril, and their combination in high-risk patients: the Ongoing Telmisartan Alone and in Combination with Ramipril Global Endpoint Trial/Telmisartan Randomized Assessment Study in ACE Intolerant Subjects with Cardiovascular Disease (ONTARGET/ TRANSCEND) trials. Am Heart J 2004;148:52– 61.
Vol. 48, No. 7, 2006 ISSN 0735-1097/06/$32.00
CORRESPONDENCE Letters to the Editor Of Statistical Significance: “Trends” Toward Significance and Optimism Bias In the study by McMurray et al. (1), the investigators discuss a post hoc analysis of the VALIANT (Valsartan in Acute Myocardial Infarction) trial and conclude: “These data, although not conclusive, also support the hypothesis that adding an ARB (angiotensin receptor blocker) to an ACE (angiotensin-converting enzyme) inhibitor may have a small additional anti-infarction effect.” This conclusion was based on a “trend” toward decreased myocardial infarction (MI) and stroke compared with monotherapy. One has to question whether p values of 0.65 and 0.187 should be considered a “trend” toward significance. By the same analogy, there seems to be a “trend” toward greater incidence of angina in patients on combination therapy (based on survival curves). Moreover, based on the survival curves, at 6 months there seems to be a “trend” toward a greater number of strokes in the combination therapy compared to valsartan alone. There should be some thought/discussion on what can reasonably be defined as a “trend.” This question has surfaced in all post hoc analyses of randomized controlled trials where there is no preset criterion for significance. If we consider p ⬍ 0.05 as statistically significant, perhaps a p value between 0.05 to 0.10 may be considered as a trend toward significance. The conclusion that “These data also suggest, but do not prove, that adding an ARB to an ACE inhibitor may have a small additional anti-infarction effect” should thus be interpreted with caution. Given these somewhat slippery slopes, we call for a consensus as to what level of significance should be identified as a “trend” in a post hoc analysis and whether the observation of such a trend should be an acceptable basis for conclusions. Unless these terms are clearly defined a priori, one cannot quite avoid a euphemism such as “optimism bias” (2) to characterize the conclusions made by McMurray et al. (1).
Sripal Bangalore, MD, MHA *Franz H. Messerli, MD *Hypertension Program Division of Cardiology Columbia University College of Physicians and Surgeons St. Luke’s-Roosevelt Hospital Center 1000 Tenth Avenue Suite 3B-30 New York, New York 10025 E-mail: [email protected] doi:10.1016/j.jacc.2006.07.011
REFERENCES 1. McMurray J, Solomon S, Pieper K, et al. The effect of valsartan, captopril, or both on atherosclerotic events after acute myocardial
infarction: an analysis of the Valsartan in Acute Myocardial Infarction Trial (VALIANT). J Am Coll Cardiol 2006;47:726 –33. 2. Chalmers I, Matthews R. What are the implications of optimism bias in clinical research? Lancet 2006;367:449 –50.
REPLY Unfortunately, Drs. Bangalore and Messerli did not cite our concluding sentence about myocardial infarction in full, omitting the final statement that our observation was “a possibility that needs to be prospectively tested” (1). A p value of 0.187 (for the comparison of combination treatment with captopril and valsartan versus captopril monotherapy) might, with the appropriate caution we applied in the Abstract Conclusions and elsewhere, reasonably be described as a trend. In addition, in supportive analyses using other classifications of events (e.g., investigator reported versus adjudicated) and other statistical techniques, the p value for combination versus monotherapy was nominally statistically significant. Overall, strokes were numerically fewer in the combination treatment group than in the captopril-only group, with a p value of 0.079, which, according to Drs. Bangalore and Messerli’s own criteria, might be described as a trend. It is not clear why one would try to segment the Kaplan-Meier curves into 6-month time periods. We clearly stated that our observations had been made in a post hoc analysis, were not conclusive, and required prospective testing. Fortunately, such a trial has been conducted and will be reported in the near future (2). *John J. V. McMurray, MD, FACC Robert M. Califf, MD, FACC Marc A. Pfeffer, MD, PhD, FACC *Western Infirmary Cardiology Dumbarton Road Glasgow G11 6NT Scotland United Kingdom E-mail: [email protected] doi:10.1016/j.jacc.2006.07.012
REFERENCES 1. McMurray J, Solomon S, Pieper K, et al. The effect of valsartan, captopril, or both on atherosclerotic events after acute myocardial infarction: an analysis of the Valsartan in Acute Myocardial Infarction Trial (VALIANT). J Am Coll Cardiol 2006;47:726 –33. 2. Teo K, Yusuf S, Sleight P, et al., for the ONTARGET/ TRANSCEND investigators. Rationale, design, and baseline characteristics of 2 large, simple, randomized trials evaluating telmisartan, ramipril, and their combination in high-risk patients: the Ongoing Telmisartan Alone and in Combination with Ramipril Global Endpoint Trial/Telmisartan Randomized Assessment Study in ACE Intolerant Subjects with Cardiovascular Disease (ONTARGET/ TRANSCEND) trials. Am Heart J 2004;148:52– 61.