- Email: [email protected]
Journal Club: Outcome Measures, Interim Analyses, and Bayesian Approaches to Randomized Trials
ANNALS OF EMERGENCY MEDICINE JOURNAL CLUB
Journal Club: Outcome Measures, Interim Analyses, and Bayesian Approaches to Randomized Trials Editor’s Capsule Summary for Ferre et al1 What is already known on this topic: Patients with ureteral calculi are often prescribed adjunctive treatment with an ␣-blocking agent to enhance spontaneous stone passage. This practice has not been validated in emergency department (ED) patients. What question this study addressed: Does the addition of a 10-day course of tamsulosin to standard therapy after discharge from the ED increase the rate of passage of distal ureteral stones? What this study adds to our knowledge: In this randomized trial of 80 patients, most of whom had stones of 4 mm or less, time to stone passage was similar in tamsulosin and control patients. How this might change clinical practice: This study does not support the routine use of tamsulosin in ED patients, though it is possible that it would be beneficial in patients with larger stones.
Aaron M. Brown, MD Tyler W. Barrett, MD
University of Pittsburgh, Affiliated Residency in Emergency Medicine, Pittsburgh, PA Vanderbilt University Medical Center, Nashville, TN
0196-0644/$-see front matter Copyright © 2009 by the American College of Emergency Physicians. doi:10.1016/j.annemergmed.2009.07.006
SEE RELATED ARTICLE, P. 432. Editor’s Note: You are reading the 11th installment of Annals of Emergency Medicine Journal Club. This bimonthly feature seeks to improve the critical appraisal skills of emergency physicians and other interested readers through a guided critique of actual Annals of Emergency Medicine articles. Each Journal Club will pose questions that encourage readers— be they clinicians, academics, residents, or medical students—to critically appraise the literature. During a 2- to 3-year cycle, we plan to ask questions that cover the main topics in research methodology and critical appraisal of the literature. To do this, we will select articles that use a variety of study designs and analytic techniques. These may or may not be the most clinically important articles in a specific issue, but they are articles that serve the mission of covering the clinical epidemiology curriculum. Journal Club entries are published in 2 phases. In the first phase, a list of questions about the article is published in the issue in which the article appears. Questions are rated “novice,” ( ) “intermediate,” ( ) and “advanced” ( ) so that individuals planning a journal club can assign the right question to the right student. The answers to this journal club will be published in the February 2010 issue. US residency directors will have immediate access to the answers through the Council of Emergency Medicine Residency Directors Share Point Web site. International residency directors can gain access to the questions by going to http://www.emergencymedicine.ucla. edu/annalsjc/ and following the directions. Thus, if a program conducts its journal club within 5 months of the publication of the questions, no one will have access to the published
440 Annals of Emergency Medicine
answers except the residency director. The purpose of delaying the publication of the answers is to promote discussion and critical review of the literature by residents and medical students and discourage regurgitation of the published answers. It is our hope that the Journal Club will broaden Annals of Emergency Medicine’s appeal to residents and medical students. We are interested in receiving feedback about this feature. Please e-mail [email protected] with your comments.
DISCUSSION POINTS 1. A well-designed clinical trial will be meaningless if the outcome measure is inappropriate, irrelevant, or unhelpful. A. What are the qualities of an ideal outcome measure? B. There are multiple types of outcome measures in clinical trials, including biomarkers, surrogate (aka intermediate) endpoints, and clinical (aka patientcentered) endpoints. Discuss the benefits and limitations of each of these outcome measures. Can you identify a recent high-profile case in which a drug with successful surrogate marker trials failed to demonstrate success when clinical endpoints were measured? C. In this small, randomized, controlled trial (RCT) of tamsulosin for the treatment of distal ureterolithiasis, the authors chose to study the primary outcome of successful spontaneous passage of renal calculi by 14 days. Is this a reasonable outcome measure to evaluate tamsulosin’s Volume , . : September
Brown & Barrett efficacy? How might a patient’s failure to recognize stone passage affect the results? Postulate why the authors used the passage of calculi by 14 days rather than time to passage as the primary outcome. D. Large clinical trials often use a composite outcome measurement. For example, a new heart failure trial might measure a composite outcome of death, emergency department visit, and/or unplanned hospitalization within 30 days. What are the advantages of using a composite endpoint? How might a composite outcome affect the trial’s conclusions? E. This small trial also examines many secondary outcomes such as days of work missed, time to stone passage, adverse events, number of pain episodes, amount of pain medication used, and the number of return visits. What are the benefits of secondary outcomes, and what does a lack of a clinically important change in secondary outcomes usually mean? 2. Clarke et al2 have argued that every published RCT should begin with a systematic review of previous studies and conclude with a revised systematic review that shows how the RCT changes our belief about the topic. They suggest that this structure would foster the conduct of meaningful research. A. Speculate why the editors deemed this study sufficiently important to warrant publication in Annals. B. How did the ED-based study population and the other methodology factors discussed in answer 2a affect this study’s results compared with previous studies? 3. In this study the authors randomized patients to a 10-day course of standard analgesic therapy or tamsulosin plus standard therapy. Study participants were not blinded to their treatment assignment because the investigators elected not to administer placebo tablets to patients randomized to the standard analgesic-only arm. A. It goes without saying that randomization is a central issue in the conduct of randomized clinical trials. Discuss the pros and cons of the following randomization techniques: simple randomization, prerandomization, block randomization, and stratified block randomization. What randomization technique did these authors select?
Volume , . : September
Journal Club How might a different randomization strategy have influenced the study results? B. Do you feel that the lack of placebo tablet in this trial altered the results? What if the primary outcome were patient ratings of their pain? Do you think it would have been more or less important to blind patients to their treatment assignment through the use of a placebo tablet? C. Imagine you are designing a placebo-controlled drug trial to treat renal colic in the ED. What practical considerations about the inclusion of the placebo do you need to weigh during the trial design? How might the study’s conclusions be affected if the participants or the investigators measuring the outcome can differentiate active drug from placebo? How might investigators measure whether study participants are truly blinded to whether the treatment is active or placebo? 4. In this study the investigators performed an interim analysis and found no issues prompting early stoppage of the trial. A. What are the reasons that investigators perform interim analyses and why might a trial be stopped? How do planned interim analyses affect the study sample size? Why is it important to specify planned interim analyses in a clinical trial that is using a classic statistical analysis? B. The Bayesian statistical approach provides an alternative, and many would argue more robust, strategy for trial design and analysis. Describe the major differences between the Bayesian statistical approach and the more commonly used frequentist statistical approach with regard to hypothesis testing, interim analyses, analyzing the data, and interpreting the study’s results. Section editors: Tyler W. Barrett, MD; David L. Schriger, MD, MPH REFERENCES 1. Ferre RM, Wasielewski JN, Strout TD, et al. Tamsulosin for uretal stones in the emergency department: a randomized, controlled trial. Ann Emerg Med. 2009;54:432-439. 2. Clarke M, Sally Hopewell S, Chalmers I. Reports of clinical trials should begin and end with up-to-date systematic reviews of other relevant evidence: a status report. J R Soc Med. 2007;100:187190.
Annals of Emergency Medicine 441
Journal Club: Outcome Measures, Interim Analyses, and Bayesian Approaches to Randomized Trials Editor’s Capsule Summary for Ferre et al1 What is already known on this topic: Patients with ureteral calculi are often prescribed adjunctive treatment with an ␣-blocking agent to enhance spontaneous stone passage. This practice has not been validated in emergency department (ED) patients. What question this study addressed: Does the addition of a 10-day course of tamsulosin to standard therapy after discharge from the ED increase the rate of passage of distal ureteral stones? What this study adds to our knowledge: In this randomized trial of 80 patients, most of whom had stones of 4 mm or less, time to stone passage was similar in tamsulosin and control patients. How this might change clinical practice: This study does not support the routine use of tamsulosin in ED patients, though it is possible that it would be beneficial in patients with larger stones.
Aaron M. Brown, MD Tyler W. Barrett, MD
University of Pittsburgh, Affiliated Residency in Emergency Medicine, Pittsburgh, PA Vanderbilt University Medical Center, Nashville, TN
0196-0644/$-see front matter Copyright © 2009 by the American College of Emergency Physicians. doi:10.1016/j.annemergmed.2009.07.006
SEE RELATED ARTICLE, P. 432. Editor’s Note: You are reading the 11th installment of Annals of Emergency Medicine Journal Club. This bimonthly feature seeks to improve the critical appraisal skills of emergency physicians and other interested readers through a guided critique of actual Annals of Emergency Medicine articles. Each Journal Club will pose questions that encourage readers— be they clinicians, academics, residents, or medical students—to critically appraise the literature. During a 2- to 3-year cycle, we plan to ask questions that cover the main topics in research methodology and critical appraisal of the literature. To do this, we will select articles that use a variety of study designs and analytic techniques. These may or may not be the most clinically important articles in a specific issue, but they are articles that serve the mission of covering the clinical epidemiology curriculum. Journal Club entries are published in 2 phases. In the first phase, a list of questions about the article is published in the issue in which the article appears. Questions are rated “novice,” ( ) “intermediate,” ( ) and “advanced” ( ) so that individuals planning a journal club can assign the right question to the right student. The answers to this journal club will be published in the February 2010 issue. US residency directors will have immediate access to the answers through the Council of Emergency Medicine Residency Directors Share Point Web site. International residency directors can gain access to the questions by going to http://www.emergencymedicine.ucla. edu/annalsjc/ and following the directions. Thus, if a program conducts its journal club within 5 months of the publication of the questions, no one will have access to the published
440 Annals of Emergency Medicine
answers except the residency director. The purpose of delaying the publication of the answers is to promote discussion and critical review of the literature by residents and medical students and discourage regurgitation of the published answers. It is our hope that the Journal Club will broaden Annals of Emergency Medicine’s appeal to residents and medical students. We are interested in receiving feedback about this feature. Please e-mail [email protected] with your comments.
DISCUSSION POINTS 1. A well-designed clinical trial will be meaningless if the outcome measure is inappropriate, irrelevant, or unhelpful. A. What are the qualities of an ideal outcome measure? B. There are multiple types of outcome measures in clinical trials, including biomarkers, surrogate (aka intermediate) endpoints, and clinical (aka patientcentered) endpoints. Discuss the benefits and limitations of each of these outcome measures. Can you identify a recent high-profile case in which a drug with successful surrogate marker trials failed to demonstrate success when clinical endpoints were measured? C. In this small, randomized, controlled trial (RCT) of tamsulosin for the treatment of distal ureterolithiasis, the authors chose to study the primary outcome of successful spontaneous passage of renal calculi by 14 days. Is this a reasonable outcome measure to evaluate tamsulosin’s Volume , . : September
Brown & Barrett efficacy? How might a patient’s failure to recognize stone passage affect the results? Postulate why the authors used the passage of calculi by 14 days rather than time to passage as the primary outcome. D. Large clinical trials often use a composite outcome measurement. For example, a new heart failure trial might measure a composite outcome of death, emergency department visit, and/or unplanned hospitalization within 30 days. What are the advantages of using a composite endpoint? How might a composite outcome affect the trial’s conclusions? E. This small trial also examines many secondary outcomes such as days of work missed, time to stone passage, adverse events, number of pain episodes, amount of pain medication used, and the number of return visits. What are the benefits of secondary outcomes, and what does a lack of a clinically important change in secondary outcomes usually mean? 2. Clarke et al2 have argued that every published RCT should begin with a systematic review of previous studies and conclude with a revised systematic review that shows how the RCT changes our belief about the topic. They suggest that this structure would foster the conduct of meaningful research. A. Speculate why the editors deemed this study sufficiently important to warrant publication in Annals. B. How did the ED-based study population and the other methodology factors discussed in answer 2a affect this study’s results compared with previous studies? 3. In this study the authors randomized patients to a 10-day course of standard analgesic therapy or tamsulosin plus standard therapy. Study participants were not blinded to their treatment assignment because the investigators elected not to administer placebo tablets to patients randomized to the standard analgesic-only arm. A. It goes without saying that randomization is a central issue in the conduct of randomized clinical trials. Discuss the pros and cons of the following randomization techniques: simple randomization, prerandomization, block randomization, and stratified block randomization. What randomization technique did these authors select?
Volume , . : September
Journal Club How might a different randomization strategy have influenced the study results? B. Do you feel that the lack of placebo tablet in this trial altered the results? What if the primary outcome were patient ratings of their pain? Do you think it would have been more or less important to blind patients to their treatment assignment through the use of a placebo tablet? C. Imagine you are designing a placebo-controlled drug trial to treat renal colic in the ED. What practical considerations about the inclusion of the placebo do you need to weigh during the trial design? How might the study’s conclusions be affected if the participants or the investigators measuring the outcome can differentiate active drug from placebo? How might investigators measure whether study participants are truly blinded to whether the treatment is active or placebo? 4. In this study the investigators performed an interim analysis and found no issues prompting early stoppage of the trial. A. What are the reasons that investigators perform interim analyses and why might a trial be stopped? How do planned interim analyses affect the study sample size? Why is it important to specify planned interim analyses in a clinical trial that is using a classic statistical analysis? B. The Bayesian statistical approach provides an alternative, and many would argue more robust, strategy for trial design and analysis. Describe the major differences between the Bayesian statistical approach and the more commonly used frequentist statistical approach with regard to hypothesis testing, interim analyses, analyzing the data, and interpreting the study’s results. Section editors: Tyler W. Barrett, MD; David L. Schriger, MD, MPH REFERENCES 1. Ferre RM, Wasielewski JN, Strout TD, et al. Tamsulosin for uretal stones in the emergency department: a randomized, controlled trial. Ann Emerg Med. 2009;54:432-439. 2. Clarke M, Sally Hopewell S, Chalmers I. Reports of clinical trials should begin and end with up-to-date systematic reviews of other relevant evidence: a status report. J R Soc Med. 2007;100:187190.
Annals of Emergency Medicine 441