- Email: [email protected]
Are we letting good stories obscure the truth—Can qualitative techniques provide clarification?
Editorial
Are we letting good stories obscure the truth—Can qualitative techniques provide clarification? John K. French, MB, PhD, and Jennifer A. Shearer, PhD
We need an occasional wake-up call in respect of citation of data collected in clinical science. Large randomized clinical trials can create opportunities for valuable opportunities to assess the significance of the data that may not be obvious at the time of publication. A case in point is the article of Siontis et al1 (p. 695-701) that analyzes citations of the OAT (Occluded Artery Trial) and COURAGE (Clinical Outcomes Utilizing Revascularization and Aggressive Drug Evaluation) trials2,3 and concludes that significant numbers of clinicians, in this instance, interventional cardiologists, may be resistant to the findings of these landmark trials, which concluded that 2 clinical scenarios of routine percutaneous coronary intervention (PCI) should not be performed. The authors independently collected data in triplicate and reported the proportion of citing articles that were “reserved” (did not agree with main study findings, trial design, interpretation of findings, etc) or “unreserved” (essentially agreed with the scientific validity and interpretation of the study findings); discrepancies were resolved by a fourth investigator. In the article in this edition of the journal, Siontis et al1 use an innovative approach to the analysis of word usage in manuscripts, often commentaries, citing two trials published in the New England Journal of Medicine in 2006 and 2007, OAT and COURAGE.1-3 These trials compared strategies of routine PCI with initial medical therapy for 2 different patient groups with coronary heart disease, in COURAGE patients with stable coronary heart disease who have coronary anatomy suitable for PCI, and stable (essentially) asymptomatic patients on days 3 to 28 post myocardial infarction (MI), with occluded infarct arteries of which 25% had late spontaneous recanalisation.4 The article evaluated whether citations with a “reserved stance” would be associated with either an interventional cardiologist author or editorialist, or with an author who had reported relationships with the device industry. They also examined the association of the stance of the author/editorialist with other available
From South West Sydney Clinical School, University of South Wales, Sydney, NSW, Australia. Submitted January 26, 2009; accepted February 3, 2009. Reprint requests: John K. French, MB, PhD, Department of Cardiology, University of New South Wales, Liverpool Hospital, Elizabeth St, Liverpool, NSW 2170, Australia. E-mail: [email protected] 0002-8703/$ - see front matter © 2009, Published by Mosby, Inc. doi:10.1016/j.ahj.2009.02.004
author and article characteristics. A total of 94 articles were cited, 59 critiquing the COURAGE trial, and 35, the OAT trial. Non-English articles and reports that were not full text articles were excluded. This technique and others, both quantitative and qualitative, have the potential to provide a measure of confidence for clinicians faced with “expert commentaries” on what is an increasingly baffling quantity and variety of information. Most well-designed studies generate data that may be useful both immediately and potentially in the long term. Smaller trials may attempt to address questions, and in fact, their methods and/or results provide hypotheses be examined more definitively in larger studies, such as those which generated the hypotheses which lead to the design of both the OAT and COURAGE trials. Large landmark trials may not be repeated and can have the implications of their findings incorporated into clinical practice guidelines without needing a second or third large trial yet, as this article indicates, may still suffer from citation bias and the associated fear of selective memory about what was originally reported. There is in this scenario a real possibility of practice drift, making this analytic technique one that may be developed, given close scrutiny of individual aspects of clinical studies, into a widespread tool of considerable importance. The systematic review of articles using expert interpretation avoids many of the semantic issues encountered by keyword-based “data-mining studies of the literature in the past.” Lest we consider such concerns unique to randomized clinical trial, Ioannidis with other coauthors have reported somewhat similar concerns about “repeatability of published microarray gene expression analyses”.5 The willingness of cardiologists to lead the way forward with meticulous testing of established practice and checks against bias, whether personal or created by lack of balance in the work of others, is a measure of success as a clinical science community. The technique will doubtlessly be taken up in other areas of clinical science as well as biomedical science and population health and, potentially, other applications. Certainly, no one disputes that both COURAGE and OAT have been lightning rods for discussion and debate. Both trials found clinical results that challenged so-called conventional wisdom, and as such, it is not surprising that there was a spectrum of professional opinion regarding the interpretation of trial results, study design, generalizability, and so on. It would
American Heart Journal April 2009
598 French and Shearer
be entirely expected that some critiques of both studies or editorials would be critical and not supportive. Given the polarity of opinion that both OAT and COURAGE have continued to create, as demonstrated by vigorous debates at major international meetings in 2008, some may have considered 15% to 20% of cited articles being categorized as reserved surprisingly small. This is even more so as some reserved comments by interventionalists, such as those relating to inadequate sample size, “do have some justification,” as the authors acknowledge. Examples include those about COURAGE: “despite the liberalized definition of MI and extended duration of follow-up, the observed rate of death or MI in medically treated patients to 3 years of follow-up was only 12% (413 end point events or 67% of the projected requirement), so that the trial remains underpowered,”6 and those about OAT: “only 301 study events (approximately 60% of the expected rate), suggesting that the report was published well before full information was available.”7 However, as a representation of the degree of deviation from the wording of these trials findings, this figure of approximately 1 of 5 is large. It is of interest that 80% to 85% of the articles which underwent citation analysis were unreserved. This indicates that the vast majority of commentaries or editorials on the COURAGE and OAT trials were supportive of these trials. Thus, a positive message could have been concluded from this article, rather than a critical appraisal of the minority of authors who expressed a reserved status. The article could serve a very useful function to objectively clarify the extent and degree to which these two important randomized controlled trials have been vetted in the published literature over the past two years. The subsequent relative impact of these publications will more clearly determine the validity of this consensus. Sionitis et al1 are rather selective in citations themselves when they quote a single registry to support their contention that medical management may be the most cost-effective strategy in stable patients. However, they use the results of 4 meta-analyses, albeit still underpowered, to examine effects on mortality to support their contention that the use of drug-eluting
stents is unlikely to improve mortality, whereas an increasingly large body of registry data in general reports the use of drug-eluting stents to be associated with increased survival.8 For consistency, if registry data are cited to support an argument, these should be discussed to provide balance to the clinician's ongoing review process by which they incorporate new information to their clinical decision making. Whether the innovative qualitative analytic approach of Sionitis et al1 will become an established part of the medical literature remains to be determined. However, it is to be hoped that others will undertake similar analyses to assess the citations of pivotal clinical trials. As with quantitative research, qualitative techniques will have limitations, and refinements should enhance the ability to discriminate between legitimate criticism, such as a trial being underpowered, and comments which misrepresent a trial's key message.
References 1. Siontis GMC, Tatsioni A, Katritsis DP, et al. Persistent reservations against contradicted percutaneous coronary intervention indications: citation content analysis. Am Heart J 2009;157:696-702. 2. Hochman JS, Lamas GA, Buller CE, et al. For the Occluded Artery Trial Investigators. Coronary intervention for persistent occlusion after myocardial infarction. N Engl J Med 2006;355:2395-407. 3. Boden WE, O'Rourke RA, Teo KK, et al. For the COURAGE Trial Research Group. Optimal medical therapy with or without PCI for stable coronary disease. N Engl J Med 2007;356:1503-16. 4. Dzavík V, Buller CE, Lamas GA, et al. For the TOSCA-2 Investigators. Randomized trial of percutaneous coronary intervention for subacute infarct-related coronary artery occlusion to achieve long-term patency and improve ventricular function: the Total Occlusion Study of Canada (TOSCA)–2 Trial. Circulation 2006;114:2449-57. 5. Ioannidis JPA, Allison DB, Ball CA, et al. Repeatability of published microarray gene expression analyses. Nat Genet 2009;41:149-55. 6. Kereiakes DJ, Teirstein PS, Sarembock IJ, et al. The truth and consequences of the COURAGE trial. J Am Coll Cardiol 2007;50: 1598-603. 7. Anderson JR. Persistent coronary occlusion after myocardial infarction. N Engl J Med 2007;356:1681. 8. Mauri L, Silbaugh TS, Wolf RE, et al. Long-term clinical outcomes after drug-eluting and bare-metal stenting in Massachusetts. Circulation 2008;118:1817-27.
Are we letting good stories obscure the truth—Can qualitative techniques provide clarification? John K. French, MB, PhD, and Jennifer A. Shearer, PhD
We need an occasional wake-up call in respect of citation of data collected in clinical science. Large randomized clinical trials can create opportunities for valuable opportunities to assess the significance of the data that may not be obvious at the time of publication. A case in point is the article of Siontis et al1 (p. 695-701) that analyzes citations of the OAT (Occluded Artery Trial) and COURAGE (Clinical Outcomes Utilizing Revascularization and Aggressive Drug Evaluation) trials2,3 and concludes that significant numbers of clinicians, in this instance, interventional cardiologists, may be resistant to the findings of these landmark trials, which concluded that 2 clinical scenarios of routine percutaneous coronary intervention (PCI) should not be performed. The authors independently collected data in triplicate and reported the proportion of citing articles that were “reserved” (did not agree with main study findings, trial design, interpretation of findings, etc) or “unreserved” (essentially agreed with the scientific validity and interpretation of the study findings); discrepancies were resolved by a fourth investigator. In the article in this edition of the journal, Siontis et al1 use an innovative approach to the analysis of word usage in manuscripts, often commentaries, citing two trials published in the New England Journal of Medicine in 2006 and 2007, OAT and COURAGE.1-3 These trials compared strategies of routine PCI with initial medical therapy for 2 different patient groups with coronary heart disease, in COURAGE patients with stable coronary heart disease who have coronary anatomy suitable for PCI, and stable (essentially) asymptomatic patients on days 3 to 28 post myocardial infarction (MI), with occluded infarct arteries of which 25% had late spontaneous recanalisation.4 The article evaluated whether citations with a “reserved stance” would be associated with either an interventional cardiologist author or editorialist, or with an author who had reported relationships with the device industry. They also examined the association of the stance of the author/editorialist with other available
From South West Sydney Clinical School, University of South Wales, Sydney, NSW, Australia. Submitted January 26, 2009; accepted February 3, 2009. Reprint requests: John K. French, MB, PhD, Department of Cardiology, University of New South Wales, Liverpool Hospital, Elizabeth St, Liverpool, NSW 2170, Australia. E-mail: [email protected] 0002-8703/$ - see front matter © 2009, Published by Mosby, Inc. doi:10.1016/j.ahj.2009.02.004
author and article characteristics. A total of 94 articles were cited, 59 critiquing the COURAGE trial, and 35, the OAT trial. Non-English articles and reports that were not full text articles were excluded. This technique and others, both quantitative and qualitative, have the potential to provide a measure of confidence for clinicians faced with “expert commentaries” on what is an increasingly baffling quantity and variety of information. Most well-designed studies generate data that may be useful both immediately and potentially in the long term. Smaller trials may attempt to address questions, and in fact, their methods and/or results provide hypotheses be examined more definitively in larger studies, such as those which generated the hypotheses which lead to the design of both the OAT and COURAGE trials. Large landmark trials may not be repeated and can have the implications of their findings incorporated into clinical practice guidelines without needing a second or third large trial yet, as this article indicates, may still suffer from citation bias and the associated fear of selective memory about what was originally reported. There is in this scenario a real possibility of practice drift, making this analytic technique one that may be developed, given close scrutiny of individual aspects of clinical studies, into a widespread tool of considerable importance. The systematic review of articles using expert interpretation avoids many of the semantic issues encountered by keyword-based “data-mining studies of the literature in the past.” Lest we consider such concerns unique to randomized clinical trial, Ioannidis with other coauthors have reported somewhat similar concerns about “repeatability of published microarray gene expression analyses”.5 The willingness of cardiologists to lead the way forward with meticulous testing of established practice and checks against bias, whether personal or created by lack of balance in the work of others, is a measure of success as a clinical science community. The technique will doubtlessly be taken up in other areas of clinical science as well as biomedical science and population health and, potentially, other applications. Certainly, no one disputes that both COURAGE and OAT have been lightning rods for discussion and debate. Both trials found clinical results that challenged so-called conventional wisdom, and as such, it is not surprising that there was a spectrum of professional opinion regarding the interpretation of trial results, study design, generalizability, and so on. It would
American Heart Journal April 2009
598 French and Shearer
be entirely expected that some critiques of both studies or editorials would be critical and not supportive. Given the polarity of opinion that both OAT and COURAGE have continued to create, as demonstrated by vigorous debates at major international meetings in 2008, some may have considered 15% to 20% of cited articles being categorized as reserved surprisingly small. This is even more so as some reserved comments by interventionalists, such as those relating to inadequate sample size, “do have some justification,” as the authors acknowledge. Examples include those about COURAGE: “despite the liberalized definition of MI and extended duration of follow-up, the observed rate of death or MI in medically treated patients to 3 years of follow-up was only 12% (413 end point events or 67% of the projected requirement), so that the trial remains underpowered,”6 and those about OAT: “only 301 study events (approximately 60% of the expected rate), suggesting that the report was published well before full information was available.”7 However, as a representation of the degree of deviation from the wording of these trials findings, this figure of approximately 1 of 5 is large. It is of interest that 80% to 85% of the articles which underwent citation analysis were unreserved. This indicates that the vast majority of commentaries or editorials on the COURAGE and OAT trials were supportive of these trials. Thus, a positive message could have been concluded from this article, rather than a critical appraisal of the minority of authors who expressed a reserved status. The article could serve a very useful function to objectively clarify the extent and degree to which these two important randomized controlled trials have been vetted in the published literature over the past two years. The subsequent relative impact of these publications will more clearly determine the validity of this consensus. Sionitis et al1 are rather selective in citations themselves when they quote a single registry to support their contention that medical management may be the most cost-effective strategy in stable patients. However, they use the results of 4 meta-analyses, albeit still underpowered, to examine effects on mortality to support their contention that the use of drug-eluting
stents is unlikely to improve mortality, whereas an increasingly large body of registry data in general reports the use of drug-eluting stents to be associated with increased survival.8 For consistency, if registry data are cited to support an argument, these should be discussed to provide balance to the clinician's ongoing review process by which they incorporate new information to their clinical decision making. Whether the innovative qualitative analytic approach of Sionitis et al1 will become an established part of the medical literature remains to be determined. However, it is to be hoped that others will undertake similar analyses to assess the citations of pivotal clinical trials. As with quantitative research, qualitative techniques will have limitations, and refinements should enhance the ability to discriminate between legitimate criticism, such as a trial being underpowered, and comments which misrepresent a trial's key message.
References 1. Siontis GMC, Tatsioni A, Katritsis DP, et al. Persistent reservations against contradicted percutaneous coronary intervention indications: citation content analysis. Am Heart J 2009;157:696-702. 2. Hochman JS, Lamas GA, Buller CE, et al. For the Occluded Artery Trial Investigators. Coronary intervention for persistent occlusion after myocardial infarction. N Engl J Med 2006;355:2395-407. 3. Boden WE, O'Rourke RA, Teo KK, et al. For the COURAGE Trial Research Group. Optimal medical therapy with or without PCI for stable coronary disease. N Engl J Med 2007;356:1503-16. 4. Dzavík V, Buller CE, Lamas GA, et al. For the TOSCA-2 Investigators. Randomized trial of percutaneous coronary intervention for subacute infarct-related coronary artery occlusion to achieve long-term patency and improve ventricular function: the Total Occlusion Study of Canada (TOSCA)–2 Trial. Circulation 2006;114:2449-57. 5. Ioannidis JPA, Allison DB, Ball CA, et al. Repeatability of published microarray gene expression analyses. Nat Genet 2009;41:149-55. 6. Kereiakes DJ, Teirstein PS, Sarembock IJ, et al. The truth and consequences of the COURAGE trial. J Am Coll Cardiol 2007;50: 1598-603. 7. Anderson JR. Persistent coronary occlusion after myocardial infarction. N Engl J Med 2007;356:1681. 8. Mauri L, Silbaugh TS, Wolf RE, et al. Long-term clinical outcomes after drug-eluting and bare-metal stenting in Massachusetts. Circulation 2008;118:1817-27.