- Email: [email protected]
Reporting of harms and adverse events in otolaryngology journals
Otolaryngology–Head and Neck Surgery (2009) 140, 241-244
ORIGINAL RESEARCH– GENERAL OTOLARYNGOLOGY
Reporting of harms and adverse events in otolaryngology journals Haidy Bibawy, MD, Anne Cossu, MD, Sophia Cogan, MD, and Richard Rosenfeld, MD, MPH, Brooklyn, NY; and Boston, MA OBJECTIVE: To identify the frequency and quality of harms and adverse events reporting in otolaryngology journals. STUDY DESIGN: A literature review. METHODS: Four major otolaryngology journals from 2006 and 1996 were reviewed. Clinical research studies offering therapeutic recommendations were evaluated for frequency and quality of harms and adverse events reporting. RESULTS: Of 1,835 total articles reviewed, 576 (31%) offered therapeutic recommendations. Sixty-five percent provided any mention of harms or adverse events, 47 percent explicitly defined the events, and 24 percent described methodology for collecting harms data. The median number of harms or adverse events reported was three. Studies concluding a beneficial effect of therapy were more likely to not mention adverse events (odds ratio 2.99, P ⫽ .007) compared with studies concluding no benefit. Studies of surgical therapy were more likely to report harms or adverse events (odds ratio 1.46, P ⫽ .046). CONCLUSIONS: Harms and adverse events are underreported and poorly described in otolaryngology journals with about one third of clinical research not mentioning adverse events at all. Most authors do not explicitly describe harms or adverse events (53%) or the methodology behind collecting adverse events data (76%). Underreporting is more likely when a therapeutic effect is found to be beneficial. No sponsorships or competing interests have been disclosed for this article. © 2009 American Academy of Otolaryngology–Head and Neck Surgery Foundation. All rights reserved.
O
ne of the oldest medical tenets is to first “do no harm.”1 This expression took on new meaning in the mid-1990s with an explosive interest in health care safety.2 Thereafter, reporting of safety conditions in hospitals, side effects of medication, and surgical complications have been receiving increased attention in medical literature and the media as a whole. Ioannidis and Lau3 and Martin et al4 both report underreporting of harms and adverse events (AEs) in medical and surgical literature. Harms and AEs have also taken a greater role in medical decision making.5 When considering the balance between risks and benefits of various therapies, harms are often the
preeminent issue in making guideline recommendations, at times superseding levels of evidence. Harms reporting is now suggested as a standard for publishing therapeutic recommendations. The Consolidated Standards of Reporting Trials (CONSORT) statement, first reported in 1996, has been revised multiple times to include more recommendations on the reporting of harms and AEs.6,7 Despite increasing importance of harms and AEs in medical decisions, their reporting has yet to be evaluated in otolaryngology literature. Therefore, the present study sought to identify the frequency and quality of harms and AEs reporting in leading peer-reviewed otolaryngology journals.
METHODS Publications from the four highest impact clinical otolaryngology journals in the years 1996 and 2006 were reviewed. These were Otolaryngology–Head and Neck Surgery, Laryngoscope, Archives of Otolaryngology–Head and Neck Surgery, and Annals of Otology, Rhinology and Laryngology. Included in the study were clinical research publications offering therapeutic recommendations, and these were evaluated for their frequency and quality of harms and AEs reporting. Any study dealing with living human subjects or directly related with patient care was defined as “clinical.”8 Studies dealing with bodily fluids, tissues, or specimens were considered “clinical” as well if they correlated with patient care, individual patient histories, or disease states. “Nonclinical” research included any animal or laboratory studies; postmortem studies not correlated with patient histories; physician questionnaires; and surveys of manpower, education, cost-effectiveness, or quality control. Therefore, “nonclinical” research includes but is not limited to basic science. Data collection was planned a priori and conducted using a standardized checklist. Articles were reviewed by one of three of the authors (HB, SK, and AC) who were not blinded as to the source and year of publication. An initial subset of 30 articles was reviewed by all reviewers to ensure interrater reliability.
Received September 3, 2008; revised October 28, 2008; accepted November 12, 2008.
0194-5998/$36.00 © 2009 American Academy of Otolaryngology–Head and Neck Surgery Foundation. All rights reserved. doi:10.1016/j.otohns.2008.11.015
242
Otolaryngology–Head and Neck Surgery, Vol 140, No 2, February 2009
Data were collected under the broad headings of study design, level of evidence, and harms/AE information. Information related to study design include patient age, sample size, control groups, and the study focus (medical therapy, surgical therapy, or nontherapeutic). For all therapy articles, the level of evidence was determined using the following definitions: level 1, randomized controlled trial; level 2, controlled prospective study; level 3, controlled retrospective study; level 4, uncontrolled case series; and level 5, expert opinion without explicit appraisal. Face validity of the questions relating to harms/AEs was based on the 2003 extension of the CONSORT statement6 and similar investigations published in peer-reviewed journals.3,4 Questions related to general reporting of harms/AEs in the article overall and subsections of the manuscript (abstract, results, tables, and figures), with emphasis on the thoroughness of reporting and the extent to which information about harms/AEs was considered in the discussion section when drawing conclusions or making recommendations, were evaluated. Data were entered into an Excel (Microsoft Corp, Redmond, WA) spreadsheet and verified for accuracy. All statistical analyses were performed by using SPSS software (SPSS Inc, Chicago, IL).9 Bivariate comparisons were made using a chi-square test for categoric data and a Fisher exact test whenever the expected count in any cell was less than five. Odds ratios were calculated, when appropriate, using a 95 percent confidence interval. All tests were performed by using a twosided alpha level of 0.05 for statistical significance.
RESULTS A total of 1,835 articles were reviewed. Of those, 1399 (76%) were clinical research studies. Of the 1,399, 53 percent included only adults, 17 percent children, and 30 percent included both or did not specify age restrictions. Only 26 percent of the clinical research studies had a control or comparison group, and the median sample size was 28. The number of studies making therapeutic recommendations was 576 (31%) of the total reviewed; 70 percent made
surgical recommendations and 30 percent were medical. Ninety-two percent of the studies concluded that the therapy recommended was beneficial. With regards to levels of evidence, it was found that the majority of clinical studies were level 4 case series. Ten percent was level 1 evidence, 8 percent fell into the category of level 2, and 7 percent were level 3. When examining harms and AEs reporting specifically, it was found that 65 percent (377/576) of articles mentioned harms or adverse events anywhere in the manuscript. These 377 articles were further examined for the quality of harms and AE reporting. Seventy-one percent of the studies defined explicitly harms and/or AEs. Approximately 27 percent mention harms in the abstract, 64 percent in the results, 70 percent in the discussion, and 29 percent in tables/figures sections. Thirty-four percent of the studies describe a methodology for collecting harms data; 29 percent provide a comparison of harms of the therapy recommended and other therapies; 7 percent state if therapy needed to be stopped due to adverse events during the study, and 10 percent mention a comparison of harms versus benefits. Factors predictive of not mentioning harms or AEs were also examined. When the investigators reported a benefit of therapy, they were almost three times more likely to not mention harms or AEs compared with studies that reported no benefit (odds ratio 2.99; 95% confidence interval [CI], 1.30-6.81; P ⫽ .007). Whereas 36 percent of “positive” studies did not report harms, only 16 percent of “negative” studies were remiss. Similarly, studies of medical interventions were less likely to report AEs than reports of surgery (odds ratio 1.46; 95% CI, 1.01-2.10; P ⫽ .046). Although a significant relationship was found, the 95 percent CI includes 1.01 as a potentially plausible value for the population parameter, which is consistent with a trivial difference. In other words, we are somewhat uncertain of how important this finding is because of limited precision (eg, a broad 95% CI). Age group and level of evidence were not associated with reporting outcomes. Table 1 summarizes the trends in AE reporting for 2006 versus 1996. Compared with 1996, articles published in 2006 more often defined explicitly the methodology for
Table 1 Trends in AE reporting for 2006 versus 1996 Factor: n/N (%) AE mentioned AE defined explicitly AE methodology defined AE mentioned in abstract AE mentioned in results section AE mentioned in tables/figures State need to stop therapy if AE Discussion section mentions AE Discussion compares AEs with other treatment options Discussion of relationship of harms vs. benefits
2006 244/384 181/245 108/245 131/245 218/245 89/245 18/245 181/245 75/245 30/245
(64) (74) (44) (53) (89) (36) (7) (74) (31) (12)
1996 133/192 89/133 32/133 60/133 119/133 26/133 7/133 82/133 35/133 8/133
(69) (67) (24) (45) (90) (19) (5) (62) (26) (6)
Total 377/576 270/378 140/378 191/378 337/378 115/378 25/378 263/378 110/378 38/378
(65) (71) (37) (50) (89) (30) (7) (70) (29) (1)
P value .193 .155 ⬍.001 .276 .460 .002 .520 .019 .408 .072
Bibawy et al
Reporting of harms and adverse events . . .
monitoring adverse events (44% vs 24%, P ⬍ .001), mentioned more events in the articles’ tables or figures (36% vs 19%, P ⫽ .002), and mentioned more events in the discussion section of the article (74% vs 62%, P ⫽ .019). The overall frequency that adverse events were mentioned in the article, abstract, and results section was unchanged. No change was observed in how information about AEs was used to compare alternative treatment options or to balance decisions about treatment benefits.
DISCUSSION As the first examination of harms and AEs reporting in otolaryngology literature, this study provides significant insights. Although 31 percent of clinical research in major otolaryngology journals offered therapeutic recommendations, only 65 percent of authors mentioned harms or AEs anywhere in the article. When events were mentioned, only 47 percent explicitly defined the events and 24 percent described methodology for collecting harms data. Studies concluding a beneficial effect of therapy were three times more likely to not mention AEs compared with studies concluding no benefit, and studies of surgical therapy were approximately one and half times more likely to report harms or AEs. Our review also shows that most of the clinical research published in the otolaryngology literature consists of uncontrolled case series of surgical interventions (70%) that nearly always (92%) report a beneficial effect for the intervention discussed. These findings are consistent with other published reviews. Bentsianov et al8 showed that 80 percent of therapy recommendations were supported by uncontrolled studies and only 7 percent were randomized trials. Similarly, Wasserman et al10 reported that only 9 percent of therapy-related articles were randomized controlled trials. One explanation for these findings is that at least some controlled or randomized trials conducted by otolaryngologists are published in otolaryngology journals other than those studied or in nonotolaryngology journals. Despite this high frequency of articles providing therapeutic recommendations, only 65 percent of the articles reviewed in the presented study mention harms or AEs anywhere in the published article, which suggests underreporting. Similar studies have been performed in other medical and surgical disciplines, also reflecting underreporting in the medical literature. Martin et al4 report 88 percent morbidity reporting when examining general surgery literature from 1990 to 2001. Papakikolaou et al11 found that nonrandomized trials are more conservative in estimating risks of harms with more than a two-fold difference, an interesting finding considering the majority of clinical studies in otolaryngology literature are case series. When the investigators do report AEs, our study shows that AEs are usually defined explicitly (71%) and stated in the results section (64%) and/or the discussion section
243
(70%) of the article. These findings are relatively higher than those of studies in other disciplines. Martin et al4 reported that 34 percent of authors defined AEs explicitly in the general surgery literature from 1990 to 2001. Ioannidis and Lau3 compared randomized trials across seven medical areas and reported that 39 percent of authors provided adequate definitions of AEs. However, several factors examined showed significant room for improvement. For example, less than one third of articles report events in the study abstract (27%). Tables or figures displaying data regarding harms or AEs were included in 29 percent of the articles. Few studies stated if therapy had to be stopped because of harms or adverse events (7%). Despite making therapeutic recommendations, only 29 percent of the studies compare risks of the therapy recommended with risks of other therapies, and a mere 10 percent mention a balance of harms versus benefits when making the recommendation. When examining the data, reviewers also looked for factors predictive of underreporting. The fact that harms are less likely to be reported when a benefit is found to the therapy recommended is a concerning one considering the increasing importance of harms and adverse events information when formulating therapeutic guidelines. Ioannidis and Lau’s study3 also found that emphasis on safety decreased when the trial found statistically significant results for efficacy. Another part of our study consisted of a comparison of harms and adverse events reporting between 1996 and 2006. Most aspects of reporting have remained unchanged for 2006 versus 1996, but some improvement has occurred; 2006 studies more often defined explicitly the harms reported and provided a methodology for monitoring adverse events when compared with 1996 (44% vs 24%, P ⬍ .001). More articles mentioned harms and adverse events in tables or figures (36% vs 19%, P ⫽ .002) as well as in the discussion section (74% vs 62%, P ⫽ .019). None of the reporting measures worsened between 1996 and 2006. No comparison of harms and adverse events reporting over time was found in the literature reviewed. Our study has several strengths. First, data were collected using a standardized approach with special forms, and the face validity of questions about reporting harms and adverse events was maximized by the CONSORT statement6 extension and by modeling our questions on reports by other researchers.3,4 Moreover, this is the first comprehensive review of harms and adverse event reporting in otolaryngology literature. However, it is not without its limitations. During a review of the literature examined, the reviewers were not blinded as to the source or year of the articles. Also, the three-reviewer system, as used here, may offer some variability with regard to data collection. Lastly, generalizability of the study to otolaryngology as a discipline is unknown because only four journals were analyzed and studies by otolaryngologists in nonotolaryngology journals were not assessed.
244
Otolaryngology–Head and Neck Surgery, Vol 140, No 2, February 2009
CONCLUSIONS
FINANCIAL DISCLOSURE
Despite the high prevalence of treatment studies in otolaryngology literature, only 65 percent of investigators mention harms or adverse events anywhere in the article, suggesting underreporting. Underreporting is more common when a benefit is found for the therapy and when the therapy studied is a medical intervention. Most aspects of reporting have remained unchanged for 2006 versus 1996, but some improvement has occurred. As information regarding harms and adverse events becomes an increasing requirement for medical decision making, the otolaryngology literature must appropriately respond to this demand with increased and higher quality of harms and AEs reporting.
None.
AUTHOR INFORMATION From the Department of Otolaryngology, SUNY Downstate Medical Center, Brooklyn, NY (Dr Bibawy); Department of Otolaryngology, Tufts Medical Center, Boston, MA (Drs Cossu and Kogan); and Department of Otolaryngology, Long Island College Hospital and SUNY Downstate Medical Center, Brooklyn, NY (Dr Rosenfeld). Corresponding author: Haidy Bibawy, MD, Department of Otolaryngology, SUNY Downstate Medical Center, 450 Clarkson Avenue, Brooklyn, NY 11203. E-mail address: [email protected].
AUTHOR CONTRIBUTIONS Haidy Bibawy, data collection, analysis, drafting of manuscript; Anne Cossu, data collection; Sophia Cogan, data collection; Richard Rosenfeld, data collection, analysis, drafting of manuscript.
REFERENCES 1. Hippocrates, Of the Epidemics Book I, 4th century BC. Available at: http://classics.mit.edu/Hippocrates/epidemics.1.i.html. Accessed 7/2008. 2. Small SD, Barach P. Patient safety and health policy: a history and review. Hematol Oncol Clin North Am 2002;16:1463– 82. 3. Ioannidis J, Lau J. Completeness of safety reporting in randomized trials: an evaluation of seven medical areas. JAMA 2001;285:437– 43. 4. Martin R, Brenna M, Jaques D. Quality of complication reporting in the surgical literature. Ann Surg 2002;236:803–13. 5. American Academy of Pediatrics Steering Committee on Quality Improvement and Management. Policy statement: classifying recommendations for clinical practice guidelines. Pediatrics 2004;114: 874 –7. 6. Begg C, Cho M, Eastwood S, et al. Improving the quality of reporting of randomized controlled trials. The CONSORT statement. JAMA 1996;276:637–9. 7. Ioannidis JP, Evans SJ, Gotzsche PC, et al. Better reporting of harms in randomized trials: an extension of the CONSORT statement. Ann Intern Med 2004;141:781– 8. 8. Bentsianov BL, Boruk M, Rosenfeld RM. Evidence-based medicine in otolaryngology journals. Otolaryngol Head Neck Surg 2002;126: 371– 6. 9. SPSS Inc. SPSS 11.5 Brief Guide. Upper Saddle River, NJ: Prentice Hall Inc; 2002. 10. Wasserman JM, Wynn R, Bash TS, et al. Levels of evidence in otolaryngology journals. Otolaryngol Head Neck Surg 2006;134:717–23. 11. Papanikolaou P, Christidi G, Ioannidis J. Comparison of evidence on harms of medical interventions in randomized and nonrandomized studies. CMAJ 2006;174:635– 41.
ORIGINAL RESEARCH– GENERAL OTOLARYNGOLOGY
Reporting of harms and adverse events in otolaryngology journals Haidy Bibawy, MD, Anne Cossu, MD, Sophia Cogan, MD, and Richard Rosenfeld, MD, MPH, Brooklyn, NY; and Boston, MA OBJECTIVE: To identify the frequency and quality of harms and adverse events reporting in otolaryngology journals. STUDY DESIGN: A literature review. METHODS: Four major otolaryngology journals from 2006 and 1996 were reviewed. Clinical research studies offering therapeutic recommendations were evaluated for frequency and quality of harms and adverse events reporting. RESULTS: Of 1,835 total articles reviewed, 576 (31%) offered therapeutic recommendations. Sixty-five percent provided any mention of harms or adverse events, 47 percent explicitly defined the events, and 24 percent described methodology for collecting harms data. The median number of harms or adverse events reported was three. Studies concluding a beneficial effect of therapy were more likely to not mention adverse events (odds ratio 2.99, P ⫽ .007) compared with studies concluding no benefit. Studies of surgical therapy were more likely to report harms or adverse events (odds ratio 1.46, P ⫽ .046). CONCLUSIONS: Harms and adverse events are underreported and poorly described in otolaryngology journals with about one third of clinical research not mentioning adverse events at all. Most authors do not explicitly describe harms or adverse events (53%) or the methodology behind collecting adverse events data (76%). Underreporting is more likely when a therapeutic effect is found to be beneficial. No sponsorships or competing interests have been disclosed for this article. © 2009 American Academy of Otolaryngology–Head and Neck Surgery Foundation. All rights reserved.
O
ne of the oldest medical tenets is to first “do no harm.”1 This expression took on new meaning in the mid-1990s with an explosive interest in health care safety.2 Thereafter, reporting of safety conditions in hospitals, side effects of medication, and surgical complications have been receiving increased attention in medical literature and the media as a whole. Ioannidis and Lau3 and Martin et al4 both report underreporting of harms and adverse events (AEs) in medical and surgical literature. Harms and AEs have also taken a greater role in medical decision making.5 When considering the balance between risks and benefits of various therapies, harms are often the
preeminent issue in making guideline recommendations, at times superseding levels of evidence. Harms reporting is now suggested as a standard for publishing therapeutic recommendations. The Consolidated Standards of Reporting Trials (CONSORT) statement, first reported in 1996, has been revised multiple times to include more recommendations on the reporting of harms and AEs.6,7 Despite increasing importance of harms and AEs in medical decisions, their reporting has yet to be evaluated in otolaryngology literature. Therefore, the present study sought to identify the frequency and quality of harms and AEs reporting in leading peer-reviewed otolaryngology journals.
METHODS Publications from the four highest impact clinical otolaryngology journals in the years 1996 and 2006 were reviewed. These were Otolaryngology–Head and Neck Surgery, Laryngoscope, Archives of Otolaryngology–Head and Neck Surgery, and Annals of Otology, Rhinology and Laryngology. Included in the study were clinical research publications offering therapeutic recommendations, and these were evaluated for their frequency and quality of harms and AEs reporting. Any study dealing with living human subjects or directly related with patient care was defined as “clinical.”8 Studies dealing with bodily fluids, tissues, or specimens were considered “clinical” as well if they correlated with patient care, individual patient histories, or disease states. “Nonclinical” research included any animal or laboratory studies; postmortem studies not correlated with patient histories; physician questionnaires; and surveys of manpower, education, cost-effectiveness, or quality control. Therefore, “nonclinical” research includes but is not limited to basic science. Data collection was planned a priori and conducted using a standardized checklist. Articles were reviewed by one of three of the authors (HB, SK, and AC) who were not blinded as to the source and year of publication. An initial subset of 30 articles was reviewed by all reviewers to ensure interrater reliability.
Received September 3, 2008; revised October 28, 2008; accepted November 12, 2008.
0194-5998/$36.00 © 2009 American Academy of Otolaryngology–Head and Neck Surgery Foundation. All rights reserved. doi:10.1016/j.otohns.2008.11.015
242
Otolaryngology–Head and Neck Surgery, Vol 140, No 2, February 2009
Data were collected under the broad headings of study design, level of evidence, and harms/AE information. Information related to study design include patient age, sample size, control groups, and the study focus (medical therapy, surgical therapy, or nontherapeutic). For all therapy articles, the level of evidence was determined using the following definitions: level 1, randomized controlled trial; level 2, controlled prospective study; level 3, controlled retrospective study; level 4, uncontrolled case series; and level 5, expert opinion without explicit appraisal. Face validity of the questions relating to harms/AEs was based on the 2003 extension of the CONSORT statement6 and similar investigations published in peer-reviewed journals.3,4 Questions related to general reporting of harms/AEs in the article overall and subsections of the manuscript (abstract, results, tables, and figures), with emphasis on the thoroughness of reporting and the extent to which information about harms/AEs was considered in the discussion section when drawing conclusions or making recommendations, were evaluated. Data were entered into an Excel (Microsoft Corp, Redmond, WA) spreadsheet and verified for accuracy. All statistical analyses were performed by using SPSS software (SPSS Inc, Chicago, IL).9 Bivariate comparisons were made using a chi-square test for categoric data and a Fisher exact test whenever the expected count in any cell was less than five. Odds ratios were calculated, when appropriate, using a 95 percent confidence interval. All tests were performed by using a twosided alpha level of 0.05 for statistical significance.
RESULTS A total of 1,835 articles were reviewed. Of those, 1399 (76%) were clinical research studies. Of the 1,399, 53 percent included only adults, 17 percent children, and 30 percent included both or did not specify age restrictions. Only 26 percent of the clinical research studies had a control or comparison group, and the median sample size was 28. The number of studies making therapeutic recommendations was 576 (31%) of the total reviewed; 70 percent made
surgical recommendations and 30 percent were medical. Ninety-two percent of the studies concluded that the therapy recommended was beneficial. With regards to levels of evidence, it was found that the majority of clinical studies were level 4 case series. Ten percent was level 1 evidence, 8 percent fell into the category of level 2, and 7 percent were level 3. When examining harms and AEs reporting specifically, it was found that 65 percent (377/576) of articles mentioned harms or adverse events anywhere in the manuscript. These 377 articles were further examined for the quality of harms and AE reporting. Seventy-one percent of the studies defined explicitly harms and/or AEs. Approximately 27 percent mention harms in the abstract, 64 percent in the results, 70 percent in the discussion, and 29 percent in tables/figures sections. Thirty-four percent of the studies describe a methodology for collecting harms data; 29 percent provide a comparison of harms of the therapy recommended and other therapies; 7 percent state if therapy needed to be stopped due to adverse events during the study, and 10 percent mention a comparison of harms versus benefits. Factors predictive of not mentioning harms or AEs were also examined. When the investigators reported a benefit of therapy, they were almost three times more likely to not mention harms or AEs compared with studies that reported no benefit (odds ratio 2.99; 95% confidence interval [CI], 1.30-6.81; P ⫽ .007). Whereas 36 percent of “positive” studies did not report harms, only 16 percent of “negative” studies were remiss. Similarly, studies of medical interventions were less likely to report AEs than reports of surgery (odds ratio 1.46; 95% CI, 1.01-2.10; P ⫽ .046). Although a significant relationship was found, the 95 percent CI includes 1.01 as a potentially plausible value for the population parameter, which is consistent with a trivial difference. In other words, we are somewhat uncertain of how important this finding is because of limited precision (eg, a broad 95% CI). Age group and level of evidence were not associated with reporting outcomes. Table 1 summarizes the trends in AE reporting for 2006 versus 1996. Compared with 1996, articles published in 2006 more often defined explicitly the methodology for
Table 1 Trends in AE reporting for 2006 versus 1996 Factor: n/N (%) AE mentioned AE defined explicitly AE methodology defined AE mentioned in abstract AE mentioned in results section AE mentioned in tables/figures State need to stop therapy if AE Discussion section mentions AE Discussion compares AEs with other treatment options Discussion of relationship of harms vs. benefits
2006 244/384 181/245 108/245 131/245 218/245 89/245 18/245 181/245 75/245 30/245
(64) (74) (44) (53) (89) (36) (7) (74) (31) (12)
1996 133/192 89/133 32/133 60/133 119/133 26/133 7/133 82/133 35/133 8/133
(69) (67) (24) (45) (90) (19) (5) (62) (26) (6)
Total 377/576 270/378 140/378 191/378 337/378 115/378 25/378 263/378 110/378 38/378
(65) (71) (37) (50) (89) (30) (7) (70) (29) (1)
P value .193 .155 ⬍.001 .276 .460 .002 .520 .019 .408 .072
Bibawy et al
Reporting of harms and adverse events . . .
monitoring adverse events (44% vs 24%, P ⬍ .001), mentioned more events in the articles’ tables or figures (36% vs 19%, P ⫽ .002), and mentioned more events in the discussion section of the article (74% vs 62%, P ⫽ .019). The overall frequency that adverse events were mentioned in the article, abstract, and results section was unchanged. No change was observed in how information about AEs was used to compare alternative treatment options or to balance decisions about treatment benefits.
DISCUSSION As the first examination of harms and AEs reporting in otolaryngology literature, this study provides significant insights. Although 31 percent of clinical research in major otolaryngology journals offered therapeutic recommendations, only 65 percent of authors mentioned harms or AEs anywhere in the article. When events were mentioned, only 47 percent explicitly defined the events and 24 percent described methodology for collecting harms data. Studies concluding a beneficial effect of therapy were three times more likely to not mention AEs compared with studies concluding no benefit, and studies of surgical therapy were approximately one and half times more likely to report harms or AEs. Our review also shows that most of the clinical research published in the otolaryngology literature consists of uncontrolled case series of surgical interventions (70%) that nearly always (92%) report a beneficial effect for the intervention discussed. These findings are consistent with other published reviews. Bentsianov et al8 showed that 80 percent of therapy recommendations were supported by uncontrolled studies and only 7 percent were randomized trials. Similarly, Wasserman et al10 reported that only 9 percent of therapy-related articles were randomized controlled trials. One explanation for these findings is that at least some controlled or randomized trials conducted by otolaryngologists are published in otolaryngology journals other than those studied or in nonotolaryngology journals. Despite this high frequency of articles providing therapeutic recommendations, only 65 percent of the articles reviewed in the presented study mention harms or AEs anywhere in the published article, which suggests underreporting. Similar studies have been performed in other medical and surgical disciplines, also reflecting underreporting in the medical literature. Martin et al4 report 88 percent morbidity reporting when examining general surgery literature from 1990 to 2001. Papakikolaou et al11 found that nonrandomized trials are more conservative in estimating risks of harms with more than a two-fold difference, an interesting finding considering the majority of clinical studies in otolaryngology literature are case series. When the investigators do report AEs, our study shows that AEs are usually defined explicitly (71%) and stated in the results section (64%) and/or the discussion section
243
(70%) of the article. These findings are relatively higher than those of studies in other disciplines. Martin et al4 reported that 34 percent of authors defined AEs explicitly in the general surgery literature from 1990 to 2001. Ioannidis and Lau3 compared randomized trials across seven medical areas and reported that 39 percent of authors provided adequate definitions of AEs. However, several factors examined showed significant room for improvement. For example, less than one third of articles report events in the study abstract (27%). Tables or figures displaying data regarding harms or AEs were included in 29 percent of the articles. Few studies stated if therapy had to be stopped because of harms or adverse events (7%). Despite making therapeutic recommendations, only 29 percent of the studies compare risks of the therapy recommended with risks of other therapies, and a mere 10 percent mention a balance of harms versus benefits when making the recommendation. When examining the data, reviewers also looked for factors predictive of underreporting. The fact that harms are less likely to be reported when a benefit is found to the therapy recommended is a concerning one considering the increasing importance of harms and adverse events information when formulating therapeutic guidelines. Ioannidis and Lau’s study3 also found that emphasis on safety decreased when the trial found statistically significant results for efficacy. Another part of our study consisted of a comparison of harms and adverse events reporting between 1996 and 2006. Most aspects of reporting have remained unchanged for 2006 versus 1996, but some improvement has occurred; 2006 studies more often defined explicitly the harms reported and provided a methodology for monitoring adverse events when compared with 1996 (44% vs 24%, P ⬍ .001). More articles mentioned harms and adverse events in tables or figures (36% vs 19%, P ⫽ .002) as well as in the discussion section (74% vs 62%, P ⫽ .019). None of the reporting measures worsened between 1996 and 2006. No comparison of harms and adverse events reporting over time was found in the literature reviewed. Our study has several strengths. First, data were collected using a standardized approach with special forms, and the face validity of questions about reporting harms and adverse events was maximized by the CONSORT statement6 extension and by modeling our questions on reports by other researchers.3,4 Moreover, this is the first comprehensive review of harms and adverse event reporting in otolaryngology literature. However, it is not without its limitations. During a review of the literature examined, the reviewers were not blinded as to the source or year of the articles. Also, the three-reviewer system, as used here, may offer some variability with regard to data collection. Lastly, generalizability of the study to otolaryngology as a discipline is unknown because only four journals were analyzed and studies by otolaryngologists in nonotolaryngology journals were not assessed.
244
Otolaryngology–Head and Neck Surgery, Vol 140, No 2, February 2009
CONCLUSIONS
FINANCIAL DISCLOSURE
Despite the high prevalence of treatment studies in otolaryngology literature, only 65 percent of investigators mention harms or adverse events anywhere in the article, suggesting underreporting. Underreporting is more common when a benefit is found for the therapy and when the therapy studied is a medical intervention. Most aspects of reporting have remained unchanged for 2006 versus 1996, but some improvement has occurred. As information regarding harms and adverse events becomes an increasing requirement for medical decision making, the otolaryngology literature must appropriately respond to this demand with increased and higher quality of harms and AEs reporting.
None.
AUTHOR INFORMATION From the Department of Otolaryngology, SUNY Downstate Medical Center, Brooklyn, NY (Dr Bibawy); Department of Otolaryngology, Tufts Medical Center, Boston, MA (Drs Cossu and Kogan); and Department of Otolaryngology, Long Island College Hospital and SUNY Downstate Medical Center, Brooklyn, NY (Dr Rosenfeld). Corresponding author: Haidy Bibawy, MD, Department of Otolaryngology, SUNY Downstate Medical Center, 450 Clarkson Avenue, Brooklyn, NY 11203. E-mail address: [email protected].
AUTHOR CONTRIBUTIONS Haidy Bibawy, data collection, analysis, drafting of manuscript; Anne Cossu, data collection; Sophia Cogan, data collection; Richard Rosenfeld, data collection, analysis, drafting of manuscript.
REFERENCES 1. Hippocrates, Of the Epidemics Book I, 4th century BC. Available at: http://classics.mit.edu/Hippocrates/epidemics.1.i.html. Accessed 7/2008. 2. Small SD, Barach P. Patient safety and health policy: a history and review. Hematol Oncol Clin North Am 2002;16:1463– 82. 3. Ioannidis J, Lau J. Completeness of safety reporting in randomized trials: an evaluation of seven medical areas. JAMA 2001;285:437– 43. 4. Martin R, Brenna M, Jaques D. Quality of complication reporting in the surgical literature. Ann Surg 2002;236:803–13. 5. American Academy of Pediatrics Steering Committee on Quality Improvement and Management. Policy statement: classifying recommendations for clinical practice guidelines. Pediatrics 2004;114: 874 –7. 6. Begg C, Cho M, Eastwood S, et al. Improving the quality of reporting of randomized controlled trials. The CONSORT statement. JAMA 1996;276:637–9. 7. Ioannidis JP, Evans SJ, Gotzsche PC, et al. Better reporting of harms in randomized trials: an extension of the CONSORT statement. Ann Intern Med 2004;141:781– 8. 8. Bentsianov BL, Boruk M, Rosenfeld RM. Evidence-based medicine in otolaryngology journals. Otolaryngol Head Neck Surg 2002;126: 371– 6. 9. SPSS Inc. SPSS 11.5 Brief Guide. Upper Saddle River, NJ: Prentice Hall Inc; 2002. 10. Wasserman JM, Wynn R, Bash TS, et al. Levels of evidence in otolaryngology journals. Otolaryngol Head Neck Surg 2006;134:717–23. 11. Papanikolaou P, Christidi G, Ioannidis J. Comparison of evidence on harms of medical interventions in randomized and nonrandomized studies. CMAJ 2006;174:635– 41.