- Email: [email protected]
The Need for Critical Analysis of Case-Series Reporting in Pediatric Surgery
The Need for Critical Analysis of Case-Series Reporting in Pediatric Surgery By Shawn J. Rangel and R. Lawrence Moss New Haven, Connecticut BACKGROUND AND SIGNIFICANCE
E
VIDENCE-BASED PRACTICE (EBP) has gained broad support from the health care community for its emphasis on making clinical decisions based on the weight of experimental evidence. The need for highquality EBP is particularly great in the field of Pediatric Surgery, where many surgical diseases in children are associated with significant morbidity, mortality, and utilization of health care resources. Currently, no clear consensus guidelines exist for the treatment of many of these conditions. 1.2 The randomized, controlled clinical trial (RCT) represents the gold standard of clinical research designs and forms the foundation of high-quality EBP. The impact of well-designed RCTs on the medical care of children has been significant. This perhaps is no better illustrated than by the dramatic decrease in pediatric cancer mortality after establishment of the Cooperative Oncology Study Groups over the last 3 decades. 3 In the field of pediatric surgery, the impact of high-quality randomized data on clinical practice has been limited. In a recent review, Hardin et ai' uncovered an alarming paucity of RCTs within the pediatric surgery literature (constituting less than 0.3% of all clinical research studies; Fig 1).' They found the majority of clinical practice in pediatric surgery to be based on nonrandomized data in the form of retrospective observational studies (largely case series and case reports), editorials, and expert opinion. For the majority of surgical conditions, very little prospective evidence was available. Together, single-institution case reports and case series data constituted over 97% of the published clinical evidence.' This reliance on relatively inferior evidence is likely attributable to the difficulty in conducting prospective clinical research in pediatric surgery. Solomon and McLeod4 -6 have characterized specific obstacles to conducting RCTs for surgical diseases, including the rarity of many surgical conditions, diminishing funding in support of clinical trials, difficulty with standardizing surgiFrom the Division of Pediatric Surge /y, Stanford University School of Medicine. Stanford. CA. . . Address reprint requests to R. Lawrence Moss, MD. Yale Umverslty School of Medicine, Division of Pediatric Surgery, 333 Cedar St, PO Box 208062, New Haven, CT 06520-8062. Copyright 2002, Elsevier Science (USA). All rights reserved. 1055-8586/02/1103-0010$35.00/0 doi: 10.1053!'psu.2002.33743 184
cal interventions, preferences of individual surgeons for specific operations, and the reluctance of patients to randomize into potentially invasive treatment arms, among others. The most significant obstacles to the design and conduct of RCTs specific to pediatric surgery are the relatively low prevalence of surgical conditions, the difficulty with standardizing complex operative procedures (and perioperative care in the neonatal intensive care units), and parental bias against invasive, potentially disfiguring interventions.,,7 These same issues present difficulties in the design of other well-conducted clinical studies such as cohort designs and prospective surgical databases. Although the cornerstone of high-quality EBP remains the randomized clinical trial, the broader concept of EBP involves synthesizing clinical decisions based on the best available clinical evidence. For pediatric surgeons, great difficulty presents when attempting to practice EBP from a literature dominated by heterogeneous case series data, often supporting differing conclusions. The Cochrane Collaboration and others interested in the development of standardized, systematic review methodology for clinical evidence have recommended the omission of case series data from such analysis. 8 This strategy is appropriate when available evidence includes a reasonable proportion of high-quality prospective data covering a sufficient range of relevant clinical topics. However, we believe this strategy is neither practical nor necessarily desirable for pediatric surgery given that case series data effectively represents the only clinical evidence for most interventions. Furthermore, we believe there will continue to exist a heavy reliance on case series data in the future given the significant difficulty in conducting prospective clinical research. We propose that a more constructive and appropriate strategy would involve developing a novel and standardized means to assess the quality of case series data in pediatric surgery. This strategy could be used to facilitate systematic reviews of existing pools of case series data (where higher quality evidence does not exist), and to maximize the utility of information provided by this study design through development of submission guidelines for pediatric surgery journals. Although the case series report represents a relatively inferior study design along the continuum of clinical evidence, incorporating this strategy could have a significant impact on the overall quality of EBP given the ubiquity and influence of case series data within the pediatric surgical literature. Seminars in Pediatric Surgery, Vol 11, No 3 (August), 2002: pp 184-189
185
CASE-SERIES REPORTING
6000
1:
5000
0
Q.
...'" "'"
4000
.J:.
.!!!
:is 3000 :::l
Q.
..... 0
...
2000
'"
.Q
E 1000 :::l
Z
Reviews
Case Reports
Case Series
Basic Prospective Science Non-ReT
ReT's
Fig 1. Relative contribution of different levels of experimental evidence to the Peditraic Surgery literature. Reprinted with permis-
sion.'
In this report, we describe the potential applications for a novel, standardized quality assessment scale for case series reporting in pediatric surgery. Specific considerations for the design and development of such a scale also are discussed. POTENTIAL APPLICATIONS OF A QUALITY ASSESSMENT TOOL FOR CASE SERIES REPORTING
Development of a quality assessment tool for case series data potentially could improve EBP within pediatric surgery through several different avenues. One potentially significant application is the development of standardized submission guidelines for use in peer-reviewed journals. Universal reporting guidelines such as those dictated by the CONSORT (Consolidated Standards of Reporting Trials) statement for randomized trials have had a positive impact on the quality of published clinical research. Y Specifically, adoption of CONSORTmandated guidelines by many influential medical journals has resulted in a clear and measurable improvement in the methodologic quality of published trials. lO •11 The Journal of Pediatric Surgery became the second Englishlanguage surgical journal to adopt the CONSORT reporting guidelines in December 2001. 12 Journals adopting similar guidelines for robust observational studies also have noted an improvement in the quality of submitted manuscripts, although this has been largely subjective. '3 Although the methodologic rigor of a well-constructed case-series report is by no means comparable with that of a randomized trial, a potentially wide range of reporting quality is possible with even relatively simple observational studies. Case series reports could be evaluated in this manner using standardized reporting guidelines derived from a quality assessment instrument. Corresponding authors subsequently could be provided with a stan-
dardized review cntIque identifying specific areas of weakness to address for the purpose of revisions. Adoption and strict adherence to qualitative guidelines designed for case series reports could have a significant impact on the overall quality of EBP given the current volume and influence of this study design in pediatric surgery. As done successfully after implementation of the CONSORT statement, periodic audits of the literature could be performed to examine for tangible improvements in the quality of case series reporting. II Subsequent modification of the guidelines for the purposes of clarification and other refinements can be done to achieve maximum utility. 14 Facilitating the systematic review of existing case series data is another potentially important application of a quality assessment instrument. This would be relevant particularly to surgical diseases in which evidence is limited to case series data characterized by conflicting results (eg, treatment of perforated necrotizing enterocolitis).IS Currently, no effective method exists for the derivation of evidence-based decisions from pools of case series data. Jenicek l6 has defined the concept of meta-analysis as a systematic, organized, and structured evaluation of a clinical problem based on the comprehensive results of independent studies. Although it is a technique most often applied to the systematic review of randomized studies, several groups recently have proposed the application of meta-analysis principles to case series data in which large homogeneous pools of such data exist. 17 - 19 The utility of this approach obviously is limited by the collective quality of data being analyzed. A systematic review of case series data in which the component studies may already contain significant bias potentially could provide even less valid conclusions. For the meta-analysis of randomized clinical trials, methodology has been developed for assigning relative weights to individual studies based on their qualitative rigor. 20 A similar strategy can be utilized in the metaanalysis of case series data. Depending on the nature of the clinical intervention, relevant criteria derived from the instrument could be used to identify case series reports of a minimum qualitative threshold for inclusion. Results of case series meta-analysis may help clarify conflicting results, and, more importantly, prioritize interventions for prospective studies based on the quality of evidence in support of specific conclusions. This application is attractive particularly given the limited resources and logistical difficulties with conducting randomized studies in pediatric surgery. Finally, a quality assessment instrument could be used as a foundation for teaching individual practitioners, residents, and medical students how to critically evaluate published case series data in a systematic fashion. Many peer-reviewed journals, including those dedicated pri-
186
marily to surgical practice, have published tutorials on how to wade through different levels of clinical evidence and associated sources of bias. 2! Unfortunately, the majority of these tutorials, including those published in surgical journals, ironically, fail to provide a useful set of guidelines for the critical interpretation of case series data. Although a relatively simple study design when compared with randomized and high-quality observational studies, effective interpretation of the results and conclusions of case-series reporting requires even greater attention to potential methodologic and reporting limitations. Such a tutorial would be useful particularly for pediatric surgeons given that the overwhelming majority of published evidence is derived from case series reporting. Although a quality assessment instrument should be developed in the context of a specific clinical focus (eg, to take into account study design issues specific to pediatric surgical diseases), the basic principles as they relate to clinical relevance, reporting methodology, and interpretation of results should be applicable to a wide range of surgical diseases from many different fields. In addition to providing surgeons with the fundamental background knowledge to critically appraise these studies, such a tutorial also will better equip them to prepare high-quality case series reports for publication. CONSIDERATIONS IN THE DESIGN OF A QUALITY ASSESSMENT SCALE FOR CASE SERIES REPORTS
Development of standardized instruments for the qualitative assessment of published clinical evidence is by no means a novel undertaking. The first scale published for this purpose was developed by Chalmers et aJ22 almost 2 decades ago and focused on the qualitative characteristics ofrandomized clinical trials. Since then, a myriad of quality scales have been developed de novo and by using modifications of Chalmer's methodology.23,24 Recent efforts also have focused on developing assessment scales for nonrandomized study designs including case control and cohort studies. 25.27 To date, however, little has been published in the way of quality assessment instruments designed specifically for case series reporting. Using a MEDLINE search dating back 30 years, we could identify no published reports describing a methodology for the quality assessment of case series data. We did find one article describing a fairly superficial set of repOiting criteria for case series data in internal medicine. 28 A similar search was conducted on the Worldwide Web using the Google metasearch engine. 29 A website sponsored by Australia's National Public Health Partnership also featured a set of recommended reporting guidelines for case series reports. 30 However, neither of these 2 resources provided a sufficiently rigorous set of guidelines to base a comprehensive quality assessment scale.
RANGEL AND MOSS
The paucity of quality assessment instruments designed specifically for case series data is perhaps not surprising given the relatively esoteric nature of this study design. In many fields in which practice is grounded firmly in rigorous clinical research (eg, pediatric oncology), the role of case series reporting is limited to describing the bizarre case or unexpected side effect. It is exceedingly rare outside of a surgical journal for case series reports to present results as a first line of evidence, even with relatively novel therapeutic interventions. Prominent epidemiologists and academic think tanks for evidence-based practice have concentrated their efforts largely on the evolution of clinical trial methodology and metaanalysis techniques. 31 ,32 Given the conventional wisdom that clinical trials and robust observational studies provide the only true approach for determining clinical efficacy, there has been little motivation for developing a methodology for the critical analysis of case series reporting. The absence of preexisting quality assessment scales for case series reporting will require creation of a new instrument de novo using appropriate clinimetric principles. As with the development of many existing instruments, this can be accomplished most efficiently using existing quality assessment scales (instruments designed for randomized data and higherquality observational studies) and clinical study design reviews as a conceptual framework. 24 ,33 Although concepts such as randomization and complex statistical analysis are not applicable to case-series methods, key principles such as patient selection, reporting methodology, and accurate interpretation of results in context of study limitations are integral to aIJ epidemiologic study designs. Relevant criteria within existing instrument can be included or modified based on specific applicability to the case series design. In defining specific quality contructs from these sources, a set of recommended clinimetric principles should be followed as outlined by Downs and Black (Table 1).33 They proposed that such scales should address components of internal and external validity in the context of study design limitations and the clinical discipline of investigation. For these purposes, internal validity refers to the quality of the study design (specifically, the ability of the study to address potential bias given the limitations of the study design itself) as well as use of appropriate statistical methods. The concept of external validity refers to the clinical relevance of a study, both in terms of the generalizability and applicability of results and conclusions outside original study parameters. Previous quality scales for observational data have been criticized for failing to include measurement schemes for external validity.26,33 This concept is impor-
187
CASE-SERIES REPORTING Table 1. Steps in Developing a Quality Assessment Instrument Step 1 Step 2 Step 3
Step 4
Define the quality construct. Consider quality aspects including statistical analysis and internal/external validity Define the scope and purpose of quality assessment. Consider the study design and clinical questions being posed in the review Develop the instrument. Consider relevant quality items and develop an objective system for their scoring. Quality issues/items may be defined as generic (items specifically related to study Design) and specific (items related specifically to the review's subject matter). Methodologic and clinical items should be defined and evaluated as distinct entities. Measure the measurement properties of the instrument. Assess the inter-rater and intra-rater reliability of the quality assessment scale in a pilot phase prior to its broad application
Reprinted with permission .33
tant particularly in the subset of literature we wish to examine given the heterogeneous nature of pediatric surgical diseases and the surgeons themselves. An equally relevant, although often neglected, category of external validity is the potential impact of a clinical report in light of cun-ently published literature. This specifically refers to the potential clinical relevance of a report in-espective of its methodologic quality (internal validity is appropriately addressed). The cun-ent body of pediatric surgery literature is characterized by a redundancy of case report and case series data, with many institutions publishing studies of similar clinical content and results. I We propose that a quality assessment scale designed for case series also should evaluate a study based on the potential usefulness of its clinical information. Using the above considerations as a conceptual framework, a comprehensive set of criteria can be developed to measure the components of internal and external validity specific to case series data. As done with existing instruments, individual criteria can be categorized in the context of generic study design components (represented as individual subscales within the instrument), including: (1) clinical relevance of results, (2) generalizability of reported experience, (3) methodology of reporting, and (4) discussion of results and study limitations. Central concepts that should be addressed systematically within the individual subs cales are detailed in Table 2. In the process of interpreting potential results using such an instrument, we believe the results from each subscale should be reported independently without an attempt to combine results to an-ive at a global quality score. Complex weighting schemes have been attempted for this purpose in the past, and this has resulted in relatively poor reliability of the quality assessment instrumentY In addition to facilitating reliability, we believe the scale would have much greater utility if the validity components are reported independently along the 4 categories described above.
CONCLUSIONS AND FUTURE DIRECTIONS
The case series report long has played an important, albeit understated, role in the progress of clinical medicine. In essentially every field, clinical case reporting has permitted discovery of new diseases and unexpected effects, as well as descriptions of medically bizan-e and fascinating observations. For pediatric surgeons, the case series report has and will continue to play a much more significant role as the first and only line of available clinical evidence for the treatment of many surgical conditions. Development of a methodology to more accurately interpret the utility of existing case series data and to set qualitative standards for future reporting therefore is crucial. Cun-ently, we are developing a comprehensive set of quality assessment criteria in an attempt to construct the
Table 2. Central Concepts to be Addressed in a Quality Assessment Scale Designed for Case Series Data in Pediatric Surgery Clinical relevance of the publiShed report How do the results and conclusions of a study impact clinical practice or current understanding of the disease process? Generalizability of reported experience Are adequate details furnished regarding the patient population studied? Is the surgical technique or intervention in question adequately described? Are adequate details provided regarding key diagnostic and follow-up criteria? Are important nonoperative aspects of care discussed in sufficient detail? Methodology of evaluation and reporting Is the timeline of the reported surgical experience clearly stated? Are important demographic variables for patients provided? Are missing data addressed accordingly? Are statistical inferences used appropriately for the study design? Are historical controls used for comparison, and if so, used appropriately? Discussion of results and limitations of study design Are results discussed in the context of study design limitations?
188
RANGEL AND MOSS
first quality assessment scale designed for case series reporting. Future directions include the task of validating and proving sufficient reliability of the instrument in a pilot phase once the instrument is sufficiently refined. Given that no gold standard methodology exists for assessing the quality of case series data, proving validity of our tool will be particularly challenging. However, we are designing the instrument to measure and report independently on the fundamental aspects of research design as detailed previously in the component subscales. Similar methods have been used in successfully validated quality scales for randomized and high-quality observational studies. 33 With respect to demonstrating adequate reliability, the relative subjectivity involved with assessing specific clinical criteria within individual subscales also may pose a challenge (eg, determining whether an adequate description of operative technique was furnished). Interpretation of these criteria may require a baseline level of sophistication, both in terms of a fundamental understanding of epidemiologic principles as well as the clinical issue of investigation. In an attempt to render clinical criteria as objective as possible, those in question will be put into context of detailed explanations and examples. However, interpretation of such criteria still may remain somewhat subjective, and this likely will be influenced by the reviewers own background. A medical student may arrive at slightly different conclusions than a resident physician, who may, in turn, arrive at slightly dif-
ferent conclusions than an attending surgeon, and so forth. The scale certainly will continue to evolve as these considerations and other refinements are taken into account. The field of pediatric surgery only recently has adopted a greater emphasis in incorporating evidencebased principles into clinical practice. This is evidenced by an increasing number of publications challenging the current state of clinical literature, adoption of the CONSORT statement by pediatric surgery journals, and the formation of committees dedicated to improving the quality of clinical research by such influential bodies as the American Pediatric Surgical Association. I ,2,12,34 Despite this growing movement, we have not witnessed a significant change in the relative proportions of published research designs in the pediatric surgical literature. This is attributable largely to the formidable barriers discussed previously, which continue to impede the application of high-quality prospective research in the field. Case-series reporting will continue to have a significant influence on the surgical care of children in the foreseeable future. Until there is a time when resources and effective collaboration allow for the broader application of high-quality prospective clinical research, a strategy to improve the quality and interpretation of case series data may have the single greatest impact on improving the state of evidence-based medicine within pediatric surgery.
REFERENCES 1. Hardin WD Jr, Stylianos S, Lally KP: Evidence-based practice in pediatric surgery. J Pediatr Surg 34:908-912; discussion 912-913,1999 2. Baraldini Y, Spitz L, Pierro A: Evidence-based operations in paediatric surgery. Pediatr Surg lnt 13:331-335, 1998 3, Ross JA, Severson RK, Pollock BH, et al: Childhood cancer in the United States. A Geographical analysis of cases from the Pediatric Cooperative Clinical Trials groups, Cancer 77:201-207, 1996 4. Solomon MJ, McLeod RS: Surgery and the randomised controlled trial: Past, present and future. Med J Aus 169:380-383, 1998 5. McLeod RS, Wright JG, Solomon MJ, H et al: Randomized controlled trials in surgery: Issues and problems. Surgery 119:483-486, 1996 6. Solomon MJ, McLeod RS: Should we be performing more randomized controlled trials evaluating surgical operations? Surgery 118: 459-467, 1995 7. Moss RL, Henry MC, Dimmitt R, et al: The role of prospective, randomized clinical trials in Pediatric Surgery: State of the art? J Pediatr Surg 36:1182-1186, 2001 8. The Cochrane Non-randomized Study Methods Group: http:// www.cochrane.dk/nrsmg/ 9. Begg C, Cho M, Eastwood S, et al: Improving the quality of reporting of randomized controlled trials. The CONSORT statement. JAMA 276:637-639, 1996 10. Moher D, Jones A, Lepage L, for the CONSORT Group: Use of the CONSORT statement and quality of reports of randomized trials: a comparative before and after evaluation? JAMA 285: 19921995 , 2001
II. Egger M, Jiini P, Bartlett C, for the CONSORT Group: The value of patient flow charts in reports of randomized controlled trials: bibliographic study. JAMA 285:1996-1999, 2001 12. Moss RL: The CONSORT statement: Progress in clinical research in Pediatric Surgery. J Pediatr Surg 36:1739-1742, 2001 13. Squires BP: Biomedical manuscripts: What editors expect from authors and peer reviewers. Can Med Assoc J 141:17-19, 1989 14. Meinert CL: Beyond CONSORT: Need for improved reporting standards for clinical trials. JAMA 279:1487-1489, 1998 15. Moss RL, Dimmitt RA, Henry MC, et al: A meta-analysis of peritoneal drainage versus laparotomy for perforated necrotizing enterocolitis. J Pediatr Surg 36:1210-1213, 2001 16. Jenicek M: Meta-analysis in medicine. Where we are and where we want to go. J Clin Epidemiol 42:35-44, 1989 17. Fraser EJ, Grimes DA, Shultz KF: Immunization as therapy for recurrent spontaneous abortion: A review and meta-analysis. Obstet Gynecol 82:854-859, 1993 18. Nordin AJ: Primary carcinoma of the fallopian tube. Obstet Gynecol Surv 49:349-361, 1994 19. Cook MW, Levin LA, Joseph MP: Traumatic optic neuropathy: A meta-analysis. Arch Otolaryngol Head Neck Surg 122:389-392, 1996 20. Jenicek M: Meta-analysis in medicine in Epidemiology. The Logic of Modern Medicine. Montreal, Canada Epidmed International, 1995, pp 267-295 21. Urschel JD, Goldsmith CH, Tandan YR, et al: Users' guide to evidence-based surgery: How to use an article evaluating surgical
CASE-SERIES REPORTING
interventions. Evidence-Based Surgery Working Group. Can J Surg 44:95-100,2001 22. Chalmers TC, Smith H Jr, Blackburn B, et al: A method for assessing the quality of a randomized control trial. Controlled Clinical Trials 2:31-49, 1981 23. DerSimonian R, Charette LJ, McPeek B, et al: Reporting on methods in clinical trials. N Engl J Med 306:1332-1337, 1982 24. Moher D, Jadad AR, Nichol G, et al: Assessing the quality of randomized controlled trials: An annotated bibliography of scales and checklists. Control Clin Trials 16:62-73, 1995 25. Bracken MB: Reporting observational studies. Br J Obstet Gynaecol 96:383-388, 1989 26. Cho MK, Bero LA: Instruments for assessing the quality of drug studies published in the medical literature. JAMA 272:101-143, 1994 27. Margetts BM, Thompson RL, Key T, et al: Development of a scoring system to judge the scientific quality of information from case-control and cohort studies of nutrition and disease. Nutrition and Cancer 24:231-239, 1995
189
28. Squires BP, Elmslie TJ: Reports of case series: What editors expect from authors and peer reviewers. Can Med Assoc J 142: 12051206, 1990 29. Google meta-search engine. Http://www.google.com 30. National Health and Medical Research Council (NHMRC): How to Use the Evidence: Assessment and Application of Scientific Evidence, Auslnfo, Canberra, 2000, p8 31. Cochrane Collaboration: Offical website. http://www.cochrane. org 32. Chalmers I: The Cochrane Collaboration: Preparing, maintaining, and disseminating systematic reviews of the effects of health care. Ann N Y Acad Sci 703:156-163, 1993 33. Downs SH, Black N: The feasibility of creating a checklist for the assessment of themethodological quality both of randomised and non-randomised studies of health care interventions. J Epidemiol Com Health 52:377-384, 1998 34. American Pediatric Surgery Association: Outcomes and clinical trials center. Http://www.eapsa.org/outcomes/index.htm
E
VIDENCE-BASED PRACTICE (EBP) has gained broad support from the health care community for its emphasis on making clinical decisions based on the weight of experimental evidence. The need for highquality EBP is particularly great in the field of Pediatric Surgery, where many surgical diseases in children are associated with significant morbidity, mortality, and utilization of health care resources. Currently, no clear consensus guidelines exist for the treatment of many of these conditions. 1.2 The randomized, controlled clinical trial (RCT) represents the gold standard of clinical research designs and forms the foundation of high-quality EBP. The impact of well-designed RCTs on the medical care of children has been significant. This perhaps is no better illustrated than by the dramatic decrease in pediatric cancer mortality after establishment of the Cooperative Oncology Study Groups over the last 3 decades. 3 In the field of pediatric surgery, the impact of high-quality randomized data on clinical practice has been limited. In a recent review, Hardin et ai' uncovered an alarming paucity of RCTs within the pediatric surgery literature (constituting less than 0.3% of all clinical research studies; Fig 1).' They found the majority of clinical practice in pediatric surgery to be based on nonrandomized data in the form of retrospective observational studies (largely case series and case reports), editorials, and expert opinion. For the majority of surgical conditions, very little prospective evidence was available. Together, single-institution case reports and case series data constituted over 97% of the published clinical evidence.' This reliance on relatively inferior evidence is likely attributable to the difficulty in conducting prospective clinical research in pediatric surgery. Solomon and McLeod4 -6 have characterized specific obstacles to conducting RCTs for surgical diseases, including the rarity of many surgical conditions, diminishing funding in support of clinical trials, difficulty with standardizing surgiFrom the Division of Pediatric Surge /y, Stanford University School of Medicine. Stanford. CA. . . Address reprint requests to R. Lawrence Moss, MD. Yale Umverslty School of Medicine, Division of Pediatric Surgery, 333 Cedar St, PO Box 208062, New Haven, CT 06520-8062. Copyright 2002, Elsevier Science (USA). All rights reserved. 1055-8586/02/1103-0010$35.00/0 doi: 10.1053!'psu.2002.33743 184
cal interventions, preferences of individual surgeons for specific operations, and the reluctance of patients to randomize into potentially invasive treatment arms, among others. The most significant obstacles to the design and conduct of RCTs specific to pediatric surgery are the relatively low prevalence of surgical conditions, the difficulty with standardizing complex operative procedures (and perioperative care in the neonatal intensive care units), and parental bias against invasive, potentially disfiguring interventions.,,7 These same issues present difficulties in the design of other well-conducted clinical studies such as cohort designs and prospective surgical databases. Although the cornerstone of high-quality EBP remains the randomized clinical trial, the broader concept of EBP involves synthesizing clinical decisions based on the best available clinical evidence. For pediatric surgeons, great difficulty presents when attempting to practice EBP from a literature dominated by heterogeneous case series data, often supporting differing conclusions. The Cochrane Collaboration and others interested in the development of standardized, systematic review methodology for clinical evidence have recommended the omission of case series data from such analysis. 8 This strategy is appropriate when available evidence includes a reasonable proportion of high-quality prospective data covering a sufficient range of relevant clinical topics. However, we believe this strategy is neither practical nor necessarily desirable for pediatric surgery given that case series data effectively represents the only clinical evidence for most interventions. Furthermore, we believe there will continue to exist a heavy reliance on case series data in the future given the significant difficulty in conducting prospective clinical research. We propose that a more constructive and appropriate strategy would involve developing a novel and standardized means to assess the quality of case series data in pediatric surgery. This strategy could be used to facilitate systematic reviews of existing pools of case series data (where higher quality evidence does not exist), and to maximize the utility of information provided by this study design through development of submission guidelines for pediatric surgery journals. Although the case series report represents a relatively inferior study design along the continuum of clinical evidence, incorporating this strategy could have a significant impact on the overall quality of EBP given the ubiquity and influence of case series data within the pediatric surgical literature. Seminars in Pediatric Surgery, Vol 11, No 3 (August), 2002: pp 184-189
185
CASE-SERIES REPORTING
6000
1:
5000
0
Q.
...'" "'"
4000
.J:.
.!!!
:is 3000 :::l
Q.
..... 0
...
2000
'"
.Q
E 1000 :::l
Z
Reviews
Case Reports
Case Series
Basic Prospective Science Non-ReT
ReT's
Fig 1. Relative contribution of different levels of experimental evidence to the Peditraic Surgery literature. Reprinted with permis-
sion.'
In this report, we describe the potential applications for a novel, standardized quality assessment scale for case series reporting in pediatric surgery. Specific considerations for the design and development of such a scale also are discussed. POTENTIAL APPLICATIONS OF A QUALITY ASSESSMENT TOOL FOR CASE SERIES REPORTING
Development of a quality assessment tool for case series data potentially could improve EBP within pediatric surgery through several different avenues. One potentially significant application is the development of standardized submission guidelines for use in peer-reviewed journals. Universal reporting guidelines such as those dictated by the CONSORT (Consolidated Standards of Reporting Trials) statement for randomized trials have had a positive impact on the quality of published clinical research. Y Specifically, adoption of CONSORTmandated guidelines by many influential medical journals has resulted in a clear and measurable improvement in the methodologic quality of published trials. lO •11 The Journal of Pediatric Surgery became the second Englishlanguage surgical journal to adopt the CONSORT reporting guidelines in December 2001. 12 Journals adopting similar guidelines for robust observational studies also have noted an improvement in the quality of submitted manuscripts, although this has been largely subjective. '3 Although the methodologic rigor of a well-constructed case-series report is by no means comparable with that of a randomized trial, a potentially wide range of reporting quality is possible with even relatively simple observational studies. Case series reports could be evaluated in this manner using standardized reporting guidelines derived from a quality assessment instrument. Corresponding authors subsequently could be provided with a stan-
dardized review cntIque identifying specific areas of weakness to address for the purpose of revisions. Adoption and strict adherence to qualitative guidelines designed for case series reports could have a significant impact on the overall quality of EBP given the current volume and influence of this study design in pediatric surgery. As done successfully after implementation of the CONSORT statement, periodic audits of the literature could be performed to examine for tangible improvements in the quality of case series reporting. II Subsequent modification of the guidelines for the purposes of clarification and other refinements can be done to achieve maximum utility. 14 Facilitating the systematic review of existing case series data is another potentially important application of a quality assessment instrument. This would be relevant particularly to surgical diseases in which evidence is limited to case series data characterized by conflicting results (eg, treatment of perforated necrotizing enterocolitis).IS Currently, no effective method exists for the derivation of evidence-based decisions from pools of case series data. Jenicek l6 has defined the concept of meta-analysis as a systematic, organized, and structured evaluation of a clinical problem based on the comprehensive results of independent studies. Although it is a technique most often applied to the systematic review of randomized studies, several groups recently have proposed the application of meta-analysis principles to case series data in which large homogeneous pools of such data exist. 17 - 19 The utility of this approach obviously is limited by the collective quality of data being analyzed. A systematic review of case series data in which the component studies may already contain significant bias potentially could provide even less valid conclusions. For the meta-analysis of randomized clinical trials, methodology has been developed for assigning relative weights to individual studies based on their qualitative rigor. 20 A similar strategy can be utilized in the metaanalysis of case series data. Depending on the nature of the clinical intervention, relevant criteria derived from the instrument could be used to identify case series reports of a minimum qualitative threshold for inclusion. Results of case series meta-analysis may help clarify conflicting results, and, more importantly, prioritize interventions for prospective studies based on the quality of evidence in support of specific conclusions. This application is attractive particularly given the limited resources and logistical difficulties with conducting randomized studies in pediatric surgery. Finally, a quality assessment instrument could be used as a foundation for teaching individual practitioners, residents, and medical students how to critically evaluate published case series data in a systematic fashion. Many peer-reviewed journals, including those dedicated pri-
186
marily to surgical practice, have published tutorials on how to wade through different levels of clinical evidence and associated sources of bias. 2! Unfortunately, the majority of these tutorials, including those published in surgical journals, ironically, fail to provide a useful set of guidelines for the critical interpretation of case series data. Although a relatively simple study design when compared with randomized and high-quality observational studies, effective interpretation of the results and conclusions of case-series reporting requires even greater attention to potential methodologic and reporting limitations. Such a tutorial would be useful particularly for pediatric surgeons given that the overwhelming majority of published evidence is derived from case series reporting. Although a quality assessment instrument should be developed in the context of a specific clinical focus (eg, to take into account study design issues specific to pediatric surgical diseases), the basic principles as they relate to clinical relevance, reporting methodology, and interpretation of results should be applicable to a wide range of surgical diseases from many different fields. In addition to providing surgeons with the fundamental background knowledge to critically appraise these studies, such a tutorial also will better equip them to prepare high-quality case series reports for publication. CONSIDERATIONS IN THE DESIGN OF A QUALITY ASSESSMENT SCALE FOR CASE SERIES REPORTS
Development of standardized instruments for the qualitative assessment of published clinical evidence is by no means a novel undertaking. The first scale published for this purpose was developed by Chalmers et aJ22 almost 2 decades ago and focused on the qualitative characteristics ofrandomized clinical trials. Since then, a myriad of quality scales have been developed de novo and by using modifications of Chalmer's methodology.23,24 Recent efforts also have focused on developing assessment scales for nonrandomized study designs including case control and cohort studies. 25.27 To date, however, little has been published in the way of quality assessment instruments designed specifically for case series reporting. Using a MEDLINE search dating back 30 years, we could identify no published reports describing a methodology for the quality assessment of case series data. We did find one article describing a fairly superficial set of repOiting criteria for case series data in internal medicine. 28 A similar search was conducted on the Worldwide Web using the Google metasearch engine. 29 A website sponsored by Australia's National Public Health Partnership also featured a set of recommended reporting guidelines for case series reports. 30 However, neither of these 2 resources provided a sufficiently rigorous set of guidelines to base a comprehensive quality assessment scale.
RANGEL AND MOSS
The paucity of quality assessment instruments designed specifically for case series data is perhaps not surprising given the relatively esoteric nature of this study design. In many fields in which practice is grounded firmly in rigorous clinical research (eg, pediatric oncology), the role of case series reporting is limited to describing the bizarre case or unexpected side effect. It is exceedingly rare outside of a surgical journal for case series reports to present results as a first line of evidence, even with relatively novel therapeutic interventions. Prominent epidemiologists and academic think tanks for evidence-based practice have concentrated their efforts largely on the evolution of clinical trial methodology and metaanalysis techniques. 31 ,32 Given the conventional wisdom that clinical trials and robust observational studies provide the only true approach for determining clinical efficacy, there has been little motivation for developing a methodology for the critical analysis of case series reporting. The absence of preexisting quality assessment scales for case series reporting will require creation of a new instrument de novo using appropriate clinimetric principles. As with the development of many existing instruments, this can be accomplished most efficiently using existing quality assessment scales (instruments designed for randomized data and higherquality observational studies) and clinical study design reviews as a conceptual framework. 24 ,33 Although concepts such as randomization and complex statistical analysis are not applicable to case-series methods, key principles such as patient selection, reporting methodology, and accurate interpretation of results in context of study limitations are integral to aIJ epidemiologic study designs. Relevant criteria within existing instrument can be included or modified based on specific applicability to the case series design. In defining specific quality contructs from these sources, a set of recommended clinimetric principles should be followed as outlined by Downs and Black (Table 1).33 They proposed that such scales should address components of internal and external validity in the context of study design limitations and the clinical discipline of investigation. For these purposes, internal validity refers to the quality of the study design (specifically, the ability of the study to address potential bias given the limitations of the study design itself) as well as use of appropriate statistical methods. The concept of external validity refers to the clinical relevance of a study, both in terms of the generalizability and applicability of results and conclusions outside original study parameters. Previous quality scales for observational data have been criticized for failing to include measurement schemes for external validity.26,33 This concept is impor-
187
CASE-SERIES REPORTING Table 1. Steps in Developing a Quality Assessment Instrument Step 1 Step 2 Step 3
Step 4
Define the quality construct. Consider quality aspects including statistical analysis and internal/external validity Define the scope and purpose of quality assessment. Consider the study design and clinical questions being posed in the review Develop the instrument. Consider relevant quality items and develop an objective system for their scoring. Quality issues/items may be defined as generic (items specifically related to study Design) and specific (items related specifically to the review's subject matter). Methodologic and clinical items should be defined and evaluated as distinct entities. Measure the measurement properties of the instrument. Assess the inter-rater and intra-rater reliability of the quality assessment scale in a pilot phase prior to its broad application
Reprinted with permission .33
tant particularly in the subset of literature we wish to examine given the heterogeneous nature of pediatric surgical diseases and the surgeons themselves. An equally relevant, although often neglected, category of external validity is the potential impact of a clinical report in light of cun-ently published literature. This specifically refers to the potential clinical relevance of a report in-espective of its methodologic quality (internal validity is appropriately addressed). The cun-ent body of pediatric surgery literature is characterized by a redundancy of case report and case series data, with many institutions publishing studies of similar clinical content and results. I We propose that a quality assessment scale designed for case series also should evaluate a study based on the potential usefulness of its clinical information. Using the above considerations as a conceptual framework, a comprehensive set of criteria can be developed to measure the components of internal and external validity specific to case series data. As done with existing instruments, individual criteria can be categorized in the context of generic study design components (represented as individual subscales within the instrument), including: (1) clinical relevance of results, (2) generalizability of reported experience, (3) methodology of reporting, and (4) discussion of results and study limitations. Central concepts that should be addressed systematically within the individual subs cales are detailed in Table 2. In the process of interpreting potential results using such an instrument, we believe the results from each subscale should be reported independently without an attempt to combine results to an-ive at a global quality score. Complex weighting schemes have been attempted for this purpose in the past, and this has resulted in relatively poor reliability of the quality assessment instrumentY In addition to facilitating reliability, we believe the scale would have much greater utility if the validity components are reported independently along the 4 categories described above.
CONCLUSIONS AND FUTURE DIRECTIONS
The case series report long has played an important, albeit understated, role in the progress of clinical medicine. In essentially every field, clinical case reporting has permitted discovery of new diseases and unexpected effects, as well as descriptions of medically bizan-e and fascinating observations. For pediatric surgeons, the case series report has and will continue to play a much more significant role as the first and only line of available clinical evidence for the treatment of many surgical conditions. Development of a methodology to more accurately interpret the utility of existing case series data and to set qualitative standards for future reporting therefore is crucial. Cun-ently, we are developing a comprehensive set of quality assessment criteria in an attempt to construct the
Table 2. Central Concepts to be Addressed in a Quality Assessment Scale Designed for Case Series Data in Pediatric Surgery Clinical relevance of the publiShed report How do the results and conclusions of a study impact clinical practice or current understanding of the disease process? Generalizability of reported experience Are adequate details furnished regarding the patient population studied? Is the surgical technique or intervention in question adequately described? Are adequate details provided regarding key diagnostic and follow-up criteria? Are important nonoperative aspects of care discussed in sufficient detail? Methodology of evaluation and reporting Is the timeline of the reported surgical experience clearly stated? Are important demographic variables for patients provided? Are missing data addressed accordingly? Are statistical inferences used appropriately for the study design? Are historical controls used for comparison, and if so, used appropriately? Discussion of results and limitations of study design Are results discussed in the context of study design limitations?
188
RANGEL AND MOSS
first quality assessment scale designed for case series reporting. Future directions include the task of validating and proving sufficient reliability of the instrument in a pilot phase once the instrument is sufficiently refined. Given that no gold standard methodology exists for assessing the quality of case series data, proving validity of our tool will be particularly challenging. However, we are designing the instrument to measure and report independently on the fundamental aspects of research design as detailed previously in the component subscales. Similar methods have been used in successfully validated quality scales for randomized and high-quality observational studies. 33 With respect to demonstrating adequate reliability, the relative subjectivity involved with assessing specific clinical criteria within individual subscales also may pose a challenge (eg, determining whether an adequate description of operative technique was furnished). Interpretation of these criteria may require a baseline level of sophistication, both in terms of a fundamental understanding of epidemiologic principles as well as the clinical issue of investigation. In an attempt to render clinical criteria as objective as possible, those in question will be put into context of detailed explanations and examples. However, interpretation of such criteria still may remain somewhat subjective, and this likely will be influenced by the reviewers own background. A medical student may arrive at slightly different conclusions than a resident physician, who may, in turn, arrive at slightly dif-
ferent conclusions than an attending surgeon, and so forth. The scale certainly will continue to evolve as these considerations and other refinements are taken into account. The field of pediatric surgery only recently has adopted a greater emphasis in incorporating evidencebased principles into clinical practice. This is evidenced by an increasing number of publications challenging the current state of clinical literature, adoption of the CONSORT statement by pediatric surgery journals, and the formation of committees dedicated to improving the quality of clinical research by such influential bodies as the American Pediatric Surgical Association. I ,2,12,34 Despite this growing movement, we have not witnessed a significant change in the relative proportions of published research designs in the pediatric surgical literature. This is attributable largely to the formidable barriers discussed previously, which continue to impede the application of high-quality prospective research in the field. Case-series reporting will continue to have a significant influence on the surgical care of children in the foreseeable future. Until there is a time when resources and effective collaboration allow for the broader application of high-quality prospective clinical research, a strategy to improve the quality and interpretation of case series data may have the single greatest impact on improving the state of evidence-based medicine within pediatric surgery.
REFERENCES 1. Hardin WD Jr, Stylianos S, Lally KP: Evidence-based practice in pediatric surgery. J Pediatr Surg 34:908-912; discussion 912-913,1999 2. Baraldini Y, Spitz L, Pierro A: Evidence-based operations in paediatric surgery. Pediatr Surg lnt 13:331-335, 1998 3, Ross JA, Severson RK, Pollock BH, et al: Childhood cancer in the United States. A Geographical analysis of cases from the Pediatric Cooperative Clinical Trials groups, Cancer 77:201-207, 1996 4. Solomon MJ, McLeod RS: Surgery and the randomised controlled trial: Past, present and future. Med J Aus 169:380-383, 1998 5. McLeod RS, Wright JG, Solomon MJ, H et al: Randomized controlled trials in surgery: Issues and problems. Surgery 119:483-486, 1996 6. Solomon MJ, McLeod RS: Should we be performing more randomized controlled trials evaluating surgical operations? Surgery 118: 459-467, 1995 7. Moss RL, Henry MC, Dimmitt R, et al: The role of prospective, randomized clinical trials in Pediatric Surgery: State of the art? J Pediatr Surg 36:1182-1186, 2001 8. The Cochrane Non-randomized Study Methods Group: http:// www.cochrane.dk/nrsmg/ 9. Begg C, Cho M, Eastwood S, et al: Improving the quality of reporting of randomized controlled trials. The CONSORT statement. JAMA 276:637-639, 1996 10. Moher D, Jones A, Lepage L, for the CONSORT Group: Use of the CONSORT statement and quality of reports of randomized trials: a comparative before and after evaluation? JAMA 285: 19921995 , 2001
II. Egger M, Jiini P, Bartlett C, for the CONSORT Group: The value of patient flow charts in reports of randomized controlled trials: bibliographic study. JAMA 285:1996-1999, 2001 12. Moss RL: The CONSORT statement: Progress in clinical research in Pediatric Surgery. J Pediatr Surg 36:1739-1742, 2001 13. Squires BP: Biomedical manuscripts: What editors expect from authors and peer reviewers. Can Med Assoc J 141:17-19, 1989 14. Meinert CL: Beyond CONSORT: Need for improved reporting standards for clinical trials. JAMA 279:1487-1489, 1998 15. Moss RL, Dimmitt RA, Henry MC, et al: A meta-analysis of peritoneal drainage versus laparotomy for perforated necrotizing enterocolitis. J Pediatr Surg 36:1210-1213, 2001 16. Jenicek M: Meta-analysis in medicine. Where we are and where we want to go. J Clin Epidemiol 42:35-44, 1989 17. Fraser EJ, Grimes DA, Shultz KF: Immunization as therapy for recurrent spontaneous abortion: A review and meta-analysis. Obstet Gynecol 82:854-859, 1993 18. Nordin AJ: Primary carcinoma of the fallopian tube. Obstet Gynecol Surv 49:349-361, 1994 19. Cook MW, Levin LA, Joseph MP: Traumatic optic neuropathy: A meta-analysis. Arch Otolaryngol Head Neck Surg 122:389-392, 1996 20. Jenicek M: Meta-analysis in medicine in Epidemiology. The Logic of Modern Medicine. Montreal, Canada Epidmed International, 1995, pp 267-295 21. Urschel JD, Goldsmith CH, Tandan YR, et al: Users' guide to evidence-based surgery: How to use an article evaluating surgical
CASE-SERIES REPORTING
interventions. Evidence-Based Surgery Working Group. Can J Surg 44:95-100,2001 22. Chalmers TC, Smith H Jr, Blackburn B, et al: A method for assessing the quality of a randomized control trial. Controlled Clinical Trials 2:31-49, 1981 23. DerSimonian R, Charette LJ, McPeek B, et al: Reporting on methods in clinical trials. N Engl J Med 306:1332-1337, 1982 24. Moher D, Jadad AR, Nichol G, et al: Assessing the quality of randomized controlled trials: An annotated bibliography of scales and checklists. Control Clin Trials 16:62-73, 1995 25. Bracken MB: Reporting observational studies. Br J Obstet Gynaecol 96:383-388, 1989 26. Cho MK, Bero LA: Instruments for assessing the quality of drug studies published in the medical literature. JAMA 272:101-143, 1994 27. Margetts BM, Thompson RL, Key T, et al: Development of a scoring system to judge the scientific quality of information from case-control and cohort studies of nutrition and disease. Nutrition and Cancer 24:231-239, 1995
189
28. Squires BP, Elmslie TJ: Reports of case series: What editors expect from authors and peer reviewers. Can Med Assoc J 142: 12051206, 1990 29. Google meta-search engine. Http://www.google.com 30. National Health and Medical Research Council (NHMRC): How to Use the Evidence: Assessment and Application of Scientific Evidence, Auslnfo, Canberra, 2000, p8 31. Cochrane Collaboration: Offical website. http://www.cochrane. org 32. Chalmers I: The Cochrane Collaboration: Preparing, maintaining, and disseminating systematic reviews of the effects of health care. Ann N Y Acad Sci 703:156-163, 1993 33. Downs SH, Black N: The feasibility of creating a checklist for the assessment of themethodological quality both of randomised and non-randomised studies of health care interventions. J Epidemiol Com Health 52:377-384, 1998 34. American Pediatric Surgery Association: Outcomes and clinical trials center. Http://www.eapsa.org/outcomes/index.htm