- Email: [email protected]
Development of a clinical definition for acute respiratory distress syndrome using the Delphi technique
Journal of Critical Care (2005) 20, 147 – 154
Development of a clinical definition for acute respiratory distress syndrome using the Delphi technique Niall D. Ferguson MD, MSca,b,d,*, Aileen M. Davis PhDc,e, Arthur S. Slutsky MDa,b,f, Thomas E. Stewart MDa,b,d a
Interdepartmental Division of Critical Care, University of Toronto, Toronto, ON, Canada Department of Medicine, Division of Respirology, University of Toronto, Toronto, ON, Canada c Department of Physiotherapy, University of Toronto, Toronto, ON, Canada d Mount Sinai Hospital and University Health Network, University of Toronto, Toronto, ON, Canada e Toronto Rehabilitation Institute, University of Toronto, Toronto, ON, Canada f St Michael’s Hospital, University of Toronto, Toronto, ON, Canada b
Received 10 November 2004; revised 25 January 2005; accepted 1 March 2005
Keywords: Diagnosis; Consensus; ARDS; Delphi technique; Definition development
Abstract Purpose: The objective of this study is to describe the implementation of formal consensus techniques in the development of a clinical definition for acute respiratory distress syndrome. Materials and Methods: A Delphi consensus process was conducted using e-mail. Sixteen panelists who were both researchers and opinion leaders were systematically recruited. The Delphi technique was performed over 4 rounds on the background of an explicit definition framework. Item generation was performed in round 1, item reduction in rounds 2 and 3, and definition evaluation in round 4. Explicit consensus thresholds were used throughout. Results: Of the 16 panelists, 11 actually participated in developing a definition that met a priori consensus rules on the third iteration. New incorporations in the Delphi definition include the use of a standardized oxygenation assessment and the documentation of either a predisposing factor or decreased thoracic compliance. The panelists rated the Delphi definition as acceptable to highly acceptable (median score, 6; range, 5-7 on a 7-point Likert scale). Conclusions: We conclude that it is feasible to consider using formal consensus in the development of future definitions of acute respiratory distress syndrome. Testing of sensibility, reliability, and validity are needed for this preliminary definition; these test results should be incorporated into future iterations of this definition. D 2005 Elsevier Inc. All rights reserved.
1. Introduction T Corresponding author. Toronto Western Hospital, F2-150-399 Bathurst Street, Toronto, ON, Canada M5T 2S8. Tel.: +1 416 603 6203; fax: +1 416 603 5375. E-mail address: [email protected] (N.D. Ferguson). 0883-9441/$ – see front matter D 2005 Elsevier Inc. All rights reserved. doi:10.1016/j.jcrc.2005.03.001
There is no available diagnostic test to confirm the presence of acute respiratory distress syndrome (ARDS), and it is, therefore, defined by clinical criteria in a similar fashion to diseases such as rheumatoid arthritis and depression.
148 Unlike many rheumatologic and psychiatric diseases, however, very little work has been done to develop and test standardized diagnostic criteria for ARDS [1]. Indeed, for many years after its original description [2], ARDS remained a gestalt diagnosis without specific criteria [3]. Translating the pathophysiologic concepts of ARDS [4-9] into a working clinical definition has proved challenging. The widely used American-European Consensus Conference (AECC) definition requires the acute onset of hypoxemia and bilateral chest x-ray infiltrates in the absence of left atrial hypertension [10]. This definition has not performed well in limited reliability testing focusing on the chest radiograph [11,12], and the agreement between this definition and the autopsy findings of ARDS is only moderate [13]. The recognition that rigorously developed and validated diagnostic criteria are vital to the performance of clinical trials in this area has led to calls for a reevaluation of this definition [14-17]. The use of formal consensus techniques in medicine is becoming increasingly frequent, but in critical care, these have typically taken the form of consensus statements used to help guide clinical care [18-24]. Formal consensus techniques may be helpful in explicating the necessary judgmental approach to definition development in situations where no gold standard exists, potentially reducing bias and resulting in a definition with improved operating characteristics. The AECC definition, however, was created without the use of formal consensus techniques. We are unaware of any reports documenting the use of such methodologies in the derivation of definitions for ARDS or other clinical entities in critical care. A variety of formal methods is available when conducting a consensus process [19,25-27]. One such method is the Delphi technique, an iterative process where experts are polled individually with a series of questionnaires, receiving anonymous group feedback between iterations [19]. In this paper, we describe the use of the Delphi technique in the development of a new ARDS definition. This represents the first phase of a strategy to develop and test a novel ARDS definition using rigorous methodologies.
N.D. Ferguson et al. Table 1
Definition framework
Concept: The phenomenon that this definition seeks to capture is ARDS. This is a specific type of pathological lung injury reflecting diffuse alveolar damage that is caused by an acute insult, originating from the airways or from the systemic circulation. Pathophysiologically, this results in widespread disruption of the barrier and gas exchange functions of the lung. Population: The intended population of this definition is composed of male and female adults who are intubated and currently cared for in an ICU setting. Purpose: The purpose of this definition is to act as a test to distinguish between patients who do and do not have ARDS, as inclusion criteria or endpoint for clinical studies of acute lung injury.
critical care symposia (Toronto 1999 and Brussels 2000) and by listing all participants in the original AECC process [10] (n = 279). Secondly, clinical researchers in ARDS were identified by listing all authors of ARDS clinical studies or review articles from 1995 to 1999 (n = 940). Individuals whose names appeared on both lists were considered eligible for inclusion in the Delphi panel (n = 73). Telephone recruitment using a standardized script proceeded down the randomly ordered lists, which were stratified by continent (Europe vs America). Nine panel members were recruited from North America and 7 from Europe. Demographic data were collected from the panelists at the time of the first Delphi questionnaire.
2.3. Definition framework Because different instruments may be needed for different purposes and populations, a framework was created to explicitly outline the concept, purpose, and target population of the new definition. The framework that was distributed to each panelist with every questionnaire is illustrated in Table 1 [29,30].
2.4. Item generation
2. Methods 2.1. Design The Delphi technique was chosen over other consensus techniques because of its ability to overcome temporal and geographic constraints, low cost, and ability to ensure all the panelists an equal voice in the proceedings. Approval from the University of Toronto research ethics board was obtained.
2.2. Participants A systematic method for identifying individuals who were both opinion leaders and clinical researchers was used to avoid bias in the selection of panelists [28]. First, opinion leaders were gathered by listing all speakers at 2 international
Throughout all rounds of the Delphi process, communication occurred by e-mail. The panelists were asked to complete each questionnaire within 2 weeks and were sent reminder e-mails 10, 15, and 21 days after the initial mailing as needed. Each Delphi round took approximately 1 month to complete. In the first round, the panelists were asked to list all possible defining characteristics that should be included in the new definition of ARDS. A defining characteristic was defined as bone of a number of essential features by which a person with a particular diagnosis can be recognisedQ [31]. Using a definition of chronic obstructive pulmonary disease as an example, one defining characteristic might be airflow limitation. They were then asked to create an operational definition for each of their defining characteristics. An operational definition was defined as ba set of directives,
ARDS definition Delphi process activities or procedures that specify how to measure, observe or record the defining characteristicQ [31]. Again, using a definition of chronic obstructive pulmonary disease as an example, the operational definition of airflow limitation might be a forced expiratory volume in 1 second to forced vital capacity ratio of less than 70%. In addition to these methods, a list of criteria used to define ARDS in previous clinical studies was compared to the items generated by the panel to avoid important omissions. This list was created by searching MedLine using a search strategy with the MeSH heading respiratory distress syndrome, adult limited to human being, English language, publication years 1985-1999, and limited to clinical trials (clinical trial, clinical trial phase 1-4, metaanalysis, controlled clinical trial, or randomized controlled trial). These papers were then reviewed, and individual defining characteristics used to define ARDS were abstracted from those papers (n = 102) that were identified as clinical trials in patients with ARDS.
149
2.5. Item reduction During rounds 2 and 3, the panelists were asked to state whether they believed that each defining characteristic was clear, feasible, and appropriate for inclusion in the new ARDS definition, using a yes/no format for each [31]. Similarly, they were asked to state whether they thought each operational definition was clear, feasible, and appropriate to adequately capture the defining characteristic that it represented [31]. When more than one potential operational definition existed for a given defining characteristic, the panelists were asked to rank these in terms of suitability for inclusion in the new definition. Suggestions for revisions were encouraged. During rounds 3 and 4, pooled anonymous feedback was presented on the basis of the opinions generated in the previous round. Comments or suggestions for revision were pasted directly into the feedback forms with a minimum of editing to minimize bias on behalf of the investigators.
ROUND 1 List all possible defining characteristics for inclusion in new ARDS definition Give a corresponding operational definition for each characteristic Answers to Round 1 collated Literature search performed ROUND 2 For each defining characteristic state whether clear, feasible and appropriate for inclusion in new ARDS definition (yes/no to each) For each operational definition state whether clear, feasible and appropriate to adequately capture attached defining characteristic Rank operational definitions where > 1 exists for a characteristic Comments and suggestions for revision encouraged Results collated and presented anonymously for next round Comments pasted directly into feedback form
3 characteristics dropped because >70% agree not appropriate / feasible 2 most popular operational definitions kept for each characteristic
ROUND 3 State clear, feasible and appropriate for characteristics and definitions as in Round 2 Rank operational definitions where > 1 exists for a characteristic Give rationale if voting against majority from Round 2, in addition to other comments Vote on methods to combine characteristics into cohesive definition Results collated and presented anonymously for next round Characteristics combined to yield functional definition Face and content validity questionnaire devised for Round 4
1 characteristic dropped because of redundancy 1 operational definition altered based on comments from Round 3
ROUND 4 Complete face and content validity questionnaire Rate overall acceptability of new definition Provide any further comments or feedback
Fig. 1 Delphi process schematic. The Delphi consensus process used in this study is outlined in this flow sheet. Tasks completed by panelists in each Delphi round are shown within boxes. Actions that occurred between rounds are shown between the boxes with methodology on the left of the arrows and results on the right of the arrows.
150
N.D. Ferguson et al.
Table 2
Delphi panel participant demographics
Number participating
11/16
Education MD MSc/MPH PhD Clinical training Critical care Internal medicine Anesthesia Pulmonary Other Time in clinical ICUa Time in researcha Primary research type Clinical Basic science Mixed ALI research ALI research focus Previous consensus experience
11 (100%) 2 (18%) 3 (27%) 9 9 2 8 1 33% 35% 8 2 1 9 7 9
(82%) (82%) (18%) (73%) (9%) (0-50) (10-75) (73%) (18%) (9%) (82%) (64%) (82%)
ALI indicates acute lung injury. a Median value (range) of percentage of working time spent in respective capacities.
An a priori consensus threshold of 70% F 5% was chosen. The error margin around this threshold was allowed because of the small number of panelists (eg, if 8 [67%] of the 12 panelists were to have been in agreement, this would have been considered adequate for consensus, because 9 [75%] of the 12 would have been well above the threshold value). To be accepted into the definition without further changes, an item needed at least 70% of the panelists to
Table 3
agree that it was appropriate, clear, and feasible for inclusion. Items that at least 70% of the panelists felt were unsuitable (ie, not clear, not feasible, or not appropriate) were removed after the second round to focus the discussion on more contentious issues. In addition, after the second round, the choice of operational definitions for a given defining characteristic was limited to the 2 highest ranked choices from the previous round to focus the discussion. In the third round, panelists voting against the majority opinion were asked to explain their reasoning in the comments sections to enhance group thinking [32]. Also, during the third round, panelists voted on methods for combining the proposed criteria into a cohesive definition. Finally, during the fourth round, a questionnaire regarding the face and content validity of the new definition was administered to gauge the panel’s satisfaction with the generated definition. A summary of the Delphi process is presented in Fig. 1.
2.6. Statistical issues On the basis of a compromise between generalizability and feasibility, 16 members were recruited to the Delphi panel. Allowing for a 25% dropout rate, it was estimated that the final panel would include 12 members. Descriptive statistics were generated using Excel 2000 (Microsoft, Redmond, Wash).
3. Results 3.1. Participants The first 16 individuals who were approached agreed to participate in the consensus process. However, 11 (69%) of
Item generation — defining characteristics
Defining characteristic
Item generation: round 1
Item reduction: round 2 (%)
Item reduction: round 3 (%)
Number suggesting (n)a
Appropriate
Feasible
Appropriate
10
100
100
Radiographic abnormalities
9
100
100
Predisposition Acute onset
7 6
100 100
89 100
No left atrial hypertension Decreased lung compliance
6 5
100 100
67 100
Increased permeability of pulmonary vasculature Evidence of lung inflammation Respiratory failure Surfactant abnormalities
4
78
33
4
89
22
3 1
67 33
89 0
Voting not repeated because of unanimity Voting not repeated because of unanimity 80 80 Voting not repeated because of unanimity 90 70 Voting not repeated because of unanimity Item eliminated after meeting infeasibility threshold Item eliminated after meeting infeasibility threshold 50 80 Item eliminated after meeting infeasibility threshold
Hypoxemia
a
Ten panelists participated in round 1, 9 sent in responses in round 2, 10 gave responses in round 3, and 11 responded in round 4.
Feasible
ARDS definition Delphi process Table 4
Modified clinical definition of ARDSa
Defining characteristic
Operational definition
1
Hypoxemia
2 3
Acute onset Radiographic abnormalities
4
Noncardiogenic in origin
5
Decreased lung compliance
6
Predisposition
Pao2/Fio2 b 200 mm Hg with PEEP z10 Rapid onset in b 72 h Bilateral airspace diseaseb involving z2 quadrants on frontal chest x-ray No clinical evidence of congestive heart failure (including use of PA catheter and/or echo if clinically indicated) Static respiratory system compliance b 50 mL/cm H2O (with patient sedated, Vt of 8 mL/kg, IBW, PEEP z10) Direct and/or indirect factor associated with lung injuryc
PA indicates pulmonary artery; Vt, tidal volume; IBW, ideal body weight. a ARDS is indicated by the presence of criteria 1- 4 and one of 5 or 6. b Airspace disease is defined as the presence of one or more of the following: (1) air bronchograms, (2) acinar shadows (nodular opacities 4-10 mm in diameter with poor margination), (3) coalescence of acinar shadows, (4) silhouette sign (loss of definition of the heart border or hemidiaphragm—excluding that caused by lobar collapse). c Clinical syndromes associated with ARDS: (1) direct lung injury: pneumonia, aspiration of gastric contents, fat emboli, near drowning, inhalational injury, reperfusion pulmonary edema after transplantation, or pulmonary embolectomy; (2) indirect lung injury: sepsis, severe trauma with shock and multiple transfusions, cardiopulmonary bypass, transfusions of blood products, and severe burns.
the 16 panelists actually participated in the definition development. Their names are listed in the acknowledgements. Demographic information on the participants is shown in Table 2.
3.2. Item generation—Delphi round 1 The first Delphi questionnaire, asking panelists to identify all possible defining characteristics for inclusion in a new ARDS definition, yielded 10 possible defining characteristics (Table 3). The review of previously published studies of ARDS did not yield any additional defining characteristics that had not already been identified by the panelists.
151 third round, the defining characteristic of respiratory failure was dropped because of redundancy, because the use of positive pressure mechanical ventilation was implicit in a number of the other criteria. After the third round, the consensus threshold of 70% was reached for all the remaining defining characteristics (Table 3). After round 3, operational definitions that met the predefined consensus thresholds (z70% F 5% agreement for appropriateness, clarity, feasibility and highest rank) were present for each of the defining characteristics except for no left atrial hypertension. Because of concerns voiced by the panelists, this defining characteristic was renamed to noncardiogenic in origin, recognizing the fact that ARDS and high left atrial pressures can coexist. The operational definition was then altered to no clinical evidence of congestive heart failure, with a supplementary statement that bclinical evidenceQ could include not only the history and physical examination, but also the use of ancillary tests, such as echocardiography, and/or the use of a pulmonary artery catheter, as clinically indicated.
3.4. Definition construction and review—Delphi round 4 When asked about ideal definition construction, 78% of the panelists suggested either requiring all criteria to be present, or requiring the presence of 4 core criteria with or without others present, to make the diagnosis of ARDS. On the basis of these suggestions, a provisional definition was created (Table 4). The definition requires the presence hypoxemia, radiographic abnormalities, acute onset, noncardiogenic in origin and predisposition, and/or decreased compliance to make the diagnosis of ARDS. This provisional definition was sent out to the development panel in the fourth questionnaire. Overall, the definition received a high degree of acceptance with a median overall acceptability score of 6.0 (range, 5-7), which corresponds to a rating of acceptable to highly acceptable. Other results (Table 5) show that in concordance with their high global acceptability rating, panelists specifically rated the content and face validity of the new definition in the good to excellent categories. On the basis of these results,
Table 5
Definition review results
3.3. Item reduction—Delphi rounds 2 and 3
Topic
Resulta
Corresponding rating
After the second round, 3 defining characteristics were eliminated because of infeasibility. Lung inflammation and increased permeability of the pulmonary vasculature were both felt to be appropriate for inclusion in a new ARDS definition by most panelists (89% and 78% appropriate, respectively), but were not felt to be feasible at the current time (67% and 78% infeasible, respectively). Surfactant abnormalities were felt to be neither appropriate nor feasible (33% appropriate, 100% infeasible) (Table 3). After the
Omissions Redundancies Grouping of criteria Clarity of wording Ability to accurately detect ARDS Global acceptability
7 6 5 6 5
(5-7) (5-7) (4-7) (4-7) (4-7)
Minimal Few to none Appropriate Good to excellent Likely
6 (5-7)
Acceptable to highly acceptable
a
Median values (range).
152 the provisional definition (Table 4) was accepted as the final product of the Delphi consensus process.
4. Discussion Definitions for difficult-to-define intensive care unit (ICU) clinical entities such as sepsis, ventilator-associated pneumonia, and ARDS have not been approached previously using formal consensus techniques. Papers labeled as bconsensus conferencesQ have been published addressing each of these examples, but in each case, the format used was an informal group discussion. Formal consensus methods, such as the Delphi technique, may be useful in reducing bias and enhancing the transparency of the judgmental approach that is often required to develop diagnostic criteria in the absence of an available gold standard. Such definitions developed with more methodological rigor ultimately may perform better than existing definitions. The exact choice of which validated formal consensus technique to use, however, is probably less important than ensuring that a number of methodological principles are met [18]. Items on a consensus methods checklist include (a) a structured and unbiased method for enrolling expert participants, (b) an explicit method for information elicitation or item generation, (c) the use of a respected moderator who has no vested interest in the results, (d) predefined consensus thresholds, and (e) an explicit feedback loop that is operative after each iteration. Another integral part of any definition development process involves the testing of the definition and the incorporation of these results to produce a more coherent revised definition. In this way, definition development and definition testing work together to complete a feedback loop that should improve the performance of the resultant definition. The results of this study represent only the first half of such a loop, and as such, this definition should be regarded as preliminary. We do not recommend, therefore, that this preliminary definition be put into widespread use at the current time. Rather, we hope that this methodological framework may be helpful in the development of future consensus processes in critical care. The sensibility, reliability, and validity of this new definition will need to be tested and potential revisions implemented before its acceptance into practice. The definition developed by the Delphi panel has important differences from the existing AECC definition. These include the requirement of positive end-expiratory pressure (PEEP) in the oxygenation criteria, the specification of airspace disease on chest radiograph, the recognition that ARDS and left atrial hypertension can coexist, and the requirement for either a predisposing factor or decreased compliance to be present. These changes may improve the new definition’s reliability and its ability to consistently select a group of patients with a more homogeneously severe form of lung injury [33-35]. The differences between definitions may have resulted from the differing development
N.D. Ferguson et al. methodologies, but the panel composition and the time interval since the publication of the AECC definition may have also played a role. The Delphi panelists had the advantage of 6 years time of working with the AECC definition and the subsequent publication of numerous papers examining its performance [11,12,16,17,33,36-42]. In addition, the panelists in this study may have had different beliefs, expectations, and biases compared to the AECC panel. These clearly may have had an impact on the resultant new definition. This raises the issue of reliability or reproducibility of the Delphi technique [19,31,43]. The only study that we identified that examined the reliability of the Delphi technique observed 93% agreement between 2 separate Delphi panels [44]. The recognition that panel membership may have a potential impact upon the results of any consensus process supports the use of a structured method for recruiting panelists into this study. Despite the advantages inherent to the Delphi technique, this method also has potential disadvantages including a potential lack of panelist responsibility and accountability, the loss of the group dynamic achieved with a face-to-face meeting, and the potential for biased feedback from the coordinating center. To try and minimize these problems, a number of steps were taken, including (a) encouraging comments and suggestions rather than just simple yes/no questions, (b) asking panelists to explain their thought process when voting against the majority, and (c) employing cut-and-paste methods for transcribing comments to avoid bias and transcription errors. A relatively small panel size of 16 members was recruited for this study. The results from a larger panel might have been more generalizable, but it was practically difficult to coordinate a significantly larger consensus group. In addition, this Delphi panel was similar in size to the AECC definition subcommittee (numbering 13 panelists) who generated the AECC definition [10]. Potentially compounding the issues of a small panel size was the lack of participation of a number of panelists. Although all 16 panelists verbally agreed to participate, only 11 sent back responses, leading to a smaller final panel size than was anticipated. The phenomenon of panel-member dropout is well recognized, however, and is likely to leave a core panel of individuals most interested and motivated to reach a consensus result [32]. The degree of consensus with the Delphi technique, as with other consensus methods, has often been poorly defined [25,43,44]. Many studies have not specified consensus thresholds a priori, instead, specifying the degree of consensus only after having seen the data [43]. The optimal consensus threshold for use with the Delphi technique remains a subject for debate. The level of consensus should clearly be specified a priori, because this will decrease the likelihood that bias would lead to the inclusion or the exclusion of marginal choices. This consensus level may be arbitrary, because it was in this study, and will likely vary significantly between studies depending on the context, topic, and questionnaire structure. Regardless, this method
ARDS definition Delphi process should lend more transparency and generalizability to the process and reduce bias [43]. We conclude that it is feasible to consider utilizing formal consensus methods in the development of future definitions of ARDS and other clinical entities in the ICU. Our specific consensus process yielded an ARDS definition with important differences from the AECC definition; however, testing of this preliminary definition in terms of sensibility, reliability, and validity, with subsequent incorporation of feedback, is required.
Acknowledgments This study was supported in part by the Canadian Institutes of Health Research/Canadian Lung Association (Ottawa, Ontario, Canada) Post-Doctoral Fellowship (Dr Ferguson) and the Health Career Award from the Canadian Institutes of Health Research (Dr Davis). We thank the following individuals who participated as members of our expert panel during the Delphi process: Drs Antonio Artigas, Jean-Daniel Chiche, Gregory Downey, Margaret Herridge, Waldemar Johanson, Marin Kollef, James Lewis, Neil MacIntyre, Umberto Meduri, Antonio Pesenti, and Jean-Louis Teboul. Thanks also to Drs Charles Chan, Allan Detsky, Ted Marras, and Matthew Stanbrook for their helpful input during the design of the study and/or the preparation of the manuscript.
References [1] Rubenfeld GD. Epidemiology of acute lung injury. Crit Care Med 2003;31(Suppl):S276 - 84. [2] Ashbaugh DG, Bigelow DB, Petty TL, et al. Acute respiratory distress in adults. Lancet 1967;2:319 - 23. [3] Garber BG, Hebert PC, Yelle JD, et al. Adult respiratory distress syndrome: a systemic overview of incidence and risk factors. Crit Care Med 1996;24:687 - 95. [4] Hudson LD, Steinberg KP. Epidemiology of acute lung injury and ARDS. Chest 1999;116:74S - 82S. [5] Marinelli WA, Ingbar DH. Diagnosis and management of acute lung injury. Clin Chest Med 1994;15:517 - 46. [6] Schuster DP, Kollef MH. Acute respiratory distress syndrome. Dis Mon 1996;42:270 - 326. [7] Lesur O, Berthiaume Y, Blaise G, et al. Acute respiratory distress syndrome: 30 years later. Can Respir J 1999;6:71 - 86. [8] Matthay MA. Conference summary: acute lung injury. Chest 1999; 116:119S - 26S. [9] Ware LB, Matthay MA. The acute respiratory distress syndrome. N Engl J Med 2000;342:1334 - 49. [10] Bernard GR, Artigas A, Brigham KL, et al. The American-European Consensus Conference on ARDS. Definitions, mechanisms, relevant outcomes, and clinical trial coordination. Am J Respir Crit Care Med 1994;149:818 - 24. [11] Meade MO, Cook RJ, Guyatt GH, et al. Interobserver variation in interpreting chest radiographs for the diagnosis of acute respiratory distress syndrome. Am J Respir Crit Care Med 2000;161:85 - 90. [12] Rubenfeld GD, Caldwell E, Granton JT, et al. Interobserver variability in applying a radiographic definition for ARDS. Chest 1999;116: 1347 - 53.
153 [13] Esteban A, Ferna´ndez-Segoviano P, Frutos-Vivar F, et al. Comparison of clinical criteria for the acute respiratory distress syndrome with autopsy findings. Ann Intern Med 2004;141:440 - 5. [14] Abraham E. Toward new definitions of acute respiratory distress syndrome. Crit Care Med 1999;27:237 - 8. [15] Abraham E, Matthay MA, Dinarello CA, et al. Consensus conference definitions for sepsis, septic shock, acute lung injury, and acute respiratory distress syndrome: time for a reevaluation. Crit Care Med 2000;28:232 - 5. [16] Lewandowski K. Epidemiological data challenge ARDS/ALI definition. Intensive Care Med 1999;25:884 - 6. [17] Schuster DP. Identifying patients with ARDS: time for a different approach. Intensive Care Med 1997;23:1197 - 203. [18] Rotondi AJ, Kvetan V, Carlet J, et al. Consensus conferences in critical care medicine. Methodologies and impact. Crit Care Clinics 1997;13: 417 - 39. [19] Fink A, Kosecoff J, Chassin M, et al. Consensus methods: characteristics and guidelines for use. Am J Public Health 1984;74:979 - 83. [20] Anonymous. International Consensus Conference in Intensive Care Medicine: ventilator-associated lung injury in ARDS. Am J Respir Crit Care Med 1999;160:2118 - 24. [21] Anonymous. Pulmonary Artery Catheter Consensus Conference: consensus statement. Crit Care Med 1997;25:910 - 24. [22] Artigas A, Bernard GR, Carlet J, et al. The American-European Consensus Conference on ARDS, Part 2: Ventilatory, pharmacologic, supportive therapy, study design strategies, and issues related to recovery and remodeling. Acute respiratory distress syndrome. Am J Respir Crit Care Med 1998;157:1332 - 47. [23] Cuthbertson BH, Dellinger P, Dyar OJ, et al. UK guidelines for the use of inhaled nitric oxide therapy in adult ICUs. American-European Consensus Conference on ALI/ARDS. Intensive Care Med 1997;23: 1212 - 8. [24] Slutsky AS. Consensus conference on mechanical ventilation— January 28-30, 1993 at Northbrook. Illinois, USA. Part 2. Intensive Care Med 1994;20:150 - 62. [25] Crisp J, Pelletier D, Duffield C, et al. The Delphi method? Nurs Res 1997;46:116 - 8. [26] Goodman CM. The Delphi technique: a critique. J Adv Nurs 1987;12: 729 - 34. [27] Carlet J, Artigas A, Bihari D, et al. The first European consensus conference in intensive care medicine: introductory remarks. Intensive Care Med 1992;18:180 - 1. [28] Naglie G, Silberfeld M, O’Rourke K, et al. Convening expert panels to identify mental capacity assessment items. Can J Aging 1995;14: 697 - 705. [29] Buchbinder R, Goel V, Bombardier C, et al. Classification systems of soft tissue disorders of the neck and upper limb: do they satisfy methodological guidelines? J Clin Epidemiol 1996;49:141 - 9. [30] Feinstein AR. The theory and evaluation of sensibility, clinimetrics. New Haven7 Yale University Press; 1987. p. 141 - 66. [31] Grant JS, Kinney MR. Using the Delphi technique to examine the content validity of nursing diagnoses. Nurs Diagn 1992;3:12 - 22. [32] Rotondi A, Gustafson D. Theoretical, methodological and practical issues arising out of the Delphi method. In: Adler M, Ziglio E, editors. Gazing into the oracle: the Delphi method and its application to social policy and public health. London7 Jessica Kingsley Publishers; 1996. p. 34 - 55. [33] Villar J, Perez-Mendez L, Kacmarek RM. Current definitions of acute lung injury and the acute respiratory distress syndrome do not reflect their true severity and outcome. Intensive Care Med 1999;25: 930 - 5. [34] Neff MJ, Rubenfeld GD, Caldwell ES, et al. Exclusion of patients with elevated pulmonary capillary wedge pressure from acute respiratory distress syndrome. Am J Respir Crit Care Med 1999; 159:A716. [35] Ferguson ND, Meade MO, Hallett DC, et al. High values of the pulmonary artery wedge pressure in patients with acute lung injury and
154
[36]
[37]
[38]
[39]
N.D. Ferguson et al. acute respiratory distress syndrome. Intensive Care Med 2002;28: 1073 - 7. Moss M, Goodman PL, Heinig M, et al. Establishing the relative accuracy of three new definitions of the adult respiratory distress syndrome. Crit Care Med 1995;23:1629 - 37. Luhr OR, Antonsen K, Karlsson M, et al. Incidence and mortality after acute respiratory failure and acute respiratory distress syndrome in Sweden, Denmark, and Iceland. The ARF Study Group. Am J Respir Crit Care Med 1999;159:1849 - 61. Goh AY, Chan PW, Lum LC, et al. Incidence of acute respiratory distress syndrome: a comparison of two definitions. Arch Dis Child 1998;79:256 - 9. Knaus WA, Sun X, Hakim RB, et al. Evaluation of definitions for adult respiratory distress syndrome. Am J Respir Crit Care Med 1994; 150:311 - 7.
[40] Schuster DP. What is acute lung injury? What is ARDS? Chest 1995; 107:1721 - 6. [41] Krafft P, Fridrich P, Pernerstorfer T, et al. The acute respiratory distress syndrome: definitions, severity and clinical outcome. An analysis of 101 clinical investigations. Intensive Care Med 1996;22: 519 - 29. [42] Roupie E, Lepage E, Wysocki M, et al. Prevalence, etiologies and outcome of the acute respiratory distress syndrome among hypoxemic ventilated patients. SRLF Collaborative Group on Mechanical Ventilation. Societe de Reanimation de Langue Francaise. Intensive Care Med 1999;25:920 - 9. [43] Williams PL, Webb C. The Delphi technique: a methodological discussion. J Adv Nurs 1994;19:180 - 6. [44] Duffield C. The Delphi technique: a comparison of results obtained using two expert panels. Int J Nurs Stud 1993;30:227 - 37.
Development of a clinical definition for acute respiratory distress syndrome using the Delphi technique Niall D. Ferguson MD, MSca,b,d,*, Aileen M. Davis PhDc,e, Arthur S. Slutsky MDa,b,f, Thomas E. Stewart MDa,b,d a
Interdepartmental Division of Critical Care, University of Toronto, Toronto, ON, Canada Department of Medicine, Division of Respirology, University of Toronto, Toronto, ON, Canada c Department of Physiotherapy, University of Toronto, Toronto, ON, Canada d Mount Sinai Hospital and University Health Network, University of Toronto, Toronto, ON, Canada e Toronto Rehabilitation Institute, University of Toronto, Toronto, ON, Canada f St Michael’s Hospital, University of Toronto, Toronto, ON, Canada b
Received 10 November 2004; revised 25 January 2005; accepted 1 March 2005
Keywords: Diagnosis; Consensus; ARDS; Delphi technique; Definition development
Abstract Purpose: The objective of this study is to describe the implementation of formal consensus techniques in the development of a clinical definition for acute respiratory distress syndrome. Materials and Methods: A Delphi consensus process was conducted using e-mail. Sixteen panelists who were both researchers and opinion leaders were systematically recruited. The Delphi technique was performed over 4 rounds on the background of an explicit definition framework. Item generation was performed in round 1, item reduction in rounds 2 and 3, and definition evaluation in round 4. Explicit consensus thresholds were used throughout. Results: Of the 16 panelists, 11 actually participated in developing a definition that met a priori consensus rules on the third iteration. New incorporations in the Delphi definition include the use of a standardized oxygenation assessment and the documentation of either a predisposing factor or decreased thoracic compliance. The panelists rated the Delphi definition as acceptable to highly acceptable (median score, 6; range, 5-7 on a 7-point Likert scale). Conclusions: We conclude that it is feasible to consider using formal consensus in the development of future definitions of acute respiratory distress syndrome. Testing of sensibility, reliability, and validity are needed for this preliminary definition; these test results should be incorporated into future iterations of this definition. D 2005 Elsevier Inc. All rights reserved.
1. Introduction T Corresponding author. Toronto Western Hospital, F2-150-399 Bathurst Street, Toronto, ON, Canada M5T 2S8. Tel.: +1 416 603 6203; fax: +1 416 603 5375. E-mail address: [email protected] (N.D. Ferguson). 0883-9441/$ – see front matter D 2005 Elsevier Inc. All rights reserved. doi:10.1016/j.jcrc.2005.03.001
There is no available diagnostic test to confirm the presence of acute respiratory distress syndrome (ARDS), and it is, therefore, defined by clinical criteria in a similar fashion to diseases such as rheumatoid arthritis and depression.
148 Unlike many rheumatologic and psychiatric diseases, however, very little work has been done to develop and test standardized diagnostic criteria for ARDS [1]. Indeed, for many years after its original description [2], ARDS remained a gestalt diagnosis without specific criteria [3]. Translating the pathophysiologic concepts of ARDS [4-9] into a working clinical definition has proved challenging. The widely used American-European Consensus Conference (AECC) definition requires the acute onset of hypoxemia and bilateral chest x-ray infiltrates in the absence of left atrial hypertension [10]. This definition has not performed well in limited reliability testing focusing on the chest radiograph [11,12], and the agreement between this definition and the autopsy findings of ARDS is only moderate [13]. The recognition that rigorously developed and validated diagnostic criteria are vital to the performance of clinical trials in this area has led to calls for a reevaluation of this definition [14-17]. The use of formal consensus techniques in medicine is becoming increasingly frequent, but in critical care, these have typically taken the form of consensus statements used to help guide clinical care [18-24]. Formal consensus techniques may be helpful in explicating the necessary judgmental approach to definition development in situations where no gold standard exists, potentially reducing bias and resulting in a definition with improved operating characteristics. The AECC definition, however, was created without the use of formal consensus techniques. We are unaware of any reports documenting the use of such methodologies in the derivation of definitions for ARDS or other clinical entities in critical care. A variety of formal methods is available when conducting a consensus process [19,25-27]. One such method is the Delphi technique, an iterative process where experts are polled individually with a series of questionnaires, receiving anonymous group feedback between iterations [19]. In this paper, we describe the use of the Delphi technique in the development of a new ARDS definition. This represents the first phase of a strategy to develop and test a novel ARDS definition using rigorous methodologies.
N.D. Ferguson et al. Table 1
Definition framework
Concept: The phenomenon that this definition seeks to capture is ARDS. This is a specific type of pathological lung injury reflecting diffuse alveolar damage that is caused by an acute insult, originating from the airways or from the systemic circulation. Pathophysiologically, this results in widespread disruption of the barrier and gas exchange functions of the lung. Population: The intended population of this definition is composed of male and female adults who are intubated and currently cared for in an ICU setting. Purpose: The purpose of this definition is to act as a test to distinguish between patients who do and do not have ARDS, as inclusion criteria or endpoint for clinical studies of acute lung injury.
critical care symposia (Toronto 1999 and Brussels 2000) and by listing all participants in the original AECC process [10] (n = 279). Secondly, clinical researchers in ARDS were identified by listing all authors of ARDS clinical studies or review articles from 1995 to 1999 (n = 940). Individuals whose names appeared on both lists were considered eligible for inclusion in the Delphi panel (n = 73). Telephone recruitment using a standardized script proceeded down the randomly ordered lists, which were stratified by continent (Europe vs America). Nine panel members were recruited from North America and 7 from Europe. Demographic data were collected from the panelists at the time of the first Delphi questionnaire.
2.3. Definition framework Because different instruments may be needed for different purposes and populations, a framework was created to explicitly outline the concept, purpose, and target population of the new definition. The framework that was distributed to each panelist with every questionnaire is illustrated in Table 1 [29,30].
2.4. Item generation
2. Methods 2.1. Design The Delphi technique was chosen over other consensus techniques because of its ability to overcome temporal and geographic constraints, low cost, and ability to ensure all the panelists an equal voice in the proceedings. Approval from the University of Toronto research ethics board was obtained.
2.2. Participants A systematic method for identifying individuals who were both opinion leaders and clinical researchers was used to avoid bias in the selection of panelists [28]. First, opinion leaders were gathered by listing all speakers at 2 international
Throughout all rounds of the Delphi process, communication occurred by e-mail. The panelists were asked to complete each questionnaire within 2 weeks and were sent reminder e-mails 10, 15, and 21 days after the initial mailing as needed. Each Delphi round took approximately 1 month to complete. In the first round, the panelists were asked to list all possible defining characteristics that should be included in the new definition of ARDS. A defining characteristic was defined as bone of a number of essential features by which a person with a particular diagnosis can be recognisedQ [31]. Using a definition of chronic obstructive pulmonary disease as an example, one defining characteristic might be airflow limitation. They were then asked to create an operational definition for each of their defining characteristics. An operational definition was defined as ba set of directives,
ARDS definition Delphi process activities or procedures that specify how to measure, observe or record the defining characteristicQ [31]. Again, using a definition of chronic obstructive pulmonary disease as an example, the operational definition of airflow limitation might be a forced expiratory volume in 1 second to forced vital capacity ratio of less than 70%. In addition to these methods, a list of criteria used to define ARDS in previous clinical studies was compared to the items generated by the panel to avoid important omissions. This list was created by searching MedLine using a search strategy with the MeSH heading respiratory distress syndrome, adult limited to human being, English language, publication years 1985-1999, and limited to clinical trials (clinical trial, clinical trial phase 1-4, metaanalysis, controlled clinical trial, or randomized controlled trial). These papers were then reviewed, and individual defining characteristics used to define ARDS were abstracted from those papers (n = 102) that were identified as clinical trials in patients with ARDS.
149
2.5. Item reduction During rounds 2 and 3, the panelists were asked to state whether they believed that each defining characteristic was clear, feasible, and appropriate for inclusion in the new ARDS definition, using a yes/no format for each [31]. Similarly, they were asked to state whether they thought each operational definition was clear, feasible, and appropriate to adequately capture the defining characteristic that it represented [31]. When more than one potential operational definition existed for a given defining characteristic, the panelists were asked to rank these in terms of suitability for inclusion in the new definition. Suggestions for revisions were encouraged. During rounds 3 and 4, pooled anonymous feedback was presented on the basis of the opinions generated in the previous round. Comments or suggestions for revision were pasted directly into the feedback forms with a minimum of editing to minimize bias on behalf of the investigators.
ROUND 1 List all possible defining characteristics for inclusion in new ARDS definition Give a corresponding operational definition for each characteristic Answers to Round 1 collated Literature search performed ROUND 2 For each defining characteristic state whether clear, feasible and appropriate for inclusion in new ARDS definition (yes/no to each) For each operational definition state whether clear, feasible and appropriate to adequately capture attached defining characteristic Rank operational definitions where > 1 exists for a characteristic Comments and suggestions for revision encouraged Results collated and presented anonymously for next round Comments pasted directly into feedback form
3 characteristics dropped because >70% agree not appropriate / feasible 2 most popular operational definitions kept for each characteristic
ROUND 3 State clear, feasible and appropriate for characteristics and definitions as in Round 2 Rank operational definitions where > 1 exists for a characteristic Give rationale if voting against majority from Round 2, in addition to other comments Vote on methods to combine characteristics into cohesive definition Results collated and presented anonymously for next round Characteristics combined to yield functional definition Face and content validity questionnaire devised for Round 4
1 characteristic dropped because of redundancy 1 operational definition altered based on comments from Round 3
ROUND 4 Complete face and content validity questionnaire Rate overall acceptability of new definition Provide any further comments or feedback
Fig. 1 Delphi process schematic. The Delphi consensus process used in this study is outlined in this flow sheet. Tasks completed by panelists in each Delphi round are shown within boxes. Actions that occurred between rounds are shown between the boxes with methodology on the left of the arrows and results on the right of the arrows.
150
N.D. Ferguson et al.
Table 2
Delphi panel participant demographics
Number participating
11/16
Education MD MSc/MPH PhD Clinical training Critical care Internal medicine Anesthesia Pulmonary Other Time in clinical ICUa Time in researcha Primary research type Clinical Basic science Mixed ALI research ALI research focus Previous consensus experience
11 (100%) 2 (18%) 3 (27%) 9 9 2 8 1 33% 35% 8 2 1 9 7 9
(82%) (82%) (18%) (73%) (9%) (0-50) (10-75) (73%) (18%) (9%) (82%) (64%) (82%)
ALI indicates acute lung injury. a Median value (range) of percentage of working time spent in respective capacities.
An a priori consensus threshold of 70% F 5% was chosen. The error margin around this threshold was allowed because of the small number of panelists (eg, if 8 [67%] of the 12 panelists were to have been in agreement, this would have been considered adequate for consensus, because 9 [75%] of the 12 would have been well above the threshold value). To be accepted into the definition without further changes, an item needed at least 70% of the panelists to
Table 3
agree that it was appropriate, clear, and feasible for inclusion. Items that at least 70% of the panelists felt were unsuitable (ie, not clear, not feasible, or not appropriate) were removed after the second round to focus the discussion on more contentious issues. In addition, after the second round, the choice of operational definitions for a given defining characteristic was limited to the 2 highest ranked choices from the previous round to focus the discussion. In the third round, panelists voting against the majority opinion were asked to explain their reasoning in the comments sections to enhance group thinking [32]. Also, during the third round, panelists voted on methods for combining the proposed criteria into a cohesive definition. Finally, during the fourth round, a questionnaire regarding the face and content validity of the new definition was administered to gauge the panel’s satisfaction with the generated definition. A summary of the Delphi process is presented in Fig. 1.
2.6. Statistical issues On the basis of a compromise between generalizability and feasibility, 16 members were recruited to the Delphi panel. Allowing for a 25% dropout rate, it was estimated that the final panel would include 12 members. Descriptive statistics were generated using Excel 2000 (Microsoft, Redmond, Wash).
3. Results 3.1. Participants The first 16 individuals who were approached agreed to participate in the consensus process. However, 11 (69%) of
Item generation — defining characteristics
Defining characteristic
Item generation: round 1
Item reduction: round 2 (%)
Item reduction: round 3 (%)
Number suggesting (n)a
Appropriate
Feasible
Appropriate
10
100
100
Radiographic abnormalities
9
100
100
Predisposition Acute onset
7 6
100 100
89 100
No left atrial hypertension Decreased lung compliance
6 5
100 100
67 100
Increased permeability of pulmonary vasculature Evidence of lung inflammation Respiratory failure Surfactant abnormalities
4
78
33
4
89
22
3 1
67 33
89 0
Voting not repeated because of unanimity Voting not repeated because of unanimity 80 80 Voting not repeated because of unanimity 90 70 Voting not repeated because of unanimity Item eliminated after meeting infeasibility threshold Item eliminated after meeting infeasibility threshold 50 80 Item eliminated after meeting infeasibility threshold
Hypoxemia
a
Ten panelists participated in round 1, 9 sent in responses in round 2, 10 gave responses in round 3, and 11 responded in round 4.
Feasible
ARDS definition Delphi process Table 4
Modified clinical definition of ARDSa
Defining characteristic
Operational definition
1
Hypoxemia
2 3
Acute onset Radiographic abnormalities
4
Noncardiogenic in origin
5
Decreased lung compliance
6
Predisposition
Pao2/Fio2 b 200 mm Hg with PEEP z10 Rapid onset in b 72 h Bilateral airspace diseaseb involving z2 quadrants on frontal chest x-ray No clinical evidence of congestive heart failure (including use of PA catheter and/or echo if clinically indicated) Static respiratory system compliance b 50 mL/cm H2O (with patient sedated, Vt of 8 mL/kg, IBW, PEEP z10) Direct and/or indirect factor associated with lung injuryc
PA indicates pulmonary artery; Vt, tidal volume; IBW, ideal body weight. a ARDS is indicated by the presence of criteria 1- 4 and one of 5 or 6. b Airspace disease is defined as the presence of one or more of the following: (1) air bronchograms, (2) acinar shadows (nodular opacities 4-10 mm in diameter with poor margination), (3) coalescence of acinar shadows, (4) silhouette sign (loss of definition of the heart border or hemidiaphragm—excluding that caused by lobar collapse). c Clinical syndromes associated with ARDS: (1) direct lung injury: pneumonia, aspiration of gastric contents, fat emboli, near drowning, inhalational injury, reperfusion pulmonary edema after transplantation, or pulmonary embolectomy; (2) indirect lung injury: sepsis, severe trauma with shock and multiple transfusions, cardiopulmonary bypass, transfusions of blood products, and severe burns.
the 16 panelists actually participated in the definition development. Their names are listed in the acknowledgements. Demographic information on the participants is shown in Table 2.
3.2. Item generation—Delphi round 1 The first Delphi questionnaire, asking panelists to identify all possible defining characteristics for inclusion in a new ARDS definition, yielded 10 possible defining characteristics (Table 3). The review of previously published studies of ARDS did not yield any additional defining characteristics that had not already been identified by the panelists.
151 third round, the defining characteristic of respiratory failure was dropped because of redundancy, because the use of positive pressure mechanical ventilation was implicit in a number of the other criteria. After the third round, the consensus threshold of 70% was reached for all the remaining defining characteristics (Table 3). After round 3, operational definitions that met the predefined consensus thresholds (z70% F 5% agreement for appropriateness, clarity, feasibility and highest rank) were present for each of the defining characteristics except for no left atrial hypertension. Because of concerns voiced by the panelists, this defining characteristic was renamed to noncardiogenic in origin, recognizing the fact that ARDS and high left atrial pressures can coexist. The operational definition was then altered to no clinical evidence of congestive heart failure, with a supplementary statement that bclinical evidenceQ could include not only the history and physical examination, but also the use of ancillary tests, such as echocardiography, and/or the use of a pulmonary artery catheter, as clinically indicated.
3.4. Definition construction and review—Delphi round 4 When asked about ideal definition construction, 78% of the panelists suggested either requiring all criteria to be present, or requiring the presence of 4 core criteria with or without others present, to make the diagnosis of ARDS. On the basis of these suggestions, a provisional definition was created (Table 4). The definition requires the presence hypoxemia, radiographic abnormalities, acute onset, noncardiogenic in origin and predisposition, and/or decreased compliance to make the diagnosis of ARDS. This provisional definition was sent out to the development panel in the fourth questionnaire. Overall, the definition received a high degree of acceptance with a median overall acceptability score of 6.0 (range, 5-7), which corresponds to a rating of acceptable to highly acceptable. Other results (Table 5) show that in concordance with their high global acceptability rating, panelists specifically rated the content and face validity of the new definition in the good to excellent categories. On the basis of these results,
Table 5
Definition review results
3.3. Item reduction—Delphi rounds 2 and 3
Topic
Resulta
Corresponding rating
After the second round, 3 defining characteristics were eliminated because of infeasibility. Lung inflammation and increased permeability of the pulmonary vasculature were both felt to be appropriate for inclusion in a new ARDS definition by most panelists (89% and 78% appropriate, respectively), but were not felt to be feasible at the current time (67% and 78% infeasible, respectively). Surfactant abnormalities were felt to be neither appropriate nor feasible (33% appropriate, 100% infeasible) (Table 3). After the
Omissions Redundancies Grouping of criteria Clarity of wording Ability to accurately detect ARDS Global acceptability
7 6 5 6 5
(5-7) (5-7) (4-7) (4-7) (4-7)
Minimal Few to none Appropriate Good to excellent Likely
6 (5-7)
Acceptable to highly acceptable
a
Median values (range).
152 the provisional definition (Table 4) was accepted as the final product of the Delphi consensus process.
4. Discussion Definitions for difficult-to-define intensive care unit (ICU) clinical entities such as sepsis, ventilator-associated pneumonia, and ARDS have not been approached previously using formal consensus techniques. Papers labeled as bconsensus conferencesQ have been published addressing each of these examples, but in each case, the format used was an informal group discussion. Formal consensus methods, such as the Delphi technique, may be useful in reducing bias and enhancing the transparency of the judgmental approach that is often required to develop diagnostic criteria in the absence of an available gold standard. Such definitions developed with more methodological rigor ultimately may perform better than existing definitions. The exact choice of which validated formal consensus technique to use, however, is probably less important than ensuring that a number of methodological principles are met [18]. Items on a consensus methods checklist include (a) a structured and unbiased method for enrolling expert participants, (b) an explicit method for information elicitation or item generation, (c) the use of a respected moderator who has no vested interest in the results, (d) predefined consensus thresholds, and (e) an explicit feedback loop that is operative after each iteration. Another integral part of any definition development process involves the testing of the definition and the incorporation of these results to produce a more coherent revised definition. In this way, definition development and definition testing work together to complete a feedback loop that should improve the performance of the resultant definition. The results of this study represent only the first half of such a loop, and as such, this definition should be regarded as preliminary. We do not recommend, therefore, that this preliminary definition be put into widespread use at the current time. Rather, we hope that this methodological framework may be helpful in the development of future consensus processes in critical care. The sensibility, reliability, and validity of this new definition will need to be tested and potential revisions implemented before its acceptance into practice. The definition developed by the Delphi panel has important differences from the existing AECC definition. These include the requirement of positive end-expiratory pressure (PEEP) in the oxygenation criteria, the specification of airspace disease on chest radiograph, the recognition that ARDS and left atrial hypertension can coexist, and the requirement for either a predisposing factor or decreased compliance to be present. These changes may improve the new definition’s reliability and its ability to consistently select a group of patients with a more homogeneously severe form of lung injury [33-35]. The differences between definitions may have resulted from the differing development
N.D. Ferguson et al. methodologies, but the panel composition and the time interval since the publication of the AECC definition may have also played a role. The Delphi panelists had the advantage of 6 years time of working with the AECC definition and the subsequent publication of numerous papers examining its performance [11,12,16,17,33,36-42]. In addition, the panelists in this study may have had different beliefs, expectations, and biases compared to the AECC panel. These clearly may have had an impact on the resultant new definition. This raises the issue of reliability or reproducibility of the Delphi technique [19,31,43]. The only study that we identified that examined the reliability of the Delphi technique observed 93% agreement between 2 separate Delphi panels [44]. The recognition that panel membership may have a potential impact upon the results of any consensus process supports the use of a structured method for recruiting panelists into this study. Despite the advantages inherent to the Delphi technique, this method also has potential disadvantages including a potential lack of panelist responsibility and accountability, the loss of the group dynamic achieved with a face-to-face meeting, and the potential for biased feedback from the coordinating center. To try and minimize these problems, a number of steps were taken, including (a) encouraging comments and suggestions rather than just simple yes/no questions, (b) asking panelists to explain their thought process when voting against the majority, and (c) employing cut-and-paste methods for transcribing comments to avoid bias and transcription errors. A relatively small panel size of 16 members was recruited for this study. The results from a larger panel might have been more generalizable, but it was practically difficult to coordinate a significantly larger consensus group. In addition, this Delphi panel was similar in size to the AECC definition subcommittee (numbering 13 panelists) who generated the AECC definition [10]. Potentially compounding the issues of a small panel size was the lack of participation of a number of panelists. Although all 16 panelists verbally agreed to participate, only 11 sent back responses, leading to a smaller final panel size than was anticipated. The phenomenon of panel-member dropout is well recognized, however, and is likely to leave a core panel of individuals most interested and motivated to reach a consensus result [32]. The degree of consensus with the Delphi technique, as with other consensus methods, has often been poorly defined [25,43,44]. Many studies have not specified consensus thresholds a priori, instead, specifying the degree of consensus only after having seen the data [43]. The optimal consensus threshold for use with the Delphi technique remains a subject for debate. The level of consensus should clearly be specified a priori, because this will decrease the likelihood that bias would lead to the inclusion or the exclusion of marginal choices. This consensus level may be arbitrary, because it was in this study, and will likely vary significantly between studies depending on the context, topic, and questionnaire structure. Regardless, this method
ARDS definition Delphi process should lend more transparency and generalizability to the process and reduce bias [43]. We conclude that it is feasible to consider utilizing formal consensus methods in the development of future definitions of ARDS and other clinical entities in the ICU. Our specific consensus process yielded an ARDS definition with important differences from the AECC definition; however, testing of this preliminary definition in terms of sensibility, reliability, and validity, with subsequent incorporation of feedback, is required.
Acknowledgments This study was supported in part by the Canadian Institutes of Health Research/Canadian Lung Association (Ottawa, Ontario, Canada) Post-Doctoral Fellowship (Dr Ferguson) and the Health Career Award from the Canadian Institutes of Health Research (Dr Davis). We thank the following individuals who participated as members of our expert panel during the Delphi process: Drs Antonio Artigas, Jean-Daniel Chiche, Gregory Downey, Margaret Herridge, Waldemar Johanson, Marin Kollef, James Lewis, Neil MacIntyre, Umberto Meduri, Antonio Pesenti, and Jean-Louis Teboul. Thanks also to Drs Charles Chan, Allan Detsky, Ted Marras, and Matthew Stanbrook for their helpful input during the design of the study and/or the preparation of the manuscript.
References [1] Rubenfeld GD. Epidemiology of acute lung injury. Crit Care Med 2003;31(Suppl):S276 - 84. [2] Ashbaugh DG, Bigelow DB, Petty TL, et al. Acute respiratory distress in adults. Lancet 1967;2:319 - 23. [3] Garber BG, Hebert PC, Yelle JD, et al. Adult respiratory distress syndrome: a systemic overview of incidence and risk factors. Crit Care Med 1996;24:687 - 95. [4] Hudson LD, Steinberg KP. Epidemiology of acute lung injury and ARDS. Chest 1999;116:74S - 82S. [5] Marinelli WA, Ingbar DH. Diagnosis and management of acute lung injury. Clin Chest Med 1994;15:517 - 46. [6] Schuster DP, Kollef MH. Acute respiratory distress syndrome. Dis Mon 1996;42:270 - 326. [7] Lesur O, Berthiaume Y, Blaise G, et al. Acute respiratory distress syndrome: 30 years later. Can Respir J 1999;6:71 - 86. [8] Matthay MA. Conference summary: acute lung injury. Chest 1999; 116:119S - 26S. [9] Ware LB, Matthay MA. The acute respiratory distress syndrome. N Engl J Med 2000;342:1334 - 49. [10] Bernard GR, Artigas A, Brigham KL, et al. The American-European Consensus Conference on ARDS. Definitions, mechanisms, relevant outcomes, and clinical trial coordination. Am J Respir Crit Care Med 1994;149:818 - 24. [11] Meade MO, Cook RJ, Guyatt GH, et al. Interobserver variation in interpreting chest radiographs for the diagnosis of acute respiratory distress syndrome. Am J Respir Crit Care Med 2000;161:85 - 90. [12] Rubenfeld GD, Caldwell E, Granton JT, et al. Interobserver variability in applying a radiographic definition for ARDS. Chest 1999;116: 1347 - 53.
153 [13] Esteban A, Ferna´ndez-Segoviano P, Frutos-Vivar F, et al. Comparison of clinical criteria for the acute respiratory distress syndrome with autopsy findings. Ann Intern Med 2004;141:440 - 5. [14] Abraham E. Toward new definitions of acute respiratory distress syndrome. Crit Care Med 1999;27:237 - 8. [15] Abraham E, Matthay MA, Dinarello CA, et al. Consensus conference definitions for sepsis, septic shock, acute lung injury, and acute respiratory distress syndrome: time for a reevaluation. Crit Care Med 2000;28:232 - 5. [16] Lewandowski K. Epidemiological data challenge ARDS/ALI definition. Intensive Care Med 1999;25:884 - 6. [17] Schuster DP. Identifying patients with ARDS: time for a different approach. Intensive Care Med 1997;23:1197 - 203. [18] Rotondi AJ, Kvetan V, Carlet J, et al. Consensus conferences in critical care medicine. Methodologies and impact. Crit Care Clinics 1997;13: 417 - 39. [19] Fink A, Kosecoff J, Chassin M, et al. Consensus methods: characteristics and guidelines for use. Am J Public Health 1984;74:979 - 83. [20] Anonymous. International Consensus Conference in Intensive Care Medicine: ventilator-associated lung injury in ARDS. Am J Respir Crit Care Med 1999;160:2118 - 24. [21] Anonymous. Pulmonary Artery Catheter Consensus Conference: consensus statement. Crit Care Med 1997;25:910 - 24. [22] Artigas A, Bernard GR, Carlet J, et al. The American-European Consensus Conference on ARDS, Part 2: Ventilatory, pharmacologic, supportive therapy, study design strategies, and issues related to recovery and remodeling. Acute respiratory distress syndrome. Am J Respir Crit Care Med 1998;157:1332 - 47. [23] Cuthbertson BH, Dellinger P, Dyar OJ, et al. UK guidelines for the use of inhaled nitric oxide therapy in adult ICUs. American-European Consensus Conference on ALI/ARDS. Intensive Care Med 1997;23: 1212 - 8. [24] Slutsky AS. Consensus conference on mechanical ventilation— January 28-30, 1993 at Northbrook. Illinois, USA. Part 2. Intensive Care Med 1994;20:150 - 62. [25] Crisp J, Pelletier D, Duffield C, et al. The Delphi method? Nurs Res 1997;46:116 - 8. [26] Goodman CM. The Delphi technique: a critique. J Adv Nurs 1987;12: 729 - 34. [27] Carlet J, Artigas A, Bihari D, et al. The first European consensus conference in intensive care medicine: introductory remarks. Intensive Care Med 1992;18:180 - 1. [28] Naglie G, Silberfeld M, O’Rourke K, et al. Convening expert panels to identify mental capacity assessment items. Can J Aging 1995;14: 697 - 705. [29] Buchbinder R, Goel V, Bombardier C, et al. Classification systems of soft tissue disorders of the neck and upper limb: do they satisfy methodological guidelines? J Clin Epidemiol 1996;49:141 - 9. [30] Feinstein AR. The theory and evaluation of sensibility, clinimetrics. New Haven7 Yale University Press; 1987. p. 141 - 66. [31] Grant JS, Kinney MR. Using the Delphi technique to examine the content validity of nursing diagnoses. Nurs Diagn 1992;3:12 - 22. [32] Rotondi A, Gustafson D. Theoretical, methodological and practical issues arising out of the Delphi method. In: Adler M, Ziglio E, editors. Gazing into the oracle: the Delphi method and its application to social policy and public health. London7 Jessica Kingsley Publishers; 1996. p. 34 - 55. [33] Villar J, Perez-Mendez L, Kacmarek RM. Current definitions of acute lung injury and the acute respiratory distress syndrome do not reflect their true severity and outcome. Intensive Care Med 1999;25: 930 - 5. [34] Neff MJ, Rubenfeld GD, Caldwell ES, et al. Exclusion of patients with elevated pulmonary capillary wedge pressure from acute respiratory distress syndrome. Am J Respir Crit Care Med 1999; 159:A716. [35] Ferguson ND, Meade MO, Hallett DC, et al. High values of the pulmonary artery wedge pressure in patients with acute lung injury and
154
[36]
[37]
[38]
[39]
N.D. Ferguson et al. acute respiratory distress syndrome. Intensive Care Med 2002;28: 1073 - 7. Moss M, Goodman PL, Heinig M, et al. Establishing the relative accuracy of three new definitions of the adult respiratory distress syndrome. Crit Care Med 1995;23:1629 - 37. Luhr OR, Antonsen K, Karlsson M, et al. Incidence and mortality after acute respiratory failure and acute respiratory distress syndrome in Sweden, Denmark, and Iceland. The ARF Study Group. Am J Respir Crit Care Med 1999;159:1849 - 61. Goh AY, Chan PW, Lum LC, et al. Incidence of acute respiratory distress syndrome: a comparison of two definitions. Arch Dis Child 1998;79:256 - 9. Knaus WA, Sun X, Hakim RB, et al. Evaluation of definitions for adult respiratory distress syndrome. Am J Respir Crit Care Med 1994; 150:311 - 7.
[40] Schuster DP. What is acute lung injury? What is ARDS? Chest 1995; 107:1721 - 6. [41] Krafft P, Fridrich P, Pernerstorfer T, et al. The acute respiratory distress syndrome: definitions, severity and clinical outcome. An analysis of 101 clinical investigations. Intensive Care Med 1996;22: 519 - 29. [42] Roupie E, Lepage E, Wysocki M, et al. Prevalence, etiologies and outcome of the acute respiratory distress syndrome among hypoxemic ventilated patients. SRLF Collaborative Group on Mechanical Ventilation. Societe de Reanimation de Langue Francaise. Intensive Care Med 1999;25:920 - 9. [43] Williams PL, Webb C. The Delphi technique: a methodological discussion. J Adv Nurs 1994;19:180 - 6. [44] Duffield C. The Delphi technique: a comparison of results obtained using two expert panels. Int J Nurs Stud 1993;30:227 - 37.