- Email: [email protected]
Expert Opinion Is Necessary: Delphi Panel Methodology Facilitates a Scientific Approach to Consensus
Editorial
Expert Opinion Is Necessary: Delphi Panel Methodology Facilitates a Scientific Approach to Consensus
Abstract: Our current trend and focus on evidence-based medicine is biased in favor of randomized controlled trials, which are ranked highest in the hierarchy of evidence while devaluing expert opinion, which is ranked lowest in the hierarchy. However, randomized controlled trials have weaknesses as well as strengths, and no research method is flawless. Moreover, stringent application of scientific research techniques, such as the Delphi Panel methodology, allows survey of experts in a high quality and scientific manner. Level V evidence (expert opinion) remains a necessary component in the armamentarium used to determine the answer to a clinical question.
I
n ancient Greece, oracles were experts who were consulted to advise on important decisions, and in modern medicine, we still seek the opinions of experts when we require guidance. Yet in the hierarchy of evidence-based medicine, expert opinion is known as Level V. And, by the way, Level V of V means the lowest level of evidence.1 Being at the bottom could feel disappointing for putative expertsdand reputed oraclesdlet alone we who seek their expertise. In contrast, at the top of the medical level of evidence hierarchy is the Level I evidence randomized controlled trial (RCT).1 Recently, however, the primacy of randomized trials has been reconsidered because of the weaknesses of RCTs,2 including the following: Potential for validity to be limited to the study population, with limited generalizability to actual practice Potential for outcome measures to not correlate with actual outcomes of interest Resource-intensive with regard to costs Risk of inadequate sample size Resource-intensive with regard to time: completion may not occur until after the introduction of new treatment methods, so that trials are not studying what is used in actual clinical practice Impractical for urgent situations or rare diseases.2 In truth, “No study design is flawless, and conflicting findings can emerge from all types of studies.”2 That said, practical alternatives to RCTs range from modern observational studies in an age of large health registries containing “big data” (to be further Ó 2017 by the Arthroscopy Association of North America 0749-8063/171461/$36.00 https://doi.org/10.1016/j.arthro.2017.11.022
considered in an upcoming editorial) to ancient methods including this month’s focus, iteration of expert opinion. We are learning that properly conducted surveys do have merit and can be critical in guiding proper acquisition of data.3 Properly designed surveys very much depend on the quality of the data collected.4,5 With any scientific researchdand this may come as no surprisedproper and rigorously detailed research methods determine the merit of a study, and studies based on expert opinion are no exception.6-9 It follows that properly designed research surveys may be a valuable means of determining the answers to clinical questions, regardless of a hierarchical and potentially demeaning numerical level of evidence assignment that fails to consider the diversity of processes performed to reach an expert opinion. Obviously, reliance on an oracle has fallen out of favor. As a result, medical journals, including ours, have only uncommonly published “survey” data.10-12 This reflects our 21st-century evidence-based medicine bias,1,13-16 and Level V evidence is treated with some disrepute. However, and we credit Angelo et al.17 for seizing our attention, we have been educated to the fact that it is possible to survey a panel of experts in a scientific manner. Today, we are aware of 2 reasonably well accepted methods available for attaining expert consensus: the Nominal Group Technique (NGT) and the Delphi method.18 The NGT is based on structured, face-to-face meetings to facilitate discussions.19,20 The limitations of the NGT20 include a requirement for participants to personally attend a meeting that may prove difficult to organize, or attendance could be limited by geography. Worse, a potential limitation of
Arthroscopy: The Journal of Arthroscopic and Related Surgery, Vol 34, No 2 (February), 2018: pp 349-351
349
350
EDITORIAL
Table 1. Summary of the Delphi Panel Method 1. Researchers define a problem and develop related questions. 2. Researchers select a panel of diverse experts (whose anonymity is generally protected). 3. Researchers distribute the questionnaire to the panel. 4. Researchers analyze and summarize the data and develop follow-up questions. 5. Repeat step 4 as required. 6. As consensus begins forming and issues are clarified, repeat step 4 as required. 7. Researchers invite panelists to revise or review consensus and specify reasons for dissenting opinion. 8. Repeat steps 4-7 as required.* 9. Researchers summarize consensus and provide feedback to the panel. 10. Researchers publish the final consensus statement. *Three to 4 iterations are sufficient to collect the needed information and to reach a consensus in most cases.28
the NGT is that more strongly opinionated or vocal participants can dominate the discussion, imposing their views. Also, during face-to-face meetings, peer pressure may result in conformity rather than potentially valuable scholarly dissent. As a result of the limitations of the NGT, we are more enamored with the Delphi technique, which, in name, bears homage to the sanctuary of the most famous ancient oracle. The Delphi technique was recently introduced to Arthroscopy editors by Angelo et al.,17 as above. The Delphi Panel methodology, a scientific method for achieved expert consensus, represents a structured process used to collect knowledge by defining a problem, developing questions for experts to resolve, selecting a panel of experts including academics and clinicians, employing open-ended questionnaires, performing controlled assessment and feedback including qualitative and quantitative analysis, and follow-up (reassessment) using a series of surveys until an accord is established and summarized.21,22 The advantages of the Delphi method include anonymity18,23 and the capability of achieving consensus when uncertainty may exist due to the inevitable lack of complete and definitive evidence.24,25 Again, no methods are flawless, and the Delphi Panel methodology has been criticized as representing a lowest common denominator,26 which could lead to oversimplification.24 Yet, criticism of the Delphi method may result from failure of investigators to adequately execute the methods, rather than the methodology itself.27 In the opinion of your editors, determination of the collective experience of experts has merit as a tool to improve the quality of treatment for our patients, and we encourage Arthroscopy authors to familiarize themselves with Delphi techniques and to attempt to be above reproach when performing research using Delphi methods. A concise summary of the Delphi Panel method may be found in Table 1. However, authors
who aim to employ this method in their future research should not rely on this summary but should study and incorporate the Delphi Panel methodology in detail. It has been said of clinicians that, at the end of the day, it is not the evidence, it is how we use it,29 and “good doctors must use both individual clinical expertise and the best available external evidence, and neither alone is enough.”30 Consequently, Arthroscopy editors will continue to judiciously solicit and publish Level V evidence and hold our experts in high esteem. Erik Hohmann, M.D., Ph.D., F.R.C.S. Associate Editor Jefferson C. Brand, M.D. Assistant Editor-in-Chief Michael J. Rossi, M.D., M.S. Assistant Editor-in-Chief James H. Lubowitz, M.D. Editor-in-Chief
References 1. Atkins D, Best D, Briss PA, et al. Grading quality of evidence and strength of recommendations. BMJ 2004;328: 1490. 2. Frieden TR. Evidence for health decision makingdBeyond randomized, controlled trials. N Engl J Med 2017;377:465-475. 3. Rossi MJ, Brand JC, Provencher MT, Lubowitz JH. The expectation game: Patient comprehension is a determinant of outcome. Arthroscopy 2015;31:2283-2284. 4. Ekhtiari S, Kay J, de Sa D, et al. What makes a successful survey? A systematic review of surveys used in anterior cruciate ligament reconstruction. Arthroscopy 2017;33: 1072-1079. 5. Yoshiya S, Rossi MJ. Editorial commentary: Still miles away from designing high-quality surveys with high response rates. Arthroscopy 2017;33:1080-1081. 6. Brand JC, Rossi MJ, Lubowitz JH. Reliable references, accountable authors, and steadfast reviewers mitigate against retracted publications and scientific misconduct. Arthroscopy 2017;33:1435-1436. 7. Dhawan A, Brand JC, Provencher MT, Rossi MJ, Lubowitz JH. Research pearls: The significance of statistics and perils of pooling. Arthroscopy 2017;33:1099-1101. 8. Lubowitz JH, Brand JC, Provencher MT, Rossi MJ. Put systems in place to better measure surgical outcomes. Arthroscopy 2017;33:1271-1272. 9. Lubowitz JH, D’Agostino RB Jr, Provencher MT, Rossi MJ, Brand JC. Can we trust knee meniscus studies? One-way crossover confounds intent-to-treat statistical methods. Arthroscopy 2016;32:2187-2190. 10. Mancuso CA, Wentzel CH, Ghomrawi HMK, Kelly BT. Hip preservation surgery expectations survey: A new method to measure patients’ preoperative expectations. Arthroscopy 2017;33:959-968. 11. Khan M, Ayeni OR, Madden K, et al. Femoroacetabular impingement: Have we hit a global tipping point in diagnosis and treatment? Results from the international
EDITORIAL
12.
13.
14.
15.
16.
17.
18. 19.
femoroacetabular impingement optimal care update survey (IN FOCUS). Arthroscopy 2016;32:779-787.e4. Ponzio DY, VanBeek C, Wong JC, et al. Profile of current opinion on arthroscopic acromioplasty: A video survey study. Arthroscopy 2016;32:1253-1262. Bhandari M, Zlowodzki M, Cole PA. From eminencebased practice to evidence-based practice: A paradigm shift. Minn Med 2004;87:51-54. Schräder P, Scharf HP, Günther KP, Puhl W, Porzsolt F. Evidence-based medicine in orthopaedicsdA sensible or unnecessary addition to clinical routine? Part 1: The diagnostics tool. Z Orthop Ihre Grenzgeb 2003;141: 386-390. Schräder P, Scharf HP, Günther KP, Puhl W, Porzsolt F. Evidence-based medicine in orthopaedicsdA sensible or unnecessary addition to clinical routine? Part 2: The therapy tool. Z Orthop Ihre Grenzgeb 2003;141: 391-394. Krahn J, Sauerland S, Rixen D, Gregor S, Bouillon B, Neugebauer EA. Applying evidence-based surgery in daily clinical routine: A feasibility study. Arch Orthop Trauma Surg 2006;126:88-92. Angelo RL, Ryu RK, Pedowitz RA, Gallagher AG. Metric development for an arthroscopic Bankart procedure: Assessment of face and content validity. Arthroscopy 2015;31:1430-1440. Jones J, Hunter D. Consensus methods for medical and health services research. BMJ 1995;311:376-380. McMillan SS, Kelly F, Sav A, et al. Using the nominal group technique: How to analyse across multiple groups. Health Serv Outcomes Res Method 2014;14:92-108.
351
20. McMillan SS, King M, Tully MP. How to use the nominal group and Delphi techniques. Int J Clin Pharm 2016;38: 655-662. 21. Sandrey MA, Bulger SM. The Delphi method. An approach for facilitating evidence based practice in athletic training. Athl Train Edu J 2008;3:135-142. 22. McKenna HP. The Delphi technique: A worthwhile research approach for nursing? J Adv Nurs 1994;19: 1221-1225. 23. Habibi A, Sarafrazi A, Izadyar S. Delphi technique theoretical framework in qualitative research. Int J Eng Sci 2014;3:8-13. 24. Powell C. The Delphi techniques: Myth and realities. J Adv Nurs 2003;41:376-382. 25. Keeney S, Hasson F, McKenna H. Consulting the oracle: Ten lessons learned from using the Delphi technique in nursing research. J Adv Nurs 2006;53:205-212. 26. Rennie D. Consensus statements. N Engl J Med 1981;304: 665-666. 27. Boulkedid R, Abdoul H, Lousta M, Sibony O, Alberti C. Using and reporting the Delphi method for selecting healthcare quality indicators: A systematic review. PLoS One 2011;6:e20476. 28. Hsu CC, Sandford BA. The Delphi technique: Making sense of consensus. Pract Assess Res Eval 2007;12:1-8. 29. Stiller K. It’s not the evidence, it’s the way you use it: Is clinical practice being tyrannised by evidence? Austral Health Rev 2008;32:204-207. 30. Sackett DL, Rosenberg WM, Gray JA, Haynes RB, Richardson WS. Evidence based medicine: What is it and what it isn’t. BMJ 1996;312:71-72.
Expert Opinion Is Necessary: Delphi Panel Methodology Facilitates a Scientific Approach to Consensus
Abstract: Our current trend and focus on evidence-based medicine is biased in favor of randomized controlled trials, which are ranked highest in the hierarchy of evidence while devaluing expert opinion, which is ranked lowest in the hierarchy. However, randomized controlled trials have weaknesses as well as strengths, and no research method is flawless. Moreover, stringent application of scientific research techniques, such as the Delphi Panel methodology, allows survey of experts in a high quality and scientific manner. Level V evidence (expert opinion) remains a necessary component in the armamentarium used to determine the answer to a clinical question.
I
n ancient Greece, oracles were experts who were consulted to advise on important decisions, and in modern medicine, we still seek the opinions of experts when we require guidance. Yet in the hierarchy of evidence-based medicine, expert opinion is known as Level V. And, by the way, Level V of V means the lowest level of evidence.1 Being at the bottom could feel disappointing for putative expertsdand reputed oraclesdlet alone we who seek their expertise. In contrast, at the top of the medical level of evidence hierarchy is the Level I evidence randomized controlled trial (RCT).1 Recently, however, the primacy of randomized trials has been reconsidered because of the weaknesses of RCTs,2 including the following: Potential for validity to be limited to the study population, with limited generalizability to actual practice Potential for outcome measures to not correlate with actual outcomes of interest Resource-intensive with regard to costs Risk of inadequate sample size Resource-intensive with regard to time: completion may not occur until after the introduction of new treatment methods, so that trials are not studying what is used in actual clinical practice Impractical for urgent situations or rare diseases.2 In truth, “No study design is flawless, and conflicting findings can emerge from all types of studies.”2 That said, practical alternatives to RCTs range from modern observational studies in an age of large health registries containing “big data” (to be further Ó 2017 by the Arthroscopy Association of North America 0749-8063/171461/$36.00 https://doi.org/10.1016/j.arthro.2017.11.022
considered in an upcoming editorial) to ancient methods including this month’s focus, iteration of expert opinion. We are learning that properly conducted surveys do have merit and can be critical in guiding proper acquisition of data.3 Properly designed surveys very much depend on the quality of the data collected.4,5 With any scientific researchdand this may come as no surprisedproper and rigorously detailed research methods determine the merit of a study, and studies based on expert opinion are no exception.6-9 It follows that properly designed research surveys may be a valuable means of determining the answers to clinical questions, regardless of a hierarchical and potentially demeaning numerical level of evidence assignment that fails to consider the diversity of processes performed to reach an expert opinion. Obviously, reliance on an oracle has fallen out of favor. As a result, medical journals, including ours, have only uncommonly published “survey” data.10-12 This reflects our 21st-century evidence-based medicine bias,1,13-16 and Level V evidence is treated with some disrepute. However, and we credit Angelo et al.17 for seizing our attention, we have been educated to the fact that it is possible to survey a panel of experts in a scientific manner. Today, we are aware of 2 reasonably well accepted methods available for attaining expert consensus: the Nominal Group Technique (NGT) and the Delphi method.18 The NGT is based on structured, face-to-face meetings to facilitate discussions.19,20 The limitations of the NGT20 include a requirement for participants to personally attend a meeting that may prove difficult to organize, or attendance could be limited by geography. Worse, a potential limitation of
Arthroscopy: The Journal of Arthroscopic and Related Surgery, Vol 34, No 2 (February), 2018: pp 349-351
349
350
EDITORIAL
Table 1. Summary of the Delphi Panel Method 1. Researchers define a problem and develop related questions. 2. Researchers select a panel of diverse experts (whose anonymity is generally protected). 3. Researchers distribute the questionnaire to the panel. 4. Researchers analyze and summarize the data and develop follow-up questions. 5. Repeat step 4 as required. 6. As consensus begins forming and issues are clarified, repeat step 4 as required. 7. Researchers invite panelists to revise or review consensus and specify reasons for dissenting opinion. 8. Repeat steps 4-7 as required.* 9. Researchers summarize consensus and provide feedback to the panel. 10. Researchers publish the final consensus statement. *Three to 4 iterations are sufficient to collect the needed information and to reach a consensus in most cases.28
the NGT is that more strongly opinionated or vocal participants can dominate the discussion, imposing their views. Also, during face-to-face meetings, peer pressure may result in conformity rather than potentially valuable scholarly dissent. As a result of the limitations of the NGT, we are more enamored with the Delphi technique, which, in name, bears homage to the sanctuary of the most famous ancient oracle. The Delphi technique was recently introduced to Arthroscopy editors by Angelo et al.,17 as above. The Delphi Panel methodology, a scientific method for achieved expert consensus, represents a structured process used to collect knowledge by defining a problem, developing questions for experts to resolve, selecting a panel of experts including academics and clinicians, employing open-ended questionnaires, performing controlled assessment and feedback including qualitative and quantitative analysis, and follow-up (reassessment) using a series of surveys until an accord is established and summarized.21,22 The advantages of the Delphi method include anonymity18,23 and the capability of achieving consensus when uncertainty may exist due to the inevitable lack of complete and definitive evidence.24,25 Again, no methods are flawless, and the Delphi Panel methodology has been criticized as representing a lowest common denominator,26 which could lead to oversimplification.24 Yet, criticism of the Delphi method may result from failure of investigators to adequately execute the methods, rather than the methodology itself.27 In the opinion of your editors, determination of the collective experience of experts has merit as a tool to improve the quality of treatment for our patients, and we encourage Arthroscopy authors to familiarize themselves with Delphi techniques and to attempt to be above reproach when performing research using Delphi methods. A concise summary of the Delphi Panel method may be found in Table 1. However, authors
who aim to employ this method in their future research should not rely on this summary but should study and incorporate the Delphi Panel methodology in detail. It has been said of clinicians that, at the end of the day, it is not the evidence, it is how we use it,29 and “good doctors must use both individual clinical expertise and the best available external evidence, and neither alone is enough.”30 Consequently, Arthroscopy editors will continue to judiciously solicit and publish Level V evidence and hold our experts in high esteem. Erik Hohmann, M.D., Ph.D., F.R.C.S. Associate Editor Jefferson C. Brand, M.D. Assistant Editor-in-Chief Michael J. Rossi, M.D., M.S. Assistant Editor-in-Chief James H. Lubowitz, M.D. Editor-in-Chief
References 1. Atkins D, Best D, Briss PA, et al. Grading quality of evidence and strength of recommendations. BMJ 2004;328: 1490. 2. Frieden TR. Evidence for health decision makingdBeyond randomized, controlled trials. N Engl J Med 2017;377:465-475. 3. Rossi MJ, Brand JC, Provencher MT, Lubowitz JH. The expectation game: Patient comprehension is a determinant of outcome. Arthroscopy 2015;31:2283-2284. 4. Ekhtiari S, Kay J, de Sa D, et al. What makes a successful survey? A systematic review of surveys used in anterior cruciate ligament reconstruction. Arthroscopy 2017;33: 1072-1079. 5. Yoshiya S, Rossi MJ. Editorial commentary: Still miles away from designing high-quality surveys with high response rates. Arthroscopy 2017;33:1080-1081. 6. Brand JC, Rossi MJ, Lubowitz JH. Reliable references, accountable authors, and steadfast reviewers mitigate against retracted publications and scientific misconduct. Arthroscopy 2017;33:1435-1436. 7. Dhawan A, Brand JC, Provencher MT, Rossi MJ, Lubowitz JH. Research pearls: The significance of statistics and perils of pooling. Arthroscopy 2017;33:1099-1101. 8. Lubowitz JH, Brand JC, Provencher MT, Rossi MJ. Put systems in place to better measure surgical outcomes. Arthroscopy 2017;33:1271-1272. 9. Lubowitz JH, D’Agostino RB Jr, Provencher MT, Rossi MJ, Brand JC. Can we trust knee meniscus studies? One-way crossover confounds intent-to-treat statistical methods. Arthroscopy 2016;32:2187-2190. 10. Mancuso CA, Wentzel CH, Ghomrawi HMK, Kelly BT. Hip preservation surgery expectations survey: A new method to measure patients’ preoperative expectations. Arthroscopy 2017;33:959-968. 11. Khan M, Ayeni OR, Madden K, et al. Femoroacetabular impingement: Have we hit a global tipping point in diagnosis and treatment? Results from the international
EDITORIAL
12.
13.
14.
15.
16.
17.
18. 19.
femoroacetabular impingement optimal care update survey (IN FOCUS). Arthroscopy 2016;32:779-787.e4. Ponzio DY, VanBeek C, Wong JC, et al. Profile of current opinion on arthroscopic acromioplasty: A video survey study. Arthroscopy 2016;32:1253-1262. Bhandari M, Zlowodzki M, Cole PA. From eminencebased practice to evidence-based practice: A paradigm shift. Minn Med 2004;87:51-54. Schräder P, Scharf HP, Günther KP, Puhl W, Porzsolt F. Evidence-based medicine in orthopaedicsdA sensible or unnecessary addition to clinical routine? Part 1: The diagnostics tool. Z Orthop Ihre Grenzgeb 2003;141: 386-390. Schräder P, Scharf HP, Günther KP, Puhl W, Porzsolt F. Evidence-based medicine in orthopaedicsdA sensible or unnecessary addition to clinical routine? Part 2: The therapy tool. Z Orthop Ihre Grenzgeb 2003;141: 391-394. Krahn J, Sauerland S, Rixen D, Gregor S, Bouillon B, Neugebauer EA. Applying evidence-based surgery in daily clinical routine: A feasibility study. Arch Orthop Trauma Surg 2006;126:88-92. Angelo RL, Ryu RK, Pedowitz RA, Gallagher AG. Metric development for an arthroscopic Bankart procedure: Assessment of face and content validity. Arthroscopy 2015;31:1430-1440. Jones J, Hunter D. Consensus methods for medical and health services research. BMJ 1995;311:376-380. McMillan SS, Kelly F, Sav A, et al. Using the nominal group technique: How to analyse across multiple groups. Health Serv Outcomes Res Method 2014;14:92-108.
351
20. McMillan SS, King M, Tully MP. How to use the nominal group and Delphi techniques. Int J Clin Pharm 2016;38: 655-662. 21. Sandrey MA, Bulger SM. The Delphi method. An approach for facilitating evidence based practice in athletic training. Athl Train Edu J 2008;3:135-142. 22. McKenna HP. The Delphi technique: A worthwhile research approach for nursing? J Adv Nurs 1994;19: 1221-1225. 23. Habibi A, Sarafrazi A, Izadyar S. Delphi technique theoretical framework in qualitative research. Int J Eng Sci 2014;3:8-13. 24. Powell C. The Delphi techniques: Myth and realities. J Adv Nurs 2003;41:376-382. 25. Keeney S, Hasson F, McKenna H. Consulting the oracle: Ten lessons learned from using the Delphi technique in nursing research. J Adv Nurs 2006;53:205-212. 26. Rennie D. Consensus statements. N Engl J Med 1981;304: 665-666. 27. Boulkedid R, Abdoul H, Lousta M, Sibony O, Alberti C. Using and reporting the Delphi method for selecting healthcare quality indicators: A systematic review. PLoS One 2011;6:e20476. 28. Hsu CC, Sandford BA. The Delphi technique: Making sense of consensus. Pract Assess Res Eval 2007;12:1-8. 29. Stiller K. It’s not the evidence, it’s the way you use it: Is clinical practice being tyrannised by evidence? Austral Health Rev 2008;32:204-207. 30. Sackett DL, Rosenberg WM, Gray JA, Haynes RB, Richardson WS. Evidence based medicine: What is it and what it isn’t. BMJ 1996;312:71-72.