- Email: [email protected]
Developing tools for the assessment of the learning colonoscopist
EDITORIAL
Developing tools for the assessment of the learning colonoscopist We are experiencing significant changes in health care delivery, health care funding, and training of medical professionals. Profound change in physician training is occurring because of several forces including focus on quality, outcomes, and accountability. Training in internal medicine and its subspecialties has responded to these forces by moving away from apprentice model training and normative evaluations to competency-based outcomes training and assessment. The adoption of competency-based education began in the late 1990s through initiatives embraced by the Accreditation Council for Graduate Medical Education (ACGME). Although most program directors agree that competency assessment has been useful for most aspects of physician training, there has been little effect on how we assess procedural skills. Most procedural assessment in gastroenterology still relies heavily on the apprenticeship model. A final and summative evaluation of a fellow’s competency to perform a procedure often focuses on the number of procedures and the “gestalt” of the observing physicians. Limited tools have been developed to objectively assess the endoscopic learner. With the more recent stress on outcomes-based training, we are moving toward milestone-based competency assessment. The next accreditation system proposed by the ACGME promises more focus on milestone-based education.1 However, the proposed reporting milestones of internal medicine and its subspecialties are woefully inadequate to assist GI fellowship programs in assessing fellow procedural skills. A fundamental question is posed to every program director on the final summative fellow assessment. Is this fellow competent to independently perform endoscopy or colonoscopy? What we need as endoscopic trainers are useful instruments that assess all of the aspects involved in training future endoscopists and colonoscopists. Procedural learning not only includes the technical aspects of the procedure but involves critical cognitive, communicative, and integrative skills. Several tools have been developed and are in development that, it is hoped, will provide trainers with objective means for assessing the various aspects of procedural learning. In ancient times, leaders and teachers could travel to Delphi in Greece and inquire of the Oracle. The wisdom
and insight of Apollo was reportedly dispensed by the high priestess to the supplicants. Because we are lacking in reliable seers and psychics in modern times, we need to rely on our collective knowledge and wisdom to facilitate training of our fellows. Delphi methodology refers to the development of collective wisdom accrued after multiple rounds of refinement.2 In this issue of GIE, one such undertaking is presented. The authors use Delphi methodology to determine which aspects of colonoscopy are critical to proficiency.3 In this study, the authors achieved consensus among a large group of experts in endoscopy by using Delphi methodology. An anonymous panel was assembled from
A fundamental question is posed to every program director on the final summative fellow assessment. Is this fellow competent to independently perform endoscopy or colonoscopy?
Copyright ª 2014 by the American Society for Gastrointestinal Endoscopy 0016-5107/$36.00 http://dx.doi.org/10.1016/j.gie.2013.12.021
recognized experts in the field based on contributions to the literature and recognition by professional societies. The experts were gastroenterologists, surgeons who perform endoscopy, and endoscopy nurses. The final panel was 55 individuals. This group was given a comprehensive list of items that were deemed useful to assess colonoscopy skills. The original list was quite broad and was developed after an extensive review of the literature. The list included checklist items and global rating items. By using a system of surveys with a grading scale and a need for 80% consensus (established a priori), the authors developed a final consensus after 5 rounds of survey. A final gradation scheme was developed, which the authors term the Gastrointestinal Endoscopy Competency Assessment Tool (GiECAT). In addition to assessing the technical and cognitive aspects of procedure performance, the GiECAT also addresses the performance of pre- and postprocedural elements. After the 5 iterations, the authors developed 19 checklist items and 7 global items that reportedly reflect the collective wisdom of the expert panel. The GiECAT is a useful tool developed in a unique method. Interestingly, it is similar to other tools in the literature including the Mayo Colonoscopy Skills Assessment Tool and the UK Joint Advisory Groups colonoscopy
808 GASTROINTESTINAL ENDOSCOPY Volume 79, No. 5 : 2014
www.giejournal.org
Coyle
Editorial
tool.4,5 The similarity should not be surprising because some of the expert panel may have been involved in development of the other tools. What is unique about the GiECAT is the assessment of a “lapse” in professionalism in the global rating items. The number of global items is also greater in the GiECAT compared with the other tools. One significant item missing from the GiECAT tool is the outcome assessment of polyp detection. The adenoma detection rate has been a prominent indicator of quality in colonoscopy and has become a quality measure in colonoscopy along with the cecal intubation rate. Practicing endoscopists now have to measure this aspect of colonoscopy. Trainees should also be assessed for this quality metric. Because the adenoma detection rate requires postprocedure follow-up, often weeks after the procedure is performed, some have advocated the polyp detection rate at the time of colonoscopy as a reliable parallel of the adenoma detection rate.6 Inclusion of polyp detection in any outcome assessment tool seems reasonable. The GiECAT focuses on the colonic withdrawal time in the checklist items. The authors chose the widely promulgated minimum of 6 minutes for colonoscope removal. However, the more important outcome is mucosal visualization. Early learners of colonoscopy could easily take 15 minutes to withdraw the colonoscope, yet only actually inspect 50% of the mucosa. Also, recent literature supports that high-quality technique upon withdrawal is more important that an actual set withdrawal time.7 In the global item list, the authors do list mucosal visualization, but this outcome is critical and should have been a checklist item. Another critical aspect of any outcome assessment tool is its use over the entire continuum of the training of fellows. As more fellows use a prospective outcomes tool, we will be 1 step closer to true procedural milestones. At present, there is no consensus or large study that defines where a fellow should be after 50 or 150 procedures. Scant data exist as to what the average cecal intubation time should be for an endoscopist who has completed 75 colonoscopies. Additionally, the separation into year group is rather arbitrary for procedures because each fellow advances on his or her own learning curve. However, if we could aggregate outcomes data by using a tool such as the GiECAT on large numbers of fellows, then we could define the milestones along the learning curve. Imagine that a program director could perform 10 consecutive assessments on any-year fellow and find that fellow’s position on a national learning curve based on aggregated data. This would allow the program directors to quickly identify fellows who need remedial or specialized attention. Using a well-designed tool would also allow endoscopic trainers to better identify the specific skills that are lacking, such as loop reduction, torqueing, and retroflexion, among others. Before we embrace any tool, even one based on collective wisdom, it should go through a vigorous validation
1. Nasca TJ, Philibert I, Brigham T, et al. The next GME accreditation systemd rationale and benefits. N Engl J Med 2012;366:1051-6. 2. Jones J, Hunter D. Consensus methods for medical and health services research. BMJ 1995;311:376-80. 3. Walsh CM, Ling SC, Khanna N, et al. Gastrointestinal Endoscopy Competency Assessment Tool: development of a procedure-specific assessment tool for colonoscopy. Gastrointest Endosc 2014;79:798-807. 4. Sedlack RE. The Mayo Colonoscopy Skills Assessment Tool: validation of a unique instrument to assess colonoscopy skills in trainees. Gastrointest Endosc 2010;72:1125-33.
www.giejournal.org
Volume 79, No. 5 : 2014 GASTROINTESTINAL ENDOSCOPY 809
evaluation. The tool should be easy to use for trainers but be granular enough to develop specific assessments for the breadth of procedural skill. The tool needs to be useful in assessing the novice through the continuum to the experienced endoscopist. The assessment should be flexible enough to be used for formative evaluations throughout the year and final summative evaluation at the end of the year or training. Hopefully, the GiECAT is a step in this direction. What are the next steps in procedural assessment for gastroenterologists and surgical endoscopists? The next accreditation system being implemented by the ACGME has only 1 reporting milestone for invasive procedures, and that milestone is supposed to cover all GI procedures. Therefore, it will fall to the GI training societies to coordinate efforts to develop reliable and easily adopted outcome assessment tools like the GiECAT for program directors to use in procedural assessment. There is a definite fork in the road for those of us who train endoscopists. We can all continue to use various homegrown or adapted tools from previous publications at each training site or we can attempt to develop a universal tool that, if adopted by most programs, would facilitate meaningful milestone development throughout the continuum of endoscopic training. One would expect that if the Oracle were consulted or if Delphi methodology were used, we would develop 1 tool for each procedure and adopt it nationally. We will see whether those of us in gastroenterology have the collective wisdom to cooperate and develop useful tools for assessing the next generation of endoscopists. DISCLOSURE The author disclosed no financial relationships relevant to this publication. Walter J. Coyle, MD, FACG, FASGE Division of Gastroenterology/Hepatology Scripps Clinic Torrey Pines La Jolla, California, USA Abbreviations: ACGME, Accreditation Council for Graduate Medical Education; GiECAT, Gastrointestinal Endoscopy Competency Assessment Tool.
REFERENCES
Editorial
Coyle
5. Joint Advisory Group on GI Endoscopy (UK). Formative DOPS assessment form-Colonoscopy and flexible sigmoidoscopy [Internet]. Available at: http://www.thejag.org.uk/downloads/JAG% 20Certification%20for%20trainees/JETS%20User%20guides%20% 20and%20site%20information/JETS%20Introduction.pdf. Accessed November 2013.
6. Francis DL, Rodriguez-Correa DT, Buchner A, et al. Application of a conversion factor to estimate the adenoma detection rate from the polyp detection rate. Gastrointest Endosc 2011;73:493-7. 7. Lee RH, Tang RS, Muthusamy VR, et al. Quality of colonoscopy withdrawal technique and variability in adenoma detection rates. Gastrointest Endosc 2011;74:128-34.
Registration of Human Clinical Trials Gastrointestinal Endoscopy follows the International Committee of Medical Journal Editors (ICMJE)’s Uniform Requirements for Manuscripts Submitted to Biomedical Journals. All prospective human clinical trials eventually submitted in GIE must have been registered through one of the registries approved by the ICMJE, and proof of that registration must be submitted to GIE along with the article. For further details and explanation of which trials need to be registered as well as a list of ICMJE-acceptable registries, please go to http://www.icmje.org.
810 GASTROINTESTINAL ENDOSCOPY Volume 79, No. 5 : 2014
www.giejournal.org
Developing tools for the assessment of the learning colonoscopist We are experiencing significant changes in health care delivery, health care funding, and training of medical professionals. Profound change in physician training is occurring because of several forces including focus on quality, outcomes, and accountability. Training in internal medicine and its subspecialties has responded to these forces by moving away from apprentice model training and normative evaluations to competency-based outcomes training and assessment. The adoption of competency-based education began in the late 1990s through initiatives embraced by the Accreditation Council for Graduate Medical Education (ACGME). Although most program directors agree that competency assessment has been useful for most aspects of physician training, there has been little effect on how we assess procedural skills. Most procedural assessment in gastroenterology still relies heavily on the apprenticeship model. A final and summative evaluation of a fellow’s competency to perform a procedure often focuses on the number of procedures and the “gestalt” of the observing physicians. Limited tools have been developed to objectively assess the endoscopic learner. With the more recent stress on outcomes-based training, we are moving toward milestone-based competency assessment. The next accreditation system proposed by the ACGME promises more focus on milestone-based education.1 However, the proposed reporting milestones of internal medicine and its subspecialties are woefully inadequate to assist GI fellowship programs in assessing fellow procedural skills. A fundamental question is posed to every program director on the final summative fellow assessment. Is this fellow competent to independently perform endoscopy or colonoscopy? What we need as endoscopic trainers are useful instruments that assess all of the aspects involved in training future endoscopists and colonoscopists. Procedural learning not only includes the technical aspects of the procedure but involves critical cognitive, communicative, and integrative skills. Several tools have been developed and are in development that, it is hoped, will provide trainers with objective means for assessing the various aspects of procedural learning. In ancient times, leaders and teachers could travel to Delphi in Greece and inquire of the Oracle. The wisdom
and insight of Apollo was reportedly dispensed by the high priestess to the supplicants. Because we are lacking in reliable seers and psychics in modern times, we need to rely on our collective knowledge and wisdom to facilitate training of our fellows. Delphi methodology refers to the development of collective wisdom accrued after multiple rounds of refinement.2 In this issue of GIE, one such undertaking is presented. The authors use Delphi methodology to determine which aspects of colonoscopy are critical to proficiency.3 In this study, the authors achieved consensus among a large group of experts in endoscopy by using Delphi methodology. An anonymous panel was assembled from
A fundamental question is posed to every program director on the final summative fellow assessment. Is this fellow competent to independently perform endoscopy or colonoscopy?
Copyright ª 2014 by the American Society for Gastrointestinal Endoscopy 0016-5107/$36.00 http://dx.doi.org/10.1016/j.gie.2013.12.021
recognized experts in the field based on contributions to the literature and recognition by professional societies. The experts were gastroenterologists, surgeons who perform endoscopy, and endoscopy nurses. The final panel was 55 individuals. This group was given a comprehensive list of items that were deemed useful to assess colonoscopy skills. The original list was quite broad and was developed after an extensive review of the literature. The list included checklist items and global rating items. By using a system of surveys with a grading scale and a need for 80% consensus (established a priori), the authors developed a final consensus after 5 rounds of survey. A final gradation scheme was developed, which the authors term the Gastrointestinal Endoscopy Competency Assessment Tool (GiECAT). In addition to assessing the technical and cognitive aspects of procedure performance, the GiECAT also addresses the performance of pre- and postprocedural elements. After the 5 iterations, the authors developed 19 checklist items and 7 global items that reportedly reflect the collective wisdom of the expert panel. The GiECAT is a useful tool developed in a unique method. Interestingly, it is similar to other tools in the literature including the Mayo Colonoscopy Skills Assessment Tool and the UK Joint Advisory Groups colonoscopy
808 GASTROINTESTINAL ENDOSCOPY Volume 79, No. 5 : 2014
www.giejournal.org
Coyle
Editorial
tool.4,5 The similarity should not be surprising because some of the expert panel may have been involved in development of the other tools. What is unique about the GiECAT is the assessment of a “lapse” in professionalism in the global rating items. The number of global items is also greater in the GiECAT compared with the other tools. One significant item missing from the GiECAT tool is the outcome assessment of polyp detection. The adenoma detection rate has been a prominent indicator of quality in colonoscopy and has become a quality measure in colonoscopy along with the cecal intubation rate. Practicing endoscopists now have to measure this aspect of colonoscopy. Trainees should also be assessed for this quality metric. Because the adenoma detection rate requires postprocedure follow-up, often weeks after the procedure is performed, some have advocated the polyp detection rate at the time of colonoscopy as a reliable parallel of the adenoma detection rate.6 Inclusion of polyp detection in any outcome assessment tool seems reasonable. The GiECAT focuses on the colonic withdrawal time in the checklist items. The authors chose the widely promulgated minimum of 6 minutes for colonoscope removal. However, the more important outcome is mucosal visualization. Early learners of colonoscopy could easily take 15 minutes to withdraw the colonoscope, yet only actually inspect 50% of the mucosa. Also, recent literature supports that high-quality technique upon withdrawal is more important that an actual set withdrawal time.7 In the global item list, the authors do list mucosal visualization, but this outcome is critical and should have been a checklist item. Another critical aspect of any outcome assessment tool is its use over the entire continuum of the training of fellows. As more fellows use a prospective outcomes tool, we will be 1 step closer to true procedural milestones. At present, there is no consensus or large study that defines where a fellow should be after 50 or 150 procedures. Scant data exist as to what the average cecal intubation time should be for an endoscopist who has completed 75 colonoscopies. Additionally, the separation into year group is rather arbitrary for procedures because each fellow advances on his or her own learning curve. However, if we could aggregate outcomes data by using a tool such as the GiECAT on large numbers of fellows, then we could define the milestones along the learning curve. Imagine that a program director could perform 10 consecutive assessments on any-year fellow and find that fellow’s position on a national learning curve based on aggregated data. This would allow the program directors to quickly identify fellows who need remedial or specialized attention. Using a well-designed tool would also allow endoscopic trainers to better identify the specific skills that are lacking, such as loop reduction, torqueing, and retroflexion, among others. Before we embrace any tool, even one based on collective wisdom, it should go through a vigorous validation
1. Nasca TJ, Philibert I, Brigham T, et al. The next GME accreditation systemd rationale and benefits. N Engl J Med 2012;366:1051-6. 2. Jones J, Hunter D. Consensus methods for medical and health services research. BMJ 1995;311:376-80. 3. Walsh CM, Ling SC, Khanna N, et al. Gastrointestinal Endoscopy Competency Assessment Tool: development of a procedure-specific assessment tool for colonoscopy. Gastrointest Endosc 2014;79:798-807. 4. Sedlack RE. The Mayo Colonoscopy Skills Assessment Tool: validation of a unique instrument to assess colonoscopy skills in trainees. Gastrointest Endosc 2010;72:1125-33.
www.giejournal.org
Volume 79, No. 5 : 2014 GASTROINTESTINAL ENDOSCOPY 809
evaluation. The tool should be easy to use for trainers but be granular enough to develop specific assessments for the breadth of procedural skill. The tool needs to be useful in assessing the novice through the continuum to the experienced endoscopist. The assessment should be flexible enough to be used for formative evaluations throughout the year and final summative evaluation at the end of the year or training. Hopefully, the GiECAT is a step in this direction. What are the next steps in procedural assessment for gastroenterologists and surgical endoscopists? The next accreditation system being implemented by the ACGME has only 1 reporting milestone for invasive procedures, and that milestone is supposed to cover all GI procedures. Therefore, it will fall to the GI training societies to coordinate efforts to develop reliable and easily adopted outcome assessment tools like the GiECAT for program directors to use in procedural assessment. There is a definite fork in the road for those of us who train endoscopists. We can all continue to use various homegrown or adapted tools from previous publications at each training site or we can attempt to develop a universal tool that, if adopted by most programs, would facilitate meaningful milestone development throughout the continuum of endoscopic training. One would expect that if the Oracle were consulted or if Delphi methodology were used, we would develop 1 tool for each procedure and adopt it nationally. We will see whether those of us in gastroenterology have the collective wisdom to cooperate and develop useful tools for assessing the next generation of endoscopists. DISCLOSURE The author disclosed no financial relationships relevant to this publication. Walter J. Coyle, MD, FACG, FASGE Division of Gastroenterology/Hepatology Scripps Clinic Torrey Pines La Jolla, California, USA Abbreviations: ACGME, Accreditation Council for Graduate Medical Education; GiECAT, Gastrointestinal Endoscopy Competency Assessment Tool.
REFERENCES
Editorial
Coyle
5. Joint Advisory Group on GI Endoscopy (UK). Formative DOPS assessment form-Colonoscopy and flexible sigmoidoscopy [Internet]. Available at: http://www.thejag.org.uk/downloads/JAG% 20Certification%20for%20trainees/JETS%20User%20guides%20% 20and%20site%20information/JETS%20Introduction.pdf. Accessed November 2013.
6. Francis DL, Rodriguez-Correa DT, Buchner A, et al. Application of a conversion factor to estimate the adenoma detection rate from the polyp detection rate. Gastrointest Endosc 2011;73:493-7. 7. Lee RH, Tang RS, Muthusamy VR, et al. Quality of colonoscopy withdrawal technique and variability in adenoma detection rates. Gastrointest Endosc 2011;74:128-34.
Registration of Human Clinical Trials Gastrointestinal Endoscopy follows the International Committee of Medical Journal Editors (ICMJE)’s Uniform Requirements for Manuscripts Submitted to Biomedical Journals. All prospective human clinical trials eventually submitted in GIE must have been registered through one of the registries approved by the ICMJE, and proof of that registration must be submitted to GIE along with the article. For further details and explanation of which trials need to be registered as well as a list of ICMJE-acceptable registries, please go to http://www.icmje.org.
810 GASTROINTESTINAL ENDOSCOPY Volume 79, No. 5 : 2014
www.giejournal.org