- Email: [email protected]
Procedural performance in gastrointestinal endoscopy: An assessment and self-appraisal tool
The American Journal of Surgery (2008) 196, 450 – 455
Surgical Education
Procedural performance in gastrointestinal endoscopy: An assessment and self-appraisal tool Sudip K. Sarker, M.B.Ch.B., F.R.C.S., Ph.D.a,*, Tark Albrani, M.B.B.S., M.R.C.S., M.Sc.b, Atiquaz Zaman, M.B.B.S., M.R.C.S., M.Sc.b, Bijen Patel, M.B.B.S., F.R.C.S.b a
University Department of Academic Surgery, Royal Free Hospital, London, United Kingdom; bAcademic Department of Upper Gastrointestinal Surgery, The Barts & Royal London Hospitals, London, United Kingdom KEYWORDS: Assessment; Endoscopy; Gastroenterology; Technical skills
Abstract AIMS: Assessing endoscopic technical skills competency in a structured manner is a topical issue, in light of several workforce factors that may affect the training of future endoscopists. To date there has been little attempt to comprehensively assess both generic and specific technical skills in lower gastrointestinal endoscopies; the current study aimed to develop and validate a tool that can assess these varied skills. METHODS: Hierarchical task analyses of generic and specific technical skills were constructed on flexible sigmoidoscopy and colonoscopy after expert panel discussions. Generic technical skills are subtasks (eg, movements of the control wheel) that allow the endoscopist to complete a main task (specific technical skills), for example, reaching the cecum. Weighted Likert scales were then constructed individually for generic and specific technical skills for each procedure. Two observers assessed each procedure independently. RESULTS: A total of 135 endoscopic procedures were performed by 9 consultants and 12 registrars. Mean inter-rater reliability Cronbach alphas were .83 and .80, P ⱕ.05, for generic and specific skills, respectively, for each procedure. Construct validity results using analysis of variance (ANOVA) for consultants and trainees were significant for each procedure, P ⫽ .005, P ⫽ .003 for generic, and P ⫽ .012, P ⫽ .004 for specific technical skills. CONCLUSIONS: This new assessment/self-appraisal tool for lower gastrointestinal endoscopies seems to have face, content, concurrent, and construct validities. The tool has the possibility of being used in training and self-appraisal. We aim to modify and apply this tool to other endoscopic procedures in the future, such as endoscopic retrograde cholangiopancreatography (ERCP), endoluminal and transluminal procedures. © 2008 Elsevier Inc. All rights reserved.
The working and training environment is entering a new era in the United Kingdom. Reduction in working hours for doctors in training1 will have a dramatic effect, with much * Corresponding author. Tel.: ⫹0207 565 8155; fax: ⫹0207 598 0285 E-mail address: [email protected] Manuscript received July 30, 2007; revised manuscript October 11, 2007
0002-9610/$ - see front matter © 2008 Elsevier Inc. All rights reserved. doi:10.1016/j.amjsurg.2007.10.024
less exposure to high volumes of operative/procedural work than previously, and the total time for training will be drastically reduced. It is therefore increasingly important that opportunities to acquire technical skills are used as effectively as possible. Healthcare is now a consultant/attending lead, which increases the safety of patient care by increasing the level of supervision of medical trainees. However, this welcome
S.K. Sarker et al.
Performance in gastrointestinal endoscopy
change reduces junior trainees to acquiring and performing technical tasks independently, which are important steps in gaining confidence in acquiring the technical skills needed to perform an operation or procedure. Evaluation of technical skills is notoriously difficult due to its subjectivity and the need for time-consuming expert analysis. No ongoing evaluation scheme exists to assess the continuing competency of endoscopists throughout their careers. The use of the endoscopists themselves to evaluate their own technique, employing objective analysis techniques, would circumvent the need for constant external analysis. In the last few years, significant developments have been made in the objective assessment of technical skills and a range of methods have been developed. The assessment of technical skills during training has been considered to be a shape of quality assurance for the future.2 The assessment of technical ability is the job of the trainers and their assessment is mainly subjective.3 Objective assessment is necessary because deficiencies in training and performance are hard to correct without objective feedback.4 Significant progress has been made to advance the methods that objectively evaluate surgical technique.5 Objective and unbiased data on procedural performance are essential during the prospective assessment in training.6,7 Checklists and global scores are some of the objective methods of assessing technical skills in endoscopic procedures. It has been said that checklists make examiners into observers, instead of interpreters, of behavior, that lead to removing the subjectivity of the evaluation process.8 Global rating scales can consist of several components of operative or procedural skills that are marked on a 5-point Likert scale, with the middle and the extreme points anchored by explicit descriptors to help in the criterion referenced assessment of performance. Generic and procedure-specific technical skills are different skills that are acquired at different times during endoscopic training. By the time the trainee reaches consultant/attending status, he or she should have mastered these aspects of technical skills in endoscopy. Generic technical skills are mainly acquired during basic endoscopic training, for example, handling of scope, handling of controls, handeye and monitor coordination, reaction to patient discom-
Table 1
451 fort, and flow of procedure and force used. These basic skills are required by the endoscopist to perform a complete procedure. They alone do not form a task or sub-task but allow the endoscopist to perform the task. Specific technical skills that are learned throughout basic and higher endoscopic training include luminal vision, therapeutic procedure, strategy for progression, and identifing end landmarks. These specific technical skills are essential to the quality and outcome of the procedure. In this present study, we aimed to develop an endoscopic technical skills assessment tool to assess generic and specific technical skills, which could be used in self-appraisal and training.
Methodology Two high-volume live elective endoscopic procedures were chosen: flexible sigmoidoscopy and colonoscopy. These procedures were selected because they are performed in large numbers in the United Kingdom and worldwide by both consultant/attending and trainees. Each full-length procedure was assessed prospectively by at least 2 observers, who evaluated each procedure independently using the developed assessment tool. The observers had more than 5 years (range 5–12 years) postgraduate clinical experience and were individually trained by an experienced endoscopist over a 3-month period. The observers were trained in using the assessment tool, as well as in various aspects of gastrointestinal endoscopy. The observers had no direct professional relationship with the attending and trainee endoscopists.
Setting The procedures were performed at the Royal Free, Hampstead and Royal London, Whitechapel hospitals, between 2005 and 2007. Consultants/attending had previously performed more than 150 lower gastrointestinal endoscopies and trainees had previously performed fewer than 50 gastrointestinal endoscopies. All patients were between American Society of Anesthesiologists (ASA) 1 and 3, body
Generic endoscopic technical skills
Generic skill
1
2
3
4
5
Handling of scope Handling of controls Force used Hand, eye & monitor coordination Patient discomfort & reaction Flow of procedure
Awkward Many unnecessary moves Rough & excessive force Poor
Competent Some unnecessary moves Gentle most of the time Good
Expert All purposeful movements Gentle at all times Expert
Severe discomfort, ignores discomfort & proceeds Unsure
Some discomfort, stops & proceeds when safe Confident with occasional hesitation
Minimal discomfort & stops & proceeds when safe Confident at all times
Generic technical skills To assess generic endoscopic technical skills a hierarchical task analysis was done as previously described.9,10 From this, a Likert generic scoring system was developed (Table 1). 10
Specific technical skills
Statistical analysis
Strategy for progression (torque/patient positioning/irrigation/scope withdrawl) Identifies end landmarks (sigmoidoscopy—splenic flexure) (colonoscopy—ileo-caecal valve)
Inadequate, always on the mucosal wall Clumsily done, inadequate specimen Has no strategy when technical difficulty arises Not clearly identified Luminal vision
Therapeutic procedure (biopsy or polypectomy)
2 Table 2
There were no major procedural complications. A total of 135 endoscopies were assessed (75 flexible sigmoidscopies, 60 colonoscopies). There were 78 male and 57 female patients. Mean age was 62 (range 18 – 80 years). There were 9 consultant/attending and 12 registrar/resident (postgraduate year 4 – 8) trainees recruited to the study. There were 18 male endoscopists and 3 female endoscopists; 19 were right-handed and 2 were left-handed. All of the trainee procedures were supervised by a consultant/attending, although the latter may not have been present throughout the entire procedure. There were 77 consultants episodes, mean number 3 (range 1– 4), and 58 registrar episodes, mean number 2 (range 1–3). Inter-rater reliabilities between independent observers using Cronbach alpha were as follows: flexible sigmoidoscopy generic skills Likert, Cronbach alpha ⫽ .81, P ⬍.05, and flexible sigmoidoscopy specific skills Likert, Cron-
Specific endoscopic technical skills
Results
Specific sill
4
6
Data were collated in an Excel database (Microsoft, Redmond, WA). Statistical analysis was performed using the SPSS software statistical package (SPSS, Chicago, IL). For reliability, Cronbach alpha was used and a value greater than .61 was deemed significant for reliability. For the non-parametric data, repeated measures analysis of variance (ANOVA) was used for comparison between the 2 groups of endoscopists for validity analysis. P ⬍.05 was regarded as statistically significant. A senior statistician from the Statistical Advisory Unit at Imperial College London along with the primary researcher analyzed the data for reliability and validity.
Uses appropriate strategy & competently progresses Adequately identified
8
To develop a procedure-specific technical skills endoscopic assessment tool, a hierarchical task analysis was constructed on flexible sigmoidoscopy and colonoscopy as previously described.9,10 From this, a weighted Likert scale was constructed focusing on key points of the procedure. The scoring system was scored double of the generic skills, as this section is more clinically relevant to the technical outcome of the procedure. Table 2 is the procedure-specific technical skills assessment tool.
Expertly done, good specimen achieved Uses appropriate strategy & progresses expertly Clearly & expertly identified
mass index (BMI) less than 30, and between 18 and 80 years old. They were all discharged on the same day.
Almost all of mucosa visualized
The American Journal of Surgery, Vol 196, No 3, September 2008
Adequate & occasionally on mucosal wall Competently done, adequate specimen
452
S.K. Sarker et al.
Performance in gastrointestinal endoscopy
453
Generic Likert Total Observer 1
30
Generic Likert Total Observer 2
28
26
24
22 Consultant
Trainee
Grade of Surgeon
Figure 1
Flexible sigmoidoscopy— generic technical skills Likert scores.
bach alpha ⫽ .79, P ⬍.05. For the colonoscopies, the values were as follows: colonoscopy generic skills Likert, Cronbach alpha ⫽ .85, P ⬍.05 and colonoscopy specific skills Likert, Cronbach alpha ⫽ .80, P ⬍ .05. Construct validities between consultants and trainees using ANOVA were as follows: flexible sigmoidoscopy generic skills Likert (Figure 1), ANOVA P ⫽ .005, flexible sigmoidoscopy specific skills Likert (Figure 2), ANOVA P ⫽ .012, colonoscopy generic skills Likert (Figure 3), ANOVA P ⫽ .003 and colonoscopy specific skills Likert (Figure 4), ANOVA P ⫽ .004.
Comments Our study was carried out to validate the performance assessment tool as a method for assessment of technical
skills in gastrointestinal endoscopy. Twenty-one subjects with different endoscopic experience performing 2 different procedures (flexible sigmoidoscopy, and colonoscopy) on live patients were assessed using the developed performance assessment tool. We found that there was good reliability and good construct validity for generic and specific Likert scores of the performance assessment tool, which means that the assessment tool can differentiate between consultants and trainees with different endoscopic experience. A limitation of the study was that the assessments were done prospectively while the procedures were being performed, so, in theory, there could be some observer bias in the assessment. This method was purposely selected as it is more practical and less time-consuming than retrospectively watching videos of the procedures. Also, current assessments of trainees are not done independently
Specific Likert Total Observer 1
40
Specific Likert Total Observer 2 38
36
34
32
30
28
26 Consultant
Trainee
Grade of Surgeon
Figure 2
Flexible sigmoidoscopy—specific technical skills Likert scores.
454
The American Journal of Surgery, Vol 196, No 3, September 2008 Generic Likert Total Observer 1
33
Generic Likert Total Observer 2
30
27
24
21
18 Consultant
Trainee
Grade of Surgeon
Figure 3
Colonoscopy— generic technical skills Likert scores.
or completely blindly, so the current study has less bias than current practice. We aim to perform a completely blinded study in the future to address the possibility of observer bias, and to conduct a series of self-appraisals of the endoscopists to demonstrate if their own assessments are as reliable as those of the independent observers. A previous study evaluated the ability of an upper gastrointestinal virtual reality simulator to assess skills in endoscopy11 and validated its metrics using a video-endoscopic (VES) technique. The VES assessment was able to discriminate performance across the groups but only on the simulators, not on real live procedures. Another study12 for objective assessment of technical skills in lower gastrointestinal endoscopy used the Likert score assessment tool. The Likert score was found to discriminate the level of
skills across the groups, with good construct validity. However, in both studies the assessment criteria concerned mainly generic technical skills in endoscopy. Task analysis has been used previously to self-appraise technical skills13 in laparoscopic surgery and also to assess technical skill errors.14 However, the latter assessment tool, although very detailed, is not practical on a day-to-day basis but is more of a research tool. Previous studies assessing technical skills in surgery using a global scoring system or motion tracking15–17 demonstrated construct validity in generic technical skills between experts and novices. A more recent, study tried to combine both assessment tools18 and other studies have aimed to assess the motion of the operator.19,20 Once again these studies showed good construct validity in generic technical skills but did not assess specific
Specific Likert Total Observer 1
40
Specific Likert Total Observer 2 38
36
34
32
30
28
20
26 Consultant
Trainee
Grade of Surgeon
Figure 4
Colonoscopy—specific technical skills Likert scores.
S.K. Sarker et al.
Performance in gastrointestinal endoscopy
technical skills that are important in the final performance and quality/outcome of the operation or procedure. It has been postulated that a structured curriculum using live patients as well as virtual reality would benefit the acquisition of generic and specific technical skills in open, laparoscopic, and endoscopic procedures or surgeries.21 Two recent studies demonstrated the benefit of a structured curriculum using virtual reality in acquiring technical skills.22,23 Therefore, it seems the optimal basis of teaching and training the future medical trainees is combining a structured curriculum, using real patients, virtual realty simulators and other simulation devices, with an assessment/ self-appraisal tool that is able to evaluate and decipher the various technical (generic and specific) skills required to complete a particular procedure or operation.
Conclusions The performance assessment tool developed in this study was able to differentiate between subjects with different endoscopic generic and specific technical skills. This indicates that the performance assessment tool evaluates skills relevant for gastrointestinal endoscopy and could be used in training programs as an assessment tool. Further studies are required to validate using the performance assessment tool for assessment of trainees with different endoscopic experience and to determine the validity of this tool in training that may be achieved by the junior endoscopists concentrating on the technical skills aspects he or she may be deficient on in the first assessment, and to determine if there is an improvement after their second assessment. It is also necessary to evaluate the amount of virtual training needed to acquire adequate basic endoscopic skills from simulator training, and whether there is a benefit from regular repetition, or whether the simulator can be used for updating skills after longer periods when endoscopy has not been practiced. This endoscopic tool has the possibility of being used in training and self-appraisal. We aim to modify and apply this tool to other endoscopic procedures in the future, for example, ERCP and endoluminal and transluminal procedures.
References 1. Cuschieri A, Francis N, Crosby J, et al. What do master surgeons think of surgical competence and revalidation? Am J Surg 2001;182:110 – 6. 2. Reznick RK. Teaching and testing technical skills. Am J Surg 1993; 165:358 – 61.
455 3. Kopta JA. An approach to the evaluation of operative skills. Surgery 1971;70:297–303. 4. Bridges M, Diamond DL. The financial impact of teaching surgical residents in the operating room. Am J Surg 1999;177:28 –32. 5. Ross DG, Harris CA, Jones DJ. A comparison of operative experience for basic surgical trainees in 1992 and 2000. Br J Surg 2002;89(suppl 1):60. 6. Skidmore FD. Junior surgeons are becoming deskilled as result of Calman proposals. BMJ 1997;314:1281. 7. Darzi A, Smith S, Taffinder N. Assessing operative skill. Needs to become more objective. BMJ 1999;318:887– 8. 8. Scott DJ, Valentine RJ, Bergen PC, et al. Evaluating surgical competency with the American board of surgery in-training examination, skill testing, and intraoperative assessment. Surgery 2000;128:613–22. 9. Sarker SK, Chang A, Albrani T, et al. Constructing a hierarchical task analysis in surgery. Surg Endosc 2008;22:107–11. 10. Sarker SK, Chang A, Vincent C. Development of assessing generic and specific technical skills in laparoscopic surgery. Am J Surg 2006; 191:238 – 44. 11. Moorthy K, Munz Y, Jiwanji M, et al. Validity and reliability of a virtual reality upper gastrointestinal simulator and cross validation using structured assessment of individual performance with video playback. Surg Endosc 2004;18:328 –33. 12. Moorthy K, Munz Y, Orchard TR, et al. An innovative method for the assessment of skills in lower gastrointestinal endoscopy. Surg Endosc 2004;18:1613–9. 13. Sarker SK, Hutchinson R, Chang A, et al. Self-appraisal hierarchical task analysis of expert surgeons performing laparoscopic surgery. Surg Endosc 2006;20:636 – 40. 14. Tang B, Hanna GB, Carter F, et al. Competence assessment of laparoscopic operative and cognitive skills: Objective Structured Clinical Examination (OSCE) or Observational Clinical Human Reliability Assessment (OCHRA). World J Surg 2006;30:527–34. 15. Martin JA, Regehr G, Reznick R, et al. Objective Structured Assessment of Technical Skill (OSATS) for surgical residents. Br J Surg 1997;84:273– 8. 16. Datta V, Chang A, Mackay S, et al. The relationship between motion analysis and surgical technical assessments. Am J Surg 2002;184:70 –3. 17. Francis NK, Hanna GB, Cuschieri A. The performance of master surgeons on the advanced Dundee endoscopic psychomotor tester: contrast validity study. Arch Surg 2002;137:841– 4. 18. Aggarwal R, Grantcharov T, Moorthy K, et al. An evaluation of the feasibility, validity, and reliability of laparoscopic skills assessment in the operating room. Ann Surg 2007;245:992–9. 19. Stylopoulos N, Vosburgh KG. Assessing technical skill in surgery and endoscopy: a set of metrics and an algorithm (C-PASS) to assess skills in surgical and endoscopic procedures. Surg Innov 2007;14:113–21. 20. Gumbs AA, Hogle NJ, Fowler DL. Evaluation of resident laparoscopic performance using global operative assessment of laparoscopic skills. J Am Coll Surg 2007;204:308 –13. 21. Sarker SK, Patel B. Simulation and surgical training. Int J Clin Pract (in press). 22. Immenroth M, Burger T, Brenner J, et al. Mental training in surgical education: a randomized controlled trial. Ann Surg 2007;245:385–91. 23. Aggarwal R, Grantcharov TP, Eriksen JR, et al. An evidence-based virtual reality training program for novice laparoscopic surgeons. Ann Surg 2006;244:310 – 4.
Surgical Education
Procedural performance in gastrointestinal endoscopy: An assessment and self-appraisal tool Sudip K. Sarker, M.B.Ch.B., F.R.C.S., Ph.D.a,*, Tark Albrani, M.B.B.S., M.R.C.S., M.Sc.b, Atiquaz Zaman, M.B.B.S., M.R.C.S., M.Sc.b, Bijen Patel, M.B.B.S., F.R.C.S.b a
University Department of Academic Surgery, Royal Free Hospital, London, United Kingdom; bAcademic Department of Upper Gastrointestinal Surgery, The Barts & Royal London Hospitals, London, United Kingdom KEYWORDS: Assessment; Endoscopy; Gastroenterology; Technical skills
Abstract AIMS: Assessing endoscopic technical skills competency in a structured manner is a topical issue, in light of several workforce factors that may affect the training of future endoscopists. To date there has been little attempt to comprehensively assess both generic and specific technical skills in lower gastrointestinal endoscopies; the current study aimed to develop and validate a tool that can assess these varied skills. METHODS: Hierarchical task analyses of generic and specific technical skills were constructed on flexible sigmoidoscopy and colonoscopy after expert panel discussions. Generic technical skills are subtasks (eg, movements of the control wheel) that allow the endoscopist to complete a main task (specific technical skills), for example, reaching the cecum. Weighted Likert scales were then constructed individually for generic and specific technical skills for each procedure. Two observers assessed each procedure independently. RESULTS: A total of 135 endoscopic procedures were performed by 9 consultants and 12 registrars. Mean inter-rater reliability Cronbach alphas were .83 and .80, P ⱕ.05, for generic and specific skills, respectively, for each procedure. Construct validity results using analysis of variance (ANOVA) for consultants and trainees were significant for each procedure, P ⫽ .005, P ⫽ .003 for generic, and P ⫽ .012, P ⫽ .004 for specific technical skills. CONCLUSIONS: This new assessment/self-appraisal tool for lower gastrointestinal endoscopies seems to have face, content, concurrent, and construct validities. The tool has the possibility of being used in training and self-appraisal. We aim to modify and apply this tool to other endoscopic procedures in the future, such as endoscopic retrograde cholangiopancreatography (ERCP), endoluminal and transluminal procedures. © 2008 Elsevier Inc. All rights reserved.
The working and training environment is entering a new era in the United Kingdom. Reduction in working hours for doctors in training1 will have a dramatic effect, with much * Corresponding author. Tel.: ⫹0207 565 8155; fax: ⫹0207 598 0285 E-mail address: [email protected] Manuscript received July 30, 2007; revised manuscript October 11, 2007
0002-9610/$ - see front matter © 2008 Elsevier Inc. All rights reserved. doi:10.1016/j.amjsurg.2007.10.024
less exposure to high volumes of operative/procedural work than previously, and the total time for training will be drastically reduced. It is therefore increasingly important that opportunities to acquire technical skills are used as effectively as possible. Healthcare is now a consultant/attending lead, which increases the safety of patient care by increasing the level of supervision of medical trainees. However, this welcome
S.K. Sarker et al.
Performance in gastrointestinal endoscopy
change reduces junior trainees to acquiring and performing technical tasks independently, which are important steps in gaining confidence in acquiring the technical skills needed to perform an operation or procedure. Evaluation of technical skills is notoriously difficult due to its subjectivity and the need for time-consuming expert analysis. No ongoing evaluation scheme exists to assess the continuing competency of endoscopists throughout their careers. The use of the endoscopists themselves to evaluate their own technique, employing objective analysis techniques, would circumvent the need for constant external analysis. In the last few years, significant developments have been made in the objective assessment of technical skills and a range of methods have been developed. The assessment of technical skills during training has been considered to be a shape of quality assurance for the future.2 The assessment of technical ability is the job of the trainers and their assessment is mainly subjective.3 Objective assessment is necessary because deficiencies in training and performance are hard to correct without objective feedback.4 Significant progress has been made to advance the methods that objectively evaluate surgical technique.5 Objective and unbiased data on procedural performance are essential during the prospective assessment in training.6,7 Checklists and global scores are some of the objective methods of assessing technical skills in endoscopic procedures. It has been said that checklists make examiners into observers, instead of interpreters, of behavior, that lead to removing the subjectivity of the evaluation process.8 Global rating scales can consist of several components of operative or procedural skills that are marked on a 5-point Likert scale, with the middle and the extreme points anchored by explicit descriptors to help in the criterion referenced assessment of performance. Generic and procedure-specific technical skills are different skills that are acquired at different times during endoscopic training. By the time the trainee reaches consultant/attending status, he or she should have mastered these aspects of technical skills in endoscopy. Generic technical skills are mainly acquired during basic endoscopic training, for example, handling of scope, handling of controls, handeye and monitor coordination, reaction to patient discom-
Table 1
451 fort, and flow of procedure and force used. These basic skills are required by the endoscopist to perform a complete procedure. They alone do not form a task or sub-task but allow the endoscopist to perform the task. Specific technical skills that are learned throughout basic and higher endoscopic training include luminal vision, therapeutic procedure, strategy for progression, and identifing end landmarks. These specific technical skills are essential to the quality and outcome of the procedure. In this present study, we aimed to develop an endoscopic technical skills assessment tool to assess generic and specific technical skills, which could be used in self-appraisal and training.
Methodology Two high-volume live elective endoscopic procedures were chosen: flexible sigmoidoscopy and colonoscopy. These procedures were selected because they are performed in large numbers in the United Kingdom and worldwide by both consultant/attending and trainees. Each full-length procedure was assessed prospectively by at least 2 observers, who evaluated each procedure independently using the developed assessment tool. The observers had more than 5 years (range 5–12 years) postgraduate clinical experience and were individually trained by an experienced endoscopist over a 3-month period. The observers were trained in using the assessment tool, as well as in various aspects of gastrointestinal endoscopy. The observers had no direct professional relationship with the attending and trainee endoscopists.
Setting The procedures were performed at the Royal Free, Hampstead and Royal London, Whitechapel hospitals, between 2005 and 2007. Consultants/attending had previously performed more than 150 lower gastrointestinal endoscopies and trainees had previously performed fewer than 50 gastrointestinal endoscopies. All patients were between American Society of Anesthesiologists (ASA) 1 and 3, body
Generic endoscopic technical skills
Generic skill
1
2
3
4
5
Handling of scope Handling of controls Force used Hand, eye & monitor coordination Patient discomfort & reaction Flow of procedure
Awkward Many unnecessary moves Rough & excessive force Poor
Competent Some unnecessary moves Gentle most of the time Good
Expert All purposeful movements Gentle at all times Expert
Severe discomfort, ignores discomfort & proceeds Unsure
Some discomfort, stops & proceeds when safe Confident with occasional hesitation
Minimal discomfort & stops & proceeds when safe Confident at all times
Generic technical skills To assess generic endoscopic technical skills a hierarchical task analysis was done as previously described.9,10 From this, a Likert generic scoring system was developed (Table 1). 10
Specific technical skills
Statistical analysis
Strategy for progression (torque/patient positioning/irrigation/scope withdrawl) Identifies end landmarks (sigmoidoscopy—splenic flexure) (colonoscopy—ileo-caecal valve)
Inadequate, always on the mucosal wall Clumsily done, inadequate specimen Has no strategy when technical difficulty arises Not clearly identified Luminal vision
Therapeutic procedure (biopsy or polypectomy)
2 Table 2
There were no major procedural complications. A total of 135 endoscopies were assessed (75 flexible sigmoidscopies, 60 colonoscopies). There were 78 male and 57 female patients. Mean age was 62 (range 18 – 80 years). There were 9 consultant/attending and 12 registrar/resident (postgraduate year 4 – 8) trainees recruited to the study. There were 18 male endoscopists and 3 female endoscopists; 19 were right-handed and 2 were left-handed. All of the trainee procedures were supervised by a consultant/attending, although the latter may not have been present throughout the entire procedure. There were 77 consultants episodes, mean number 3 (range 1– 4), and 58 registrar episodes, mean number 2 (range 1–3). Inter-rater reliabilities between independent observers using Cronbach alpha were as follows: flexible sigmoidoscopy generic skills Likert, Cronbach alpha ⫽ .81, P ⬍.05, and flexible sigmoidoscopy specific skills Likert, Cron-
Specific endoscopic technical skills
Results
Specific sill
4
6
Data were collated in an Excel database (Microsoft, Redmond, WA). Statistical analysis was performed using the SPSS software statistical package (SPSS, Chicago, IL). For reliability, Cronbach alpha was used and a value greater than .61 was deemed significant for reliability. For the non-parametric data, repeated measures analysis of variance (ANOVA) was used for comparison between the 2 groups of endoscopists for validity analysis. P ⬍.05 was regarded as statistically significant. A senior statistician from the Statistical Advisory Unit at Imperial College London along with the primary researcher analyzed the data for reliability and validity.
Uses appropriate strategy & competently progresses Adequately identified
8
To develop a procedure-specific technical skills endoscopic assessment tool, a hierarchical task analysis was constructed on flexible sigmoidoscopy and colonoscopy as previously described.9,10 From this, a weighted Likert scale was constructed focusing on key points of the procedure. The scoring system was scored double of the generic skills, as this section is more clinically relevant to the technical outcome of the procedure. Table 2 is the procedure-specific technical skills assessment tool.
Expertly done, good specimen achieved Uses appropriate strategy & progresses expertly Clearly & expertly identified
mass index (BMI) less than 30, and between 18 and 80 years old. They were all discharged on the same day.
Almost all of mucosa visualized
The American Journal of Surgery, Vol 196, No 3, September 2008
Adequate & occasionally on mucosal wall Competently done, adequate specimen
452
S.K. Sarker et al.
Performance in gastrointestinal endoscopy
453
Generic Likert Total Observer 1
30
Generic Likert Total Observer 2
28
26
24
22 Consultant
Trainee
Grade of Surgeon
Figure 1
Flexible sigmoidoscopy— generic technical skills Likert scores.
bach alpha ⫽ .79, P ⬍.05. For the colonoscopies, the values were as follows: colonoscopy generic skills Likert, Cronbach alpha ⫽ .85, P ⬍.05 and colonoscopy specific skills Likert, Cronbach alpha ⫽ .80, P ⬍ .05. Construct validities between consultants and trainees using ANOVA were as follows: flexible sigmoidoscopy generic skills Likert (Figure 1), ANOVA P ⫽ .005, flexible sigmoidoscopy specific skills Likert (Figure 2), ANOVA P ⫽ .012, colonoscopy generic skills Likert (Figure 3), ANOVA P ⫽ .003 and colonoscopy specific skills Likert (Figure 4), ANOVA P ⫽ .004.
Comments Our study was carried out to validate the performance assessment tool as a method for assessment of technical
skills in gastrointestinal endoscopy. Twenty-one subjects with different endoscopic experience performing 2 different procedures (flexible sigmoidoscopy, and colonoscopy) on live patients were assessed using the developed performance assessment tool. We found that there was good reliability and good construct validity for generic and specific Likert scores of the performance assessment tool, which means that the assessment tool can differentiate between consultants and trainees with different endoscopic experience. A limitation of the study was that the assessments were done prospectively while the procedures were being performed, so, in theory, there could be some observer bias in the assessment. This method was purposely selected as it is more practical and less time-consuming than retrospectively watching videos of the procedures. Also, current assessments of trainees are not done independently
Specific Likert Total Observer 1
40
Specific Likert Total Observer 2 38
36
34
32
30
28
26 Consultant
Trainee
Grade of Surgeon
Figure 2
Flexible sigmoidoscopy—specific technical skills Likert scores.
454
The American Journal of Surgery, Vol 196, No 3, September 2008 Generic Likert Total Observer 1
33
Generic Likert Total Observer 2
30
27
24
21
18 Consultant
Trainee
Grade of Surgeon
Figure 3
Colonoscopy— generic technical skills Likert scores.
or completely blindly, so the current study has less bias than current practice. We aim to perform a completely blinded study in the future to address the possibility of observer bias, and to conduct a series of self-appraisals of the endoscopists to demonstrate if their own assessments are as reliable as those of the independent observers. A previous study evaluated the ability of an upper gastrointestinal virtual reality simulator to assess skills in endoscopy11 and validated its metrics using a video-endoscopic (VES) technique. The VES assessment was able to discriminate performance across the groups but only on the simulators, not on real live procedures. Another study12 for objective assessment of technical skills in lower gastrointestinal endoscopy used the Likert score assessment tool. The Likert score was found to discriminate the level of
skills across the groups, with good construct validity. However, in both studies the assessment criteria concerned mainly generic technical skills in endoscopy. Task analysis has been used previously to self-appraise technical skills13 in laparoscopic surgery and also to assess technical skill errors.14 However, the latter assessment tool, although very detailed, is not practical on a day-to-day basis but is more of a research tool. Previous studies assessing technical skills in surgery using a global scoring system or motion tracking15–17 demonstrated construct validity in generic technical skills between experts and novices. A more recent, study tried to combine both assessment tools18 and other studies have aimed to assess the motion of the operator.19,20 Once again these studies showed good construct validity in generic technical skills but did not assess specific
Specific Likert Total Observer 1
40
Specific Likert Total Observer 2 38
36
34
32
30
28
20
26 Consultant
Trainee
Grade of Surgeon
Figure 4
Colonoscopy—specific technical skills Likert scores.
S.K. Sarker et al.
Performance in gastrointestinal endoscopy
technical skills that are important in the final performance and quality/outcome of the operation or procedure. It has been postulated that a structured curriculum using live patients as well as virtual reality would benefit the acquisition of generic and specific technical skills in open, laparoscopic, and endoscopic procedures or surgeries.21 Two recent studies demonstrated the benefit of a structured curriculum using virtual reality in acquiring technical skills.22,23 Therefore, it seems the optimal basis of teaching and training the future medical trainees is combining a structured curriculum, using real patients, virtual realty simulators and other simulation devices, with an assessment/ self-appraisal tool that is able to evaluate and decipher the various technical (generic and specific) skills required to complete a particular procedure or operation.
Conclusions The performance assessment tool developed in this study was able to differentiate between subjects with different endoscopic generic and specific technical skills. This indicates that the performance assessment tool evaluates skills relevant for gastrointestinal endoscopy and could be used in training programs as an assessment tool. Further studies are required to validate using the performance assessment tool for assessment of trainees with different endoscopic experience and to determine the validity of this tool in training that may be achieved by the junior endoscopists concentrating on the technical skills aspects he or she may be deficient on in the first assessment, and to determine if there is an improvement after their second assessment. It is also necessary to evaluate the amount of virtual training needed to acquire adequate basic endoscopic skills from simulator training, and whether there is a benefit from regular repetition, or whether the simulator can be used for updating skills after longer periods when endoscopy has not been practiced. This endoscopic tool has the possibility of being used in training and self-appraisal. We aim to modify and apply this tool to other endoscopic procedures in the future, for example, ERCP and endoluminal and transluminal procedures.
References 1. Cuschieri A, Francis N, Crosby J, et al. What do master surgeons think of surgical competence and revalidation? Am J Surg 2001;182:110 – 6. 2. Reznick RK. Teaching and testing technical skills. Am J Surg 1993; 165:358 – 61.
455 3. Kopta JA. An approach to the evaluation of operative skills. Surgery 1971;70:297–303. 4. Bridges M, Diamond DL. The financial impact of teaching surgical residents in the operating room. Am J Surg 1999;177:28 –32. 5. Ross DG, Harris CA, Jones DJ. A comparison of operative experience for basic surgical trainees in 1992 and 2000. Br J Surg 2002;89(suppl 1):60. 6. Skidmore FD. Junior surgeons are becoming deskilled as result of Calman proposals. BMJ 1997;314:1281. 7. Darzi A, Smith S, Taffinder N. Assessing operative skill. Needs to become more objective. BMJ 1999;318:887– 8. 8. Scott DJ, Valentine RJ, Bergen PC, et al. Evaluating surgical competency with the American board of surgery in-training examination, skill testing, and intraoperative assessment. Surgery 2000;128:613–22. 9. Sarker SK, Chang A, Albrani T, et al. Constructing a hierarchical task analysis in surgery. Surg Endosc 2008;22:107–11. 10. Sarker SK, Chang A, Vincent C. Development of assessing generic and specific technical skills in laparoscopic surgery. Am J Surg 2006; 191:238 – 44. 11. Moorthy K, Munz Y, Jiwanji M, et al. Validity and reliability of a virtual reality upper gastrointestinal simulator and cross validation using structured assessment of individual performance with video playback. Surg Endosc 2004;18:328 –33. 12. Moorthy K, Munz Y, Orchard TR, et al. An innovative method for the assessment of skills in lower gastrointestinal endoscopy. Surg Endosc 2004;18:1613–9. 13. Sarker SK, Hutchinson R, Chang A, et al. Self-appraisal hierarchical task analysis of expert surgeons performing laparoscopic surgery. Surg Endosc 2006;20:636 – 40. 14. Tang B, Hanna GB, Carter F, et al. Competence assessment of laparoscopic operative and cognitive skills: Objective Structured Clinical Examination (OSCE) or Observational Clinical Human Reliability Assessment (OCHRA). World J Surg 2006;30:527–34. 15. Martin JA, Regehr G, Reznick R, et al. Objective Structured Assessment of Technical Skill (OSATS) for surgical residents. Br J Surg 1997;84:273– 8. 16. Datta V, Chang A, Mackay S, et al. The relationship between motion analysis and surgical technical assessments. Am J Surg 2002;184:70 –3. 17. Francis NK, Hanna GB, Cuschieri A. The performance of master surgeons on the advanced Dundee endoscopic psychomotor tester: contrast validity study. Arch Surg 2002;137:841– 4. 18. Aggarwal R, Grantcharov T, Moorthy K, et al. An evaluation of the feasibility, validity, and reliability of laparoscopic skills assessment in the operating room. Ann Surg 2007;245:992–9. 19. Stylopoulos N, Vosburgh KG. Assessing technical skill in surgery and endoscopy: a set of metrics and an algorithm (C-PASS) to assess skills in surgical and endoscopic procedures. Surg Innov 2007;14:113–21. 20. Gumbs AA, Hogle NJ, Fowler DL. Evaluation of resident laparoscopic performance using global operative assessment of laparoscopic skills. J Am Coll Surg 2007;204:308 –13. 21. Sarker SK, Patel B. Simulation and surgical training. Int J Clin Pract (in press). 22. Immenroth M, Burger T, Brenner J, et al. Mental training in surgical education: a randomized controlled trial. Ann Surg 2007;245:385–91. 23. Aggarwal R, Grantcharov TP, Eriksen JR, et al. An evidence-based virtual reality training program for novice laparoscopic surgeons. Ann Surg 2006;244:310 – 4.