- Email: [email protected]
Development of proficiency-based knot-tying and suturing curriculum for otolaryngology residents: A pilot study
ARTICLE IN PRESS
JID: ANL
[mNS;December 21, 2019;0:45]
Auris Nasus Larynx xxx (xxxx) xxx Contents lists available at ScienceDirect
Auris Nasus Larynx journal homepage: www.elsevier.com/locate/anl
Development of proficiency-based knot-tying and suturing curriculum for otolaryngology residents: A pilot study Eriko Sato a,b,1, Sohei Mitani a,c,1,∗, Naoki Nishio c,d, Takashi Kitani a, Tomoyoshi Sanada a, Toru Ugumori a,e, F. Christopher Holsinger c, Fred M. Baik c, Naohito Hato a a Department
of Otolaryngology-Head and Neck Surgery, Ehime University Graduate School of Medicine, Shitsukawa, Toon, Ehime 791−0295, Japan b Division of Gastrointestinal Oncology, Shizuoka Cancer Center, 1007 Shimonagakubo, Nagaizumi, Sunto District, Shizuoka 411-0934, Japan c Division of Head and Neck Surgery, Department of Otolaryngology, Stanford University, 875 Blake Wilbur Drive, Palo Alto, CA 94305, United States d Department of Otorhinolaryngology, Nagoya University Graduate School of Medicine, 65 Tsurumai-cho, Showa Ward, Nagoya, Aichi 466-8560, Japan e Ugumori ENT Clinic, 3 Chome-10-25 Yougonishi, Matsuyama, Ehime 790-0046, Japan
A R T I C L E
I N F O
Article history: Received 1 October 2019 Accepted 27 November 2019 Available online xxx Keywords: Basic surgical skill Knot-tying and suturing curriculum RAND/UCLA method Simulation training Proficiency-based training
A B S T R A C T
Objective: Basic surgical skills such as knot-tying and suturing are important for all otolaryngologists, regardless of subspecialty. The present study was undertaken in order to assess basic surgical techniques such as knot-tying and suturing required for novice otolaryngology residents with taking the variety of subspecialties into consideration, and evaluate the impact of a proficiency-based training curriculum based on these techniques. Methods: A prospective study was performed for developing of proficiency-based knot-tying and suturing curriculum for otolaryngology residents in the third post-graduate year (PGY3). The proficiency-based training curriculum was developed based on the tasks selected by RAND/UCLA method with expert panel, which is an iterative and anonymous survey used to establish consensus among participants. Expert panelists were selected from various divisions to reflect variety of their subspecialties. PGY-3 residents trained with the developed curriculum that included proctored pre-test, self-training to proficiency, and proctored post-test. Visual analogue scale (VAS) of trainees’ overall competence in the operating room was self-assessed by each resident, before and after completing the training curriculum. Results: Nine PGY-3 residents were enrolled as trainees. Eleven experts chosen as panelists had various subspecialty, including 2 from otology, 2 from rhinology, 2 from laryngology, 2 from head and neck surgery, and 3 from general otolaryngology. Seven tasks were selected from RAND/UCLA method and used to develop the curriculum. Trainee scores at pre-test were significantly lower than expert scores for all 7 tasks (p < 0.01) and each coefficient of variation of trainee score was larger than that of expert score (p < 0.05), supporting construct validity. The mean of composite scores between pre-test and post-test had statistical significance
∗
Corresponding author at: Department of Otolaryngology-Head and Neck Surgery, Ehime University Graduate School of Medicine, Shitsukawa, Toon, Ehime 791−0295, Japan. E-mail address: [email protected] (S. Mitani). 1 These authors contributed equally to this work. https://doi.org/10.1016/j.anl.2019.11.005 0385-8146/© 2019 Oto-Rhino-Laryngological Society of Japan Inc. Published by Elsevier B.V. All rights reserved. Please cite this article as: E. Sato, S. Mitani and N. Nishio et al., Development of proficiency-based knot-tying and suturing curriculum for otolaryngology residents: A pilot study, Auris Nasus Larynx, https:// doi.org/ 10.1016/ j.anl.2019.11.005
JID: ANL
ARTICLE IN PRESS
[mNS;December 21, 2019;0:45]
E. Sato, S. Mitani and N. Nishio et al. / Auris Nasus Larynx xxx (xxxx) xxx
2
(68.6 ± 11.6 vs 95.9 ± 3.6, p < 0.01), documenting substantial improvement after training. Self-assessment VAS was also improved pre- to post-training (1.2 ± 0.9 vs 4.5 ± 1.4, p < 0.01). A follow-up questionnaire showed that trainees felt the educational curriculum to be beneficial. Conclusion: In the present study, seven basic technical skills were selected using the RAND/UCLA method and used to create a proficiency-based training curriculum. Our results indicate that this curriculum significantly improves proficiency of basic surgical skills of junior otolaryngology residents. © 2019 Oto-Rhino-Laryngological Society of Japan Inc. Published by Elsevier B.V. All rights reserved.
1 Introduction
2. Materials and method
Surgeons are required to move their hands fluently and efficiently at an instinctive level, together with accuracy, speed and flow [1]. As work hours have declined during residency, simulation training has become a more important and valuable means to achieve proficiency by supporting early acquisition of complex skills, knowledge and behaviors. Ultimately, this may improve patient-related outcomes, patient safety, cost savings [2–4], as well enhance skill acquisition among other trainees [5–8]. Proficiency-based simulation training curriculums have been developed in many general surgical fields, e.g. for laparoscopic suturing [9]. This proficiency – based training using expert – derived performance goals is currently considered the optimal simulation training method, because simulated tasks allow for objective assessment of technical skill [10]. Surgical field in otolaryngology-head and neck surgery (OHNS) is often narrow or close to the surface of body, in sharp contrast to that of “open” or laparoscopic general surgery. Thus, several different skill-sets are required. Moreover, these basic OHNS skills span not only “open” head and neck surgery, but also otology, rhinology and laryngology. One national survey reported that basic surgical skills (knot-tying and suturing) were the most prioritized technical procedures amongst a group of 50 unique procedures and technical skills (such as Flexible fiber pharyngo-laryngoscopy, Fine needle aspiration) [11]. Recent trends in otolaryngology reflect an increasing focus on surgical education and most articles are currently published mainly in otology (mastoidectomy) with 40%, and in rhinology (endoscopic sinus surgery) with 22% [12]. Regardless of variety of subspecialties in otolaryngology, all otolaryngologists need basic surgical skills and appropriately perform these techniques in serious and critical situations, such as rapid bleeding after tonsillectomy, tracheostomy, or hemorrhage after neck surgeries. However, there is no report mentioning open surgical skills for otolaryngology surgeries. The objective of this study is to estimate basic surgical techniques such as knot-tying and suturing for novice otolaryngology residents by using expert panel, in which several experts were selected from whole otolaryngology. Second aim was to evaluate the educational benefit for novice otolaryngology residents of the curriculum based on these techniques.
2.1. Study design A prospective study was performed in an academic university hospital of a proficiency-based knot-tying and suturing curriculum. The curriculum was developed based on the tasks selected by the RAND/UCLA method with expert panel (Fig. 1). PGY-3 residents trained with the developed curriculum that included proctored pre-test, selftraining to proficiency, and proctored post-test. Visual analogue scale (VAS) of trainees’ overall competence in operating room (OR) was rated as self-assessment by themselves before and after training. A questionnaire was taken after training to evaluate perceived benefit, burden, contents of curriculums, their experience in OR as “on-the-job” training and another “off-the-job” training during implementation of the curriculum. This study was approved by Institutional Review Board at the University (registration number: 1903014). PGY-3 were enrolled as trainees in this study after obtaining written consent. 2.2. Expert panel and RAND/UCLA method Fig. 1 is the process flow-chart to develop the curriculum by using RAND/UCLA method [13] with expert panel. The RAND/UCLA method is an iterative and anonymous survey used to establish consensus among participants. After several tasks related to basic surgical technique were selected [14– 21], panelists were asked to anonymously rank each task in terms of importance and necessity for novice residents using a Likert scale of 1 (no importance) to 9 (greatest possible importance). Each round of survey was administered online (https://www.surveymonkey.com, San Mateo, CA). The voting rounds were announced to consider the tasks from following three viewpoints; 1. The surgeon should be able to perform the technique as an otolaryngologist, 2. Inadequate technique can lead to patient harm, 3. The task is commensurate to a PGY-3 proficiency level. After the first voting round, an expert panel discussion whose moderator was the author was held to share results of the first voting round. Following discussion, the second anonymous voting round was held. The tasks whose median Likert scale were 7 to 9 (ap-
Please cite this article as: E. Sato, S. Mitani and N. Nishio et al., Development of proficiency-based knot-tying and suturing curriculum for otolaryngology residents: A pilot study, Auris Nasus Larynx, https:// doi.org/ 10.1016/ j.anl.2019.11.005
JID: ANL
ARTICLE IN PRESS
[mNS;December 21, 2019;0:45]
E. Sato, S. Mitani and N. Nishio et al. / Auris Nasus Larynx xxx (xxxx) xxx
3
Fig. 1. The process of developing curriculum. (A) Expert panel from various fields in otolaryngology department. General otolaryngologists were experts who perform any surgeries in otolaryngology. (B) Workflow of RAND/UCLA method process.
propriate) with disagreement index (DI) < 1 on the second voting round were decided as the final tasks. 2.3. Scoring of proficiency-based training curriculum Objective scoring was conducted on a time- and errorbased system, as developed and validated for general surgery skill exercises by previous reports [14,15,17]. Based on these previous studies, discrete errors were defined for each task, and cutoff times (maximal allowable task duration) were assigned. Performance goals were determined on the expert panel discussion, using the combined score of three board certificated head and neck surgeons as an expert score. The following scoring formula was used: score = (cutoff time) − (completion time) −10∗ (sum of errors) were converted to the scale of possible 100 points, as the mean score of the two repetitions of experts was defined as 100 points. Negative values were assigned a score of “0.” The composite score for each trainee was defined as the mean score of all tasks. Errors included as follows: accuracy (the ligature was tied outside of the colored segment(s) or that the needle was passed outside of inked targets), gap (distance between the ligature and tubing segment(s) or between incision edges), slippage (measured by cutting the tails to 1 cm, inserting a pointed pair of scissors within the loop of suture, and spreading; secure knots move > 3 mm), breakage (ligature is broken), movement (the base of the can is moved outside of the circular template) [14,15,17]. 2.4. Proficiency-based training Nine otolaryngology residents (PGY-3) trained with the developed curriculum and were evaluated. They completed an individually proctored pre-test by performing two consecu-
tive repetitions of each task. The validity of each task was confirmed by comparing the mean score of the two repetitions of trainees as a proctored pre-test and that of the three experts. After a simple lecture for each task by an attending head and neck surgeon, participants self-trained to proficiency with free access to the video tutorial (videos were installed on each trainee’s computer). Trainees self-scored each repetition during training and recorded all scores, and trainees were instructed to practice each task in order (from Task 1), until proficiency (100 points) was achieved on two consecutive repetitions, or for a maximum of 80 repetitions. The highest mean score of two consecutive repetition was recorded as the score during training, whose upper limit was determined to be 100 points. The skills lab educator was available for assistance with materials and models during training, attending head and neck surgeon checked their state of progress and answered their questions if they had once a week. The proctored post-test by completing two consecutive repetitions of each task for each trainee was separately underwent within one week after training of all tasks, and he or she underwent VAS as a self-assessment and completed a questionnaire. 2.5. Statistical analysis All data were tabulated, and statistical tests were performed with the JMP version 12 statistical software package (SAS Institute Inc., Cary, NC). In RAND/UCLA method, for each task, the median rating for appropriateness, interpercentile range (IPR), interpercentile range adjusted for symmetry (IPRAS), and disagreement index (DI) were calculated (DI = IPR/IPRAS) [13]. A median rating of 1 to 3 was considered to be “inappropriate,” 4 to 6 to be “uncertain,” and 7 to 9 to be “appropriate.” A DI value greater than or equal to 1 (DI ≥ 1) indicated a lack of consensus among the pan-
Please cite this article as: E. Sato, S. Mitani and N. Nishio et al., Development of proficiency-based knot-tying and suturing curriculum for otolaryngology residents: A pilot study, Auris Nasus Larynx, https:// doi.org/ 10.1016/ j.anl.2019.11.005
ARTICLE IN PRESS
JID: ANL
[mNS;December 21, 2019;0:45]
E. Sato, S. Mitani and N. Nishio et al. / Auris Nasus Larynx xxx (xxxx) xxx
4
Table 1 Task descriptions. Task1:
Task2:
Task3:
Task4:
Task5:
Task6:
Task7:
1-Handed knot-tying without tension Description
Errors Cutoff time Proficiency level Knot-tying with tension Description
Errors Cutoff time Proficiency level Atraumatic tie Description
Errors Cutoff time Proficiency level Tie at depth Description
Errors Cutoff time Proficiency level Tie on a pass Description
Errors Cutoff time Proficiency level Simple interrupted suturing Description
Errors Cutoff time Proficiency level Subcuticular interrupted suturing Description
Errors Cutoff time Proficiency level
Tie a 40 cm 3–0 silk ligature around 5 mm colored segment on thick single rubber tubing using only a 1-handed technique, 3 knots. Time starts with 1 end of the tie in each hand; time stops with completion of the final knot. Accuracy, gap, slippage, breakage 60 s 7 s with no errors Tie a 40 cm 3–0 silk ligature around 5 mm colored segments to approximate thick double rubber tubing set using 3 knots (either 1- or 2-handed). Time starts with 1 end of the tie in each hand; time stops with completion of the final knot. Accuracy, gap, slippage, breakage 60 s 8 s with no errors 3–0 Silk ligature is passed through the pop up tab ring (which is epoxied in the “up” position) on the 12 oz can filled to achieve a total weight of 90 g;. Using either a 1- or 2-handed technique, the trainee ties 3 knots within the 5 mm marked target area on the tab. Time starts with 1 end of the tie in each hand; time stops with completion of the final knot. Accuracy, gap, slippage, breakage, movement 60 s 9 s with no errors 3–0 silk ligature attached to the tip of a curved hemostat. The trainee first wraps the tie around a post (enclosed in a plastic cup) on the Ethicon knot-tying model; 1 knot is tied (either 1- or 2-handed) on the 2-mm marked target on the hook and the hemostat is released. The trainee then ties 2 additional knots. Time starts with the tie secured in the curved hemostat, outside the cup; time stops with completion of the final knot. Accuracy, gap, slippage, breakage 60 s 13 s with no errors 3–0 silk ligature attached to the tip of a curved hemostat. The trainee first wraps the tie around on the hook of the suture board; 1 knot is tied (either 1- or 2-handed) on the 2-mm marked target under the clip on the hook and the hemostat is released. The trainee then ties 2 additional knots. Time starts with the tie secured in the curved hemostat; time stops with the completion of the final knot. Accuracy, gap, slippage, breakage 60 s 10 s with no errors 45 cm, 3–0 synthetic surgical suture on tapered needle, pass needle through 2 inked targets (5 mm each from the wound) for simple interrupted suture, instrument tie surgeon’s knot then 2 squares (3 knots total). Time starts with the needle loaded in the needle driver in the dominant hand and the forceps held in the non-dominant hand; time stops with completion of the final knot. Accuracy, gap, slippage, breakage 60 s 21 s with no errors
45 cm, 3–0 synthetic surgical suture on tapered needle, model oriented parallel to bench edge, 1.5 cm long full thickness incision in the model (no inked targets), place a single buried subcuticular suture in a vertical fashion (perpendicular to the incision) with an instrument tie [surgeon’s knot then 2 squares (3 knots total)]. Time starts with the needle loaded in the needle driver in the dominant hand and the forceps held in the non-dominant hand; time stops with completion of the final knot. Gap, breakage 120 s 40 s with no errors
elists regarding the appropriateness of the tasks. The scale for VAS is 10-cm lines with descriptive anchors at each end. The evaluator places an ‘X’ on the line to indicate the score he or she assigns for the particular construct. The scores are
determined numerically by measuring where along the 10-cm line the mark was placed. Mann–Whitney U test was used to assess the relationship between trainee score and expert score of each task. Coefficient of variation (CV) was defined
Please cite this article as: E. Sato, S. Mitani and N. Nishio et al., Development of proficiency-based knot-tying and suturing curriculum for otolaryngology residents: A pilot study, Auris Nasus Larynx, https:// doi.org/ 10.1016/ j.anl.2019.11.005
JID: ANL
ARTICLE IN PRESS
[mNS;December 21, 2019;0:45]
E. Sato, S. Mitani and N. Nishio et al. / Auris Nasus Larynx xxx (xxxx) xxx
5
Fig. 2. Decided tasks. (A). A Suture board (Ethicon Inc., Cincinnati, OH) with 5 mm marked segments of tubing for Tasks 1: 1-Handed knot-tying without tension and 2: Knot-tying with tension. (B) Task3: Atraumatic tie. (C)※ Task4: Tie at depth. (D) The marked target for Task 5. (E)※ Task 5: Tie on a pass. (F) A wound closure pad (Ethicon Inc., Cincinnati, OH) with inked target for Task 6: Simple interrupted suturing and Task 7: Subcuticular interrupted suturing. ※ An inked string was used to demonstrate tasks clearly instead of 3–0 silk used during implementation of the curriculum.
as standard deviation (SD) divided by the mean value. The f-test was used to assess the difference between the CV of expert and that of trainee on each task. A paired t-test was used to assess the change between pre-test score and post-test score of trainees and the change between VAS before training and that after training. pValues of < 0.05 were regarded as statistically significant. All values reported are mean ± SD, unless otherwise noted. 3. Results 3.1. Seven tasks selected by RAND/UCLA method Eleven experts chosen as panelists had various subspecialty, i.e. 2 from otology, 2 from rhinology, 2 from laryngology, 2 from head and neck surgery and 3 were general otolaryngologists without specialty to reflect their opinion fairly (Fig. 1(A)). They joined expert panel from 3 institutions and all were board certificated otolaryngologists. Eighteen tasks of knot-tying or suturing were enumerated from general open surgical skills, and 3 tasks from otolaryngology surgery were added in this study. These 21 tasks, including 13 for knot-tying and 8 for suturing, were nominated and subjected to the anonymous voting rounds by panelists. At the first voting round, 11 tasks were evaluated as “appropriate” and no additional tasks were proposed. On the expert panel discussion, 3 tasks were integrated to 1 task and another
task was deleted because it was interpreted as the task had duplicated content with other tasks. This modification was approved at the second anonymous voting round. At the second voting round, from the previous 18 tasks, 7 tasks were evaluated as “appropriate” and DI < 1 and finally decided as tasks to use for training (Fig. 1(B)). Seven tasks were as follows: Task 1: 1-Handed knot-tying without tension, Task 2: Knot-tying with tension, Task 3: Atraumatic tie, Task 4: Tie at depth, Task 5: Tie on a pass, Task 6: Simple interrupted suturing, Task 7: Subcuticular interrupted suturing (Fig. 2). Donated suture board kits and wound closure pad (Ethicon Inc., Cincinnati, OH) were used with 40 cm 3–0 silk and 3–0 synthetic surgical suture for tasks. Table 1 shows the details of each task. An edited video illustrating appropriate technique as well as pitfalls to avoid was created for use as a tutorial (Supplemental Digital Data). 3.2. Trainees characteristics Nine novice otolaryngology residents (PGY-3) were enrolled as trainees between 2018 and 2019. The mean age was 27.2 (range: 26–30) years old, they were 6 males and 3 females, and 8 were right-handed and 1 was left-handed. The mean VAS before training of trainees as self-assessment was 1.2 ± 0.9. Trainees reported having served as surgeon of record in the mean 1.3 (range: 0–5) cases and first assistant in the mean 1.9 (range: 0–5) cases before training.
Please cite this article as: E. Sato, S. Mitani and N. Nishio et al., Development of proficiency-based knot-tying and suturing curriculum for otolaryngology residents: A pilot study, Auris Nasus Larynx, https:// doi.org/ 10.1016/ j.anl.2019.11.005
ARTICLE IN PRESS
JID: ANL
[mNS;December 21, 2019;0:45]
E. Sato, S. Mitani and N. Nishio et al. / Auris Nasus Larynx xxx (xxxx) xxx
6
Table 2 Expert scores vs trainee scores at pre-test. CV: coefficient of validation. p < 0.05 indicates statistical significance. Mean score (points)
Task1 Task2 Task3 Task4 Task5 Task6 Task7
CV
Expert
Trainee
p
Expert
Trainee
p
100 100 100 100 100 100 100
83 77 73 70 77 55 46
0.0015 0.0019 0.0009 0.0013 <0.0001 <0.0001 <0.0001
0.04 0.06 0.04 0.07 0.04 0.07 0.07
0.23 0.25 0.19 0.28 0.19 0.42 0.86
0.0042 0.0145 0.0065 0.0259 0.0002 0.0002 <0.0001
3.3. Validation of the tasks in experts vs. trainees Expert score on each task was derived by averaging the scores of 3 head and neck surgery specialists during twice consecutive repetitions and the score was defined as 100 points (Table 2). The data was suitably homogeneous and there were no outliers (> 2 SD). No trainee achieved expert score at pre-test for all 7 tasks. Scores of trainees were significantly lower in all 7 tasks (p < 0.01) compared to those of experts, in particular, approximately half score of tasks 7 (subcuticular interrupted suturing) was obtained in trainee (the mean score: 46 vs 100). Moreover, CV of trainee was significantly higher than that of expert (p < 0.05), which indicated the heterogeneity of the training score in trainee. Therefore, the validity has been proved as basic surgical trainings for these 7 tasks in PGY-3 otolaryngology residents. 3.4. Training results During training, trainees performed 187 ± 51 repetitions for all 7 tasks; the median training duration was 8.0 h (range: 2.0 to 52.5) during the training period whose median was 2.4 months (range: 1.5 to 2.7). On average, proficiency was reached after 25.0 ± 7.3 repetitions per task. Task 3 needed the most repetitions for 34. Trainee 8 reached the maximum number of 80 repetitions on Task 2. Trainee 9 reached the maximum number of 80 repetitions on Task 1, 2, 3, 5 and 7. The trainee score at post-test was significantly higher than that at pre-test for each task (p < 0.01 for all comparisons) (Fig. 3). The mean of composite scores between pre-test and posttest has statistical significance (68.6 ± 11.6 vs 95.9 ± 3.6, p < 0.01), suggesting a substantial improvement in training technique even at post-test; shown in Fig. 4(A). VAS as self-assessment was improved from pre-test to post-test (1.2 ± 0.9 vs 4.5 ± 1.4, p < 0.01) (Fig. 4(B)). According to questionnaire of self-ratings, 9 of 9 trainees (100%) answered that training was “beneficial”, “more comfortable with their surgical skills in OR”, and “training improved their operative skill”. On a 5-point scale, 5 trainees (55.6%) answered that the proficiency levels were “4: a little difficult”, 4 trainees (44.4%) felt “3: appropriate”, and none felt easy. One trainee (11.1%) felt that the burden of this curriculum was “4: a little”, 4 trainees (44.4%) felt “3: normal” and 4 trainees (44.4%) felt “2: less”. Additional comments from the trainees indicated that the video tutorial was useful
Fig. 3. Skill acquisition for each task. All tasks have significantly difference between pre-test score and post-test score. (p < 0.01 for all comparisons).
(the mean: 4.6 out of 5 points), and that additional help during their training was useful (the mean: 4.8 out of 5 points). Trainees served as surgeon of record in the mean 1.8 (range: 0–6) cases and first assistant in the mean 2.4 (range: 0–10) cases in OR as “on-the-job” training, and they did not take another “off-the-job” training during the implementation of this curriculum. 4. Discussion Basic surgical skills, such as knot-tying and suturing, are required for OHNS, essential for otolaryngologists’ career and critical for patients’ safety. In this study, 7 tasks were determined as basic surgical techniques for PGY-3 otolaryngology residents by using RAND/UCLA method with expert panel organized from 11 otolaryngology experts. Moreover, proctored post-test score and the VAS after training were significantly improved compared to pre-test score and the VAS before training in the curriculum based on these 7 tasks. These data indicated that curriculum with specific surgical training for novice otolaryngology residents was required and useful. Within the field of general surgery, there have been many reports about training for basic surgical skills such as knot-tying and suturing [14,15,17,19,20]. However, no study has focused on the specific surgical skills for OHNS. The RAND/UCLA method has been widely used as a technique to measure the appropriateness of medical and surgical interventions [22–26], and combines expert opinion with sci-
Please cite this article as: E. Sato, S. Mitani and N. Nishio et al., Development of proficiency-based knot-tying and suturing curriculum for otolaryngology residents: A pilot study, Auris Nasus Larynx, https:// doi.org/ 10.1016/ j.anl.2019.11.005
JID: ANL
ARTICLE IN PRESS
[mNS;December 21, 2019;0:45]
E. Sato, S. Mitani and N. Nishio et al. / Auris Nasus Larynx xxx (xxxx) xxx
7
Fig. 4. Skill acquisition for each trainee. (A) Transition of score. The mean of composite trainee scores at post-test was significantly improved compared to that at pre-test (68.6 ± 11.6 vs 95.9 ± 3.6, p < 0.01) (B) VAS (visual analogue scale) after training was significantly improved compared to that before training to 4.5 ± 1.4 after training (1.2 ± 0.9 vs 1.2 ± 0.9, p < 0.01) as self-assessment for their overall competence of knot-tying and suturing in operating room.
entific evidence by multiple voting round and expert panel discussions (Fig. 1). The benefit of this method is the process of determination for required surgical tasks using multidisciplinary panel as expert panel from various persons who actually are involved in patient treatment [13]. In this method, the anonymous voting rounds of participants are useful in the aspect of fairness for selecting training tasks, because their decisions are not influenced by one person’s opinion. For example, simple interrupted suturing (Task 6) or subcuticular interrupted suturing (Task 7) were considered as important tasks for novice otolaryngology residents and were finally selected as training tasks from the several suturing techniques in our study, whereas the other tasks, such as mattress suturing, were not considered as appropriate tasks which were important for general surgeons as previously reported [17,21]. This indicates that the strength of tension in head and neck region is lower than that in abdominal cavity and thus suturing techniques with strong tension are not required much for novice otolaryngology resident. Therefore, RAND/UCLA method would allow us to develop surgical education curriculum for novice otolaryngology residents and implementation of the present curriculum adjusted to otolaryngology would be great help for novice otolaryngology residents to obtain techniques required efficiently. In this study, we evaluated the surgical skills using proficiency-based training. Proficiency-based training has displayed greater effectiveness in providing training tailored to individual needs and allows for the acquisition of a more uniform skillset by learners when compared to time- or repetition-based training [27,28]. Significant improvement of
proficiency and the score of tasks was found in this study between pre-test and post-test (mean: 68.6 vs 95.9, p < 0.01) (Fig. 4). Furthermore, the average VAS score as selfassessment in OR also rose from 1.2 to 4.5 (p < 0.01) and all trainees realized the educational benefit according to the questionnaire. Most importantly, achievement of basic surgical principles is necessary, even for junior residents, to improve patient safety and prevent surgical complications. Junior residents who do not demonstrate adequate surgical skills and selfconfidence commonly lose their opportunity for “on-the-job” training, owing to patient safety or time constraints. Even in all 9 participates, they got a chance to perform surgery as surgeons with only the mean 1.8 cases and as first assistants with the mean 2.4 cases in OR as an “on-the-job” training. Despite the fact that “on-the-job” training (coincidental experience) is the basis of surgical skills for novice residents [1], “off-the-job” training (planned experience) with a proficiencybased method is essential for novice residents to acquire experienced surgical skills and to motivate them to learn surgical techniques. In the United States, as control of the otolaryngology internship training has shifted away from general surgery, many internships no longer include basic skills trainings that were routinely part of the general surgery curriculum. We believe that education of these fundamental skills remains crucial and can result in improving surgical efficiency and preventing unnecessary surgical complications. Further studies in multiple institutions should be conducted and the curriculum needs repetition of implement, evaluation and improvement.
Please cite this article as: E. Sato, S. Mitani and N. Nishio et al., Development of proficiency-based knot-tying and suturing curriculum for otolaryngology residents: A pilot study, Auris Nasus Larynx, https:// doi.org/ 10.1016/ j.anl.2019.11.005
ARTICLE IN PRESS
JID: ANL
[mNS;December 21, 2019;0:45]
E. Sato, S. Mitani and N. Nishio et al. / Auris Nasus Larynx xxx (xxxx) xxx
8
5. Conclusion In the present study, 7 tasks required for otolaryngologists were estimated in the basic technical skills by using RAND/UCLA method with expert panel. A proficiency-based training curriculum was developed based on the 7 tasks and was implemented with educational benefit. Declaration of Competing Interest All authors declare no conflict of interest. Acknowledgment We want to thank Naoya Nishida, Kaori Tanaka, Masato Teraoka, Masahiro Okada, Hirofumi Sei, Takashi Mukaigawa, Eriko Nishihara, and Yuki Irifune for performing this study. Supplementary material Supplementary material associated with this article can be found, in the online version, at doi:10.1016/j.anl.2019.11.005. References [1] Yokoyama H. Introducing off-the-job training to cardiovascular surgical residency training: a new era of developing competent cardiovascular surgeons. Surg Today 2019;49:300–10. [2] Wiet GJ, Stredney D, Kerwin T, Hittle B, Fernandez SA, Abdel-Rasoul M, et al. Virtual temporal bone dissection system: OSU virtual temporal bone system: development and testing. Laryngoscope 2012;122(Suppl 1):S1–12. [3] Fried MP, Sadoughi B, Gibber MJ, Jacobs JB, Lebowitz RA, Ross DA, et al. From virtual reality to the operating room: the endoscopic sinus surgery simulator experiment. Otolaryngol Head Neck Surg 2010;142:202–7. [4] Deutsch ES, Wiet GJ, Seidman M, Hussey HM, Malekzadeh S, Fried MP. Simulation activity in otolaryngology residencies. Otolaryngol Head Neck Surg 2015;153:193–201. [5] Newe A, Ganslandt T. Simplified generation of biomedical 3D surface model data for embedding into 3D portable document format (PDF) files for publication and education. PLoS One 2013;8:e79004. [6] Barsuk JH, Cohen ER, Feinglass J, McGaghie WC, Wayne DB. Unexpected collateral effects of simulation-based medical education. Acad Med 2011;86:1513–17. [7] Cohen ER, Feinglass J, Barsuk JH, Barnard C, O’Donnell A, McGaghie WC, et al. Cost savings from reduced catheter-related bloodstream infection after simulation-based education for residents in a medical intensive care unit. Simul Healthc 2010;5:98–102. [8] Lopreiato JO, Sawyer T. Simulation-based medical education in pediatrics. Acad Pediatr 2015;15:134–42. [9] Sroka G, Feldman LS, Vassiliou MC, Kaneva PA, Fayez R, Fried GM. Fundamentals of laparoscopic surgery simulator training to proficiency improves laparoscopic performance in the operating room-a randomized controlled trial. Am J Surg 2010;199:115–20. [10] Mizota T, Anton NE, Huffman EM, Guzman MJ, Lane F, Choi JN, et al. Development of a fundamentals of endoscopic surgery proficiency-based skills curriculum for general surgery residents. Surg Endosc 2019. In press.
[11] Andersen SAW, Nayahangan LJ, Konge L, Melchiors J. Identifying and prioritizing technical procedures in otorhinolaryngology for simulation-based training: a national needs assessment in Denmark. Eur Arch Otorhinolaryngol 2019;276:1517–24. [12] Cass ND, Okland TS, Rodriguez K, Mann SE. Otolaryngology education: recent trends in publication. Otolaryngol Head Neck Surg 2017;156:1124–9. [13] Fitch K, Bernstein SJ, Aguilar MD, Burnand B, LaCalle JR, Lazaro P, et al. The RAND/UCLA appropriateness method user’s manual. RAND; 2001. [14] Scott DJ, Goova MT, Tesfay ST. A cost-effective proficiency-based knot-tying and suturing curriculum for residency programs. J Surg Res 2007;141:7–15. [15] Goova MT, Hollett LA, Tesfay ST, Gala RB, Puzziferri N, Kehdy FJ, et al. Implementation, construct validity, and benefit of a proficiency-based knot-tying and suturing curriculum. J Surg Educ 2008;65:309–15. [16] Gershuni V, Woodhouse J, Brunt LM. Retention of suturing and knot– tying skills in senior medical students after proficiency-based training: results of a prospective, randomized trial. Surgery 2013;154:823–9 discussion 9-30. [17] Mashaud LB, Arain NA, Hogg DC, Scott DJ. Development, validation, and implementation of a cost-effective intermediate-level proficiency-based knot-tying and suturing curriculum for surgery residents. J Surg Educ 2013;70:193–9. [18] Huang E, Chern H, O’Sullivan P, Cook B, McDonald E, Palmer B, et al. A better way to teach knot tying: a randomized controlled trial comparing the kinesthetic and traditional methods. Am J Surg 2014;208:690–4. [19] Huang E, Vaughn CJ, Chern H, O’Sullivan P, Kim E. An objective assessment tool for basic surgical knot-tying skills. J Surg Educ 2015;72:572–6. [20] Wade TJ, Lorbeer K, Awad MM, Woodhouse J, DeClue A, Brunt LM. Outcomes of a proficiency-based skills curriculum at the beginning of the fourth year for senior medical students entering surgery. Surgery 2015;158:962–9 discussion 9-71. [21] Pender C, Kiselov V, Yu Q, Mooney J, Greiffenstein P, Paige JT. All for knots: evaluating the effectiveness of a proficiency-driven, simulation-based knot tying and suturing curriculum for medical students during their third-year surgery clerkship. Am J Surg 2017;213:362–70. [22] Shekelle PG, Park RE, Kahan JP, Leape LL, Kamberg CJ, Bernstein SJ. Sensitivity and specificity of the RAND/UCLA appropriateness method to identify the overuse and underuse of coronary revascularization and hysterectomy. J Clin Epidemiol 2001;54:1004–10. [23] Bennett S, Tinmouth A, McIsaac DI, English S, Hebert PC, Karanicolas AJ, et al. Ottawa criteria for appropriate transfusions in hepatectomy using the RAND/UCLA appropriateness method. Ann Surg 2018;267:766–74. [24] Teoh AYB, Dhir V, Kida M, Yasuda I, Jin ZD, Seo DW, et al. Consensus guidelines on the optimal management in interventional EUS procedures: results from the Asian EUS group RAND/UCLA expert panel. Gut 2018;67:1209–28. [25] Maverakis E, Ma C, Shinkai K, Fiorentino D, Callen JP, Wollina U, et al. Diagnostic criteria of ulcerative pyoderma gangrenosum: a Delphi consensus of international experts. JAMA Dermatol 2018;154:461–6. [26] Lawson EH, Gibbons MM, Ko CY, Shekelle PG. The appropriateness method has acceptable reliability and validity for assessing overuse and underuse of surgical procedures. J Clin Epidemiol 2012;65:1133–43. [27] Ritter EM, Scott DJ. Design of a proficiency-based skills training curriculum for the fundamentals of laparoscopic surgery. Surg Innov 2007;14:107–12. [28] Stefanidis D, Acker C, Heniford BT. Proficiency-based laparoscopic simulator training leads to improved operating room skill that is resistant to decay. Surg Innov 2008;15:69–73.
Please cite this article as: E. Sato, S. Mitani and N. Nishio et al., Development of proficiency-based knot-tying and suturing curriculum for otolaryngology residents: A pilot study, Auris Nasus Larynx, https:// doi.org/ 10.1016/ j.anl.2019.11.005
JID: ANL
[mNS;December 21, 2019;0:45]
Auris Nasus Larynx xxx (xxxx) xxx Contents lists available at ScienceDirect
Auris Nasus Larynx journal homepage: www.elsevier.com/locate/anl
Development of proficiency-based knot-tying and suturing curriculum for otolaryngology residents: A pilot study Eriko Sato a,b,1, Sohei Mitani a,c,1,∗, Naoki Nishio c,d, Takashi Kitani a, Tomoyoshi Sanada a, Toru Ugumori a,e, F. Christopher Holsinger c, Fred M. Baik c, Naohito Hato a a Department
of Otolaryngology-Head and Neck Surgery, Ehime University Graduate School of Medicine, Shitsukawa, Toon, Ehime 791−0295, Japan b Division of Gastrointestinal Oncology, Shizuoka Cancer Center, 1007 Shimonagakubo, Nagaizumi, Sunto District, Shizuoka 411-0934, Japan c Division of Head and Neck Surgery, Department of Otolaryngology, Stanford University, 875 Blake Wilbur Drive, Palo Alto, CA 94305, United States d Department of Otorhinolaryngology, Nagoya University Graduate School of Medicine, 65 Tsurumai-cho, Showa Ward, Nagoya, Aichi 466-8560, Japan e Ugumori ENT Clinic, 3 Chome-10-25 Yougonishi, Matsuyama, Ehime 790-0046, Japan
A R T I C L E
I N F O
Article history: Received 1 October 2019 Accepted 27 November 2019 Available online xxx Keywords: Basic surgical skill Knot-tying and suturing curriculum RAND/UCLA method Simulation training Proficiency-based training
A B S T R A C T
Objective: Basic surgical skills such as knot-tying and suturing are important for all otolaryngologists, regardless of subspecialty. The present study was undertaken in order to assess basic surgical techniques such as knot-tying and suturing required for novice otolaryngology residents with taking the variety of subspecialties into consideration, and evaluate the impact of a proficiency-based training curriculum based on these techniques. Methods: A prospective study was performed for developing of proficiency-based knot-tying and suturing curriculum for otolaryngology residents in the third post-graduate year (PGY3). The proficiency-based training curriculum was developed based on the tasks selected by RAND/UCLA method with expert panel, which is an iterative and anonymous survey used to establish consensus among participants. Expert panelists were selected from various divisions to reflect variety of their subspecialties. PGY-3 residents trained with the developed curriculum that included proctored pre-test, self-training to proficiency, and proctored post-test. Visual analogue scale (VAS) of trainees’ overall competence in the operating room was self-assessed by each resident, before and after completing the training curriculum. Results: Nine PGY-3 residents were enrolled as trainees. Eleven experts chosen as panelists had various subspecialty, including 2 from otology, 2 from rhinology, 2 from laryngology, 2 from head and neck surgery, and 3 from general otolaryngology. Seven tasks were selected from RAND/UCLA method and used to develop the curriculum. Trainee scores at pre-test were significantly lower than expert scores for all 7 tasks (p < 0.01) and each coefficient of variation of trainee score was larger than that of expert score (p < 0.05), supporting construct validity. The mean of composite scores between pre-test and post-test had statistical significance
∗
Corresponding author at: Department of Otolaryngology-Head and Neck Surgery, Ehime University Graduate School of Medicine, Shitsukawa, Toon, Ehime 791−0295, Japan. E-mail address: [email protected] (S. Mitani). 1 These authors contributed equally to this work. https://doi.org/10.1016/j.anl.2019.11.005 0385-8146/© 2019 Oto-Rhino-Laryngological Society of Japan Inc. Published by Elsevier B.V. All rights reserved. Please cite this article as: E. Sato, S. Mitani and N. Nishio et al., Development of proficiency-based knot-tying and suturing curriculum for otolaryngology residents: A pilot study, Auris Nasus Larynx, https:// doi.org/ 10.1016/ j.anl.2019.11.005
JID: ANL
ARTICLE IN PRESS
[mNS;December 21, 2019;0:45]
E. Sato, S. Mitani and N. Nishio et al. / Auris Nasus Larynx xxx (xxxx) xxx
2
(68.6 ± 11.6 vs 95.9 ± 3.6, p < 0.01), documenting substantial improvement after training. Self-assessment VAS was also improved pre- to post-training (1.2 ± 0.9 vs 4.5 ± 1.4, p < 0.01). A follow-up questionnaire showed that trainees felt the educational curriculum to be beneficial. Conclusion: In the present study, seven basic technical skills were selected using the RAND/UCLA method and used to create a proficiency-based training curriculum. Our results indicate that this curriculum significantly improves proficiency of basic surgical skills of junior otolaryngology residents. © 2019 Oto-Rhino-Laryngological Society of Japan Inc. Published by Elsevier B.V. All rights reserved.
1 Introduction
2. Materials and method
Surgeons are required to move their hands fluently and efficiently at an instinctive level, together with accuracy, speed and flow [1]. As work hours have declined during residency, simulation training has become a more important and valuable means to achieve proficiency by supporting early acquisition of complex skills, knowledge and behaviors. Ultimately, this may improve patient-related outcomes, patient safety, cost savings [2–4], as well enhance skill acquisition among other trainees [5–8]. Proficiency-based simulation training curriculums have been developed in many general surgical fields, e.g. for laparoscopic suturing [9]. This proficiency – based training using expert – derived performance goals is currently considered the optimal simulation training method, because simulated tasks allow for objective assessment of technical skill [10]. Surgical field in otolaryngology-head and neck surgery (OHNS) is often narrow or close to the surface of body, in sharp contrast to that of “open” or laparoscopic general surgery. Thus, several different skill-sets are required. Moreover, these basic OHNS skills span not only “open” head and neck surgery, but also otology, rhinology and laryngology. One national survey reported that basic surgical skills (knot-tying and suturing) were the most prioritized technical procedures amongst a group of 50 unique procedures and technical skills (such as Flexible fiber pharyngo-laryngoscopy, Fine needle aspiration) [11]. Recent trends in otolaryngology reflect an increasing focus on surgical education and most articles are currently published mainly in otology (mastoidectomy) with 40%, and in rhinology (endoscopic sinus surgery) with 22% [12]. Regardless of variety of subspecialties in otolaryngology, all otolaryngologists need basic surgical skills and appropriately perform these techniques in serious and critical situations, such as rapid bleeding after tonsillectomy, tracheostomy, or hemorrhage after neck surgeries. However, there is no report mentioning open surgical skills for otolaryngology surgeries. The objective of this study is to estimate basic surgical techniques such as knot-tying and suturing for novice otolaryngology residents by using expert panel, in which several experts were selected from whole otolaryngology. Second aim was to evaluate the educational benefit for novice otolaryngology residents of the curriculum based on these techniques.
2.1. Study design A prospective study was performed in an academic university hospital of a proficiency-based knot-tying and suturing curriculum. The curriculum was developed based on the tasks selected by the RAND/UCLA method with expert panel (Fig. 1). PGY-3 residents trained with the developed curriculum that included proctored pre-test, selftraining to proficiency, and proctored post-test. Visual analogue scale (VAS) of trainees’ overall competence in operating room (OR) was rated as self-assessment by themselves before and after training. A questionnaire was taken after training to evaluate perceived benefit, burden, contents of curriculums, their experience in OR as “on-the-job” training and another “off-the-job” training during implementation of the curriculum. This study was approved by Institutional Review Board at the University (registration number: 1903014). PGY-3 were enrolled as trainees in this study after obtaining written consent. 2.2. Expert panel and RAND/UCLA method Fig. 1 is the process flow-chart to develop the curriculum by using RAND/UCLA method [13] with expert panel. The RAND/UCLA method is an iterative and anonymous survey used to establish consensus among participants. After several tasks related to basic surgical technique were selected [14– 21], panelists were asked to anonymously rank each task in terms of importance and necessity for novice residents using a Likert scale of 1 (no importance) to 9 (greatest possible importance). Each round of survey was administered online (https://www.surveymonkey.com, San Mateo, CA). The voting rounds were announced to consider the tasks from following three viewpoints; 1. The surgeon should be able to perform the technique as an otolaryngologist, 2. Inadequate technique can lead to patient harm, 3. The task is commensurate to a PGY-3 proficiency level. After the first voting round, an expert panel discussion whose moderator was the author was held to share results of the first voting round. Following discussion, the second anonymous voting round was held. The tasks whose median Likert scale were 7 to 9 (ap-
Please cite this article as: E. Sato, S. Mitani and N. Nishio et al., Development of proficiency-based knot-tying and suturing curriculum for otolaryngology residents: A pilot study, Auris Nasus Larynx, https:// doi.org/ 10.1016/ j.anl.2019.11.005
JID: ANL
ARTICLE IN PRESS
[mNS;December 21, 2019;0:45]
E. Sato, S. Mitani and N. Nishio et al. / Auris Nasus Larynx xxx (xxxx) xxx
3
Fig. 1. The process of developing curriculum. (A) Expert panel from various fields in otolaryngology department. General otolaryngologists were experts who perform any surgeries in otolaryngology. (B) Workflow of RAND/UCLA method process.
propriate) with disagreement index (DI) < 1 on the second voting round were decided as the final tasks. 2.3. Scoring of proficiency-based training curriculum Objective scoring was conducted on a time- and errorbased system, as developed and validated for general surgery skill exercises by previous reports [14,15,17]. Based on these previous studies, discrete errors were defined for each task, and cutoff times (maximal allowable task duration) were assigned. Performance goals were determined on the expert panel discussion, using the combined score of three board certificated head and neck surgeons as an expert score. The following scoring formula was used: score = (cutoff time) − (completion time) −10∗ (sum of errors) were converted to the scale of possible 100 points, as the mean score of the two repetitions of experts was defined as 100 points. Negative values were assigned a score of “0.” The composite score for each trainee was defined as the mean score of all tasks. Errors included as follows: accuracy (the ligature was tied outside of the colored segment(s) or that the needle was passed outside of inked targets), gap (distance between the ligature and tubing segment(s) or between incision edges), slippage (measured by cutting the tails to 1 cm, inserting a pointed pair of scissors within the loop of suture, and spreading; secure knots move > 3 mm), breakage (ligature is broken), movement (the base of the can is moved outside of the circular template) [14,15,17]. 2.4. Proficiency-based training Nine otolaryngology residents (PGY-3) trained with the developed curriculum and were evaluated. They completed an individually proctored pre-test by performing two consecu-
tive repetitions of each task. The validity of each task was confirmed by comparing the mean score of the two repetitions of trainees as a proctored pre-test and that of the three experts. After a simple lecture for each task by an attending head and neck surgeon, participants self-trained to proficiency with free access to the video tutorial (videos were installed on each trainee’s computer). Trainees self-scored each repetition during training and recorded all scores, and trainees were instructed to practice each task in order (from Task 1), until proficiency (100 points) was achieved on two consecutive repetitions, or for a maximum of 80 repetitions. The highest mean score of two consecutive repetition was recorded as the score during training, whose upper limit was determined to be 100 points. The skills lab educator was available for assistance with materials and models during training, attending head and neck surgeon checked their state of progress and answered their questions if they had once a week. The proctored post-test by completing two consecutive repetitions of each task for each trainee was separately underwent within one week after training of all tasks, and he or she underwent VAS as a self-assessment and completed a questionnaire. 2.5. Statistical analysis All data were tabulated, and statistical tests were performed with the JMP version 12 statistical software package (SAS Institute Inc., Cary, NC). In RAND/UCLA method, for each task, the median rating for appropriateness, interpercentile range (IPR), interpercentile range adjusted for symmetry (IPRAS), and disagreement index (DI) were calculated (DI = IPR/IPRAS) [13]. A median rating of 1 to 3 was considered to be “inappropriate,” 4 to 6 to be “uncertain,” and 7 to 9 to be “appropriate.” A DI value greater than or equal to 1 (DI ≥ 1) indicated a lack of consensus among the pan-
Please cite this article as: E. Sato, S. Mitani and N. Nishio et al., Development of proficiency-based knot-tying and suturing curriculum for otolaryngology residents: A pilot study, Auris Nasus Larynx, https:// doi.org/ 10.1016/ j.anl.2019.11.005
ARTICLE IN PRESS
JID: ANL
[mNS;December 21, 2019;0:45]
E. Sato, S. Mitani and N. Nishio et al. / Auris Nasus Larynx xxx (xxxx) xxx
4
Table 1 Task descriptions. Task1:
Task2:
Task3:
Task4:
Task5:
Task6:
Task7:
1-Handed knot-tying without tension Description
Errors Cutoff time Proficiency level Knot-tying with tension Description
Errors Cutoff time Proficiency level Atraumatic tie Description
Errors Cutoff time Proficiency level Tie at depth Description
Errors Cutoff time Proficiency level Tie on a pass Description
Errors Cutoff time Proficiency level Simple interrupted suturing Description
Errors Cutoff time Proficiency level Subcuticular interrupted suturing Description
Errors Cutoff time Proficiency level
Tie a 40 cm 3–0 silk ligature around 5 mm colored segment on thick single rubber tubing using only a 1-handed technique, 3 knots. Time starts with 1 end of the tie in each hand; time stops with completion of the final knot. Accuracy, gap, slippage, breakage 60 s 7 s with no errors Tie a 40 cm 3–0 silk ligature around 5 mm colored segments to approximate thick double rubber tubing set using 3 knots (either 1- or 2-handed). Time starts with 1 end of the tie in each hand; time stops with completion of the final knot. Accuracy, gap, slippage, breakage 60 s 8 s with no errors 3–0 Silk ligature is passed through the pop up tab ring (which is epoxied in the “up” position) on the 12 oz can filled to achieve a total weight of 90 g;. Using either a 1- or 2-handed technique, the trainee ties 3 knots within the 5 mm marked target area on the tab. Time starts with 1 end of the tie in each hand; time stops with completion of the final knot. Accuracy, gap, slippage, breakage, movement 60 s 9 s with no errors 3–0 silk ligature attached to the tip of a curved hemostat. The trainee first wraps the tie around a post (enclosed in a plastic cup) on the Ethicon knot-tying model; 1 knot is tied (either 1- or 2-handed) on the 2-mm marked target on the hook and the hemostat is released. The trainee then ties 2 additional knots. Time starts with the tie secured in the curved hemostat, outside the cup; time stops with completion of the final knot. Accuracy, gap, slippage, breakage 60 s 13 s with no errors 3–0 silk ligature attached to the tip of a curved hemostat. The trainee first wraps the tie around on the hook of the suture board; 1 knot is tied (either 1- or 2-handed) on the 2-mm marked target under the clip on the hook and the hemostat is released. The trainee then ties 2 additional knots. Time starts with the tie secured in the curved hemostat; time stops with the completion of the final knot. Accuracy, gap, slippage, breakage 60 s 10 s with no errors 45 cm, 3–0 synthetic surgical suture on tapered needle, pass needle through 2 inked targets (5 mm each from the wound) for simple interrupted suture, instrument tie surgeon’s knot then 2 squares (3 knots total). Time starts with the needle loaded in the needle driver in the dominant hand and the forceps held in the non-dominant hand; time stops with completion of the final knot. Accuracy, gap, slippage, breakage 60 s 21 s with no errors
45 cm, 3–0 synthetic surgical suture on tapered needle, model oriented parallel to bench edge, 1.5 cm long full thickness incision in the model (no inked targets), place a single buried subcuticular suture in a vertical fashion (perpendicular to the incision) with an instrument tie [surgeon’s knot then 2 squares (3 knots total)]. Time starts with the needle loaded in the needle driver in the dominant hand and the forceps held in the non-dominant hand; time stops with completion of the final knot. Gap, breakage 120 s 40 s with no errors
elists regarding the appropriateness of the tasks. The scale for VAS is 10-cm lines with descriptive anchors at each end. The evaluator places an ‘X’ on the line to indicate the score he or she assigns for the particular construct. The scores are
determined numerically by measuring where along the 10-cm line the mark was placed. Mann–Whitney U test was used to assess the relationship between trainee score and expert score of each task. Coefficient of variation (CV) was defined
Please cite this article as: E. Sato, S. Mitani and N. Nishio et al., Development of proficiency-based knot-tying and suturing curriculum for otolaryngology residents: A pilot study, Auris Nasus Larynx, https:// doi.org/ 10.1016/ j.anl.2019.11.005
JID: ANL
ARTICLE IN PRESS
[mNS;December 21, 2019;0:45]
E. Sato, S. Mitani and N. Nishio et al. / Auris Nasus Larynx xxx (xxxx) xxx
5
Fig. 2. Decided tasks. (A). A Suture board (Ethicon Inc., Cincinnati, OH) with 5 mm marked segments of tubing for Tasks 1: 1-Handed knot-tying without tension and 2: Knot-tying with tension. (B) Task3: Atraumatic tie. (C)※ Task4: Tie at depth. (D) The marked target for Task 5. (E)※ Task 5: Tie on a pass. (F) A wound closure pad (Ethicon Inc., Cincinnati, OH) with inked target for Task 6: Simple interrupted suturing and Task 7: Subcuticular interrupted suturing. ※ An inked string was used to demonstrate tasks clearly instead of 3–0 silk used during implementation of the curriculum.
as standard deviation (SD) divided by the mean value. The f-test was used to assess the difference between the CV of expert and that of trainee on each task. A paired t-test was used to assess the change between pre-test score and post-test score of trainees and the change between VAS before training and that after training. pValues of < 0.05 were regarded as statistically significant. All values reported are mean ± SD, unless otherwise noted. 3. Results 3.1. Seven tasks selected by RAND/UCLA method Eleven experts chosen as panelists had various subspecialty, i.e. 2 from otology, 2 from rhinology, 2 from laryngology, 2 from head and neck surgery and 3 were general otolaryngologists without specialty to reflect their opinion fairly (Fig. 1(A)). They joined expert panel from 3 institutions and all were board certificated otolaryngologists. Eighteen tasks of knot-tying or suturing were enumerated from general open surgical skills, and 3 tasks from otolaryngology surgery were added in this study. These 21 tasks, including 13 for knot-tying and 8 for suturing, were nominated and subjected to the anonymous voting rounds by panelists. At the first voting round, 11 tasks were evaluated as “appropriate” and no additional tasks were proposed. On the expert panel discussion, 3 tasks were integrated to 1 task and another
task was deleted because it was interpreted as the task had duplicated content with other tasks. This modification was approved at the second anonymous voting round. At the second voting round, from the previous 18 tasks, 7 tasks were evaluated as “appropriate” and DI < 1 and finally decided as tasks to use for training (Fig. 1(B)). Seven tasks were as follows: Task 1: 1-Handed knot-tying without tension, Task 2: Knot-tying with tension, Task 3: Atraumatic tie, Task 4: Tie at depth, Task 5: Tie on a pass, Task 6: Simple interrupted suturing, Task 7: Subcuticular interrupted suturing (Fig. 2). Donated suture board kits and wound closure pad (Ethicon Inc., Cincinnati, OH) were used with 40 cm 3–0 silk and 3–0 synthetic surgical suture for tasks. Table 1 shows the details of each task. An edited video illustrating appropriate technique as well as pitfalls to avoid was created for use as a tutorial (Supplemental Digital Data). 3.2. Trainees characteristics Nine novice otolaryngology residents (PGY-3) were enrolled as trainees between 2018 and 2019. The mean age was 27.2 (range: 26–30) years old, they were 6 males and 3 females, and 8 were right-handed and 1 was left-handed. The mean VAS before training of trainees as self-assessment was 1.2 ± 0.9. Trainees reported having served as surgeon of record in the mean 1.3 (range: 0–5) cases and first assistant in the mean 1.9 (range: 0–5) cases before training.
Please cite this article as: E. Sato, S. Mitani and N. Nishio et al., Development of proficiency-based knot-tying and suturing curriculum for otolaryngology residents: A pilot study, Auris Nasus Larynx, https:// doi.org/ 10.1016/ j.anl.2019.11.005
ARTICLE IN PRESS
JID: ANL
[mNS;December 21, 2019;0:45]
E. Sato, S. Mitani and N. Nishio et al. / Auris Nasus Larynx xxx (xxxx) xxx
6
Table 2 Expert scores vs trainee scores at pre-test. CV: coefficient of validation. p < 0.05 indicates statistical significance. Mean score (points)
Task1 Task2 Task3 Task4 Task5 Task6 Task7
CV
Expert
Trainee
p
Expert
Trainee
p
100 100 100 100 100 100 100
83 77 73 70 77 55 46
0.0015 0.0019 0.0009 0.0013 <0.0001 <0.0001 <0.0001
0.04 0.06 0.04 0.07 0.04 0.07 0.07
0.23 0.25 0.19 0.28 0.19 0.42 0.86
0.0042 0.0145 0.0065 0.0259 0.0002 0.0002 <0.0001
3.3. Validation of the tasks in experts vs. trainees Expert score on each task was derived by averaging the scores of 3 head and neck surgery specialists during twice consecutive repetitions and the score was defined as 100 points (Table 2). The data was suitably homogeneous and there were no outliers (> 2 SD). No trainee achieved expert score at pre-test for all 7 tasks. Scores of trainees were significantly lower in all 7 tasks (p < 0.01) compared to those of experts, in particular, approximately half score of tasks 7 (subcuticular interrupted suturing) was obtained in trainee (the mean score: 46 vs 100). Moreover, CV of trainee was significantly higher than that of expert (p < 0.05), which indicated the heterogeneity of the training score in trainee. Therefore, the validity has been proved as basic surgical trainings for these 7 tasks in PGY-3 otolaryngology residents. 3.4. Training results During training, trainees performed 187 ± 51 repetitions for all 7 tasks; the median training duration was 8.0 h (range: 2.0 to 52.5) during the training period whose median was 2.4 months (range: 1.5 to 2.7). On average, proficiency was reached after 25.0 ± 7.3 repetitions per task. Task 3 needed the most repetitions for 34. Trainee 8 reached the maximum number of 80 repetitions on Task 2. Trainee 9 reached the maximum number of 80 repetitions on Task 1, 2, 3, 5 and 7. The trainee score at post-test was significantly higher than that at pre-test for each task (p < 0.01 for all comparisons) (Fig. 3). The mean of composite scores between pre-test and posttest has statistical significance (68.6 ± 11.6 vs 95.9 ± 3.6, p < 0.01), suggesting a substantial improvement in training technique even at post-test; shown in Fig. 4(A). VAS as self-assessment was improved from pre-test to post-test (1.2 ± 0.9 vs 4.5 ± 1.4, p < 0.01) (Fig. 4(B)). According to questionnaire of self-ratings, 9 of 9 trainees (100%) answered that training was “beneficial”, “more comfortable with their surgical skills in OR”, and “training improved their operative skill”. On a 5-point scale, 5 trainees (55.6%) answered that the proficiency levels were “4: a little difficult”, 4 trainees (44.4%) felt “3: appropriate”, and none felt easy. One trainee (11.1%) felt that the burden of this curriculum was “4: a little”, 4 trainees (44.4%) felt “3: normal” and 4 trainees (44.4%) felt “2: less”. Additional comments from the trainees indicated that the video tutorial was useful
Fig. 3. Skill acquisition for each task. All tasks have significantly difference between pre-test score and post-test score. (p < 0.01 for all comparisons).
(the mean: 4.6 out of 5 points), and that additional help during their training was useful (the mean: 4.8 out of 5 points). Trainees served as surgeon of record in the mean 1.8 (range: 0–6) cases and first assistant in the mean 2.4 (range: 0–10) cases in OR as “on-the-job” training, and they did not take another “off-the-job” training during the implementation of this curriculum. 4. Discussion Basic surgical skills, such as knot-tying and suturing, are required for OHNS, essential for otolaryngologists’ career and critical for patients’ safety. In this study, 7 tasks were determined as basic surgical techniques for PGY-3 otolaryngology residents by using RAND/UCLA method with expert panel organized from 11 otolaryngology experts. Moreover, proctored post-test score and the VAS after training were significantly improved compared to pre-test score and the VAS before training in the curriculum based on these 7 tasks. These data indicated that curriculum with specific surgical training for novice otolaryngology residents was required and useful. Within the field of general surgery, there have been many reports about training for basic surgical skills such as knot-tying and suturing [14,15,17,19,20]. However, no study has focused on the specific surgical skills for OHNS. The RAND/UCLA method has been widely used as a technique to measure the appropriateness of medical and surgical interventions [22–26], and combines expert opinion with sci-
Please cite this article as: E. Sato, S. Mitani and N. Nishio et al., Development of proficiency-based knot-tying and suturing curriculum for otolaryngology residents: A pilot study, Auris Nasus Larynx, https:// doi.org/ 10.1016/ j.anl.2019.11.005
JID: ANL
ARTICLE IN PRESS
[mNS;December 21, 2019;0:45]
E. Sato, S. Mitani and N. Nishio et al. / Auris Nasus Larynx xxx (xxxx) xxx
7
Fig. 4. Skill acquisition for each trainee. (A) Transition of score. The mean of composite trainee scores at post-test was significantly improved compared to that at pre-test (68.6 ± 11.6 vs 95.9 ± 3.6, p < 0.01) (B) VAS (visual analogue scale) after training was significantly improved compared to that before training to 4.5 ± 1.4 after training (1.2 ± 0.9 vs 1.2 ± 0.9, p < 0.01) as self-assessment for their overall competence of knot-tying and suturing in operating room.
entific evidence by multiple voting round and expert panel discussions (Fig. 1). The benefit of this method is the process of determination for required surgical tasks using multidisciplinary panel as expert panel from various persons who actually are involved in patient treatment [13]. In this method, the anonymous voting rounds of participants are useful in the aspect of fairness for selecting training tasks, because their decisions are not influenced by one person’s opinion. For example, simple interrupted suturing (Task 6) or subcuticular interrupted suturing (Task 7) were considered as important tasks for novice otolaryngology residents and were finally selected as training tasks from the several suturing techniques in our study, whereas the other tasks, such as mattress suturing, were not considered as appropriate tasks which were important for general surgeons as previously reported [17,21]. This indicates that the strength of tension in head and neck region is lower than that in abdominal cavity and thus suturing techniques with strong tension are not required much for novice otolaryngology resident. Therefore, RAND/UCLA method would allow us to develop surgical education curriculum for novice otolaryngology residents and implementation of the present curriculum adjusted to otolaryngology would be great help for novice otolaryngology residents to obtain techniques required efficiently. In this study, we evaluated the surgical skills using proficiency-based training. Proficiency-based training has displayed greater effectiveness in providing training tailored to individual needs and allows for the acquisition of a more uniform skillset by learners when compared to time- or repetition-based training [27,28]. Significant improvement of
proficiency and the score of tasks was found in this study between pre-test and post-test (mean: 68.6 vs 95.9, p < 0.01) (Fig. 4). Furthermore, the average VAS score as selfassessment in OR also rose from 1.2 to 4.5 (p < 0.01) and all trainees realized the educational benefit according to the questionnaire. Most importantly, achievement of basic surgical principles is necessary, even for junior residents, to improve patient safety and prevent surgical complications. Junior residents who do not demonstrate adequate surgical skills and selfconfidence commonly lose their opportunity for “on-the-job” training, owing to patient safety or time constraints. Even in all 9 participates, they got a chance to perform surgery as surgeons with only the mean 1.8 cases and as first assistants with the mean 2.4 cases in OR as an “on-the-job” training. Despite the fact that “on-the-job” training (coincidental experience) is the basis of surgical skills for novice residents [1], “off-the-job” training (planned experience) with a proficiencybased method is essential for novice residents to acquire experienced surgical skills and to motivate them to learn surgical techniques. In the United States, as control of the otolaryngology internship training has shifted away from general surgery, many internships no longer include basic skills trainings that were routinely part of the general surgery curriculum. We believe that education of these fundamental skills remains crucial and can result in improving surgical efficiency and preventing unnecessary surgical complications. Further studies in multiple institutions should be conducted and the curriculum needs repetition of implement, evaluation and improvement.
Please cite this article as: E. Sato, S. Mitani and N. Nishio et al., Development of proficiency-based knot-tying and suturing curriculum for otolaryngology residents: A pilot study, Auris Nasus Larynx, https:// doi.org/ 10.1016/ j.anl.2019.11.005
ARTICLE IN PRESS
JID: ANL
[mNS;December 21, 2019;0:45]
E. Sato, S. Mitani and N. Nishio et al. / Auris Nasus Larynx xxx (xxxx) xxx
8
5. Conclusion In the present study, 7 tasks required for otolaryngologists were estimated in the basic technical skills by using RAND/UCLA method with expert panel. A proficiency-based training curriculum was developed based on the 7 tasks and was implemented with educational benefit. Declaration of Competing Interest All authors declare no conflict of interest. Acknowledgment We want to thank Naoya Nishida, Kaori Tanaka, Masato Teraoka, Masahiro Okada, Hirofumi Sei, Takashi Mukaigawa, Eriko Nishihara, and Yuki Irifune for performing this study. Supplementary material Supplementary material associated with this article can be found, in the online version, at doi:10.1016/j.anl.2019.11.005. References [1] Yokoyama H. Introducing off-the-job training to cardiovascular surgical residency training: a new era of developing competent cardiovascular surgeons. Surg Today 2019;49:300–10. [2] Wiet GJ, Stredney D, Kerwin T, Hittle B, Fernandez SA, Abdel-Rasoul M, et al. Virtual temporal bone dissection system: OSU virtual temporal bone system: development and testing. Laryngoscope 2012;122(Suppl 1):S1–12. [3] Fried MP, Sadoughi B, Gibber MJ, Jacobs JB, Lebowitz RA, Ross DA, et al. From virtual reality to the operating room: the endoscopic sinus surgery simulator experiment. Otolaryngol Head Neck Surg 2010;142:202–7. [4] Deutsch ES, Wiet GJ, Seidman M, Hussey HM, Malekzadeh S, Fried MP. Simulation activity in otolaryngology residencies. Otolaryngol Head Neck Surg 2015;153:193–201. [5] Newe A, Ganslandt T. Simplified generation of biomedical 3D surface model data for embedding into 3D portable document format (PDF) files for publication and education. PLoS One 2013;8:e79004. [6] Barsuk JH, Cohen ER, Feinglass J, McGaghie WC, Wayne DB. Unexpected collateral effects of simulation-based medical education. Acad Med 2011;86:1513–17. [7] Cohen ER, Feinglass J, Barsuk JH, Barnard C, O’Donnell A, McGaghie WC, et al. Cost savings from reduced catheter-related bloodstream infection after simulation-based education for residents in a medical intensive care unit. Simul Healthc 2010;5:98–102. [8] Lopreiato JO, Sawyer T. Simulation-based medical education in pediatrics. Acad Pediatr 2015;15:134–42. [9] Sroka G, Feldman LS, Vassiliou MC, Kaneva PA, Fayez R, Fried GM. Fundamentals of laparoscopic surgery simulator training to proficiency improves laparoscopic performance in the operating room-a randomized controlled trial. Am J Surg 2010;199:115–20. [10] Mizota T, Anton NE, Huffman EM, Guzman MJ, Lane F, Choi JN, et al. Development of a fundamentals of endoscopic surgery proficiency-based skills curriculum for general surgery residents. Surg Endosc 2019. In press.
[11] Andersen SAW, Nayahangan LJ, Konge L, Melchiors J. Identifying and prioritizing technical procedures in otorhinolaryngology for simulation-based training: a national needs assessment in Denmark. Eur Arch Otorhinolaryngol 2019;276:1517–24. [12] Cass ND, Okland TS, Rodriguez K, Mann SE. Otolaryngology education: recent trends in publication. Otolaryngol Head Neck Surg 2017;156:1124–9. [13] Fitch K, Bernstein SJ, Aguilar MD, Burnand B, LaCalle JR, Lazaro P, et al. The RAND/UCLA appropriateness method user’s manual. RAND; 2001. [14] Scott DJ, Goova MT, Tesfay ST. A cost-effective proficiency-based knot-tying and suturing curriculum for residency programs. J Surg Res 2007;141:7–15. [15] Goova MT, Hollett LA, Tesfay ST, Gala RB, Puzziferri N, Kehdy FJ, et al. Implementation, construct validity, and benefit of a proficiency-based knot-tying and suturing curriculum. J Surg Educ 2008;65:309–15. [16] Gershuni V, Woodhouse J, Brunt LM. Retention of suturing and knot– tying skills in senior medical students after proficiency-based training: results of a prospective, randomized trial. Surgery 2013;154:823–9 discussion 9-30. [17] Mashaud LB, Arain NA, Hogg DC, Scott DJ. Development, validation, and implementation of a cost-effective intermediate-level proficiency-based knot-tying and suturing curriculum for surgery residents. J Surg Educ 2013;70:193–9. [18] Huang E, Chern H, O’Sullivan P, Cook B, McDonald E, Palmer B, et al. A better way to teach knot tying: a randomized controlled trial comparing the kinesthetic and traditional methods. Am J Surg 2014;208:690–4. [19] Huang E, Vaughn CJ, Chern H, O’Sullivan P, Kim E. An objective assessment tool for basic surgical knot-tying skills. J Surg Educ 2015;72:572–6. [20] Wade TJ, Lorbeer K, Awad MM, Woodhouse J, DeClue A, Brunt LM. Outcomes of a proficiency-based skills curriculum at the beginning of the fourth year for senior medical students entering surgery. Surgery 2015;158:962–9 discussion 9-71. [21] Pender C, Kiselov V, Yu Q, Mooney J, Greiffenstein P, Paige JT. All for knots: evaluating the effectiveness of a proficiency-driven, simulation-based knot tying and suturing curriculum for medical students during their third-year surgery clerkship. Am J Surg 2017;213:362–70. [22] Shekelle PG, Park RE, Kahan JP, Leape LL, Kamberg CJ, Bernstein SJ. Sensitivity and specificity of the RAND/UCLA appropriateness method to identify the overuse and underuse of coronary revascularization and hysterectomy. J Clin Epidemiol 2001;54:1004–10. [23] Bennett S, Tinmouth A, McIsaac DI, English S, Hebert PC, Karanicolas AJ, et al. Ottawa criteria for appropriate transfusions in hepatectomy using the RAND/UCLA appropriateness method. Ann Surg 2018;267:766–74. [24] Teoh AYB, Dhir V, Kida M, Yasuda I, Jin ZD, Seo DW, et al. Consensus guidelines on the optimal management in interventional EUS procedures: results from the Asian EUS group RAND/UCLA expert panel. Gut 2018;67:1209–28. [25] Maverakis E, Ma C, Shinkai K, Fiorentino D, Callen JP, Wollina U, et al. Diagnostic criteria of ulcerative pyoderma gangrenosum: a Delphi consensus of international experts. JAMA Dermatol 2018;154:461–6. [26] Lawson EH, Gibbons MM, Ko CY, Shekelle PG. The appropriateness method has acceptable reliability and validity for assessing overuse and underuse of surgical procedures. J Clin Epidemiol 2012;65:1133–43. [27] Ritter EM, Scott DJ. Design of a proficiency-based skills training curriculum for the fundamentals of laparoscopic surgery. Surg Innov 2007;14:107–12. [28] Stefanidis D, Acker C, Heniford BT. Proficiency-based laparoscopic simulator training leads to improved operating room skill that is resistant to decay. Surg Innov 2008;15:69–73.
Please cite this article as: E. Sato, S. Mitani and N. Nishio et al., Development of proficiency-based knot-tying and suturing curriculum for otolaryngology residents: A pilot study, Auris Nasus Larynx, https:// doi.org/ 10.1016/ j.anl.2019.11.005