- Email: [email protected]
Use of an Objective Structured Assessment of Technical Skill After a Sports Medicine Rotation
Use of an Objective Structured Assessment of Technical Skill (OSATS) After a Sports Medicine Rotation Tim Dwyer, M.B.B.S., F.R.A.C.S., F.R.C.S.C., Ph.D., Jesse Slade Shantz, M.D., Kulamakan Mahan Kulasegaram, Ph.D., Jaskarndip Chahal, M.D., M.Sc., F.R.C.S.C., David Wasserstein, M.D., M.Sc., F.R.C.S.C., Rachel Schachar, M.D., Brian Devitt, M.D., John Theodoropoulos, M.D., M.Sc., F.R.C.S.C., Brian Hodges, M.D., Ph.D., F.R.C.P.C., and Darrell Ogilvie-Harris, M.D., M.Sc., F.R.C.S.C.
Purpose: The purpose of this study was to determine if the use of an Objective Structured Assessment of Technical skill (OSATS), using dry models, would be a valid method of assessing residents’ ability to perform sports medicine procedures after training in a competency-based model. Methods: Over 18 months, 27 residents (19 junior [postgraduate year (PGY) 1-3] and 8 senior [PGY 4-5]) sat the OSATS after their rotation, in addition to 14 sports medicine staff and fellows. Each resident was provided a list of 10 procedures in which they were expected to show competence. At the end of the rotation, each resident undertook an OSATS composed of 6 stations sampled from the 10 procedures using dry modelsdfaculty used the Arthroscopic Surgical Skill Evaluation Tool (ASSET), task-specific checklists, as well as an overall 5-point global rating scale (GRS) to score each resident. Each procedure was videotaped for blinded review. Results: The overall reliability of the OSATS (0.9) and the inter-rater reliability (0.9) were both high. A significant difference by year in training was seen for the overall GRS, the total ASSET score, and the total checklist score, as well as for each technical procedure (P < .001). Further analysis revealed a significant difference in the total ASSET score between junior (mean 18.4, 95% confidence interval [CI] 16.8 to 19.9) and senior residents (24.2, 95% CI 22.7 to 25.6), senior residents and fellows (30.1, 95% CI 28.2 to 31.9), as well as between fellows and faculty (37, 95% CI 36.1 to 27.8) (P < .05). Conclusions: The results of this study show that an OSATS using dry models shows evidence of validity when used to assess performance of technical procedures after a sports medicine rotation. However, junior residents were not able to perform as well as senior residents, suggesting that overall surgical experience is as important as intensive teaching. Clinical Relevance: As postgraduate medical training shifts to a competency-based model, methods of assessing performance of technical procedures become necessary.
T
he acquisition of competent surgical skill is a defining characteristic of surgery, but is not measured systematically in residency.1 As competencybased medical education (CBME) continues to become more prevalent across North America, objective assessment tools are required to show competency. By
From the Women’s College Hospital (T.D., J.C., J.T., D.O-H.); Mt. Sinai Hospital (T.D., J.T.); University of Toronto (J.S.S., R.S., B.D.); The Wilson Centre (K.M.K., B.H.); and Sunnybrook Health Sciences Centre (D.W.), Toronto, Ontario, Canada. The authors report that they have no conflicts of interest in the authorship and publication of this article. Received January 15, 2016; accepted May 5, 2016. Address correspondence to Tim Dwyer, M.B.B.S., F.R.A.C.S., F.R.C.S.C., Ph.D., Women’s College Hospital, 76 Grenville St., Toronto, Ontario M5S 1B1, Canada. E-mail: [email protected] Ó 2016 by the Arthroscopy Association of North America 0749-8063/1629/$36.00 http://dx.doi.org/10.1016/j.arthro.2016.05.037
using regular assessment to ensure a minimum level of competence is achieved by the conclusion of a resident’s rotation (e.g., sports medicine) before progression to the next rotation, orthopaedic education can change from time-based programs to ones based on observable and measurable outcomes.2-4 Previous work has validated methods for assessing medical knowledge5 and communication skills6 in orthopaedic residents. However, valid, reliable, and objective methods of assessing competence in the performance of technical procedures have yet to be fully defined in orthopaedics. Although the gold standard will likely always be in the operating room, difficulty standardizing operations,7 issues of patient safety, clinical outcomes, as well as time restraints8 mean that residents often cannot perform complex procedures to completion in that environment. As such, we believe that simulation-based assessment may offer part of the solution to this complex problem.9
Arthroscopy: The Journal of Arthroscopic and Related Surgery, Vol
-,
No
-
(Month), 2016: pp 1-10
1
2
T. DWYER ET AL.
At our university, residents undertake two 3-month orthopaedic sports medicine rotations during their training, once as a junior (postgraduate year [PGY] 1-3) and once as a senior (PGY 4 and 5). During these rotations, their education is composed of clinical and operating room exposure, regular didactic lectures based on a set curriculum, and the teaching of technical procedures in a laboratory using anatomic knee and shoulder dry models. The skills laboratory is also available for use on an unlimited, independent basis throughout the rotation. At the end of the rotation, medical knowledge is assessed with an Objective Structured Clinical Examination (OSCE)5 and technical skill with an Objective Structured Assessment of Technical Skills (OSATS) using dry models. The OSATS was originally developed at the University of Toronto as an objective measurement format used to assess the technical competency of surgical trainees.10 An OSATS consists of a multistation, OSCE-like examination, in which candidates perform a series of standardized surgical tasks.11 Held either with highfidelity models (cadavers, animals) or low-fidelity bench models (Sawbones), assessment usually takes the form of a task-specific checklist in combination with a global rating scale (GRS) and is typically combined with a final rating such as shows proficiency or requires further practice.11 At this time, it is thought that an OSATS is the most accepted gold standard for objective skills assessment.12 The purpose of this study was to determine if the use of an OSATS, using dry models, would be a valid method of assessing residents’ ability to perform sports
medicine procedures after training in a competencybased model. We hypothesized that a simulated OSATS using dry models would show evidence of validity, suitable for use as an in-training assessment of competence after a sports medicine rotation. We also hypothesized that after training within a CBME model, junior and senior residents would be able to perform technical procedures at a similar level.
Methods In a prospective study, all residents who undertook the 3-month sports medicine rotation under the training of 6 staff surgeons were included in this study. At the beginning of the rotation, each resident was provided with a list of 10 technical procedures that they were expected to show competency in at the end of the rotation (Table 1). The surgical procedures were selected by a focus group of academic orthopaedic sports surgeons and corresponded with the sports medicine curriculum. At the end of each rotation, 6 of these technical procedures were selected for assessment in an OSATS, which was composed of 6 stations, each 10 minutes long. Stations were selected by examiners after each rotation, to allow reasonable representation of all stations across all groups of participants over the course of this study. Stations were not selected randomly as some of the stations (drilling of tibial tunnel/drilling of transtibial femoral tunnel, insertion glenoid anchor/passage labral suture, insertion rotator cuff anchor/passage rotator cuff suture) were best performed together from a practical point of view. All of
Table 1. List of Technical Procedures That Residents Were Expected to Show a Minimal Level of Competence at After a Sports Rotation Task Perform partial meniscectomy Whipstitch hamstring graft
Drill anteromedial ACL femoral tunnel Drill transtibial ACL femoral tunnel Drill ACL tibial tunnel Insert glenoid anchors Pass labral sutures Insert rotator cuff anchors Pass rotator cuff suture Tie sliding, locking arthroscopic knot ACL, anterior cruciate ligament.
Component of Task Radial tear created in the posterior horn of medial meniscus Resident directed to perform partial medical meniscectomy Two equal lengths of shoestring provided Resident directed to whipstitch each end and pretension graft on Smith & Nephew Graftmaster II (Andover, MA) Resident directed to drill femoral tunnel using anteromedial portal, with 8-mm tunnel and with 15-mm Endobutton (Smith & Nephew) femoral fixation Resident directed to drill femoral tunnel using transtibial tunnel (tibial predrilled with size 10-mm tunnel) Resident directed to drill tibial tunnel Resident directed to insert 2.3-mm glenoid anchor (Bioraptor; Smith & Nephew) at the 5 o’clock position Resident directed to pass labral sutures using Accu-Pass (Smith & Nephew) suture passing device Resident directed to insert 5-mm metal Twinfix anchor (Smith & Nephew) into footprint of humeral head Resident directed to pass 2 sutures through tendon; both Scorpion and Scorpion Fast Pass available for use (Arthrex) Faculty insert rotator cuff anchor and pass sutures prior to performance of knot tying (Shoulder Arthroscopy Model)
USE OF AN OSATS AFTER A SPORTS MEDICINE ROTATION
the residents at the end of each rotation were assessed on the same 6 procedures. Most shoulder procedures (insertion glenoid anchors, labral suture passage, insertion rotator cuff anchors, rotator cuff suture passage) were performed on the Arthroscopic SOFT Shoulder Model (model DS-012; Arthrex, Naples, FL) bench-top simulator using replacement joint capsule assembly with cuff tear (DS-0123SST) (Fig 1). Faculty inserted arthroscopic cannulas into the subacromial space or the glenohumeral joint prior to the assessment. Knot tying was performed on a Shoulder Arthroscopy Model (S.A.M., www.arthrodemo.com); faculty placed a rotator cuff anchor and passed both suture limbs through the tissue prior to performance of a sliding, locking arthroscopic knot. Components of anterior cruciate ligament (ACL) reconstruction (drilling anteromedial femoral tunnel, transtibial femoral tunnel, and tibial tunnel) and partial meniscectomy were performed using an ACL Sawbones model (Sawbones, Vashon, WA), using an encapsulated knee insert (model 1414-1) inside a soft tissue with skin (model 1413-1) (Fig 2). All arthroscopic portals (anterolateral, anteromedial, accessory anteromedial) were made prior to the assessment. Standard 30 arthroscopes with high-definition video systems were used for all procedures, in association with disposable implants and standard equipment (Smith & Nephew, Andover, MA, and Arthrex). Whipstitching of a hamstring graft was simulated using 2 equal lengths of shoestring and the Smith & Nephew Graftmaster II. Residents were evaluated using a combination of task-specific checklists (see Appendix Tables 1-10, available at www.arthroscopyjournal.org), a previously validated GRS (the Arthroscopic Surgical Skill Evaluation Tool [ASSET]),13 and an overall final 5-point GRS. Task-specific checklists were created for each technical procedure using established arthroscopy texts, and by a panel of experienced sports medicine staff
3
surgeonsdconsensus was achieved using a modified Delphi procedure conducted by way of multiple surveys.13 The ASSET is made up of 8 domains (safety, field of view, camera dexterity, instrument dexterity, bimanual dexterity, flow of procedure, quality of procedure, and autonomy) based on the Dreyfus model of skill acquisition and is scored to a maximum of 38.13 The ASSET was designed to be generalizable to multiple procedures and settings (operating room and simulation) and has been found to be reliable and valid in the setting of diagnostic knee arthroscopy in cadaveric specimens.13 Finally, independent of the ASSET, each examiner allocated an overall 5-point GRS corresponding to the Dreyfus model of skill acquisition (novice, advanced beginner, competent, proficient, expert).14,15 Examiners were instructed to designate a resident performance as competent if they performed at the level of a graduating orthopaedic surgeon. A pilot OSATS was run using 2 junior residents, to establish the practicality of each station (sufficient time to complete each task) and optimize the checklist for each station. All residents at the end of each sports rotation undertook the OSATS; the year of training of the participants did not affect the procedure selection. All staff surgeons with fellowship training in sports medicine at our institution and all sports medicine fellows also sat the OSATS. A single examiner was present at each station; examiners were orthopaedic sports medicine surgeons who were involved in the training of the residents and fellows. All the examiners had clinical practices encompassing knee and shoulder surgery. All examiners were familiar with the checklists (having been involved in their creation), the ASSET, and the overall GRS. Examiners were available to assist as directed by or requested by residents, but no guidance with regards to the procedure was provided at any time. There was no time limit for each procedure, nor was the amount of time to complete each task recorded.
Fig 1. (A) The Arthroscopic SOFT Shoulder Model (Arthrex) bench-top simulator used for insertion of glenoid anchor and labral repair. Anterosuperior and anterior-inferior cannulas have been inserted into the rotator interval before performance of the technical procedure. (B) Instrumentation available for performance of this skill (Smith & Nephew).
4
T. DWYER ET AL.
Fig 2. (A) The knee model (Sawbones) used for performance of partial medial meniscectomy, drilling of the tibial tunnel, and drilling of the femoral tunnel using both the anteromedial and transtibial technique. Again all portals were made prior to the assessment. (B) Setup provided for whipstitch of hamstring graft.
Each participant performed each procedure once only. Videotaping of all procedures (both hand movements and arthroscopic component) was performed, which was then reviewed by a blinded, independent sports medicine fellow. Statistical Analysis Stata, version 13 (College Station, TX), was used to perform all analyses. The total score of the ASSET was calculated as the sum of the 8 domains, with a maximum score of 38. The internal consistency/ reliability of the total ASSET score, total checklist score, and the overall GRS was assessed using Cronbach alpha. Construct validity (the ability to discriminate between different years of training) was assessed using 1-way analysis of variance, with year of training (junior resident, senior resident, fellow, staff) as the independent variable. Unpaired t tests were used to examine for differences between groups. Concurrent validity was sought by examining the correlation between the end of rotation OSCE results for each of the residents, and the OSATS, using Pearson correlation. Inter-rater reliability between the examiners and blinded video reviewer was calculated using the intraclass correlation coefficient for total ASSET score. Reliability was also calculated using generalizability theory, analyzed with G-String IV software (Bloch & Norman, Hamilton, Ontario, Canada) using common station combinations and estimating the variance created by participants, their training levels, stations, as well as items within stations. A sample size calculation was performed; a minimum of 8 residents was required in each group (junior, senior) to detect a difference of 1.0 in the overall GRS, using a mean expected global rating of 3 (competent), standard deviation of 1.0, alpha of 0.05, and a power of 0.8. Costs for equipment and models were also calculated per participant, excluding implants that were donated by industry.
This study was conducted after approval by the institutional research ethics board. This study was supported by a Core Competency Innovation Grant from the OMeGA Medical Grants Association.
Results Between September 2013 and March 2015, a total of 27 residents (19 junior [postgraduate year (PGY) 1-3] and 8 senior [PGY 4 and 5]) sat an OSATS after their rotation, in addition to 7 sports medicine staff and 7 fellows, for a total of 41 participants (Table 2). No resident sat the examination both as a junior and as a senior, and no residents were excluded. The internal consistency/reliability (Cronbach alpha) of the OSATS was excellent (0.95). A good correlation was seen between the overall checklist score and the overall global rating (0.71). For no station did the Cronbach alpha if Item Deleted increase, indicating that all stations were performing well. Analysis of variance revealed a significant difference by groups for the overall GRS, the total ASSET score, and the total checklist score for all stations combined (P < .001) (Fig 3). A significant difference (analysis of variance) was also seen for the ASSET score for each individual station by level of training (Fig 4) (all P < .001). Further analysis using a t test revealed a Table 2. Demographics of Participants Undertaking the Objective Structured Assessment of Technical Skills Year of Training PGY 1 PGY 2 PGY 3 PGY 4 PGY 5 Fellows Staff PGY, postgraduate year.
Number of Participants 13 2 4 4 4 7 7
USE OF AN OSATS AFTER A SPORTS MEDICINE ROTATION
5
Fig 3. (A) Box plot for the overall checklist scores calculated as a percentage. (B) Box plot for the total ASSET global rating. (C) Box plot for the overall global rating. There was a significant difference by year of training for each (P < .001). (ASSET, Arthroscopic Surgical Skill Evaluation Tool.)
significant difference in the total ASSET score between junior (mean 18.4, 95% confidence interval [CI] 16.8 to 19.9) and senior residents (24.2, 95% CI 22.7 to 25.6), senior residents and fellows (30.1, 95% CI 28.2 to 31.9), and between fellows and faculty (37, 95% CI 36.1 to 27.8) (P < .05). The number of participants deemed competent (overall GRS of competent/proficient/expert) for each technical procedure is seen in Tables 3 and 4, whereas an example of scores from 4 residents at the end of their rotation is seen in Table 5. The inter-rater reliability between the examiners ratings, and the blinded video reviewer for the total ASSET score using intraclass correlation coefficient was 0.9. A good correlation was seen between performance on the end of rotation OSCE (overall GRS summed over the 6 stations) and each resident’s performance total ASSET score on the OSATS (0.76). A generalizability coefficient was calculated using the knot tying, partial meniscectomy, and whip stitching stations as well as the stations of glenoid anchor insertion and labral suturing; the relative error coefficients (interstation reliability) were 0.64 and 0.57, respectively. Using these calculations, it was estimated that a 6-station examination would confer an overall reliability of 0.78. The major source of variance (50%) was training level of the participant. During this study, 39 Sawbones knees ($69.75 each) with 5 knee skins ($83.70 each) were used for components of ACL reconstruction and meniscectomy. For components of rotator cuff and labral repair, 4 Arthroscopic SOFT Shoulder Models ($325) and 25 replacement inserts ($85) were required, whereas knot tying was performed using a single S.A.M model ($495) with 18 basic rotator cuff felt components ($6) and 18 humeral head replacements ($25). The total cost was $7,616.75, which averaged $185 for the OSATS per participant.
Discussion
The most important finding of this study is that the scores from the OSATS showed evidence of validity
when used to assess the performance of technical procedures by residents on dry models after a sports medicine rotation. Evidence of validity included excellent internal consistency, high inter-rater reliability, evidence of novice-expert differentiation, as well as a good correlation between performance on the OSATS and the end of rotation OSCE. The results of this study also showed that junior residents are not able to perform these simulated procedures as well as senior residents, suggesting that a combination of increased exposure to sports medicine, and overall surgical experience is a factor in the acquisition of these technical skills. A recent survey of orthopaedic program directors and residents identified that a significant number believed that surgical skills simulation should become a required part of training.16 Simulation-based training and assessment allows the opportunity for residents to attempt complex surgical tasks, without risk to patients. Simulation also allows residents the opportunity to make independent, intraoperative decisions, taking responsibility for and managing the consequences of their actionsdtangled sutures and improperly inserted anchors are one example. Simulation provides a staff surgeon with valuable insight into each resident’s knowledge of procedures, and their familiarity with instrumentation prior to performance in the operating room. Options for simulation in orthopaedics include highfidelity models (cadavers), virtual reality, and lowfidelity models (Sawbones). Cadavers are thought to be the gold standard for simulation training,7,17 with studies looking at the performance of knee arthroscopy in cadavers showing good reliability, as well as the ability to differentiate between different years of training.18,19 However, cadaver labs require advanced facilities and have significant costs and may not be a feasible option for regular assessment of technical procedures. Virtual reality simulation is commercially available for knee and shoulder arthroscopy20-24;
6
T. DWYER ET AL.
Fig 4. Box plots for the total ASSET score for each of the stations, with the ASSET score presented as a percentage. A significant difference by year of training was seen for all technical procedures. (ASSET, Arthroscopic Surgical Skill Evaluation Tool.)
however, virtual reality has limited ability to allow the performance of complex procedures such as the drilling of ACL tunnels. The advantages of dry models include relative ease of preparation, reduced cost, and the opportunity for unsupervised practice.25 Most comparative studies have shown that low-fidelity simulators are similarly effective but less expensive than high-fidelity simulators with regard to the acquisition of surgical skills.26-33 The finding that many residents were unable to perform the sports medicine procedures in the
simulated setting warrants attention. This finding is task dependent: although most of the procedures were difficult for the junior residents, some procedures were also difficult for senior residents to complete. This was especially noticeable with the simulated arthroscopic shoulder procedures; one reason may be that these are advanced skills, requiring significant experience or fellowship training. It may also highlight deficiencies in our training program. In the operating room, staff surgeons may be reluctant to allow residents to perform these procedures, because of the complexity of the
7
USE OF AN OSATS AFTER A SPORTS MEDICINE ROTATION
Table 3. Number and Percentage of Participants Deemed Competent or Better in Each Knee Station Using the Overall Global Rating Scale (Novice, Advanced Beginner, Competent, Proficient, Expert) Participants Junior residents Competent Mean ASSET score (95% Senior residents Competent Mean ASSET score (95% Fellows Competent Mean ASSET score (95% Staff Competent Mean ASSET score (95%
Anteromedial Femoral Tunnel
Transtibial Femoral Tunnel
Tibial Tunnel
CI)
1/9 (11%) 18.6 (15.2-22)
1/8 (12.5%) 15 (6.4-23.6)
4/14 (29%) 19.5 (13.9-25.1)
CI)
3/5 (60%) 24.5 (18.5-30.5)
3/3 (100%) 23 (20.8-25.2)
5/6 (83%) 27.5 (20-35)
CI)
4/4 (100%) 32.8 (27.1-38.4)
3/3 (100%) 27.7 (18.9-34.4)
CI)
5/5 (100%) 36.6 (35.3-37.9)
3/3 (100%) 38 (0-0)
Partial Meniscectomy 4/9 (44%) 19.4 (15-23.8)
Whipstitch Hamstring Graft 5/12 (42%) 20.6 (15.8-25.4)
5/6 (83%) 22 (18.7-25.3)
5/5 (100%) 27.36 (25.1-29.6)
3/4 (75%) 27 (18.9-35.1)
7/7 (100%) 29.7 (25.1-34.3)
6/7 (86%) 28 (25.2-30.8)
7/7 (100%) 38 (0-0)
7/7 (100%) 35.9 (34.9-37)
4/4 (100%) 33.3 (30.3-36.6)
ASSET, Arthroscopic Surgical Skill Evaluation Tool; CI, confidence interval.
individual steps, as opposed to simpler procedures such as knee arthroscopy.34 It was also interesting to note that a small number of fellows were not competent at procedures such as drilling the tibial tunnel or whipstitching a hamstring graft; this may highlight a deficiency in fellowship training, such as limited exposure to hamstring ACL reconstruction in some rotations. The finding that there is a significant difference in performance between junior and senior residents is at odds with the concept of CBME, which in theory demands a minimal level of competence prior to residents’ proceeding to the next rotation. The results of this study, and other studies looking at the acquisition of clinical skills in orthopaedic residents,6,35,36 suggest that there is an effect of overall clinical experience gained throughout training that has an effect on the performance of all technical procedures. How to best deal with these results is unknown. One approach would be to increase the frequency of teaching in the simulation laboratory; however, these are already being held twice monthly over the 3-month rotation. These training sessions are not without cost,
both in terms of faculty time and costs of models and implants; for this reason the number of skills sessions cannot be easily increased. Alternatively, a milestones approach, as recommended by the Accreditation Council for Graduate Medical Education, could be implemented.37 In this way, although teaching and training sessions would continue to cover the surgical procedures listed, junior residents would be expected to show a minimal level of competency at less complex tasks (partial meniscectomy, whipstitch hamstring graft), whereas it seems that senior residents can be expected to perform aspects of ACL reconstruction and rotator cuff repair. Importantly, this study does not provide any correlation between the procedures performed in the simulated setting and operative performance. Although there are some studies showing that bench model training improves operative performance in orthopaedics,25,33 the evidence that simulation-based assessment correlates with actual surgical performance is more limited,38 although studies in other surgical specialties have consistently identified good correlations.39-42 The
Table 4. Number and Percentage of Participants Deemed Competent or Better in Each Shoulder Station Using the Overall Global Rating Scale (Novice, Advanced Beginner, Competent, Proficient, Expert) Participants Junior residents Competent Mean ASSET score (95% Senior residents Competent Mean ASSET score (95% Fellows Competent Mean ASSET score (95% Staff Competent Mean ASSET score (95%
Insertion Glenoid Anchors
Pass Labral Sutures
Insertion Rotator Cuff Anchors
Pass Rotator Cuff Suture
Tie Arthroscopic Knot
CI)
4/11 (36%) 19.4 (15.6-23.1)
2/11 (18%) 18.1 (15.2-21.1)
5/13 (38%) 16.8 (11.5-22.1)
5/13 (38%) 16.5 (10.7-22.3)
3/16 (19%) 17.2 (14.4-20.2)
CI)
2/5 (40%) 25.7 (21.5-29.8)
2/5 (40%) 24 (20.1-27.9)
4/4 (100%) 24.8 (23.2-26.3)
2/4 (50%) 20.5 (18.3-22.7)
5/7 (71%) 22.3 (17.8-26.9)
CI)
3/3 (100%) 33 (28.8-37.1)
3/3 (100%) 31.3 (28.3-34.4)
4/4 (100%) 29 (24.7-33.2)
4/4 (100%) 29 (24.8-33.2)
7/7 (100%) 27.1 (25-29.3)
CI)
3/3 (100%) 34 (27.7-40.3)
3/3 (100%) 34.3 (26.7-42)
4/4 (100%) 37 (34.9-39.1)
4/4 (100%) 37.5 (36.4-38.6)
6/6 (100%) 34.4 (33-35.8)
ASSET, Arthroscopic Surgical Skill Evaluation Tool; CI, confidence interval.
8
T. DWYER ET AL.
Table 5. Example Ratings From 4 Residents at the End of Their 3-Month Rotation Participants PGY 4 PGY 4 PGY 1 PGY 1
Tibial Tunnel Competent Competent Novice Novice
Transtibial Femoral Tunnel Competent Competent Advanced beginner Novice
Partial Meniscectomy Competent Competent Advanced beginner Novice
Insertion Glenoid Anchor Proficient Advanced beginner Novice Advanced beginner
Passage Labral Suture Advanced beginner Proficient Novice Novice
Arthroscopic Knot Tie Advanced beginner Competent Advanced beginner Advanced beginner
PGY, postgraduate year.
reasons for this deficiency in the literature is likely the difficulty standardizing operations, time restraints, and issues of patient safety.7,8 Although there is no reason to believe that residents who cannot perform these procedures in the simulated setting would be able to in the workplace, this is an important area of future research in CBME. The OSATS study identified a correlation between performance of technical procedures, and the OSCE scores for each resident. However, the results of other studies seeking a correlation between medical knowledge and technical skills have been mixed.43 A recent study looking at simulated surgery on a distal radius fracture failed to identify a correlation between knowledge testing and biomechanical testing of the surgical construct.44 In 2009, Van Heest et al.45 reported on orthopaedic residents performing a carpal tunnel release on a cadaver, in association with a knowledge test; although a knowledge score <70/100 predicted failure on the OSATS, a score of >70/100 did not ensure that the examinee would pass the test. These findings are plausible: although there is a threshold of knowledge that is required, above that threshold, a combination of technical abilities and surgical experience has an important role. Limitations The most important limitation of this study is that we are unable to show a correlation between performance on this OSATS study, and performance in the operating room, as intra-operative assessments are not at this time routinely performed for all residents. Although all residents were also assessed with an In-Training Evaluation Report, these results were not available for correlation with performance on the OSATS. Previous research from the author has identified a correlation between a resident’s In-Training Evaluation Report and skills such as communication, professionalism, collaboration, etc.6 Furthermore, although residents were exposed to training on dry models, inevitably each will have had different clinical exposures during their rotations. It is also unknown how much each resident took advantage of the skills lab outside of teaching sessions. The number of sports medicine and other procedures each resident had experience with is unknown; for this reason, we cannot say whether these results are a result of sports medicine experience,
overall surgical experience, or a combination of both. We are unable to provide an intrarater reliability, as no examiner reviewed the video recordings of participants. No baseline testing of residents was performed, which would have allowed for evaluation of each resident’s knowledge and skill level prior to commencing the rotation. Stations were not selected randomly but rather with a view to establishing even representation of all stations throughout the study period, while maximizing practicality. Finally, it is important to note that the assessment of competency always requires the exercise of judgment by another competent, proficient, or expert practitioner; for this reason, the development of an objective assessment, or assessment of competency independent of the assessor, is not always possible.46 An attempt to minimize these subjective effects is made via a process of standardization, and by reaching consent with regards to what is being evaluated and the standards expected.47
Conclusions The results of this study reveal that an OSATS using dry models shows evidence of validity when used to assess performance of technical procedures after a sports medicine rotation. However, junior residents were not able to perform as well as senior residents, suggesting that overall surgical experience is as important as intensive teaching.
References 1. Williams RG, Verhulst S, Colliver JA, Sanfey H, Chen X, Dunnington GL. A template for reliable assessment of resident operative performance: Assessment intervals, numbers of cases and raters. Surgery 2012;152:517-524. discussion 24-27. 2. Hodges BD. A tea-steeping or i-Doc model for medical education? Acad Med 2010;85:S34-S44 (suppl). 3. Pugh D, Touchie C, Wood TJ, Humphrey-Murto S. Progress testing: Is there a role for the OSCE? Med Educ 2014;48:623-631. 4. Long DM. Competency-based residency training: The next advance in graduate medical education. Acad Med 2000;75:1178-1183. 5. Dwyer T, Theodoropoulos JS, Herold J, et al. Assessing competence of orthopaedic residents: The reliability and validity of an objective structured clinical examination
USE OF AN OSATS AFTER A SPORTS MEDICINE ROTATION
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.
22.
after a sports medicine rotation. J Bone Joint Surg Am 2013;95:e177. Dwyer T, Glover Takahashi S, Kennedy Hynes M, et al. How to assess communication, professionalism, collaboration and the other intrinsic CanMEDS roles in orthopedic residents: Use of an objective structured clinical examination (OSCE). Can J Surg 2014;57:230-236. Reznick R, Regehr G, MacRae H, Martin J, McCulloch W. Testing technical skill via an innovative “bench station” examination. Am J Surg 1997;173:226-230. Farnworth LR, Lemay DE, Wooldridge T, et al. A comparison of operative times in arthroscopic ACL reconstruction between orthopaedic faculty and residents: The financial impact of orthopaedic surgical training in the operating room. Iowa Orthop J 2001;21:31-35. Dwyer T, Slade Shantz J, Chahal J, et al. Simulation of anterior cruciate ligament reconstruction in a dry model. Am J Sports Med 2015;43:2997-3004. Martin JA, Regehr G, Reznick R, et al. Objective Structured Assessment of Technical Skill (OSATS) for surgical residents. Br J Surg 1997;84:273-278. Sidhu RS, Grober ED, Musselman LJ, Reznick RK. Assessing competency in surgery: Where to begin? Surgery 2004;135:6-20. van Hove PD, Tuijthof GJ, Verdaasdonk EG, Stassen LP, Dankelman J. Objective assessment of technical surgical skills. Br J Surg 2010;97:972-987. Koehler RJ, Amsdell S, Arendt EA, et al. The Arthroscopic Surgical Skill Evaluation Tool (ASSET). Am J Sports Med 2013;41:1229-1237. Carraccio CL, Benson BJ, Nixon LJ, Derstine PL. From the educational bench to the clinical bedside: Translating the Dreyfus developmental model to the learning of clinical skills. Acad Med 2008;83:761-767. Batalden P, Leach D, Swing S, Dreyfus H, Dreyfus S. General competencies and accreditation in graduate medical education. Health Aff (Millwood) 2002;21:103-111. Karam MD, Pedowitz RA, Natividad H, Murray J, Marsh JL. Current and future use of surgical skills training laboratories in orthopaedic resident education: A national survey. J Bone Joint Surg Am 2013;95:e4. Anastakis DJ, Regehr G, Reznick RK, et al. Assessment of technical skills transfer from the bench training model to the human model. Am J Surg 1999;177:167-170. Olson T, Koehler R, Butler A, Amsdell S, Nicandri G. Is there a valid and reliable assessment of diagnostic knee arthroscopy skill? Clin Orthop Relat Res 2013;471: 1670-1676. Insel A, Carofino B, Leger R, Arciero R, Mazzocca AD. The development of an objective model to assess arthroscopic performance. J Bone Joint Surg Am 2009;91:2287-2295. Gomoll AH, O’Toole RV, Czarnecki J, Warner JJ. Surgical experience correlates with performance on a virtual reality simulator for shoulder arthroscopy. Am J Sports Med 2007;35:883-888. Gomoll AH, Pappas G, Forsythe B, Warner JJ. Individual skill progression on a virtual reality simulator for shoulder arthroscopy: A 3-year follow-up study. Am J Sports Med 2008;36:1139-1142. McCarthy AD, Moody L, Waterworth AR, Bickerstaff DR. Passive haptics in a knee arthroscopy simulator: Is it valid
23.
24.
25.
26.
27.
28.
29.
30.
31.
32.
33.
34.
35.
36.
37. 38.
9
for core skills training? Clin Orthop Relat Res 2006;442: 13-20. Pedowitz RA, Esch J, Snyder S. Evaluation of a virtual reality simulator for arthroscopy skills development. Arthroscopy 2002;18:E29. Srivastava S, Youngblood PL, Rawn C, Hariri S, Heinrichs WL, Ladd AL. Initial evaluation of a shoulder arthroscopy simulator: Establishing construct validity. J Shoulder Elbow Surg 2004;13:196-205. Butler A, Olson T, Koehler R, Nicandri G. Do the skills acquired by novice surgeons using anatomic dry models transfer effectively to the task of diagnostic knee arthroscopy performed on cadaveric specimens? J Bone Joint Surg Am 2013;95. e15(1-8). Chandra DB, Savoldelli GL, Joo HS, Weiss ID, Naik VN. Fiberoptic oral intubation: The effect of model fidelity on training for transfer to patient care. Anesthesiology 2008;109:1007-1013. Grober ED, Hamstra SJ, Wanzel KR, et al. The educational impact of bench model fidelity on the acquisition of technical skill: The use of clinically relevant outcome measures. Ann Surg 2004;240:374-381. Matsumoto ED, Hamstra SJ, Radomski SB, Cusimano MD. The effect of bench model fidelity on endourological skills: A randomized controlled study. J Urol 2002;167:1243-1247. McDougall EM, Kolla SB, Santos RT, et al. Preliminary study of virtual reality and model simulation for learning laparoscopic suturing skills. J Urol 2009;182:1018-1025. Zendejas B, Wang AT, Brydges R, Hamstra SJ, Cook DA. Cost: The missing outcome in simulation-based medical education research; a systematic review. Surgery 2013;153:160-176. Alvand A, Auplish S, Gill H, Rees J. Innate arthroscopic skills in medical students and variation in learning curves. J Bone Joint Surg Am 2011;93:e115(1-9). Howells NR, Auplish S, Hand GC, Gill HS, Carr AJ, Rees JL. Retention of arthroscopic shoulder skills learned with use of a simulator. Demonstration of a learning curve and loss of performance level after a time delay. J Bone Joint Surg Am 2009;91:1207-1213. Howells NR, Gill HS, Carr AJ, Price AJ, Rees JL. Transferring simulated arthroscopic skills to the operating theatre: A randomised blinded study. J Bone Joint Surg Br 2008;90:494-499. Hodgins JL, Veillette C, Biau D, Sonnadara R. The knee arthroscopy learning curve: Quantitative assessment of surgical skills. Arthroscopy 2014;30:613-621. Dwyer T, Wright S, Kulasegaram KM, et al. Competencybased medical education: Can both junior residents and senior residents achieve competence after a sports medicine training module? J Bone Joint Surg Am 2015;97: 1985-1991. Dwyer T, Wadey V, Archibald D, et al. Cognitive and psychomotor entrustable professional activities: Can simulators help assess competency in trainees? Clin Orthop Relat Res 2016;474:926-934. ACGME. The Orthopaedic Surgery Milestone Project 2013. July 20, 2015. Korndorffer JR Jr, Kasten SJ, Downing SM. A call for the utilization of consensus standards in the surgical education literature. Am J Surg 2010;199:99-104.
10
T. DWYER ET AL.
39. Datta V, Bann S, Beard J, Mandalia M, Darzi A. Comparison of bench test evaluations of surgical skill with live operating performance assessments. J Am Coll Surg 2004;199:603-606. 40. Ghaderi I, Vaillancourt M, Sroka G, et al. Performance of simulated laparoscopic incisional hernia repair correlates with operating room performance. Am J Surg 2011;201: 40-45. 41. McCluney AL, Vassiliou MC, Kaneva PA, et al. FLS simulator performance predicts intraoperative laparoscopic skill. Surg Endosc 2007;21:1991-1995. 42. Beard JD, Jolly BC, Newble DI, Thomas WE, Donnelly J, Southgate LJ. Assessing the technical skills of surgical trainees. Br J Surg 2005;92:778-782. 43. Bann S, Khan MS, Datta VK, Darzi AW. Technical performance: Relation between surgical dexterity and
44.
45.
46. 47.
technical knowledge. World J Surg 2004;28:142-146. discussion 6-7. Putnam MD, Kinnucan E, Adams JE, Van Heest AE, Nuckley DJ, Shanedling J. On orthopedic surgical skill predictiondthe limited value of traditional testing. J Surg Educ 2015;72:458-470. Van Heest A, Putnam M, Agel J, Shanedling J, McPherson S, Schmitz C. Assessment of technical skills of orthopaedic surgery residents performing open carpal tunnel release surgery. J Bone Joint Surg Am 2009;91: 2811-2817. Brooks MA. Medical education and the tyranny of competency. Perspect Biol Med 2009;52:90-102. Ginsburg S, McIlroy J, Oulanova O, Eva K, Regehr G. Toward authentic clinical evaluation: pitfalls in the pursuit of competency. Acad Med 2010;85:780-786.
10.e1
USE OF AN OSATS AFTER A SPORTS MEDICINE ROTATION Appendix Table 1. Task-Specific Checklist: Anteromedial Femoral Tunnel
Appendix Table 3. Task-Specific Checklist: Tibial Tunnel Correct
Correct
Incorrect
Appropriate portal Correct knee position for technique, i.e., hyperflexed Uses femoral offset guide Appropriate offset, i.e., 6 mm Correct position on wall, i.e., between 10 and 11 Estimates guide pin exit before drilling Correct guide pin exit, i.e., between PFJ and lateral IM septum Passes Endobutton drill Measures depth Drills to correct depth (i.e., within 10 mm) Checks back wall Back wall intact
Incorrect
Knee in flexion Knowledge of landmarks Correct tibial jig angle Places in center of tibia shaft Discussion of ML landmarks Discussion of AP landmarks Care of guide pin when drilling Appropriate drill diameter NOTE. Resident is to drill a tibial tunnel for a 10-mm boneepatellar tendonebone graft. AP, anteroposterior; ML, mediolateral.
NOTE. Resident is to drill femoral tunnel for anterior cruciate ligament ACL reconstruction, using AM technique, 8-mm hamstring graft, and fixation with 20-mm Endobutton. ACL, anterior cruciate ligament; AM, anteromedial; IM, intermuscular; PFJ, patellofemoral joint.
Appendix Table 2. Task-Specific Checklist: Transtibial Femoral Tunnel Correct
Incorrect
Uses tibial tunnel Correct knee position for technique, i.e., 90 degrees Uses femoral offset guide Appropriate offset, i.e., 7 mm Correct position on wall, i.e., between 10 and 11 Estimates guide pin exit before drilling Correct guide pin exit, i.e., between PFJ and lateral IM septum Line to line drill diameter, i.e., 10-mm drill Drills to depth of 20-25 mm Checks back wall Back wall intact NOTE. Resident is to drill a femoral tunnel using previous tibial tunnel, for a 10-mm boneepatellar tendonebone graft, with a 20-mm bone block. IM, intermuscular; PFJ, patellofemoral joint.
Appendix Table 4. Task-Specific Checklist: Whipstitch Hamstring Graft Correct
Incorrect
Uses Graftmaster Able to set up Graftmaster Correct suture choice Stitches both tendons Able to whipstitch Places over Endobutton Able to pretension NOTE. Resident is to whipstitch the 2 tendons together, pass over Endobutton, and pretension on Graftmaster.
10.e2
T. DWYER ET AL.
Appendix Table 5. Task-Specific Checklist: Knee Arthroscopy and Partial Meniscectomy
Appendix Table 7. Task-Specific Checklist: Labral Suture Correct
Correct
Incorrect
Inspection of suprapatellar pouch Inspection of patellofemoral compartment Inspection of lateral gutter Inspection of medial gutter Inspection of medial compartment including anterior and posterior horns Inspection of intercondylar notch including the ACL and PCL Inspection of lateral compartment including anterior and posterior horn Identifies tear Correctly identifies tear site, i.e., posterior horn Correctly names type of tear Describes size of tear Able to resect tear Smooth margins Reassess debridement with probe
Incorrect
Separates sutures Gets assistant to hold arthroscope Correct instrument angulation, right v left Uses low anterior portal Bite inferior to anchor Good portion of labral tissue Able to suture shuttle NOTE. Resident is to used Accu-Pass to pass a labral suture, and then suture shuttle, from a glenoid anchor that is already inserted.
NOTE. Resident is to resect a radial tear in the posterior horn of the medial meniscus. ACL, anterior cruciate ligament; PCL, posterior cruciate ligament.
Appendix Table 6. Task-Specific Checklist: Insertion Glenoid Anchor Correct Uses drill guide Asks assistant to hold arthroscope Uses low anterior portal Places at 5-5.30 45 Perpendicular to clockface Correct drill insertion Maintains drill guide while placing anchor Notes is a tap in anchor Anchor inserted to correct depth Tests anchor security Ensures sutures are running
Incorrect
Appendix Table 8. Task-Specific Checklist: Rotator Cuff Anchor
‘
NOTE. Resident is to insert a tap-in anchor in the anterior glenoid.
Correct
Incorrect
Uses appropriate portal Places the trochar on footprint Insertion angle at 45 Assistant uses mallet or holds camera Able to insert anchor into pilot hole Inserts to correct depth Tests anchor security Ensures sutures are running NOTE. Resident is directed to place a rotator cuff anchor, and prepare for suture passage.
10.e3
USE OF AN OSATS AFTER A SPORTS MEDICINE ROTATION Appendix Table 9. Task-Specific Checklist: Rotator Cuff Suture Correct
Appendix Table 10. Task-Specific Checklist Arthroscopic Knot Tie
Incorrect
Suture managementdthrough separate portals Loads suture correctly Able to pass suture Adequate tissue bite Retrieves sutures through alternative portal Appropriate separation between the 2 sutures, i.e., 1 cm NOTE. Resident is to use a suture passer to pass 2 rotator cuff sutures from a preinserted anchor.
Correct
Incorrect
Knows name of knot Using a sliding knot Aware is a self-locking knot Shortens the post Correctly ties knot Dresses the knot Places knot pusher on post Snap/clip placed on post Knot tensioned down onto soft tissue Past pointing to lock knot Knot security with additional knots Able to flip knot/tie a square knot NOTE. Resident is to tie an arthroscopic sliding, self-locking knot and then tie additional secure knots.
Purpose: The purpose of this study was to determine if the use of an Objective Structured Assessment of Technical skill (OSATS), using dry models, would be a valid method of assessing residents’ ability to perform sports medicine procedures after training in a competency-based model. Methods: Over 18 months, 27 residents (19 junior [postgraduate year (PGY) 1-3] and 8 senior [PGY 4-5]) sat the OSATS after their rotation, in addition to 14 sports medicine staff and fellows. Each resident was provided a list of 10 procedures in which they were expected to show competence. At the end of the rotation, each resident undertook an OSATS composed of 6 stations sampled from the 10 procedures using dry modelsdfaculty used the Arthroscopic Surgical Skill Evaluation Tool (ASSET), task-specific checklists, as well as an overall 5-point global rating scale (GRS) to score each resident. Each procedure was videotaped for blinded review. Results: The overall reliability of the OSATS (0.9) and the inter-rater reliability (0.9) were both high. A significant difference by year in training was seen for the overall GRS, the total ASSET score, and the total checklist score, as well as for each technical procedure (P < .001). Further analysis revealed a significant difference in the total ASSET score between junior (mean 18.4, 95% confidence interval [CI] 16.8 to 19.9) and senior residents (24.2, 95% CI 22.7 to 25.6), senior residents and fellows (30.1, 95% CI 28.2 to 31.9), as well as between fellows and faculty (37, 95% CI 36.1 to 27.8) (P < .05). Conclusions: The results of this study show that an OSATS using dry models shows evidence of validity when used to assess performance of technical procedures after a sports medicine rotation. However, junior residents were not able to perform as well as senior residents, suggesting that overall surgical experience is as important as intensive teaching. Clinical Relevance: As postgraduate medical training shifts to a competency-based model, methods of assessing performance of technical procedures become necessary.
T
he acquisition of competent surgical skill is a defining characteristic of surgery, but is not measured systematically in residency.1 As competencybased medical education (CBME) continues to become more prevalent across North America, objective assessment tools are required to show competency. By
From the Women’s College Hospital (T.D., J.C., J.T., D.O-H.); Mt. Sinai Hospital (T.D., J.T.); University of Toronto (J.S.S., R.S., B.D.); The Wilson Centre (K.M.K., B.H.); and Sunnybrook Health Sciences Centre (D.W.), Toronto, Ontario, Canada. The authors report that they have no conflicts of interest in the authorship and publication of this article. Received January 15, 2016; accepted May 5, 2016. Address correspondence to Tim Dwyer, M.B.B.S., F.R.A.C.S., F.R.C.S.C., Ph.D., Women’s College Hospital, 76 Grenville St., Toronto, Ontario M5S 1B1, Canada. E-mail: [email protected] Ó 2016 by the Arthroscopy Association of North America 0749-8063/1629/$36.00 http://dx.doi.org/10.1016/j.arthro.2016.05.037
using regular assessment to ensure a minimum level of competence is achieved by the conclusion of a resident’s rotation (e.g., sports medicine) before progression to the next rotation, orthopaedic education can change from time-based programs to ones based on observable and measurable outcomes.2-4 Previous work has validated methods for assessing medical knowledge5 and communication skills6 in orthopaedic residents. However, valid, reliable, and objective methods of assessing competence in the performance of technical procedures have yet to be fully defined in orthopaedics. Although the gold standard will likely always be in the operating room, difficulty standardizing operations,7 issues of patient safety, clinical outcomes, as well as time restraints8 mean that residents often cannot perform complex procedures to completion in that environment. As such, we believe that simulation-based assessment may offer part of the solution to this complex problem.9
Arthroscopy: The Journal of Arthroscopic and Related Surgery, Vol
-,
No
-
(Month), 2016: pp 1-10
1
2
T. DWYER ET AL.
At our university, residents undertake two 3-month orthopaedic sports medicine rotations during their training, once as a junior (postgraduate year [PGY] 1-3) and once as a senior (PGY 4 and 5). During these rotations, their education is composed of clinical and operating room exposure, regular didactic lectures based on a set curriculum, and the teaching of technical procedures in a laboratory using anatomic knee and shoulder dry models. The skills laboratory is also available for use on an unlimited, independent basis throughout the rotation. At the end of the rotation, medical knowledge is assessed with an Objective Structured Clinical Examination (OSCE)5 and technical skill with an Objective Structured Assessment of Technical Skills (OSATS) using dry models. The OSATS was originally developed at the University of Toronto as an objective measurement format used to assess the technical competency of surgical trainees.10 An OSATS consists of a multistation, OSCE-like examination, in which candidates perform a series of standardized surgical tasks.11 Held either with highfidelity models (cadavers, animals) or low-fidelity bench models (Sawbones), assessment usually takes the form of a task-specific checklist in combination with a global rating scale (GRS) and is typically combined with a final rating such as shows proficiency or requires further practice.11 At this time, it is thought that an OSATS is the most accepted gold standard for objective skills assessment.12 The purpose of this study was to determine if the use of an OSATS, using dry models, would be a valid method of assessing residents’ ability to perform sports
medicine procedures after training in a competencybased model. We hypothesized that a simulated OSATS using dry models would show evidence of validity, suitable for use as an in-training assessment of competence after a sports medicine rotation. We also hypothesized that after training within a CBME model, junior and senior residents would be able to perform technical procedures at a similar level.
Methods In a prospective study, all residents who undertook the 3-month sports medicine rotation under the training of 6 staff surgeons were included in this study. At the beginning of the rotation, each resident was provided with a list of 10 technical procedures that they were expected to show competency in at the end of the rotation (Table 1). The surgical procedures were selected by a focus group of academic orthopaedic sports surgeons and corresponded with the sports medicine curriculum. At the end of each rotation, 6 of these technical procedures were selected for assessment in an OSATS, which was composed of 6 stations, each 10 minutes long. Stations were selected by examiners after each rotation, to allow reasonable representation of all stations across all groups of participants over the course of this study. Stations were not selected randomly as some of the stations (drilling of tibial tunnel/drilling of transtibial femoral tunnel, insertion glenoid anchor/passage labral suture, insertion rotator cuff anchor/passage rotator cuff suture) were best performed together from a practical point of view. All of
Table 1. List of Technical Procedures That Residents Were Expected to Show a Minimal Level of Competence at After a Sports Rotation Task Perform partial meniscectomy Whipstitch hamstring graft
Drill anteromedial ACL femoral tunnel Drill transtibial ACL femoral tunnel Drill ACL tibial tunnel Insert glenoid anchors Pass labral sutures Insert rotator cuff anchors Pass rotator cuff suture Tie sliding, locking arthroscopic knot ACL, anterior cruciate ligament.
Component of Task Radial tear created in the posterior horn of medial meniscus Resident directed to perform partial medical meniscectomy Two equal lengths of shoestring provided Resident directed to whipstitch each end and pretension graft on Smith & Nephew Graftmaster II (Andover, MA) Resident directed to drill femoral tunnel using anteromedial portal, with 8-mm tunnel and with 15-mm Endobutton (Smith & Nephew) femoral fixation Resident directed to drill femoral tunnel using transtibial tunnel (tibial predrilled with size 10-mm tunnel) Resident directed to drill tibial tunnel Resident directed to insert 2.3-mm glenoid anchor (Bioraptor; Smith & Nephew) at the 5 o’clock position Resident directed to pass labral sutures using Accu-Pass (Smith & Nephew) suture passing device Resident directed to insert 5-mm metal Twinfix anchor (Smith & Nephew) into footprint of humeral head Resident directed to pass 2 sutures through tendon; both Scorpion and Scorpion Fast Pass available for use (Arthrex) Faculty insert rotator cuff anchor and pass sutures prior to performance of knot tying (Shoulder Arthroscopy Model)
USE OF AN OSATS AFTER A SPORTS MEDICINE ROTATION
the residents at the end of each rotation were assessed on the same 6 procedures. Most shoulder procedures (insertion glenoid anchors, labral suture passage, insertion rotator cuff anchors, rotator cuff suture passage) were performed on the Arthroscopic SOFT Shoulder Model (model DS-012; Arthrex, Naples, FL) bench-top simulator using replacement joint capsule assembly with cuff tear (DS-0123SST) (Fig 1). Faculty inserted arthroscopic cannulas into the subacromial space or the glenohumeral joint prior to the assessment. Knot tying was performed on a Shoulder Arthroscopy Model (S.A.M., www.arthrodemo.com); faculty placed a rotator cuff anchor and passed both suture limbs through the tissue prior to performance of a sliding, locking arthroscopic knot. Components of anterior cruciate ligament (ACL) reconstruction (drilling anteromedial femoral tunnel, transtibial femoral tunnel, and tibial tunnel) and partial meniscectomy were performed using an ACL Sawbones model (Sawbones, Vashon, WA), using an encapsulated knee insert (model 1414-1) inside a soft tissue with skin (model 1413-1) (Fig 2). All arthroscopic portals (anterolateral, anteromedial, accessory anteromedial) were made prior to the assessment. Standard 30 arthroscopes with high-definition video systems were used for all procedures, in association with disposable implants and standard equipment (Smith & Nephew, Andover, MA, and Arthrex). Whipstitching of a hamstring graft was simulated using 2 equal lengths of shoestring and the Smith & Nephew Graftmaster II. Residents were evaluated using a combination of task-specific checklists (see Appendix Tables 1-10, available at www.arthroscopyjournal.org), a previously validated GRS (the Arthroscopic Surgical Skill Evaluation Tool [ASSET]),13 and an overall final 5-point GRS. Task-specific checklists were created for each technical procedure using established arthroscopy texts, and by a panel of experienced sports medicine staff
3
surgeonsdconsensus was achieved using a modified Delphi procedure conducted by way of multiple surveys.13 The ASSET is made up of 8 domains (safety, field of view, camera dexterity, instrument dexterity, bimanual dexterity, flow of procedure, quality of procedure, and autonomy) based on the Dreyfus model of skill acquisition and is scored to a maximum of 38.13 The ASSET was designed to be generalizable to multiple procedures and settings (operating room and simulation) and has been found to be reliable and valid in the setting of diagnostic knee arthroscopy in cadaveric specimens.13 Finally, independent of the ASSET, each examiner allocated an overall 5-point GRS corresponding to the Dreyfus model of skill acquisition (novice, advanced beginner, competent, proficient, expert).14,15 Examiners were instructed to designate a resident performance as competent if they performed at the level of a graduating orthopaedic surgeon. A pilot OSATS was run using 2 junior residents, to establish the practicality of each station (sufficient time to complete each task) and optimize the checklist for each station. All residents at the end of each sports rotation undertook the OSATS; the year of training of the participants did not affect the procedure selection. All staff surgeons with fellowship training in sports medicine at our institution and all sports medicine fellows also sat the OSATS. A single examiner was present at each station; examiners were orthopaedic sports medicine surgeons who were involved in the training of the residents and fellows. All the examiners had clinical practices encompassing knee and shoulder surgery. All examiners were familiar with the checklists (having been involved in their creation), the ASSET, and the overall GRS. Examiners were available to assist as directed by or requested by residents, but no guidance with regards to the procedure was provided at any time. There was no time limit for each procedure, nor was the amount of time to complete each task recorded.
Fig 1. (A) The Arthroscopic SOFT Shoulder Model (Arthrex) bench-top simulator used for insertion of glenoid anchor and labral repair. Anterosuperior and anterior-inferior cannulas have been inserted into the rotator interval before performance of the technical procedure. (B) Instrumentation available for performance of this skill (Smith & Nephew).
4
T. DWYER ET AL.
Fig 2. (A) The knee model (Sawbones) used for performance of partial medial meniscectomy, drilling of the tibial tunnel, and drilling of the femoral tunnel using both the anteromedial and transtibial technique. Again all portals were made prior to the assessment. (B) Setup provided for whipstitch of hamstring graft.
Each participant performed each procedure once only. Videotaping of all procedures (both hand movements and arthroscopic component) was performed, which was then reviewed by a blinded, independent sports medicine fellow. Statistical Analysis Stata, version 13 (College Station, TX), was used to perform all analyses. The total score of the ASSET was calculated as the sum of the 8 domains, with a maximum score of 38. The internal consistency/ reliability of the total ASSET score, total checklist score, and the overall GRS was assessed using Cronbach alpha. Construct validity (the ability to discriminate between different years of training) was assessed using 1-way analysis of variance, with year of training (junior resident, senior resident, fellow, staff) as the independent variable. Unpaired t tests were used to examine for differences between groups. Concurrent validity was sought by examining the correlation between the end of rotation OSCE results for each of the residents, and the OSATS, using Pearson correlation. Inter-rater reliability between the examiners and blinded video reviewer was calculated using the intraclass correlation coefficient for total ASSET score. Reliability was also calculated using generalizability theory, analyzed with G-String IV software (Bloch & Norman, Hamilton, Ontario, Canada) using common station combinations and estimating the variance created by participants, their training levels, stations, as well as items within stations. A sample size calculation was performed; a minimum of 8 residents was required in each group (junior, senior) to detect a difference of 1.0 in the overall GRS, using a mean expected global rating of 3 (competent), standard deviation of 1.0, alpha of 0.05, and a power of 0.8. Costs for equipment and models were also calculated per participant, excluding implants that were donated by industry.
This study was conducted after approval by the institutional research ethics board. This study was supported by a Core Competency Innovation Grant from the OMeGA Medical Grants Association.
Results Between September 2013 and March 2015, a total of 27 residents (19 junior [postgraduate year (PGY) 1-3] and 8 senior [PGY 4 and 5]) sat an OSATS after their rotation, in addition to 7 sports medicine staff and 7 fellows, for a total of 41 participants (Table 2). No resident sat the examination both as a junior and as a senior, and no residents were excluded. The internal consistency/reliability (Cronbach alpha) of the OSATS was excellent (0.95). A good correlation was seen between the overall checklist score and the overall global rating (0.71). For no station did the Cronbach alpha if Item Deleted increase, indicating that all stations were performing well. Analysis of variance revealed a significant difference by groups for the overall GRS, the total ASSET score, and the total checklist score for all stations combined (P < .001) (Fig 3). A significant difference (analysis of variance) was also seen for the ASSET score for each individual station by level of training (Fig 4) (all P < .001). Further analysis using a t test revealed a Table 2. Demographics of Participants Undertaking the Objective Structured Assessment of Technical Skills Year of Training PGY 1 PGY 2 PGY 3 PGY 4 PGY 5 Fellows Staff PGY, postgraduate year.
Number of Participants 13 2 4 4 4 7 7
USE OF AN OSATS AFTER A SPORTS MEDICINE ROTATION
5
Fig 3. (A) Box plot for the overall checklist scores calculated as a percentage. (B) Box plot for the total ASSET global rating. (C) Box plot for the overall global rating. There was a significant difference by year of training for each (P < .001). (ASSET, Arthroscopic Surgical Skill Evaluation Tool.)
significant difference in the total ASSET score between junior (mean 18.4, 95% confidence interval [CI] 16.8 to 19.9) and senior residents (24.2, 95% CI 22.7 to 25.6), senior residents and fellows (30.1, 95% CI 28.2 to 31.9), and between fellows and faculty (37, 95% CI 36.1 to 27.8) (P < .05). The number of participants deemed competent (overall GRS of competent/proficient/expert) for each technical procedure is seen in Tables 3 and 4, whereas an example of scores from 4 residents at the end of their rotation is seen in Table 5. The inter-rater reliability between the examiners ratings, and the blinded video reviewer for the total ASSET score using intraclass correlation coefficient was 0.9. A good correlation was seen between performance on the end of rotation OSCE (overall GRS summed over the 6 stations) and each resident’s performance total ASSET score on the OSATS (0.76). A generalizability coefficient was calculated using the knot tying, partial meniscectomy, and whip stitching stations as well as the stations of glenoid anchor insertion and labral suturing; the relative error coefficients (interstation reliability) were 0.64 and 0.57, respectively. Using these calculations, it was estimated that a 6-station examination would confer an overall reliability of 0.78. The major source of variance (50%) was training level of the participant. During this study, 39 Sawbones knees ($69.75 each) with 5 knee skins ($83.70 each) were used for components of ACL reconstruction and meniscectomy. For components of rotator cuff and labral repair, 4 Arthroscopic SOFT Shoulder Models ($325) and 25 replacement inserts ($85) were required, whereas knot tying was performed using a single S.A.M model ($495) with 18 basic rotator cuff felt components ($6) and 18 humeral head replacements ($25). The total cost was $7,616.75, which averaged $185 for the OSATS per participant.
Discussion
The most important finding of this study is that the scores from the OSATS showed evidence of validity
when used to assess the performance of technical procedures by residents on dry models after a sports medicine rotation. Evidence of validity included excellent internal consistency, high inter-rater reliability, evidence of novice-expert differentiation, as well as a good correlation between performance on the OSATS and the end of rotation OSCE. The results of this study also showed that junior residents are not able to perform these simulated procedures as well as senior residents, suggesting that a combination of increased exposure to sports medicine, and overall surgical experience is a factor in the acquisition of these technical skills. A recent survey of orthopaedic program directors and residents identified that a significant number believed that surgical skills simulation should become a required part of training.16 Simulation-based training and assessment allows the opportunity for residents to attempt complex surgical tasks, without risk to patients. Simulation also allows residents the opportunity to make independent, intraoperative decisions, taking responsibility for and managing the consequences of their actionsdtangled sutures and improperly inserted anchors are one example. Simulation provides a staff surgeon with valuable insight into each resident’s knowledge of procedures, and their familiarity with instrumentation prior to performance in the operating room. Options for simulation in orthopaedics include highfidelity models (cadavers), virtual reality, and lowfidelity models (Sawbones). Cadavers are thought to be the gold standard for simulation training,7,17 with studies looking at the performance of knee arthroscopy in cadavers showing good reliability, as well as the ability to differentiate between different years of training.18,19 However, cadaver labs require advanced facilities and have significant costs and may not be a feasible option for regular assessment of technical procedures. Virtual reality simulation is commercially available for knee and shoulder arthroscopy20-24;
6
T. DWYER ET AL.
Fig 4. Box plots for the total ASSET score for each of the stations, with the ASSET score presented as a percentage. A significant difference by year of training was seen for all technical procedures. (ASSET, Arthroscopic Surgical Skill Evaluation Tool.)
however, virtual reality has limited ability to allow the performance of complex procedures such as the drilling of ACL tunnels. The advantages of dry models include relative ease of preparation, reduced cost, and the opportunity for unsupervised practice.25 Most comparative studies have shown that low-fidelity simulators are similarly effective but less expensive than high-fidelity simulators with regard to the acquisition of surgical skills.26-33 The finding that many residents were unable to perform the sports medicine procedures in the
simulated setting warrants attention. This finding is task dependent: although most of the procedures were difficult for the junior residents, some procedures were also difficult for senior residents to complete. This was especially noticeable with the simulated arthroscopic shoulder procedures; one reason may be that these are advanced skills, requiring significant experience or fellowship training. It may also highlight deficiencies in our training program. In the operating room, staff surgeons may be reluctant to allow residents to perform these procedures, because of the complexity of the
7
USE OF AN OSATS AFTER A SPORTS MEDICINE ROTATION
Table 3. Number and Percentage of Participants Deemed Competent or Better in Each Knee Station Using the Overall Global Rating Scale (Novice, Advanced Beginner, Competent, Proficient, Expert) Participants Junior residents Competent Mean ASSET score (95% Senior residents Competent Mean ASSET score (95% Fellows Competent Mean ASSET score (95% Staff Competent Mean ASSET score (95%
Anteromedial Femoral Tunnel
Transtibial Femoral Tunnel
Tibial Tunnel
CI)
1/9 (11%) 18.6 (15.2-22)
1/8 (12.5%) 15 (6.4-23.6)
4/14 (29%) 19.5 (13.9-25.1)
CI)
3/5 (60%) 24.5 (18.5-30.5)
3/3 (100%) 23 (20.8-25.2)
5/6 (83%) 27.5 (20-35)
CI)
4/4 (100%) 32.8 (27.1-38.4)
3/3 (100%) 27.7 (18.9-34.4)
CI)
5/5 (100%) 36.6 (35.3-37.9)
3/3 (100%) 38 (0-0)
Partial Meniscectomy 4/9 (44%) 19.4 (15-23.8)
Whipstitch Hamstring Graft 5/12 (42%) 20.6 (15.8-25.4)
5/6 (83%) 22 (18.7-25.3)
5/5 (100%) 27.36 (25.1-29.6)
3/4 (75%) 27 (18.9-35.1)
7/7 (100%) 29.7 (25.1-34.3)
6/7 (86%) 28 (25.2-30.8)
7/7 (100%) 38 (0-0)
7/7 (100%) 35.9 (34.9-37)
4/4 (100%) 33.3 (30.3-36.6)
ASSET, Arthroscopic Surgical Skill Evaluation Tool; CI, confidence interval.
individual steps, as opposed to simpler procedures such as knee arthroscopy.34 It was also interesting to note that a small number of fellows were not competent at procedures such as drilling the tibial tunnel or whipstitching a hamstring graft; this may highlight a deficiency in fellowship training, such as limited exposure to hamstring ACL reconstruction in some rotations. The finding that there is a significant difference in performance between junior and senior residents is at odds with the concept of CBME, which in theory demands a minimal level of competence prior to residents’ proceeding to the next rotation. The results of this study, and other studies looking at the acquisition of clinical skills in orthopaedic residents,6,35,36 suggest that there is an effect of overall clinical experience gained throughout training that has an effect on the performance of all technical procedures. How to best deal with these results is unknown. One approach would be to increase the frequency of teaching in the simulation laboratory; however, these are already being held twice monthly over the 3-month rotation. These training sessions are not without cost,
both in terms of faculty time and costs of models and implants; for this reason the number of skills sessions cannot be easily increased. Alternatively, a milestones approach, as recommended by the Accreditation Council for Graduate Medical Education, could be implemented.37 In this way, although teaching and training sessions would continue to cover the surgical procedures listed, junior residents would be expected to show a minimal level of competency at less complex tasks (partial meniscectomy, whipstitch hamstring graft), whereas it seems that senior residents can be expected to perform aspects of ACL reconstruction and rotator cuff repair. Importantly, this study does not provide any correlation between the procedures performed in the simulated setting and operative performance. Although there are some studies showing that bench model training improves operative performance in orthopaedics,25,33 the evidence that simulation-based assessment correlates with actual surgical performance is more limited,38 although studies in other surgical specialties have consistently identified good correlations.39-42 The
Table 4. Number and Percentage of Participants Deemed Competent or Better in Each Shoulder Station Using the Overall Global Rating Scale (Novice, Advanced Beginner, Competent, Proficient, Expert) Participants Junior residents Competent Mean ASSET score (95% Senior residents Competent Mean ASSET score (95% Fellows Competent Mean ASSET score (95% Staff Competent Mean ASSET score (95%
Insertion Glenoid Anchors
Pass Labral Sutures
Insertion Rotator Cuff Anchors
Pass Rotator Cuff Suture
Tie Arthroscopic Knot
CI)
4/11 (36%) 19.4 (15.6-23.1)
2/11 (18%) 18.1 (15.2-21.1)
5/13 (38%) 16.8 (11.5-22.1)
5/13 (38%) 16.5 (10.7-22.3)
3/16 (19%) 17.2 (14.4-20.2)
CI)
2/5 (40%) 25.7 (21.5-29.8)
2/5 (40%) 24 (20.1-27.9)
4/4 (100%) 24.8 (23.2-26.3)
2/4 (50%) 20.5 (18.3-22.7)
5/7 (71%) 22.3 (17.8-26.9)
CI)
3/3 (100%) 33 (28.8-37.1)
3/3 (100%) 31.3 (28.3-34.4)
4/4 (100%) 29 (24.7-33.2)
4/4 (100%) 29 (24.8-33.2)
7/7 (100%) 27.1 (25-29.3)
CI)
3/3 (100%) 34 (27.7-40.3)
3/3 (100%) 34.3 (26.7-42)
4/4 (100%) 37 (34.9-39.1)
4/4 (100%) 37.5 (36.4-38.6)
6/6 (100%) 34.4 (33-35.8)
ASSET, Arthroscopic Surgical Skill Evaluation Tool; CI, confidence interval.
8
T. DWYER ET AL.
Table 5. Example Ratings From 4 Residents at the End of Their 3-Month Rotation Participants PGY 4 PGY 4 PGY 1 PGY 1
Tibial Tunnel Competent Competent Novice Novice
Transtibial Femoral Tunnel Competent Competent Advanced beginner Novice
Partial Meniscectomy Competent Competent Advanced beginner Novice
Insertion Glenoid Anchor Proficient Advanced beginner Novice Advanced beginner
Passage Labral Suture Advanced beginner Proficient Novice Novice
Arthroscopic Knot Tie Advanced beginner Competent Advanced beginner Advanced beginner
PGY, postgraduate year.
reasons for this deficiency in the literature is likely the difficulty standardizing operations, time restraints, and issues of patient safety.7,8 Although there is no reason to believe that residents who cannot perform these procedures in the simulated setting would be able to in the workplace, this is an important area of future research in CBME. The OSATS study identified a correlation between performance of technical procedures, and the OSCE scores for each resident. However, the results of other studies seeking a correlation between medical knowledge and technical skills have been mixed.43 A recent study looking at simulated surgery on a distal radius fracture failed to identify a correlation between knowledge testing and biomechanical testing of the surgical construct.44 In 2009, Van Heest et al.45 reported on orthopaedic residents performing a carpal tunnel release on a cadaver, in association with a knowledge test; although a knowledge score <70/100 predicted failure on the OSATS, a score of >70/100 did not ensure that the examinee would pass the test. These findings are plausible: although there is a threshold of knowledge that is required, above that threshold, a combination of technical abilities and surgical experience has an important role. Limitations The most important limitation of this study is that we are unable to show a correlation between performance on this OSATS study, and performance in the operating room, as intra-operative assessments are not at this time routinely performed for all residents. Although all residents were also assessed with an In-Training Evaluation Report, these results were not available for correlation with performance on the OSATS. Previous research from the author has identified a correlation between a resident’s In-Training Evaluation Report and skills such as communication, professionalism, collaboration, etc.6 Furthermore, although residents were exposed to training on dry models, inevitably each will have had different clinical exposures during their rotations. It is also unknown how much each resident took advantage of the skills lab outside of teaching sessions. The number of sports medicine and other procedures each resident had experience with is unknown; for this reason, we cannot say whether these results are a result of sports medicine experience,
overall surgical experience, or a combination of both. We are unable to provide an intrarater reliability, as no examiner reviewed the video recordings of participants. No baseline testing of residents was performed, which would have allowed for evaluation of each resident’s knowledge and skill level prior to commencing the rotation. Stations were not selected randomly but rather with a view to establishing even representation of all stations throughout the study period, while maximizing practicality. Finally, it is important to note that the assessment of competency always requires the exercise of judgment by another competent, proficient, or expert practitioner; for this reason, the development of an objective assessment, or assessment of competency independent of the assessor, is not always possible.46 An attempt to minimize these subjective effects is made via a process of standardization, and by reaching consent with regards to what is being evaluated and the standards expected.47
Conclusions The results of this study reveal that an OSATS using dry models shows evidence of validity when used to assess performance of technical procedures after a sports medicine rotation. However, junior residents were not able to perform as well as senior residents, suggesting that overall surgical experience is as important as intensive teaching.
References 1. Williams RG, Verhulst S, Colliver JA, Sanfey H, Chen X, Dunnington GL. A template for reliable assessment of resident operative performance: Assessment intervals, numbers of cases and raters. Surgery 2012;152:517-524. discussion 24-27. 2. Hodges BD. A tea-steeping or i-Doc model for medical education? Acad Med 2010;85:S34-S44 (suppl). 3. Pugh D, Touchie C, Wood TJ, Humphrey-Murto S. Progress testing: Is there a role for the OSCE? Med Educ 2014;48:623-631. 4. Long DM. Competency-based residency training: The next advance in graduate medical education. Acad Med 2000;75:1178-1183. 5. Dwyer T, Theodoropoulos JS, Herold J, et al. Assessing competence of orthopaedic residents: The reliability and validity of an objective structured clinical examination
USE OF AN OSATS AFTER A SPORTS MEDICINE ROTATION
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.
22.
after a sports medicine rotation. J Bone Joint Surg Am 2013;95:e177. Dwyer T, Glover Takahashi S, Kennedy Hynes M, et al. How to assess communication, professionalism, collaboration and the other intrinsic CanMEDS roles in orthopedic residents: Use of an objective structured clinical examination (OSCE). Can J Surg 2014;57:230-236. Reznick R, Regehr G, MacRae H, Martin J, McCulloch W. Testing technical skill via an innovative “bench station” examination. Am J Surg 1997;173:226-230. Farnworth LR, Lemay DE, Wooldridge T, et al. A comparison of operative times in arthroscopic ACL reconstruction between orthopaedic faculty and residents: The financial impact of orthopaedic surgical training in the operating room. Iowa Orthop J 2001;21:31-35. Dwyer T, Slade Shantz J, Chahal J, et al. Simulation of anterior cruciate ligament reconstruction in a dry model. Am J Sports Med 2015;43:2997-3004. Martin JA, Regehr G, Reznick R, et al. Objective Structured Assessment of Technical Skill (OSATS) for surgical residents. Br J Surg 1997;84:273-278. Sidhu RS, Grober ED, Musselman LJ, Reznick RK. Assessing competency in surgery: Where to begin? Surgery 2004;135:6-20. van Hove PD, Tuijthof GJ, Verdaasdonk EG, Stassen LP, Dankelman J. Objective assessment of technical surgical skills. Br J Surg 2010;97:972-987. Koehler RJ, Amsdell S, Arendt EA, et al. The Arthroscopic Surgical Skill Evaluation Tool (ASSET). Am J Sports Med 2013;41:1229-1237. Carraccio CL, Benson BJ, Nixon LJ, Derstine PL. From the educational bench to the clinical bedside: Translating the Dreyfus developmental model to the learning of clinical skills. Acad Med 2008;83:761-767. Batalden P, Leach D, Swing S, Dreyfus H, Dreyfus S. General competencies and accreditation in graduate medical education. Health Aff (Millwood) 2002;21:103-111. Karam MD, Pedowitz RA, Natividad H, Murray J, Marsh JL. Current and future use of surgical skills training laboratories in orthopaedic resident education: A national survey. J Bone Joint Surg Am 2013;95:e4. Anastakis DJ, Regehr G, Reznick RK, et al. Assessment of technical skills transfer from the bench training model to the human model. Am J Surg 1999;177:167-170. Olson T, Koehler R, Butler A, Amsdell S, Nicandri G. Is there a valid and reliable assessment of diagnostic knee arthroscopy skill? Clin Orthop Relat Res 2013;471: 1670-1676. Insel A, Carofino B, Leger R, Arciero R, Mazzocca AD. The development of an objective model to assess arthroscopic performance. J Bone Joint Surg Am 2009;91:2287-2295. Gomoll AH, O’Toole RV, Czarnecki J, Warner JJ. Surgical experience correlates with performance on a virtual reality simulator for shoulder arthroscopy. Am J Sports Med 2007;35:883-888. Gomoll AH, Pappas G, Forsythe B, Warner JJ. Individual skill progression on a virtual reality simulator for shoulder arthroscopy: A 3-year follow-up study. Am J Sports Med 2008;36:1139-1142. McCarthy AD, Moody L, Waterworth AR, Bickerstaff DR. Passive haptics in a knee arthroscopy simulator: Is it valid
23.
24.
25.
26.
27.
28.
29.
30.
31.
32.
33.
34.
35.
36.
37. 38.
9
for core skills training? Clin Orthop Relat Res 2006;442: 13-20. Pedowitz RA, Esch J, Snyder S. Evaluation of a virtual reality simulator for arthroscopy skills development. Arthroscopy 2002;18:E29. Srivastava S, Youngblood PL, Rawn C, Hariri S, Heinrichs WL, Ladd AL. Initial evaluation of a shoulder arthroscopy simulator: Establishing construct validity. J Shoulder Elbow Surg 2004;13:196-205. Butler A, Olson T, Koehler R, Nicandri G. Do the skills acquired by novice surgeons using anatomic dry models transfer effectively to the task of diagnostic knee arthroscopy performed on cadaveric specimens? J Bone Joint Surg Am 2013;95. e15(1-8). Chandra DB, Savoldelli GL, Joo HS, Weiss ID, Naik VN. Fiberoptic oral intubation: The effect of model fidelity on training for transfer to patient care. Anesthesiology 2008;109:1007-1013. Grober ED, Hamstra SJ, Wanzel KR, et al. The educational impact of bench model fidelity on the acquisition of technical skill: The use of clinically relevant outcome measures. Ann Surg 2004;240:374-381. Matsumoto ED, Hamstra SJ, Radomski SB, Cusimano MD. The effect of bench model fidelity on endourological skills: A randomized controlled study. J Urol 2002;167:1243-1247. McDougall EM, Kolla SB, Santos RT, et al. Preliminary study of virtual reality and model simulation for learning laparoscopic suturing skills. J Urol 2009;182:1018-1025. Zendejas B, Wang AT, Brydges R, Hamstra SJ, Cook DA. Cost: The missing outcome in simulation-based medical education research; a systematic review. Surgery 2013;153:160-176. Alvand A, Auplish S, Gill H, Rees J. Innate arthroscopic skills in medical students and variation in learning curves. J Bone Joint Surg Am 2011;93:e115(1-9). Howells NR, Auplish S, Hand GC, Gill HS, Carr AJ, Rees JL. Retention of arthroscopic shoulder skills learned with use of a simulator. Demonstration of a learning curve and loss of performance level after a time delay. J Bone Joint Surg Am 2009;91:1207-1213. Howells NR, Gill HS, Carr AJ, Price AJ, Rees JL. Transferring simulated arthroscopic skills to the operating theatre: A randomised blinded study. J Bone Joint Surg Br 2008;90:494-499. Hodgins JL, Veillette C, Biau D, Sonnadara R. The knee arthroscopy learning curve: Quantitative assessment of surgical skills. Arthroscopy 2014;30:613-621. Dwyer T, Wright S, Kulasegaram KM, et al. Competencybased medical education: Can both junior residents and senior residents achieve competence after a sports medicine training module? J Bone Joint Surg Am 2015;97: 1985-1991. Dwyer T, Wadey V, Archibald D, et al. Cognitive and psychomotor entrustable professional activities: Can simulators help assess competency in trainees? Clin Orthop Relat Res 2016;474:926-934. ACGME. The Orthopaedic Surgery Milestone Project 2013. July 20, 2015. Korndorffer JR Jr, Kasten SJ, Downing SM. A call for the utilization of consensus standards in the surgical education literature. Am J Surg 2010;199:99-104.
10
T. DWYER ET AL.
39. Datta V, Bann S, Beard J, Mandalia M, Darzi A. Comparison of bench test evaluations of surgical skill with live operating performance assessments. J Am Coll Surg 2004;199:603-606. 40. Ghaderi I, Vaillancourt M, Sroka G, et al. Performance of simulated laparoscopic incisional hernia repair correlates with operating room performance. Am J Surg 2011;201: 40-45. 41. McCluney AL, Vassiliou MC, Kaneva PA, et al. FLS simulator performance predicts intraoperative laparoscopic skill. Surg Endosc 2007;21:1991-1995. 42. Beard JD, Jolly BC, Newble DI, Thomas WE, Donnelly J, Southgate LJ. Assessing the technical skills of surgical trainees. Br J Surg 2005;92:778-782. 43. Bann S, Khan MS, Datta VK, Darzi AW. Technical performance: Relation between surgical dexterity and
44.
45.
46. 47.
technical knowledge. World J Surg 2004;28:142-146. discussion 6-7. Putnam MD, Kinnucan E, Adams JE, Van Heest AE, Nuckley DJ, Shanedling J. On orthopedic surgical skill predictiondthe limited value of traditional testing. J Surg Educ 2015;72:458-470. Van Heest A, Putnam M, Agel J, Shanedling J, McPherson S, Schmitz C. Assessment of technical skills of orthopaedic surgery residents performing open carpal tunnel release surgery. J Bone Joint Surg Am 2009;91: 2811-2817. Brooks MA. Medical education and the tyranny of competency. Perspect Biol Med 2009;52:90-102. Ginsburg S, McIlroy J, Oulanova O, Eva K, Regehr G. Toward authentic clinical evaluation: pitfalls in the pursuit of competency. Acad Med 2010;85:780-786.
10.e1
USE OF AN OSATS AFTER A SPORTS MEDICINE ROTATION Appendix Table 1. Task-Specific Checklist: Anteromedial Femoral Tunnel
Appendix Table 3. Task-Specific Checklist: Tibial Tunnel Correct
Correct
Incorrect
Appropriate portal Correct knee position for technique, i.e., hyperflexed Uses femoral offset guide Appropriate offset, i.e., 6 mm Correct position on wall, i.e., between 10 and 11 Estimates guide pin exit before drilling Correct guide pin exit, i.e., between PFJ and lateral IM septum Passes Endobutton drill Measures depth Drills to correct depth (i.e., within 10 mm) Checks back wall Back wall intact
Incorrect
Knee in flexion Knowledge of landmarks Correct tibial jig angle Places in center of tibia shaft Discussion of ML landmarks Discussion of AP landmarks Care of guide pin when drilling Appropriate drill diameter NOTE. Resident is to drill a tibial tunnel for a 10-mm boneepatellar tendonebone graft. AP, anteroposterior; ML, mediolateral.
NOTE. Resident is to drill femoral tunnel for anterior cruciate ligament ACL reconstruction, using AM technique, 8-mm hamstring graft, and fixation with 20-mm Endobutton. ACL, anterior cruciate ligament; AM, anteromedial; IM, intermuscular; PFJ, patellofemoral joint.
Appendix Table 2. Task-Specific Checklist: Transtibial Femoral Tunnel Correct
Incorrect
Uses tibial tunnel Correct knee position for technique, i.e., 90 degrees Uses femoral offset guide Appropriate offset, i.e., 7 mm Correct position on wall, i.e., between 10 and 11 Estimates guide pin exit before drilling Correct guide pin exit, i.e., between PFJ and lateral IM septum Line to line drill diameter, i.e., 10-mm drill Drills to depth of 20-25 mm Checks back wall Back wall intact NOTE. Resident is to drill a femoral tunnel using previous tibial tunnel, for a 10-mm boneepatellar tendonebone graft, with a 20-mm bone block. IM, intermuscular; PFJ, patellofemoral joint.
Appendix Table 4. Task-Specific Checklist: Whipstitch Hamstring Graft Correct
Incorrect
Uses Graftmaster Able to set up Graftmaster Correct suture choice Stitches both tendons Able to whipstitch Places over Endobutton Able to pretension NOTE. Resident is to whipstitch the 2 tendons together, pass over Endobutton, and pretension on Graftmaster.
10.e2
T. DWYER ET AL.
Appendix Table 5. Task-Specific Checklist: Knee Arthroscopy and Partial Meniscectomy
Appendix Table 7. Task-Specific Checklist: Labral Suture Correct
Correct
Incorrect
Inspection of suprapatellar pouch Inspection of patellofemoral compartment Inspection of lateral gutter Inspection of medial gutter Inspection of medial compartment including anterior and posterior horns Inspection of intercondylar notch including the ACL and PCL Inspection of lateral compartment including anterior and posterior horn Identifies tear Correctly identifies tear site, i.e., posterior horn Correctly names type of tear Describes size of tear Able to resect tear Smooth margins Reassess debridement with probe
Incorrect
Separates sutures Gets assistant to hold arthroscope Correct instrument angulation, right v left Uses low anterior portal Bite inferior to anchor Good portion of labral tissue Able to suture shuttle NOTE. Resident is to used Accu-Pass to pass a labral suture, and then suture shuttle, from a glenoid anchor that is already inserted.
NOTE. Resident is to resect a radial tear in the posterior horn of the medial meniscus. ACL, anterior cruciate ligament; PCL, posterior cruciate ligament.
Appendix Table 6. Task-Specific Checklist: Insertion Glenoid Anchor Correct Uses drill guide Asks assistant to hold arthroscope Uses low anterior portal Places at 5-5.30 45 Perpendicular to clockface Correct drill insertion Maintains drill guide while placing anchor Notes is a tap in anchor Anchor inserted to correct depth Tests anchor security Ensures sutures are running
Incorrect
Appendix Table 8. Task-Specific Checklist: Rotator Cuff Anchor
‘
NOTE. Resident is to insert a tap-in anchor in the anterior glenoid.
Correct
Incorrect
Uses appropriate portal Places the trochar on footprint Insertion angle at 45 Assistant uses mallet or holds camera Able to insert anchor into pilot hole Inserts to correct depth Tests anchor security Ensures sutures are running NOTE. Resident is directed to place a rotator cuff anchor, and prepare for suture passage.
10.e3
USE OF AN OSATS AFTER A SPORTS MEDICINE ROTATION Appendix Table 9. Task-Specific Checklist: Rotator Cuff Suture Correct
Appendix Table 10. Task-Specific Checklist Arthroscopic Knot Tie
Incorrect
Suture managementdthrough separate portals Loads suture correctly Able to pass suture Adequate tissue bite Retrieves sutures through alternative portal Appropriate separation between the 2 sutures, i.e., 1 cm NOTE. Resident is to use a suture passer to pass 2 rotator cuff sutures from a preinserted anchor.
Correct
Incorrect
Knows name of knot Using a sliding knot Aware is a self-locking knot Shortens the post Correctly ties knot Dresses the knot Places knot pusher on post Snap/clip placed on post Knot tensioned down onto soft tissue Past pointing to lock knot Knot security with additional knots Able to flip knot/tie a square knot NOTE. Resident is to tie an arthroscopic sliding, self-locking knot and then tie additional secure knots.