- Email: [email protected]
Impact of an Intensive 2-day Endovascular Training Course on Technical Performance of Trainees
Impact of an Intensive 2-day Endovascular Training Course on Technical Performance of Trainees Rachel Rosenthal,1 Edin Mujagic,1 Augustinus Ludwig Jacob,2 Robert Seelos,3 Juliane Sch€ afer,1,4 and Lorenz G€ urke,1 Basel, Zurich, and Lucerne, Switzerland
Background: The objective of this study was to evaluate the effect of a 2-day international endovascular training course on the performance of trainees as compared with a control group, assessed in a bench modelebased task using an objective structured evaluation protocol. Methods: A total of 50 trainees, 28 course participants of 2 consecutive identical courses and a control group of 22 participants with a similar level of experience without course attendance, underwent baseline and final assessment (simulated arterial access task). The evaluation form consisted of a global assessment (GA), task-specific checklist percentage score (CL), and global rating scale percentage score (GR), with both percentage scores ranging from 0% (worst performance) to 100% (best performance). Results: Course participants were more likely to pass the GA at final testing than the control group (odds ratio ¼ 59; 95% confidence interval [CI] 9.5e656; P < 0.001). The estimated difference in percentage score at final testing between course participants and the control group was 26% (95% CI 18e34; P < 0.001) for the CL and 29% (95% CI 19e40; P < 0.001) for the GR. Conclusions: A 2-day structured endovascular training course significantly improves endovascular performance in a simulated environment. These results are important for the design of endovascular training curricula with the ultimate goal of contributing to patient safety.
INTRODUCTION Minimally invasive vascular procedures represent the core treatment option for many cerebrovascular, aortoiliac, and peripheral occlusive diseases. The growth of these procedures is expected to continue, 1 Department of Vascular Surgery, University Hospital Basel, Basel, Switzerland. 2
Center for Microtherapy, Klinik Hirslanden, Zurich, Switzerland.
3
Department of Vascular Surgery, Cantonal Hospital, Lucerne, Switzerland. 4 Basel Institute for Clinical Epidemiology and Biostatistics, University Hospital Basel, Basel, Switzerland.
Correspondence to: Lorenz G€ urke, MD, Department of Vascular Surgery, Vascular International School, University Hospital Basel, Spitalstrasse 21, 4031 Basel, Switzerland; E-mail: [email protected] Ann Vasc Surg 2013; 27: 1173–1181 http://dx.doi.org/10.1016/j.avsg.2013.01.015 Ó 2013 Elsevier Inc. All rights reserved. Manuscript received: October 2, 2012; manuscript accepted: January 28, 2013; published online: August 22, 2013.
and teaching programs must meet the demand for skilled interventional experts to conduct these operations.1 Skills essential for performing open vascular surgery are different from those required for interventional radiologic procedures or minimally invasive vascular procedures, respectively. Therefore, specific minimally invasive vascular training is essential. Because of increasing time constraints, cost, and the increasing complexity of interventions, the modern minimally invasive suite is not the optimal learning environment and the patient is not the optimal training object to gain initial experience in endovascular interventions. Using bench models and computer-based virtual reality (VR) simulators, trainees can practice standardized interventional tasks repeatedly with feedback on performance from their instructors and the simulators without jeopardizing patient safety. Whereas the assessment using simulators is 1173
1174 Rosenthal et al.
objective, assessment by experts may be subjective. To address this shortcoming, objective assessment instruments have been developed and validated (i.e., objective structured assessment of technical skills [OSATS]).2 OSATS have been used successfully for the assessment of endovascular tasks. Endovascular task performance in a VR environment assessed using OSATS by an expert interventionalist has been shown to correlate well with the trainees’ prior experience in endovascular procedures.3 In addition, OSATS have been used to evaluate the effect of training programs. Medical students participating in an 8-week endovascular simulation training program significantly improved their technical skills, patient safety measures, and structured global assessments.4 For advanced endovascular procedures, such as carotid artery stenting, a 2-day course using a VR simulator has been shown to significantly improve performance of experienced interventionalists as measured using simulator-derived metrics.5 The goal of this study was to evaluate the effect of a 2-day endovascular training course on the performance of trainees in a bench model task using objective criteria. In particular, we evaluated whether course participants perform better than trainees with similar experience who have not attended the course.
METHODS Study Participants All participants of 2 identical consecutive endovascular training courses organized by the Vascular International School (VIS [http://www.vascularinternational.org/]) and SwissIntervention (http:// www.swissintervention.ch) held in Lucerne, Switzerland, at the Academy for Medical Training and Simulation (AMTS) on November 4e5, 2010 and November 1e2, 2011, took part in the study and represent the intervention group. This course focuses on endovascular interventions, whereas other courses at the VIS offer vascular (main focus) and some endovascular training options. This addresses instructors’ and participants’ request for courses not exceeding 2e 3 days (organizational demand from the respective clinics) and allows trainees from specialties focusing on endovascular treatment, such as interventional radiologists or angiologists, to visit a course without open vascular training. The course is not mandatory for application to the Fellow of European Board of Vascular Surgery
Annals of Vascular Surgery
(FEBVS) examinations (www.uemsvascular.com), as long as it is not required for the certificate of completion of training of the applicants’ country. The study was designed as a controlled beforee after study with observations before and after the intervention in a group receiving the intervention and in a similar (without individual matching) control group not receiving the intervention. The control group consisted of a cohort of trainees recruited at the radiology and surgery department of the Basel University Hospital, Switzerland, with similar experience and not having attended the course. Due to organizational reasons, the control group underwent assessment at their home institution. All study participants were medical doctors with specialization or aimed specialization in radiology or surgery. Written informed consent was obtained from all study participants and the study was approved by the local ethics committee. Upon informed consent, the participants completed a questionnaire on baseline information and underwent baseline assessment (on day 1 immediately before the beginning of the course), attended the course if they were part of the intervention group, and underwent final assessment on day 2 immediately after the end of the course (intervention group). All participants underwent both examinations within 30 minutes, resulting in comparable time lags between the two assessments. The control group underwent the same assessment within the same schedule and time lag without course attendance (final assessment on the day after baseline assessment). Questionnaire on Baseline Characteristics Participants completed a questionnaire on baseline characteristics, including age, gender, postgraduate year, years of radiology/surgery training, and experience in interventional radiology (as interventionalist and as first/second-hand assistant). Baseline Assessment Next, all participants performed a standardized vascular procedure on a bench model at the VIS. It consisted of a percutaneous arterial puncture of a perfused pulsatile box model and introduction of a sheath over a guidewire applying the Seldinger technique. All study participants were given standardized instructions on draping the puncture site, performing a percutaneous arterial puncture, and introducing a sheath over a guidewire. The trainees were rated by experienced vascular
Vol. 27, No. 8, November 2013
interventionalists/surgeons who were briefed about the objective rating system prior to their first rating. To not delay the course, multiple assessors were necessary. Assessment was performed according to a structured assessment form based on the OSATS previously developed and validated by Reznick,2 and adapted for endovascular procedures according to recent literature3e5 (Fig. 1). The OSATS consists of 3 outcome measures: global assessment (pass/ fail) (GA); task-specific checklist percentage score (CL); and global rating scale percentage score (GR), with both percentage scores ranging from 0% (worst performance) to 100% (best performance). The CL corresponds to the percentage of items judged as correctly done. To obtain percentages, the GR was calculated by summing the marks of the 7 dimensions as converted from 1 (poor performance) to 5 (good performance) into 0 (poor performance) to 4 (good performance) and dividing by 28 (7 4). Two course participants did not have an assistant at final assessment. We calculated their GR accordingly out of 6 (instead of 7) dimensions. Intervention The intervention consisted of a 2-day international endovascular training course. This course focuses on endovascular interventions. In the training course, standardized bench models were used to train standard interventional and endovascular techniques, such as vascular puncture, antegrade, retrograde, and Seldinger techniques; arterial dilation; and application of arterial stents for arterial occlusive disease and aneurysm treatment. The focus was on practical exercises, which were accompanied by brief theoretical introductions to use of radiology suites, radiation safety, and relevant endovascular devices. A real angio-suite environment was simulated using the Pontresina model (Fig. 2). It consists of a silicon model of the aortoiliacofemoral arterial system. After injection of a simulated contrast medium, the arterial system is depicted taking advantage of a change in acid and base properties (pH value) and displayed on a screen as an arteriogram with the aid of a video camera. The control group did not attend the course. Final Assessment At the end of the course, all participants of both courses repeated the same standardized procedure on a bench model as described for the baseline assessment and were rated in the same structured manner. The control group repeated assessment at the same time interval after baseline assessment.
Endovascular training course 1175
Statistical Analyses The sample size of this study was given by the number of participants in the two consecutive identical training courses, whereas the control group had a comparable, slightly smaller sample size. For each assessment parameter (GA, CL, and GR), we employed an analysis of covariance model to estimate the difference in final assessment between course participants and the control group using the baseline assessment as a covariate. We used multiple linear regression for the continuous assessment parameters (CL and GR), and for the binary assessment parameter (GA) we used multiple logistic regression. For each analysis, we report an estimate (and 95% confidence interval) of the difference in final assessment between course participants and the control group, with adjustment for baseline assessment. For the GA, this estimate is on the odds ratio scale. All data were entered into an Excel spreadsheet (Microsoft Office XP; Microsoft Corp., Redmond, WA) and handled anonymously. We used R software (version 2.15.0; R Foundation for Statistical Computing, Vienna, Austria)6 and the R add-on package Lattice (version 0.20-6)7 for our analyses and for graphics, respectively.
RESULTS General Characteristics A total of 50 trainees participated in the study, 28 in the intervention group and 22 in the control group; there were 22 (44%) females and 28 (56%) males. One course participant of the first course had to leave the course prior to the final assessment for unanticipated personal reasons unrelated to the course and the study. One course participant of the second course did not participate in the study, because he did not attend the second course day. Five course participants of the second course were technicians/nurses and were excluded from analysis. With the exception of 1 participant, none in the control group had any opportunity to be involved in endovascular procedures between baseline and final assessment and the control group did not assist any faculty lectures or bedside teaching between the two assessment time-points. Table 1 presents an overview of the participant characteristics. The detailed results at baseline and final assessment of the GA are presented in Table 2 and of the CL and GR in Figure 3. The participant in the control group who was involved in endovascular procedures between baseline and final
1176 Rosenthal et al.
Annals of Vascular Surgery
Participant number
________
1. Global assessment:
□ pass
□ fail
2. Procedure-related metrics Total procedure time in minutes: _________
3. Task-specific checklist:
Item
1.
Not done/ Incorrectly
Done correctly
O
O
Puncture with correct needle orientationO
O
Draping (disinfection / correct localisation)
2.
(cut of the needle pointing upwards, position 45-60° as referred to the skin)
3.
Insertion of the guidewire
O
O
(under fluoroscopic control (as mentioned by trainee))
4.
Retraction of the needle
O
O
(under fixation of the guidewire)
5.
Introduction of the sheath
O
O
6.
Retraction of the dilatation device
O
O
7.
Flushing of the sheath
O
O
O
O
(first aspiration, then flushing, checks that no air is in the system, plunger upward-oriented)
8.
Heparin flushing
4. Global rating scale: 1 Respect for tissue
Time and motion Instrument handling
Knowledge of instruments Flow of operation
Frequently used unnecessary force on tissue or caused damage by inappropriate use of instruments Many unnecessary moves
Repeatedly makes tentative or awkward moves with instruments by inappropriate use of instruments Frequently asked for wrong instrument or used inappropriate instrument Frequently stopped operating and seemed unsure of next move
Use of Assistants
Consistently placed assistants poorly or failed to use assistants
Knowledge of Specific Procedure
Deficient knowledge. Needed specific instruction at most Steps
2
3
4
5
Careful handling of tissue but occasionally caused inadvertent damage Efficient time/motion but some unnecessary moves Competent use of instruments but occasionally appeared stiff or awkward
Consistently handled tissues appropriately with minimal damage
Knew names of most instruments and used appropriate instrument
Obviously familiar with the instruments and their names
Demonstrated some forward planning with reasonable progression of procedure Appropriate use of assistants most of the time
Obviously planned course of operation with effortless flow from one move to the next Strategically used assistants to the best advantage at all times
Knew all important Steps of operation
Demonstrated familiarity with all aspects of operation
Clear economy of movement and maximum efficiency Fluid moves with instruments and no awkwardness
Fig. 1. Structured assessment form, adapted from Reznick,2 Tedesco et al.,3 Lee et al.,4 and van Herzeele et al.5
Vol. 27, No. 8, November 2013
Endovascular training course 1177
DISCUSSION
Fig. 2. Example of a bench model used at the course: angio-suite.
assessment scored identically in both assessments. Of the 6 participants (27%) in the control group with previous experience as interventionalists, 4 (67%) passed both baseline and final assessment (global assessment) and 2 (33%) failed both assessments. One participant in the control group subsequently underwent the second course with a time lapse of several months. This participant failed both assessments of the control group and passed both assessments of the intervention group. Concerning all 3 outcome parameters, participants of the course, not of the control group, performed better at final as compared with baseline assessment. For example, of 17 course participants who failed the GA at baseline testing, 16 passed it at final testing.
Comparison of Course Participants and the Control Group Relative to the control group, course participants were more likely to pass the GA at final testing (odds ratio ¼ 59; 95% confidence interval [CI] 9.5e656; P < 0.001). The estimated difference in percentage score at final testing (adjusted for baseline) between course participants and the control group was 26% (95% CI 18e34; P < 0.001) for the CL and 29% (95% CI 19e40; P < 0.001) for the GR.
This prospective, interventional study demonstrates that participants of a 2-day endovascular training course perform significantly better at final assessment of a simulated arterial access task (adjusted for baseline assessment) concerning all 3 evaluation parameters, GA, CL, and GR, than a control group with similar experience not having attended the course. Vascular surgery has significantly evolved in recent decades. For years, it was part of general and cardiothoracic surgery.1,8 In the USA, the first Certificates of Special Qualifications in General Vascular Surgery were issued in 1982.9 Whereas in some countries a certification in general surgery is still required before specializing in vascular surgery, in 2005 the Amercian Board of Medical Specialties approved a primary certificate in vascular surgery, which fundamentally changed vascular education.1,8 In countries with independent vascular surgery certification, the overall residency time is shorter, but the specific vascular surgery training is longer.10 Similar considerations apply to interventional radiology (IR). Although it continues to require a prior certification in radiology in almost all countries, the European Union of Medical Specialists (UEMS) recognized IR as a subspecialty of radiology in 2009 and, consequently, the Cardiovascular and Interventional Radiological Society of Europe (CIRSE) introduced a European Board of Interventional Radiology (EBIR) in 2010. In the USA, IR is a recognized subspecialty with its own board certification examination. In the ‘‘Global Statement Defining Interventional Radiology,’’11 CIRSE and the Society of Interventional Radiology, USA (SIR), backed by a large number of national interventional societies, required ‘‘formal training and testing in image-guided minimally invasive and related procedures and techniques’’ to be included in IR training programs. In parallel with this change in IR and vascular surgery training, there have been dramatic changes associated with endovascular approaches, allowing treatment of sicker and elderly patients with minimally invasive procedures. Subsequently, minimum numbers of endovascular interventions for vascular fellowships to be accredited have been established.12 Moreover, ethical and educational considerations, restricted working hours, and the increasing complexity of interventions have led to the introduction of simulated working environments for additional training. Trainees can practice standardized interventional tasks repeatedly on
1178 Rosenthal et al.
Annals of Vascular Surgery
Table I. Participants’ baseline characteristics
Characteristic
Gender Age, years Previous participation at any endovascular course (Aimed) speciality of board certification Number of endovascular interventions previously carried out as interventionalist
Number of endovascular interventions previously carried out as assistant
Overall years since graduation from medical school Specialty years since graduation from medical school
Measure or category
Course participants (n ¼ 28)
Control group (n ¼ 22)
Female Male Median (IQR) No Yes Radiology Surgery 0 1e5 6e20 21e100 >100 0 1e5 6e20 21e100 >100 Median (IQR) Median (IQR)
12 16 34 24 4 11 17 9 8 7 4 d 5 3 10 6 4 4 4
10 12 32 20 2 4 18 16 4 d 2 d 9 5 4 4 d 5 4
(43) (57) (31e36) (86) (14) (39) (61) (32) (29) (25) (14) (18) (11) (36) (21) (14) (3e8) (3e7)
(45) (55%) (29e34) (91) (9) (18) (82) (73) (18) (9) (41) (23) (18) (18) (3e6) (3e6)
Data expressed as number (%), unless noted otherwise. IQR, interquartile range.
Table II. Global assessment at baseline and final assessment among course participants and the control group Final assessment Baseline assessment
Fail
Course participants (n ¼ 27) Fail 1 Pass 1 Control group (n ¼ 22) Fail 16 Pass 0
Pass
16 9 1 5
bench models or VR simulators.13 Notably, they may repeatedly be assessed using either structured objective feedback forms or simulator metrics. The successful transfer of improved performance from the simulated (computer-based haptic simulator) to the real setting could be shown for lower extremity occlusive disease in a randomized, controlled trial involving a total of 20 participants.14 Because case experience of today’s trainees tends to be limited, advanced endovascular training courses and workshops are useful settings for additional training, with tutoring by a number of highly specialized experts.12 The actual course, incorporating tutors from IR and vascular surgery, assembles a panel of interdisciplinary experts to provide advanced training.
For the purpose of the basic and final assessment within this study, OSATS have been used, which were previously shown to demonstrate high reliability and construct validity.2 For the endovascular setting, task performance assessed using OSATS has been shown to correlate with the experience level of trainees.3 The results of all 3 assessment parameters, GA, CL, and GR, uniformly demonstrate the positive effect on performance of the training course. This finding is in line with previous investigations. An 8-week endovascular simulation training program significantly improved performance of medical students4 and a 2-day course using a VR simulator has been shown to significantly improve performance of experienced interventionalists in carotid artery stenting.5 However, these studies, using a mere paired design, lacked a control group. With our study setting being a controlled beforeeafter design, we are able to exclude the effect of improvement due to mere repetition and thus could show the pure effect of the course. One randomized, controlled trial has previously been conducted to evaluate the effect of simulation-based training on endovascular performance.14 In contrast to our study, in that study the intervention consisted not of bench models, but of computer-based haptic simulation. The results of the cited study are encouraging, demonstrating skills transfer to the OR/ angiography suite; however, they refer to their
Vol. 27, No. 8, November 2013
Endovascular training course 1179
Fig. 3. Boxplots of the distribution of percentage scores at baseline and final assessment among course participants and the control group. (A) Task-specific checklist percentage score (CL). (B) Global rating scale percentage score (GR).
1180 Rosenthal et al.
work as a ‘‘preliminary study,’’ referring to the low participant number of 10 per group.14 Nevertheless, the study14 is among the few to include outcomes in the operating room/angio-suite. One study used live animal modelsdspecifically a randomized, controlled trial comparing training of endovascular skills using virtual reality simulation versus live pig models.15 In that work, the endovascular skills learned with virtual reality simulation were shown to be transferable to the pig model. Another study including real operating room performance was a prospective, randomized, controlled trial of endovascular skills training in surgical residents. It showed that faculty-mentored simulation sessions carried out on a weekly basis improved technical skills on a high-fidelity simulator and also translated to a better performance in a live operating room assessment at the end of the rotation.16 Our study has some limitations. First, we did not randomize participants, but recruited course participants and a control group. Therefore, a certain selection bias may not be excluded and an imbalance concerning some baseline characteristics has been noted. We addressed this shortcoming accounting for the imbalance between the groups by adjusting for baseline performance. Notably, even in the slightly more experienced course participants, the majority did not pass the baseline test. Moreover, in the more experienced participants of the control group there were no changes noted in global assessment between baseline and final examination. Because the group sizes were relatively small, we were not able to carry out further covariate adjustment. Because the control group was assessed at their home institution, we cannot completely exclude a performance assessment bias due to lack of blinding. We addressed this shortcoming by using an objective assessment tool. Second, the number of participants included in this study is limited. This may be reflected by the wide confidence intervals. However, because the results are highly significant, conclusions may be drawn despite the small sample size. Third, the bench model test used was rather simple as compared with the contents of the training course. However, the fact that the majority of study participants did not pass the baseline test confirms that there was no initial ceiling effect. Moreover, the GR allows for objective evaluation of some general performance characteristics of the trainee,
Annals of Vascular Surgery
such as respect for tissue, flow of intervention, and instrument handling, yielding important information that potentially applies to more complex procedures as well. Last, the results are limited to the bench model test. Subsequent studies need to be undertaken to evaluate the ability of the course participants to translate the skills of this simulated arterial access task into higher endovascular skills and into a true clinical endovascular arena and to evaluate whether an improvement may be observed long term. In conclusion, a 2-day bench model endovascular training course with a structured and dense training program for endovascular interventions significantly improves endovascular performance in a simulated arterial access task. It addresses the increasing need for specific training in endovascular procedures and demonstrates the feasibility of using objective, structured evaluation of an endovascular procedure. These results are highly relevant for the design of optimal training curricula and will ultimately contribute to patient safety.
The authors thank the faculty of the Vascular International School, SwissIntervention, all course instructors, and Salome Dell-Kuster and Jim Young for their support. REFERENCES 1. Kwolek CJ, Crawford RS. Training the next generation of vascular specialists: current status and future perspectives. J Endovasc Ther 2009;16(Suppl. 1)):I42e52. 2. Reznick R. Testing technical skill via an innovative ‘‘bench station’’ examination. Am J Surg 1997;172:226e30. 3. Tedesco MM, Pak JJ, Harris EJ Jr, et al. Simulation-based endovascular skills assessment: the future of credentialing? J Vasc Surg 2008;47. 1008e1. 4. Lee JT, Qiu M, Teshome M, et al. The utility of endovascular simulation to improve technical performance and stimulate continued interest of preclinical medical students in vascular surgery. J Surg Educ 2009;66:367e73. 5. van Herzeele I, Aggarwal R, Neequaye S, et al. Experienced endovascular interventionalists objectively improve their skills by attending carotid artery stent training courses. Eur J Vasc Endovasc Surg 2008;35:541e50. 6. R Development Core Team. R: A Language and Environment for Statistical Computing. Vienna, Austria:: R Foundation for Statistical Computing, http://www.R-project.org/; 2012. 7. Sarkar D. Lattice: Multivariate Data Visualization with R. New York: Springer, 2008. 8. Mitchell EL, Arora S, Moneta GL. Ensuring vascular surgical training is on the right track. J Vasc Surg 2011;53: 517e25. 9. Mills JL Sr. Vascular surgery training in the United States: a half-century of evolution. J Vasc Surg 2008;48(Suppl.): 90Se7.
Vol. 27, No. 8, November 2013
10. Cronenwett JL, Liapis CD. Vascular surgery training and certification: an international perspective. J Vasc Surg 2007;46:621e9. 11. Kaufman JA, Reekers JA, Burnes JP, et al. Global statement defining interventional radiology. J Vasc Interv Radiol 2010;21:1147e9. 12. Johnson CM, Hodgson KJ. Advanced endovascular training for vascular residents: what more do we need? Semin Vasc Surg 2006;19:194e9. 13. Hsu JH, Younan D, Pandalai S, et al. Use of computer simulation for determining endovascular skill levels in a carotid stenting model. J Vasc Surg 2004;40:1118e25.
Endovascular training course 1181
14. Chaer RA, Derubertis BG, Lin SC, et al. Simulation improves resident performance in catheter-based intervention: results of a randomized, controlled study. Ann Surg 2006;244: 343e52. 15. Berry M, Lystig T, Beard J, et al. Porcine transfer study: virtual reality simulator training compared with porcine training in endovascular novices. Cardiovasc Intervent Radiol 2007;30:455e61. 16. Lee JT, Peruzzaro A, Krummel T, et al. See one, sim one, do one, teach one: results of a prospective randomized trial of endovascular skills training for surgical residents. J Vasc Surg 2012;55(Suppl.):27S.
Background: The objective of this study was to evaluate the effect of a 2-day international endovascular training course on the performance of trainees as compared with a control group, assessed in a bench modelebased task using an objective structured evaluation protocol. Methods: A total of 50 trainees, 28 course participants of 2 consecutive identical courses and a control group of 22 participants with a similar level of experience without course attendance, underwent baseline and final assessment (simulated arterial access task). The evaluation form consisted of a global assessment (GA), task-specific checklist percentage score (CL), and global rating scale percentage score (GR), with both percentage scores ranging from 0% (worst performance) to 100% (best performance). Results: Course participants were more likely to pass the GA at final testing than the control group (odds ratio ¼ 59; 95% confidence interval [CI] 9.5e656; P < 0.001). The estimated difference in percentage score at final testing between course participants and the control group was 26% (95% CI 18e34; P < 0.001) for the CL and 29% (95% CI 19e40; P < 0.001) for the GR. Conclusions: A 2-day structured endovascular training course significantly improves endovascular performance in a simulated environment. These results are important for the design of endovascular training curricula with the ultimate goal of contributing to patient safety.
INTRODUCTION Minimally invasive vascular procedures represent the core treatment option for many cerebrovascular, aortoiliac, and peripheral occlusive diseases. The growth of these procedures is expected to continue, 1 Department of Vascular Surgery, University Hospital Basel, Basel, Switzerland. 2
Center for Microtherapy, Klinik Hirslanden, Zurich, Switzerland.
3
Department of Vascular Surgery, Cantonal Hospital, Lucerne, Switzerland. 4 Basel Institute for Clinical Epidemiology and Biostatistics, University Hospital Basel, Basel, Switzerland.
Correspondence to: Lorenz G€ urke, MD, Department of Vascular Surgery, Vascular International School, University Hospital Basel, Spitalstrasse 21, 4031 Basel, Switzerland; E-mail: [email protected] Ann Vasc Surg 2013; 27: 1173–1181 http://dx.doi.org/10.1016/j.avsg.2013.01.015 Ó 2013 Elsevier Inc. All rights reserved. Manuscript received: October 2, 2012; manuscript accepted: January 28, 2013; published online: August 22, 2013.
and teaching programs must meet the demand for skilled interventional experts to conduct these operations.1 Skills essential for performing open vascular surgery are different from those required for interventional radiologic procedures or minimally invasive vascular procedures, respectively. Therefore, specific minimally invasive vascular training is essential. Because of increasing time constraints, cost, and the increasing complexity of interventions, the modern minimally invasive suite is not the optimal learning environment and the patient is not the optimal training object to gain initial experience in endovascular interventions. Using bench models and computer-based virtual reality (VR) simulators, trainees can practice standardized interventional tasks repeatedly with feedback on performance from their instructors and the simulators without jeopardizing patient safety. Whereas the assessment using simulators is 1173
1174 Rosenthal et al.
objective, assessment by experts may be subjective. To address this shortcoming, objective assessment instruments have been developed and validated (i.e., objective structured assessment of technical skills [OSATS]).2 OSATS have been used successfully for the assessment of endovascular tasks. Endovascular task performance in a VR environment assessed using OSATS by an expert interventionalist has been shown to correlate well with the trainees’ prior experience in endovascular procedures.3 In addition, OSATS have been used to evaluate the effect of training programs. Medical students participating in an 8-week endovascular simulation training program significantly improved their technical skills, patient safety measures, and structured global assessments.4 For advanced endovascular procedures, such as carotid artery stenting, a 2-day course using a VR simulator has been shown to significantly improve performance of experienced interventionalists as measured using simulator-derived metrics.5 The goal of this study was to evaluate the effect of a 2-day endovascular training course on the performance of trainees in a bench model task using objective criteria. In particular, we evaluated whether course participants perform better than trainees with similar experience who have not attended the course.
METHODS Study Participants All participants of 2 identical consecutive endovascular training courses organized by the Vascular International School (VIS [http://www.vascularinternational.org/]) and SwissIntervention (http:// www.swissintervention.ch) held in Lucerne, Switzerland, at the Academy for Medical Training and Simulation (AMTS) on November 4e5, 2010 and November 1e2, 2011, took part in the study and represent the intervention group. This course focuses on endovascular interventions, whereas other courses at the VIS offer vascular (main focus) and some endovascular training options. This addresses instructors’ and participants’ request for courses not exceeding 2e 3 days (organizational demand from the respective clinics) and allows trainees from specialties focusing on endovascular treatment, such as interventional radiologists or angiologists, to visit a course without open vascular training. The course is not mandatory for application to the Fellow of European Board of Vascular Surgery
Annals of Vascular Surgery
(FEBVS) examinations (www.uemsvascular.com), as long as it is not required for the certificate of completion of training of the applicants’ country. The study was designed as a controlled beforee after study with observations before and after the intervention in a group receiving the intervention and in a similar (without individual matching) control group not receiving the intervention. The control group consisted of a cohort of trainees recruited at the radiology and surgery department of the Basel University Hospital, Switzerland, with similar experience and not having attended the course. Due to organizational reasons, the control group underwent assessment at their home institution. All study participants were medical doctors with specialization or aimed specialization in radiology or surgery. Written informed consent was obtained from all study participants and the study was approved by the local ethics committee. Upon informed consent, the participants completed a questionnaire on baseline information and underwent baseline assessment (on day 1 immediately before the beginning of the course), attended the course if they were part of the intervention group, and underwent final assessment on day 2 immediately after the end of the course (intervention group). All participants underwent both examinations within 30 minutes, resulting in comparable time lags between the two assessments. The control group underwent the same assessment within the same schedule and time lag without course attendance (final assessment on the day after baseline assessment). Questionnaire on Baseline Characteristics Participants completed a questionnaire on baseline characteristics, including age, gender, postgraduate year, years of radiology/surgery training, and experience in interventional radiology (as interventionalist and as first/second-hand assistant). Baseline Assessment Next, all participants performed a standardized vascular procedure on a bench model at the VIS. It consisted of a percutaneous arterial puncture of a perfused pulsatile box model and introduction of a sheath over a guidewire applying the Seldinger technique. All study participants were given standardized instructions on draping the puncture site, performing a percutaneous arterial puncture, and introducing a sheath over a guidewire. The trainees were rated by experienced vascular
Vol. 27, No. 8, November 2013
interventionalists/surgeons who were briefed about the objective rating system prior to their first rating. To not delay the course, multiple assessors were necessary. Assessment was performed according to a structured assessment form based on the OSATS previously developed and validated by Reznick,2 and adapted for endovascular procedures according to recent literature3e5 (Fig. 1). The OSATS consists of 3 outcome measures: global assessment (pass/ fail) (GA); task-specific checklist percentage score (CL); and global rating scale percentage score (GR), with both percentage scores ranging from 0% (worst performance) to 100% (best performance). The CL corresponds to the percentage of items judged as correctly done. To obtain percentages, the GR was calculated by summing the marks of the 7 dimensions as converted from 1 (poor performance) to 5 (good performance) into 0 (poor performance) to 4 (good performance) and dividing by 28 (7 4). Two course participants did not have an assistant at final assessment. We calculated their GR accordingly out of 6 (instead of 7) dimensions. Intervention The intervention consisted of a 2-day international endovascular training course. This course focuses on endovascular interventions. In the training course, standardized bench models were used to train standard interventional and endovascular techniques, such as vascular puncture, antegrade, retrograde, and Seldinger techniques; arterial dilation; and application of arterial stents for arterial occlusive disease and aneurysm treatment. The focus was on practical exercises, which were accompanied by brief theoretical introductions to use of radiology suites, radiation safety, and relevant endovascular devices. A real angio-suite environment was simulated using the Pontresina model (Fig. 2). It consists of a silicon model of the aortoiliacofemoral arterial system. After injection of a simulated contrast medium, the arterial system is depicted taking advantage of a change in acid and base properties (pH value) and displayed on a screen as an arteriogram with the aid of a video camera. The control group did not attend the course. Final Assessment At the end of the course, all participants of both courses repeated the same standardized procedure on a bench model as described for the baseline assessment and were rated in the same structured manner. The control group repeated assessment at the same time interval after baseline assessment.
Endovascular training course 1175
Statistical Analyses The sample size of this study was given by the number of participants in the two consecutive identical training courses, whereas the control group had a comparable, slightly smaller sample size. For each assessment parameter (GA, CL, and GR), we employed an analysis of covariance model to estimate the difference in final assessment between course participants and the control group using the baseline assessment as a covariate. We used multiple linear regression for the continuous assessment parameters (CL and GR), and for the binary assessment parameter (GA) we used multiple logistic regression. For each analysis, we report an estimate (and 95% confidence interval) of the difference in final assessment between course participants and the control group, with adjustment for baseline assessment. For the GA, this estimate is on the odds ratio scale. All data were entered into an Excel spreadsheet (Microsoft Office XP; Microsoft Corp., Redmond, WA) and handled anonymously. We used R software (version 2.15.0; R Foundation for Statistical Computing, Vienna, Austria)6 and the R add-on package Lattice (version 0.20-6)7 for our analyses and for graphics, respectively.
RESULTS General Characteristics A total of 50 trainees participated in the study, 28 in the intervention group and 22 in the control group; there were 22 (44%) females and 28 (56%) males. One course participant of the first course had to leave the course prior to the final assessment for unanticipated personal reasons unrelated to the course and the study. One course participant of the second course did not participate in the study, because he did not attend the second course day. Five course participants of the second course were technicians/nurses and were excluded from analysis. With the exception of 1 participant, none in the control group had any opportunity to be involved in endovascular procedures between baseline and final assessment and the control group did not assist any faculty lectures or bedside teaching between the two assessment time-points. Table 1 presents an overview of the participant characteristics. The detailed results at baseline and final assessment of the GA are presented in Table 2 and of the CL and GR in Figure 3. The participant in the control group who was involved in endovascular procedures between baseline and final
1176 Rosenthal et al.
Annals of Vascular Surgery
Participant number
________
1. Global assessment:
□ pass
□ fail
2. Procedure-related metrics Total procedure time in minutes: _________
3. Task-specific checklist:
Item
1.
Not done/ Incorrectly
Done correctly
O
O
Puncture with correct needle orientationO
O
Draping (disinfection / correct localisation)
2.
(cut of the needle pointing upwards, position 45-60° as referred to the skin)
3.
Insertion of the guidewire
O
O
(under fluoroscopic control (as mentioned by trainee))
4.
Retraction of the needle
O
O
(under fixation of the guidewire)
5.
Introduction of the sheath
O
O
6.
Retraction of the dilatation device
O
O
7.
Flushing of the sheath
O
O
O
O
(first aspiration, then flushing, checks that no air is in the system, plunger upward-oriented)
8.
Heparin flushing
4. Global rating scale: 1 Respect for tissue
Time and motion Instrument handling
Knowledge of instruments Flow of operation
Frequently used unnecessary force on tissue or caused damage by inappropriate use of instruments Many unnecessary moves
Repeatedly makes tentative or awkward moves with instruments by inappropriate use of instruments Frequently asked for wrong instrument or used inappropriate instrument Frequently stopped operating and seemed unsure of next move
Use of Assistants
Consistently placed assistants poorly or failed to use assistants
Knowledge of Specific Procedure
Deficient knowledge. Needed specific instruction at most Steps
2
3
4
5
Careful handling of tissue but occasionally caused inadvertent damage Efficient time/motion but some unnecessary moves Competent use of instruments but occasionally appeared stiff or awkward
Consistently handled tissues appropriately with minimal damage
Knew names of most instruments and used appropriate instrument
Obviously familiar with the instruments and their names
Demonstrated some forward planning with reasonable progression of procedure Appropriate use of assistants most of the time
Obviously planned course of operation with effortless flow from one move to the next Strategically used assistants to the best advantage at all times
Knew all important Steps of operation
Demonstrated familiarity with all aspects of operation
Clear economy of movement and maximum efficiency Fluid moves with instruments and no awkwardness
Fig. 1. Structured assessment form, adapted from Reznick,2 Tedesco et al.,3 Lee et al.,4 and van Herzeele et al.5
Vol. 27, No. 8, November 2013
Endovascular training course 1177
DISCUSSION
Fig. 2. Example of a bench model used at the course: angio-suite.
assessment scored identically in both assessments. Of the 6 participants (27%) in the control group with previous experience as interventionalists, 4 (67%) passed both baseline and final assessment (global assessment) and 2 (33%) failed both assessments. One participant in the control group subsequently underwent the second course with a time lapse of several months. This participant failed both assessments of the control group and passed both assessments of the intervention group. Concerning all 3 outcome parameters, participants of the course, not of the control group, performed better at final as compared with baseline assessment. For example, of 17 course participants who failed the GA at baseline testing, 16 passed it at final testing.
Comparison of Course Participants and the Control Group Relative to the control group, course participants were more likely to pass the GA at final testing (odds ratio ¼ 59; 95% confidence interval [CI] 9.5e656; P < 0.001). The estimated difference in percentage score at final testing (adjusted for baseline) between course participants and the control group was 26% (95% CI 18e34; P < 0.001) for the CL and 29% (95% CI 19e40; P < 0.001) for the GR.
This prospective, interventional study demonstrates that participants of a 2-day endovascular training course perform significantly better at final assessment of a simulated arterial access task (adjusted for baseline assessment) concerning all 3 evaluation parameters, GA, CL, and GR, than a control group with similar experience not having attended the course. Vascular surgery has significantly evolved in recent decades. For years, it was part of general and cardiothoracic surgery.1,8 In the USA, the first Certificates of Special Qualifications in General Vascular Surgery were issued in 1982.9 Whereas in some countries a certification in general surgery is still required before specializing in vascular surgery, in 2005 the Amercian Board of Medical Specialties approved a primary certificate in vascular surgery, which fundamentally changed vascular education.1,8 In countries with independent vascular surgery certification, the overall residency time is shorter, but the specific vascular surgery training is longer.10 Similar considerations apply to interventional radiology (IR). Although it continues to require a prior certification in radiology in almost all countries, the European Union of Medical Specialists (UEMS) recognized IR as a subspecialty of radiology in 2009 and, consequently, the Cardiovascular and Interventional Radiological Society of Europe (CIRSE) introduced a European Board of Interventional Radiology (EBIR) in 2010. In the USA, IR is a recognized subspecialty with its own board certification examination. In the ‘‘Global Statement Defining Interventional Radiology,’’11 CIRSE and the Society of Interventional Radiology, USA (SIR), backed by a large number of national interventional societies, required ‘‘formal training and testing in image-guided minimally invasive and related procedures and techniques’’ to be included in IR training programs. In parallel with this change in IR and vascular surgery training, there have been dramatic changes associated with endovascular approaches, allowing treatment of sicker and elderly patients with minimally invasive procedures. Subsequently, minimum numbers of endovascular interventions for vascular fellowships to be accredited have been established.12 Moreover, ethical and educational considerations, restricted working hours, and the increasing complexity of interventions have led to the introduction of simulated working environments for additional training. Trainees can practice standardized interventional tasks repeatedly on
1178 Rosenthal et al.
Annals of Vascular Surgery
Table I. Participants’ baseline characteristics
Characteristic
Gender Age, years Previous participation at any endovascular course (Aimed) speciality of board certification Number of endovascular interventions previously carried out as interventionalist
Number of endovascular interventions previously carried out as assistant
Overall years since graduation from medical school Specialty years since graduation from medical school
Measure or category
Course participants (n ¼ 28)
Control group (n ¼ 22)
Female Male Median (IQR) No Yes Radiology Surgery 0 1e5 6e20 21e100 >100 0 1e5 6e20 21e100 >100 Median (IQR) Median (IQR)
12 16 34 24 4 11 17 9 8 7 4 d 5 3 10 6 4 4 4
10 12 32 20 2 4 18 16 4 d 2 d 9 5 4 4 d 5 4
(43) (57) (31e36) (86) (14) (39) (61) (32) (29) (25) (14) (18) (11) (36) (21) (14) (3e8) (3e7)
(45) (55%) (29e34) (91) (9) (18) (82) (73) (18) (9) (41) (23) (18) (18) (3e6) (3e6)
Data expressed as number (%), unless noted otherwise. IQR, interquartile range.
Table II. Global assessment at baseline and final assessment among course participants and the control group Final assessment Baseline assessment
Fail
Course participants (n ¼ 27) Fail 1 Pass 1 Control group (n ¼ 22) Fail 16 Pass 0
Pass
16 9 1 5
bench models or VR simulators.13 Notably, they may repeatedly be assessed using either structured objective feedback forms or simulator metrics. The successful transfer of improved performance from the simulated (computer-based haptic simulator) to the real setting could be shown for lower extremity occlusive disease in a randomized, controlled trial involving a total of 20 participants.14 Because case experience of today’s trainees tends to be limited, advanced endovascular training courses and workshops are useful settings for additional training, with tutoring by a number of highly specialized experts.12 The actual course, incorporating tutors from IR and vascular surgery, assembles a panel of interdisciplinary experts to provide advanced training.
For the purpose of the basic and final assessment within this study, OSATS have been used, which were previously shown to demonstrate high reliability and construct validity.2 For the endovascular setting, task performance assessed using OSATS has been shown to correlate with the experience level of trainees.3 The results of all 3 assessment parameters, GA, CL, and GR, uniformly demonstrate the positive effect on performance of the training course. This finding is in line with previous investigations. An 8-week endovascular simulation training program significantly improved performance of medical students4 and a 2-day course using a VR simulator has been shown to significantly improve performance of experienced interventionalists in carotid artery stenting.5 However, these studies, using a mere paired design, lacked a control group. With our study setting being a controlled beforeeafter design, we are able to exclude the effect of improvement due to mere repetition and thus could show the pure effect of the course. One randomized, controlled trial has previously been conducted to evaluate the effect of simulation-based training on endovascular performance.14 In contrast to our study, in that study the intervention consisted not of bench models, but of computer-based haptic simulation. The results of the cited study are encouraging, demonstrating skills transfer to the OR/ angiography suite; however, they refer to their
Vol. 27, No. 8, November 2013
Endovascular training course 1179
Fig. 3. Boxplots of the distribution of percentage scores at baseline and final assessment among course participants and the control group. (A) Task-specific checklist percentage score (CL). (B) Global rating scale percentage score (GR).
1180 Rosenthal et al.
work as a ‘‘preliminary study,’’ referring to the low participant number of 10 per group.14 Nevertheless, the study14 is among the few to include outcomes in the operating room/angio-suite. One study used live animal modelsdspecifically a randomized, controlled trial comparing training of endovascular skills using virtual reality simulation versus live pig models.15 In that work, the endovascular skills learned with virtual reality simulation were shown to be transferable to the pig model. Another study including real operating room performance was a prospective, randomized, controlled trial of endovascular skills training in surgical residents. It showed that faculty-mentored simulation sessions carried out on a weekly basis improved technical skills on a high-fidelity simulator and also translated to a better performance in a live operating room assessment at the end of the rotation.16 Our study has some limitations. First, we did not randomize participants, but recruited course participants and a control group. Therefore, a certain selection bias may not be excluded and an imbalance concerning some baseline characteristics has been noted. We addressed this shortcoming accounting for the imbalance between the groups by adjusting for baseline performance. Notably, even in the slightly more experienced course participants, the majority did not pass the baseline test. Moreover, in the more experienced participants of the control group there were no changes noted in global assessment between baseline and final examination. Because the group sizes were relatively small, we were not able to carry out further covariate adjustment. Because the control group was assessed at their home institution, we cannot completely exclude a performance assessment bias due to lack of blinding. We addressed this shortcoming by using an objective assessment tool. Second, the number of participants included in this study is limited. This may be reflected by the wide confidence intervals. However, because the results are highly significant, conclusions may be drawn despite the small sample size. Third, the bench model test used was rather simple as compared with the contents of the training course. However, the fact that the majority of study participants did not pass the baseline test confirms that there was no initial ceiling effect. Moreover, the GR allows for objective evaluation of some general performance characteristics of the trainee,
Annals of Vascular Surgery
such as respect for tissue, flow of intervention, and instrument handling, yielding important information that potentially applies to more complex procedures as well. Last, the results are limited to the bench model test. Subsequent studies need to be undertaken to evaluate the ability of the course participants to translate the skills of this simulated arterial access task into higher endovascular skills and into a true clinical endovascular arena and to evaluate whether an improvement may be observed long term. In conclusion, a 2-day bench model endovascular training course with a structured and dense training program for endovascular interventions significantly improves endovascular performance in a simulated arterial access task. It addresses the increasing need for specific training in endovascular procedures and demonstrates the feasibility of using objective, structured evaluation of an endovascular procedure. These results are highly relevant for the design of optimal training curricula and will ultimately contribute to patient safety.
The authors thank the faculty of the Vascular International School, SwissIntervention, all course instructors, and Salome Dell-Kuster and Jim Young for their support. REFERENCES 1. Kwolek CJ, Crawford RS. Training the next generation of vascular specialists: current status and future perspectives. J Endovasc Ther 2009;16(Suppl. 1)):I42e52. 2. Reznick R. Testing technical skill via an innovative ‘‘bench station’’ examination. Am J Surg 1997;172:226e30. 3. Tedesco MM, Pak JJ, Harris EJ Jr, et al. Simulation-based endovascular skills assessment: the future of credentialing? J Vasc Surg 2008;47. 1008e1. 4. Lee JT, Qiu M, Teshome M, et al. The utility of endovascular simulation to improve technical performance and stimulate continued interest of preclinical medical students in vascular surgery. J Surg Educ 2009;66:367e73. 5. van Herzeele I, Aggarwal R, Neequaye S, et al. Experienced endovascular interventionalists objectively improve their skills by attending carotid artery stent training courses. Eur J Vasc Endovasc Surg 2008;35:541e50. 6. R Development Core Team. R: A Language and Environment for Statistical Computing. Vienna, Austria:: R Foundation for Statistical Computing, http://www.R-project.org/; 2012. 7. Sarkar D. Lattice: Multivariate Data Visualization with R. New York: Springer, 2008. 8. Mitchell EL, Arora S, Moneta GL. Ensuring vascular surgical training is on the right track. J Vasc Surg 2011;53: 517e25. 9. Mills JL Sr. Vascular surgery training in the United States: a half-century of evolution. J Vasc Surg 2008;48(Suppl.): 90Se7.
Vol. 27, No. 8, November 2013
10. Cronenwett JL, Liapis CD. Vascular surgery training and certification: an international perspective. J Vasc Surg 2007;46:621e9. 11. Kaufman JA, Reekers JA, Burnes JP, et al. Global statement defining interventional radiology. J Vasc Interv Radiol 2010;21:1147e9. 12. Johnson CM, Hodgson KJ. Advanced endovascular training for vascular residents: what more do we need? Semin Vasc Surg 2006;19:194e9. 13. Hsu JH, Younan D, Pandalai S, et al. Use of computer simulation for determining endovascular skill levels in a carotid stenting model. J Vasc Surg 2004;40:1118e25.
Endovascular training course 1181
14. Chaer RA, Derubertis BG, Lin SC, et al. Simulation improves resident performance in catheter-based intervention: results of a randomized, controlled study. Ann Surg 2006;244: 343e52. 15. Berry M, Lystig T, Beard J, et al. Porcine transfer study: virtual reality simulator training compared with porcine training in endovascular novices. Cardiovasc Intervent Radiol 2007;30:455e61. 16. Lee JT, Peruzzaro A, Krummel T, et al. See one, sim one, do one, teach one: results of a prospective randomized trial of endovascular skills training for surgical residents. J Vasc Surg 2012;55(Suppl.):27S.