- Email: [email protected]
Evaluating the Construct Validity of a Pulsatile Fresh Frozen Human Cadaver Circulation Model for Endovascular Training
Accepted Manuscript Evaluating the Construct Validity of a Pulsatile Fresh Frozen Human Cadaver Circulation Model for Endovascular Training Craig Nesbitt, MRCS, MD., Samuel James Tingle, Robin Williams, FRCR., James McCaslin, FRCS, MD., Roger Searle, PhD., Sebastian Mafeld, FRCR., Gerard Stansby, M.A. (Catab), M.B., M.Chir, F.R.C.S. PII:
S0890-5096(18)30386-8
DOI:
10.1016/j.avsg.2018.03.041
Reference:
AVSG 3876
To appear in:
Annals of Vascular Surgery
Received Date: 3 January 2018 Revised Date:
1 March 2018
Accepted Date: 2 March 2018
Please cite this article as: Nesbitt C, James Tingle S, Williams R, McCaslin J, Searle R, Mafeld S, Stansby G, Evaluating the Construct Validity of a Pulsatile Fresh Frozen Human Cadaver Circulation Model for Endovascular Training, Annals of Vascular Surgery (2018), doi: 10.1016/j.avsg.2018.03.041. This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
ACCEPTED MANUSCRIPT 1
Title: Evaluating the Construct Validity of a Pulsatile Fresh Frozen Human Cadaver Circulation Model for Endovascular Training First author: Mr Craig Nesbitt
4 5
Order of authors: Craig Nesbitt1, Samuel James Tingle2, Robin Williams3, James McCaslin4, Roger Searle5, Sebastian Mafeld6, Gerard Stansby7
6
Author affiliations:
7 8
1. MRCS, MD. Northern Deanery Vascular Surgical Registrar and corresponding author, Middlesbrough James Cook University Hospital, Middlesbrough.
RI PT
2 3
2. Faculty of Medical Sciences, Newcastle Medical School, Newcastle, Tyne and Wear NE2 4HH, United Kingdom.
11 12
3. FRCR. Consultant Interventional Radiologist. Northern Vascular Centre, Department of Interventional Radiology, Freeman Hospital, Newcastle Upon Tyne.
13 14 15
4. FRCS, MD. Consultant Vascular and Endovascular Surgery. Department of Vascular Surgery. Northern Vascular Centre, Department of Vascular Surgery, Freeman Hospital, Newcastle Upon Tyne.
16 17
5. PhD. Head of School & Director of Anatomy and Clinical Skills & Director of Excellence in Learning and Teaching, Newcastle University, Newcastle Upon Tyne.
18 19
6. FRCR. Specialty Resgistrar in Interventional Radiology. Northern Vascular Centre, Department of Interventional Radiology, Freeman Hospital, Newcastle Upon Tyne.
20 21
7. M.A. (Catab), M.B., M.Chir, F.R.C.S. Professor of Vascular Surgery. Northern Vascular Centre, Department of Vascular Surgery, Freeman Hospital, Newcastle Upon Tyne
22
Submission category: Basic Science Research (new investigations, experimental work)
23
Key words: Endovascular Training, Human Cadaver, Pulsatile Model, Simulation
M AN U
TE D
EP
24
SC
9 10
Corresponding Author: Mr. Craig Nesbitt, MRCS, MD
26
Address: Healeyhope Barn, Waskerley, Consett, Co Durham, DH8 9DB, UK.
27
Telephone: 07969223061
28
Email: [email protected]
29
AC C
25
ACCEPTED MANUSCRIPT Abstract Objectives: We recently described a pulsatile fresh frozen human cadaver model (PHCM) for training
32
endovascular practitioners. This current study aims to assess the construct validity of PHCM; its
33
ability to differentiate between participants of varying expertise.
34
Methods: 23 participants with varying endovascular experience (12 novice, 4 intermediate, 7 expert)
35
were recruited. Each attempted catheterisation of the left renal artery on PHCM within 10 minutes
36
under exam conditions. Performances were video recorded and scored using a validated scoring tool
37
by two independent endovascular experts, blinded to performer status. Each participant was given a
38
task specific checklist score (TSC), global rating score (GRS), and overall procedure score (OPS).
39
Finally, examiners were asked whether they would be happy to supervise the participant in theatre,
40
with each participant graded as “fail”, “borderline” or “pass”.
41
Results: All expert and intermediate participants completed the index procedure within the allotted
42
10 minutes, however only one of the 12 novice participants achieved this (p<0.0005). Endovascular
43
novices had significantly lower TSC, GRS and OPS than both intermediate participants and
44
endovascular experts. There were no significant differences in TSC, GRS or OPS between
45
intermediate participants and endovascular experts. When participants were graded as “fail”,
46
“borderline” or “pass” there were significant differences between groups (p=0.001). All of the
47
intermediate and expert participants received a pass. Out of the 12 novice participants, 2 received a
48
pass, 6 received a borderline and 4 were failed.
49
Conclusion: The PHCM demonstrates construct validity. Further work is required to determine its
50
educational impact in endovascular training.
51
Introduction
52
The introduction of endovascular intervention has transformed the specialty of vascular surgery. The
53
key attraction of endovascular surgery is the minimally invasive nature of the techniques, which
54
offers reduced morbidity and mortality when compared to their equivalent open procedure
AC C
EP
TE D
M AN U
SC
RI PT
30 31
ACCEPTED MANUSCRIPT options.[1] For these reasons there has been a rapid increase in the number of endovascular
56
procedures being performed.[2]
57
As the number of endovascular procedures being completed continues to increase, robust training
58
methods must be developed. One attempt to fulfil this training need is the use of endovascular
59
virtual reality simulators.[3] Whilst virtual reality is now becoming integrated into endovascular
60
training across Europe and America, it is not without its limitations. Simulators lack the tactile
61
feedback found in real patient vessels, are unable to simulate arterial puncture, and units cost in
62
excess of £100,000.
63
The last decade has also seen an increasing number of human cadaveric (HC) based workshops in
64
higher surgical training.[4] The suitability of HC for training open vascular surgical procedures is
65
recognised.[5] Garrett et al described a technique for creating isolated pulsatile segments in a HC
66
model,[6] and the use of cadavers for stent graft development has also been reported.[7-9] However,
67
there is a lack of literature investigating the use of HC for endovascular training, despite the
68
increased use of HC for training in other fields.
69
We have published a technical note detailing a method for establishing a pulsatile fresh-frozen
70
human cadaver model (PHCM) which has potential to be used for endovascular training.[10] Following
71
this we published a paper demonstrating the model’s face validity;[11] a simple form of validity where
72
participants judge the degree of resemblance between a model and a real-life situation.[12] Whilst
73
the feasibility and face validity of this PHCM has been demonstrated, further work is required to
74
assess its construct validity.
75
Construct validity is the extent to which a test measures the trait it purports to measure. In the case
76
of training adjuncts such as the PHCM, a model is said to demonstrate construct validity if experts in
77
the actual procedure are able to outperform novices.[12] The aim of this study was to assess whether
78
our PHCM was able to demonstrate construct validity.
AC C
EP
TE D
M AN U
SC
RI PT
55
ACCEPTED MANUSCRIPT
Methods
80
Participants with a range of endovascular experience were invited to take part in the trial. This
81
included a number of participants with no previous endovascular experience of any kind (“novices”).
82
Participants who had endovascular experience but had performed fewer than 50 procedures were
83
considered “intermediate”, and those who had performed more than 50 procedures were
84
considered “expert”. None of the included participants took part in our previous trial of face
85
validity.[11]
86
Before candidates began training they completed a pre-trial questionnaire to determine certain
87
candidate demographics; level of seniority, endovascular experience, previous exposure to both
88
human cadaver and VR simulators, handedness, musical instrument experience, exposure to video
89
games, and use of correctional glasses.
90
A single index procedure was selected to compare the performances of the participants: cannulation
91
of the left renal artery and confirmatory angiogram from access through the right common femoral
92
artery. Prior to performing this procedure all participants attended an introductory lecture covering
93
details of the present study, key points about the PHCM, available equipment and the steps involved
94
in completing the index procedure. As the “novices” had no endovascular experience, they attended
95
an additional lecture. This included information on basic endovascular concepts, and a video of an
96
expert performing the procedure on a virtual reality simulator and a PHCM. Without this additional
97
training the endovascular “novices” would have been unable to proceed.
98
Participants were video recorded performing the index procedure under standard exam conditions,
99
with a time limit of 10 minutes in place. As seen in Figure 1, these recordings included both the
AC C
EP
TE D
M AN U
SC
RI PT
79
100
participant’s hands and the fluoroscopy screen (to ensure anonymity for subsequent marking). Each
101
participant signed a consent form to allow analysis of these video recordings, and was given a
102
unique random number to allow blinding during subsequent assessment.
ACCEPTED MANUSCRIPT These videos were edited to remove any identifying details, ordered randomly, copied to compact
104
discs and assessed separately by two independent endovascular experts who were fully blinded to
105
the participant status. These scorers recorded whether participants completed the procedure in the
106
allotted time. They also gave a task specific checklist score (TSC) and a global rating score (GRS).
107
Checklists for generating these scores are shown in Table 1 and Table 2 respectively. Averaging the
108
scores from each examiner gave a final TSC and GRS for each participant, which were then added to
109
give an overall procedure score. Finally, examiners were asked to “pass” or “fail” each participant
110
depending on whether they would be happy to supervise them performing the procedure on a real
111
patient. Where examiners disagreed the participant was given a grade of “borderline”.
112
To ensure both scorers understood the scoring procedure, they were initially shown a compilation of
113
clips from ten videos of edited performances of the index procedure being performed on the PHCM.
114
They openly discussed these video recordings to establish joint standards and promote concordance
115
when scoring.
116
Details of the PHCM model evaluated in this trial has been previously published as a technical note
117
in this journal.[10] Briefly, this model uses a pulsatile blood pump (1405 Harvard Apparatus™,
118
Massachusetts, USA) to perfuse a fresh frozen cadaver, with inflow through the right common
119
carotid artery and outflow through the left common femoral and right superficial femoral arteries.
120
All candidates performed their procedures on the same model, within a 48 hour period. The PHCM
121
did not demonstrate clinically significant degradation during the course of the trial.
122
Formal ethical approval was not required as it was deemed that the proposed trial represented
123
‘technical development and training’.
124
Statistical analysis
125
Statistical analysis was undertaken using the Statistical Package for the Social Sciences version 19
126
(SPSS, Chicago). Graphs were generated using GraphPad PrismTM 6.01.
AC C
EP
TE D
M AN U
SC
RI PT
103
ACCEPTED MANUSCRIPT One-way ANOVA plus post-hoc tests with Bonferroni correction were used to assess differences in
128
expert scores between groups. Cronbach’s alpha was used as a measure of inter-rater variability.
129
Categorical variables were analysed using Fisher’s Exact Test.
130
P <0.05 was considered statistically significant.
131
Results
132
In total, 23 participants were recruited to the study. Of these, there were; 7 consultant
133
interventional radiologists, 4 senior trainees (3 vascular surgery, 1 interventional radiology) and 12
134
junior trainees (medical students or foundation year doctors). Additional demographic information
135
for the candidates can be found in Table 3.
136
There were no differences between groups in terms of their baseline demographics, except for
137
previous exposure to virtual reality training; participants classed as intermediate or expert were
138
more likely to have had experience of virtual reality endovascular training than novice participants
139
(Table 3; p=0.002). Whilst none of the participants had experience of cadaver based endovascular
140
training, all participants had experience of cadaver training during undergraduate anatomy teaching.
141
In addition; one of the experts had completed a percutaneous cadaveric nephrolithotomy course,
142
one of the intermediate participants had completed a cadaveric trauma course, and another of the
143
intermediate participants had completed an advanced cadaveric vascular skills course.
144
Our first outcome was whether participants could complete the procedure in the allotted time. All
145
expert and intermediate participants completed the index procedure within 10 minutes. Only one of
146
the 12 novice participants was able to complete the procedure in the allotted time. Fisher’s Exact
147
Test found these differences to be statistically significant (p<0.0005).
148
As another point of comparison, video recordings of participants’ performances were scored by
149
experts. Measuring inter-rater variability, Cronbach’s alpha was 0.966 indicating strong agreement
150
between our two blinded scorers. As described above, each performance was given a task specific
AC C
EP
TE D
M AN U
SC
RI PT
127
ACCEPTED MANUSCRIPT checklist score (TSC) and a global rating score (GRS), which were combined to give an overall
152
procedure score (OPS). The average OPS for the expert candidates’ was 43.00 (TSC 15.00, GRS
153
27.21). The average OPS for the intermediate candidates was 39.63 (TSC 15.00, GRS 24.63). The
154
average OPS for the novice candidates was 19.88 (TSC 8.75, GRS 11.13). These results are
155
summarised in Figure 2A-C. Endovascular novices had significantly lower TSC, GRS and OPS than
156
both intermediate participants and endovascular experts. There were no significant differences in
157
TSC, GRS or OPS between intermediate participants and endovascular experts.
158
Finally, the expert scorers were asked whether they would be happy to supervise the participant
159
performing the index procedure on a real patient in theatre. A participant was given a pass if both
160
examiners would be happy supervising them and a fail if neither examiners were happy to supervise
161
them. Where examiners disagreed a participant was scored as borderline. As shown in Figure 2D all
162
of the intermediate and expert participants received a pass. Out of the 12 novice participants, 2
163
received a pass, 6 received a borderline and 4 received a fail. Fisher’s Exact Test found the
164
differences between groups of different endovascular experience to be statistically significant
165
(p=0.001).
166
Discussion
167
In the present trial the construct validity of a previously reported PHCM was assessed in the setting
168
of basic endovascular training.[10] To achieve this participants with various endovascular experience
169
were recruited, and their ability to perform an index procedure on PHCM was assessed. Participants
170
with no previous endovascular experience (“novice”) performed significantly worse than both those
171
with intermediate endovascular experience and endovascular experts in the following domains;
172
ability to complete the procedure in the allotted time, task specific checklists score (Figure 2A),
173
global rating score (Figure 2B), overall procedure score (Figure 2C) and whether examiners would be
174
happy to supervise the participants in an operating theatre (Figure 2D). Thus the PHCM
175
demonstrated construct validity.
AC C
EP
TE D
M AN U
SC
RI PT
151
ACCEPTED MANUSCRIPT Assessing the validity of a model is clearly vital before its use for training. The ability of a model to
177
demonstrate construct validity is a key part of this process. A model where novices can initially
178
perform as well as experts suggests that the model will be less useful for training. However, if the
179
model demonstrates construct validity, as is the case for PHCM, then continued training on the
180
model could allow novices to improve their performance to that of the experts. These improved
181
skills should then extrapolate to the real procedure, however further studies investigating the
182
educational impact of our PHCM are required to confirm this.
183
The methodology employed in this study is similar to that of previous trials aiming to assess
184
construct validity of a surgical training adjunct.[13-15] The combination of task specific checklist scores
185
and global rating scores as a reliable way to assess surgical skill was first reported two decades
186
ago.[16] Since then it has become the gold standard in reliably assessing technical skills in
187
endovascular literature, and for assessing construct validity in many fields.[14, 15, 17, 18] The reliability of
188
these scoring systems was further supported by the high level of inter-rater reliability reported in
189
this study (Cronbach’s alpha 0.966).
190
This study did not demonstrate that PHCM has the ability to differentiate between endovascular
191
practitioners with intermediate experience versus endovascular experts. This may well be a type 2
192
error, resulting from the two main limitations of this study. Firstly, only four intermediate
193
participants could be recruited. This limitation was due to low numbers of intermediate
194
endovascular trainees within our training deanery. The number of endovascular experts was also
195
relatively low (n=7). These low numbers meant that statistical analysis comparing these two groups
196
lacked statistical power, increasing the likelihood of type two errors. Secondly, the index training
197
procedure selected for this study was a relatively simple procedure, which is likely to be mastered at
198
an early stage in endovascular training. A procedure requiring greater skill may be required to
199
differentiate intermediate versus expert participants. However, it was not possible to reliably and
AC C
EP
TE D
M AN U
SC
RI PT
176
ACCEPTED MANUSCRIPT uniformly repeat a more challenging procedure such as stenting or angioplasty using PHCM in the
201
context of this trial.
202
Furthermore, it is suggested that the higher fidelity of PHCM is more appropriate for intermediate or
203
expert level practitioners exploring new techniques. Novice practitioners may benefit less when
204
honing basic guidewire handling skills. Further work is necessary comparing training on PMCH
205
versus a virtual reality simulator to shed more light on the model’s efficacy for training practitioners
206
at different levels.
207
Conclusion
208
This study has revealed that PHCM demonstrates construct validity; it can discriminate between
209
participants with previous endovascular experience from those without. This complements previous
210
work which demonstrated face validity of PHCM. Further work should assess the educational impact
211
of PHCM, to confirm that the model is a useful training tool.
212 213 214 215
Acknowledgments
TE D
M AN U
SC
RI PT
200
The authors would like to thank the staff of the Newcastle Surgical Training Centre for their support throughout the project, Medtronic Ltd for their generous educational grant and the radiology department of the Freeman Hospital for their expertise.
Conflict of Interest
218 219 220 221
Declaration of Funding
The authors declare no conflicts of interest.
AC C
222 223 224 225 226 227 228 229
EP
216 217
The authors declare a £2000 educational grant from Medtronic Ltd. This was donated to part fund the original purchase of the pulsatile pump. We can confirm that Medtronic had no further role in this or any subsequent trials
References 1. 2.
3.
Van Herzeele I. Virtual Reality Endovascular Simulation: Ready for Training. Nautilus Academic Books. 1st ed. 2009. Schanzer, A., et al., Vascular surgery training trends from 2001-2007: A substantial increase in total procedure volume is driven by escalating endovascular procedure volume and stable open procedure volume. J Vasc Surg, 2009. 49(5): p. 1339-44. Neequaye, S.K., et al., Endovascular skills training and assessment. Journal of Vascular Surgery, 2007. 46(5): p. 1055-1064.
ACCEPTED MANUSCRIPT 4. 5. 6.
10. 11. 12. 13. 14. 15. 16. 17. 18.
AC C
263 264 265
M AN U
9.
TE D
8.
SC
RI PT
7.
Gilbody, J., et al., The use and effectiveness of cadaveric workshops in higher surgical training: a systematic review. Ann R Coll Surg Engl, 2011. 93(5): p. 347-52. Reed, A.B., et al., Back to basics: use of fresh cadavers in vascular surgery training. Surgery, 2009. 146(4): p. 757-62; discussion 762-3. Garrett, H.E., Jr., A human cadaveric circulation model. Journal of Vascular Surgery. 33(5): p. 1128-1130. Linsen, M.A., et al., Modular branched endograft system for aortic aneurysm repair: evaluation in a human cadaver circulation model. Vasc Endovascular Surg, 2007. 41(2): p. 126-9. Arbatli, H., et al., Dynamic human cadaver model for testing the feasibility of new endovascular techniques and tools. Ann Vasc Surg, 2010. 24(3): p. 419-22. Jongkind, V., et al., Direct videoscopic approach to the thoracic aorta for aortic endograft delivery: evaluation in a human cadaver circulation model. J Endovasc Ther, 2010. 17(1): p. 12-8. Nesbitt, C., et al., Design of a Pulsatile Fresh Frozen Human Cadaver Circulation Model for Endovascular Training. Ann Vasc Surg, 2017. 44: p. 425-430. Nesbitt, C., et al., A Pulsatile Fresh Frozen Human Cadaver Circulation Model for Endovascular Training: A Trial of Face Validity. Ann Vasc Surg, 2018. 46: p. 345-350. Schreuder, H.W., et al., Face and construct validity of virtual reality simulation of laparoscopic gynecologic surgery. Am J Obstet Gynecol, 2009. 200(5): p. 540.e1-8. Piromchai, P., et al., The construct validity and reliability of an assessment tool for competency in cochlear implant surgery. Biomed Res Int, 2014. 2014: p. 192741. Awad, Z., et al., Construct validity of the ovine model in endoscopic sinus surgery training. Laryngoscope, 2015. 125(3): p. 539-43. Sharma, M., et al., Construct validity of fresh frozen human cadaver as a training model in minimal access surgery. Jsls, 2012. 16(3): p. 345-52. Martin, J.A., et al., Objective structured assessment of technical skill (OSATS) for surgical residents. Br J Surg, 1997. 84(2): p. 273-8. Hislop, S.J., et al., Simulator assessment of innate endovascular aptitude versus empirically correct performance. Journal of Vascular Surgery. 43(1): p. 47-55. Berger, P., et al., Validation of the Simulator for Testing and Rating Endovascular SkillS (STRESS)-machine in a setting of competence testing. J Cardiovasc Surg (Torino), 2010. 51(2): p. 253-6.
EP
230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262
Tables
Table 1 – Task Specific Checklist scoring tool
1. Selects Standard J-Tip Wire 2. Inserts J-Tip wire safely 3. Selects Pigtail catheter 4. Inserts Pigtail catheter safely 5. Removes J-Tip wire safely 6. Performs adequate aortic angiogram 7. Re-inserts J-Tip wire safely 8. Removes Pigtail catheter safely
Not done or incorrect 0 0 0 0 0 0 0 0
Done correctly 1 1 1 1 1 1 1 1
ACCEPTED MANUSCRIPT 0 0 0 0 0 0 0 0
1 1 1 1 1 1 1 1
266 Table 2 – Global Rating Score assessment tool
1
2
Respect for Tissue 3
4
Careful handling of tissues and/or lesion, but occasional potential for inadvertent tissue damage
1
2
M AN U
Frequently used unnecessary force on tissue and/or lesion, potential for tissue damage
SC
267
Time and Motion 3
Make unnecessary moves and/or excessive time
4
Efficient time and moves but some unnecessary moves and/or excessive time
1
2
Instrument Handling 3
4
Competent use of instruments, but occasionally appeared stiff or awkward
TE D
Repeated tentative, awkward, and/or inappropriate moves with instruments
2
Flow of Operation 3
1
2
AC C
Frequently stopped operating and seemed unsure of next move; demonstrated imprecise and/or wrong operative technique
EP
1
Very poor
1
2
Unacceptable quality
RI PT
9. Selects Cobra catheter 10. Inserts Cobra catheter safely 11. Cannulates left renal artery safely 12. Selects hydrophilic wire 13. Inserts hydrophilic wire safely 14. Advances Cobra catheter safely 15. Removes hydrophilic wire safely 16. Angiographic confirmation of left renal artery catheterisation
4
Demonstrated some forward planning with reasonable progression of procedure; careful operative technique with occasional errors
5
Consistently handled tissues and/or lesion appropriately with minimal tissue damage
5 Clear economy of moves and time with maximum efficiency
5 Fluid movements with instruments and no stiffness or awkwardness
5 Planned course of operation with effortless flow throughout; fluent, secure, and correct operative technique in all stages of procedure
Overall Performance 3
4
Competent
5 Clearly superior
Final Product 3
4
Average quality
5 Superior quality
268 269 270
Table 3 - Participant demographics. The absolute numbers of participants are given, and comparisons performed using the Fisher’s Exact Test. VR: virtual reality simulator.
Wears glasses Left handed
Novice (n=12) 1 0
Intermediate (n=4) 0 0
Expert (n=7) 0 1
Fisher’s Exact Test (p-value) 1.000 0.478
ACCEPTED MANUSCRIPT 6
2
4
1.000
7
1
1
0.169
0
3
4
0.002
0
0
0
N/A – zero participants
12
4
7
271
SC
Figures
N/A - all participants
EP
274 275
Figure 1 - Screenshot from an exemplar video of a participant performing cannulation of the left renal artery and confirmatory angiogram from access in the right femoral artery.
AC C
273
TE D
M AN U
272
RI PT
Plays a musical instrument Plays video games regularly Previous VR endovascular training Previous cadaver endovascular training Any previous cadaver training
M AN U
SC
RI PT
ACCEPTED MANUSCRIPT
276
TE D
EP
284
Figure 2 - Ability to perform the index procedure by participants with varying endovascular experience. Two independent endovascular experts marked anonymised video recordings of participants using the human cadaver model. A), B) and C) show box and whisker plot s demonstrating the task specific checklist score, global rating score, and overall procedure score respectively. ANOVA was used for statistical analysis and the results of post-hoc tests with Bonferroni correction are shown (NS: not significant; ***: P≤0.001; ****: P≤0.0001). D) Displays whether the examiners would be happy to supervise the participant performing the index procedure in theatre on a real patient (Pass = both examiners would be happy to supervise, Borderline = one examiner would be happy to supervise, Fail = neither examiner would be happy to supervise).
AC C
277 278 279 280 281 282 283
S0890-5096(18)30386-8
DOI:
10.1016/j.avsg.2018.03.041
Reference:
AVSG 3876
To appear in:
Annals of Vascular Surgery
Received Date: 3 January 2018 Revised Date:
1 March 2018
Accepted Date: 2 March 2018
Please cite this article as: Nesbitt C, James Tingle S, Williams R, McCaslin J, Searle R, Mafeld S, Stansby G, Evaluating the Construct Validity of a Pulsatile Fresh Frozen Human Cadaver Circulation Model for Endovascular Training, Annals of Vascular Surgery (2018), doi: 10.1016/j.avsg.2018.03.041. This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
ACCEPTED MANUSCRIPT 1
Title: Evaluating the Construct Validity of a Pulsatile Fresh Frozen Human Cadaver Circulation Model for Endovascular Training First author: Mr Craig Nesbitt
4 5
Order of authors: Craig Nesbitt1, Samuel James Tingle2, Robin Williams3, James McCaslin4, Roger Searle5, Sebastian Mafeld6, Gerard Stansby7
6
Author affiliations:
7 8
1. MRCS, MD. Northern Deanery Vascular Surgical Registrar and corresponding author, Middlesbrough James Cook University Hospital, Middlesbrough.
RI PT
2 3
2. Faculty of Medical Sciences, Newcastle Medical School, Newcastle, Tyne and Wear NE2 4HH, United Kingdom.
11 12
3. FRCR. Consultant Interventional Radiologist. Northern Vascular Centre, Department of Interventional Radiology, Freeman Hospital, Newcastle Upon Tyne.
13 14 15
4. FRCS, MD. Consultant Vascular and Endovascular Surgery. Department of Vascular Surgery. Northern Vascular Centre, Department of Vascular Surgery, Freeman Hospital, Newcastle Upon Tyne.
16 17
5. PhD. Head of School & Director of Anatomy and Clinical Skills & Director of Excellence in Learning and Teaching, Newcastle University, Newcastle Upon Tyne.
18 19
6. FRCR. Specialty Resgistrar in Interventional Radiology. Northern Vascular Centre, Department of Interventional Radiology, Freeman Hospital, Newcastle Upon Tyne.
20 21
7. M.A. (Catab), M.B., M.Chir, F.R.C.S. Professor of Vascular Surgery. Northern Vascular Centre, Department of Vascular Surgery, Freeman Hospital, Newcastle Upon Tyne
22
Submission category: Basic Science Research (new investigations, experimental work)
23
Key words: Endovascular Training, Human Cadaver, Pulsatile Model, Simulation
M AN U
TE D
EP
24
SC
9 10
Corresponding Author: Mr. Craig Nesbitt, MRCS, MD
26
Address: Healeyhope Barn, Waskerley, Consett, Co Durham, DH8 9DB, UK.
27
Telephone: 07969223061
28
Email: [email protected]
29
AC C
25
ACCEPTED MANUSCRIPT Abstract Objectives: We recently described a pulsatile fresh frozen human cadaver model (PHCM) for training
32
endovascular practitioners. This current study aims to assess the construct validity of PHCM; its
33
ability to differentiate between participants of varying expertise.
34
Methods: 23 participants with varying endovascular experience (12 novice, 4 intermediate, 7 expert)
35
were recruited. Each attempted catheterisation of the left renal artery on PHCM within 10 minutes
36
under exam conditions. Performances were video recorded and scored using a validated scoring tool
37
by two independent endovascular experts, blinded to performer status. Each participant was given a
38
task specific checklist score (TSC), global rating score (GRS), and overall procedure score (OPS).
39
Finally, examiners were asked whether they would be happy to supervise the participant in theatre,
40
with each participant graded as “fail”, “borderline” or “pass”.
41
Results: All expert and intermediate participants completed the index procedure within the allotted
42
10 minutes, however only one of the 12 novice participants achieved this (p<0.0005). Endovascular
43
novices had significantly lower TSC, GRS and OPS than both intermediate participants and
44
endovascular experts. There were no significant differences in TSC, GRS or OPS between
45
intermediate participants and endovascular experts. When participants were graded as “fail”,
46
“borderline” or “pass” there were significant differences between groups (p=0.001). All of the
47
intermediate and expert participants received a pass. Out of the 12 novice participants, 2 received a
48
pass, 6 received a borderline and 4 were failed.
49
Conclusion: The PHCM demonstrates construct validity. Further work is required to determine its
50
educational impact in endovascular training.
51
Introduction
52
The introduction of endovascular intervention has transformed the specialty of vascular surgery. The
53
key attraction of endovascular surgery is the minimally invasive nature of the techniques, which
54
offers reduced morbidity and mortality when compared to their equivalent open procedure
AC C
EP
TE D
M AN U
SC
RI PT
30 31
ACCEPTED MANUSCRIPT options.[1] For these reasons there has been a rapid increase in the number of endovascular
56
procedures being performed.[2]
57
As the number of endovascular procedures being completed continues to increase, robust training
58
methods must be developed. One attempt to fulfil this training need is the use of endovascular
59
virtual reality simulators.[3] Whilst virtual reality is now becoming integrated into endovascular
60
training across Europe and America, it is not without its limitations. Simulators lack the tactile
61
feedback found in real patient vessels, are unable to simulate arterial puncture, and units cost in
62
excess of £100,000.
63
The last decade has also seen an increasing number of human cadaveric (HC) based workshops in
64
higher surgical training.[4] The suitability of HC for training open vascular surgical procedures is
65
recognised.[5] Garrett et al described a technique for creating isolated pulsatile segments in a HC
66
model,[6] and the use of cadavers for stent graft development has also been reported.[7-9] However,
67
there is a lack of literature investigating the use of HC for endovascular training, despite the
68
increased use of HC for training in other fields.
69
We have published a technical note detailing a method for establishing a pulsatile fresh-frozen
70
human cadaver model (PHCM) which has potential to be used for endovascular training.[10] Following
71
this we published a paper demonstrating the model’s face validity;[11] a simple form of validity where
72
participants judge the degree of resemblance between a model and a real-life situation.[12] Whilst
73
the feasibility and face validity of this PHCM has been demonstrated, further work is required to
74
assess its construct validity.
75
Construct validity is the extent to which a test measures the trait it purports to measure. In the case
76
of training adjuncts such as the PHCM, a model is said to demonstrate construct validity if experts in
77
the actual procedure are able to outperform novices.[12] The aim of this study was to assess whether
78
our PHCM was able to demonstrate construct validity.
AC C
EP
TE D
M AN U
SC
RI PT
55
ACCEPTED MANUSCRIPT
Methods
80
Participants with a range of endovascular experience were invited to take part in the trial. This
81
included a number of participants with no previous endovascular experience of any kind (“novices”).
82
Participants who had endovascular experience but had performed fewer than 50 procedures were
83
considered “intermediate”, and those who had performed more than 50 procedures were
84
considered “expert”. None of the included participants took part in our previous trial of face
85
validity.[11]
86
Before candidates began training they completed a pre-trial questionnaire to determine certain
87
candidate demographics; level of seniority, endovascular experience, previous exposure to both
88
human cadaver and VR simulators, handedness, musical instrument experience, exposure to video
89
games, and use of correctional glasses.
90
A single index procedure was selected to compare the performances of the participants: cannulation
91
of the left renal artery and confirmatory angiogram from access through the right common femoral
92
artery. Prior to performing this procedure all participants attended an introductory lecture covering
93
details of the present study, key points about the PHCM, available equipment and the steps involved
94
in completing the index procedure. As the “novices” had no endovascular experience, they attended
95
an additional lecture. This included information on basic endovascular concepts, and a video of an
96
expert performing the procedure on a virtual reality simulator and a PHCM. Without this additional
97
training the endovascular “novices” would have been unable to proceed.
98
Participants were video recorded performing the index procedure under standard exam conditions,
99
with a time limit of 10 minutes in place. As seen in Figure 1, these recordings included both the
AC C
EP
TE D
M AN U
SC
RI PT
79
100
participant’s hands and the fluoroscopy screen (to ensure anonymity for subsequent marking). Each
101
participant signed a consent form to allow analysis of these video recordings, and was given a
102
unique random number to allow blinding during subsequent assessment.
ACCEPTED MANUSCRIPT These videos were edited to remove any identifying details, ordered randomly, copied to compact
104
discs and assessed separately by two independent endovascular experts who were fully blinded to
105
the participant status. These scorers recorded whether participants completed the procedure in the
106
allotted time. They also gave a task specific checklist score (TSC) and a global rating score (GRS).
107
Checklists for generating these scores are shown in Table 1 and Table 2 respectively. Averaging the
108
scores from each examiner gave a final TSC and GRS for each participant, which were then added to
109
give an overall procedure score. Finally, examiners were asked to “pass” or “fail” each participant
110
depending on whether they would be happy to supervise them performing the procedure on a real
111
patient. Where examiners disagreed the participant was given a grade of “borderline”.
112
To ensure both scorers understood the scoring procedure, they were initially shown a compilation of
113
clips from ten videos of edited performances of the index procedure being performed on the PHCM.
114
They openly discussed these video recordings to establish joint standards and promote concordance
115
when scoring.
116
Details of the PHCM model evaluated in this trial has been previously published as a technical note
117
in this journal.[10] Briefly, this model uses a pulsatile blood pump (1405 Harvard Apparatus™,
118
Massachusetts, USA) to perfuse a fresh frozen cadaver, with inflow through the right common
119
carotid artery and outflow through the left common femoral and right superficial femoral arteries.
120
All candidates performed their procedures on the same model, within a 48 hour period. The PHCM
121
did not demonstrate clinically significant degradation during the course of the trial.
122
Formal ethical approval was not required as it was deemed that the proposed trial represented
123
‘technical development and training’.
124
Statistical analysis
125
Statistical analysis was undertaken using the Statistical Package for the Social Sciences version 19
126
(SPSS, Chicago). Graphs were generated using GraphPad PrismTM 6.01.
AC C
EP
TE D
M AN U
SC
RI PT
103
ACCEPTED MANUSCRIPT One-way ANOVA plus post-hoc tests with Bonferroni correction were used to assess differences in
128
expert scores between groups. Cronbach’s alpha was used as a measure of inter-rater variability.
129
Categorical variables were analysed using Fisher’s Exact Test.
130
P <0.05 was considered statistically significant.
131
Results
132
In total, 23 participants were recruited to the study. Of these, there were; 7 consultant
133
interventional radiologists, 4 senior trainees (3 vascular surgery, 1 interventional radiology) and 12
134
junior trainees (medical students or foundation year doctors). Additional demographic information
135
for the candidates can be found in Table 3.
136
There were no differences between groups in terms of their baseline demographics, except for
137
previous exposure to virtual reality training; participants classed as intermediate or expert were
138
more likely to have had experience of virtual reality endovascular training than novice participants
139
(Table 3; p=0.002). Whilst none of the participants had experience of cadaver based endovascular
140
training, all participants had experience of cadaver training during undergraduate anatomy teaching.
141
In addition; one of the experts had completed a percutaneous cadaveric nephrolithotomy course,
142
one of the intermediate participants had completed a cadaveric trauma course, and another of the
143
intermediate participants had completed an advanced cadaveric vascular skills course.
144
Our first outcome was whether participants could complete the procedure in the allotted time. All
145
expert and intermediate participants completed the index procedure within 10 minutes. Only one of
146
the 12 novice participants was able to complete the procedure in the allotted time. Fisher’s Exact
147
Test found these differences to be statistically significant (p<0.0005).
148
As another point of comparison, video recordings of participants’ performances were scored by
149
experts. Measuring inter-rater variability, Cronbach’s alpha was 0.966 indicating strong agreement
150
between our two blinded scorers. As described above, each performance was given a task specific
AC C
EP
TE D
M AN U
SC
RI PT
127
ACCEPTED MANUSCRIPT checklist score (TSC) and a global rating score (GRS), which were combined to give an overall
152
procedure score (OPS). The average OPS for the expert candidates’ was 43.00 (TSC 15.00, GRS
153
27.21). The average OPS for the intermediate candidates was 39.63 (TSC 15.00, GRS 24.63). The
154
average OPS for the novice candidates was 19.88 (TSC 8.75, GRS 11.13). These results are
155
summarised in Figure 2A-C. Endovascular novices had significantly lower TSC, GRS and OPS than
156
both intermediate participants and endovascular experts. There were no significant differences in
157
TSC, GRS or OPS between intermediate participants and endovascular experts.
158
Finally, the expert scorers were asked whether they would be happy to supervise the participant
159
performing the index procedure on a real patient in theatre. A participant was given a pass if both
160
examiners would be happy supervising them and a fail if neither examiners were happy to supervise
161
them. Where examiners disagreed a participant was scored as borderline. As shown in Figure 2D all
162
of the intermediate and expert participants received a pass. Out of the 12 novice participants, 2
163
received a pass, 6 received a borderline and 4 received a fail. Fisher’s Exact Test found the
164
differences between groups of different endovascular experience to be statistically significant
165
(p=0.001).
166
Discussion
167
In the present trial the construct validity of a previously reported PHCM was assessed in the setting
168
of basic endovascular training.[10] To achieve this participants with various endovascular experience
169
were recruited, and their ability to perform an index procedure on PHCM was assessed. Participants
170
with no previous endovascular experience (“novice”) performed significantly worse than both those
171
with intermediate endovascular experience and endovascular experts in the following domains;
172
ability to complete the procedure in the allotted time, task specific checklists score (Figure 2A),
173
global rating score (Figure 2B), overall procedure score (Figure 2C) and whether examiners would be
174
happy to supervise the participants in an operating theatre (Figure 2D). Thus the PHCM
175
demonstrated construct validity.
AC C
EP
TE D
M AN U
SC
RI PT
151
ACCEPTED MANUSCRIPT Assessing the validity of a model is clearly vital before its use for training. The ability of a model to
177
demonstrate construct validity is a key part of this process. A model where novices can initially
178
perform as well as experts suggests that the model will be less useful for training. However, if the
179
model demonstrates construct validity, as is the case for PHCM, then continued training on the
180
model could allow novices to improve their performance to that of the experts. These improved
181
skills should then extrapolate to the real procedure, however further studies investigating the
182
educational impact of our PHCM are required to confirm this.
183
The methodology employed in this study is similar to that of previous trials aiming to assess
184
construct validity of a surgical training adjunct.[13-15] The combination of task specific checklist scores
185
and global rating scores as a reliable way to assess surgical skill was first reported two decades
186
ago.[16] Since then it has become the gold standard in reliably assessing technical skills in
187
endovascular literature, and for assessing construct validity in many fields.[14, 15, 17, 18] The reliability of
188
these scoring systems was further supported by the high level of inter-rater reliability reported in
189
this study (Cronbach’s alpha 0.966).
190
This study did not demonstrate that PHCM has the ability to differentiate between endovascular
191
practitioners with intermediate experience versus endovascular experts. This may well be a type 2
192
error, resulting from the two main limitations of this study. Firstly, only four intermediate
193
participants could be recruited. This limitation was due to low numbers of intermediate
194
endovascular trainees within our training deanery. The number of endovascular experts was also
195
relatively low (n=7). These low numbers meant that statistical analysis comparing these two groups
196
lacked statistical power, increasing the likelihood of type two errors. Secondly, the index training
197
procedure selected for this study was a relatively simple procedure, which is likely to be mastered at
198
an early stage in endovascular training. A procedure requiring greater skill may be required to
199
differentiate intermediate versus expert participants. However, it was not possible to reliably and
AC C
EP
TE D
M AN U
SC
RI PT
176
ACCEPTED MANUSCRIPT uniformly repeat a more challenging procedure such as stenting or angioplasty using PHCM in the
201
context of this trial.
202
Furthermore, it is suggested that the higher fidelity of PHCM is more appropriate for intermediate or
203
expert level practitioners exploring new techniques. Novice practitioners may benefit less when
204
honing basic guidewire handling skills. Further work is necessary comparing training on PMCH
205
versus a virtual reality simulator to shed more light on the model’s efficacy for training practitioners
206
at different levels.
207
Conclusion
208
This study has revealed that PHCM demonstrates construct validity; it can discriminate between
209
participants with previous endovascular experience from those without. This complements previous
210
work which demonstrated face validity of PHCM. Further work should assess the educational impact
211
of PHCM, to confirm that the model is a useful training tool.
212 213 214 215
Acknowledgments
TE D
M AN U
SC
RI PT
200
The authors would like to thank the staff of the Newcastle Surgical Training Centre for their support throughout the project, Medtronic Ltd for their generous educational grant and the radiology department of the Freeman Hospital for their expertise.
Conflict of Interest
218 219 220 221
Declaration of Funding
The authors declare no conflicts of interest.
AC C
222 223 224 225 226 227 228 229
EP
216 217
The authors declare a £2000 educational grant from Medtronic Ltd. This was donated to part fund the original purchase of the pulsatile pump. We can confirm that Medtronic had no further role in this or any subsequent trials
References 1. 2.
3.
Van Herzeele I. Virtual Reality Endovascular Simulation: Ready for Training. Nautilus Academic Books. 1st ed. 2009. Schanzer, A., et al., Vascular surgery training trends from 2001-2007: A substantial increase in total procedure volume is driven by escalating endovascular procedure volume and stable open procedure volume. J Vasc Surg, 2009. 49(5): p. 1339-44. Neequaye, S.K., et al., Endovascular skills training and assessment. Journal of Vascular Surgery, 2007. 46(5): p. 1055-1064.
ACCEPTED MANUSCRIPT 4. 5. 6.
10. 11. 12. 13. 14. 15. 16. 17. 18.
AC C
263 264 265
M AN U
9.
TE D
8.
SC
RI PT
7.
Gilbody, J., et al., The use and effectiveness of cadaveric workshops in higher surgical training: a systematic review. Ann R Coll Surg Engl, 2011. 93(5): p. 347-52. Reed, A.B., et al., Back to basics: use of fresh cadavers in vascular surgery training. Surgery, 2009. 146(4): p. 757-62; discussion 762-3. Garrett, H.E., Jr., A human cadaveric circulation model. Journal of Vascular Surgery. 33(5): p. 1128-1130. Linsen, M.A., et al., Modular branched endograft system for aortic aneurysm repair: evaluation in a human cadaver circulation model. Vasc Endovascular Surg, 2007. 41(2): p. 126-9. Arbatli, H., et al., Dynamic human cadaver model for testing the feasibility of new endovascular techniques and tools. Ann Vasc Surg, 2010. 24(3): p. 419-22. Jongkind, V., et al., Direct videoscopic approach to the thoracic aorta for aortic endograft delivery: evaluation in a human cadaver circulation model. J Endovasc Ther, 2010. 17(1): p. 12-8. Nesbitt, C., et al., Design of a Pulsatile Fresh Frozen Human Cadaver Circulation Model for Endovascular Training. Ann Vasc Surg, 2017. 44: p. 425-430. Nesbitt, C., et al., A Pulsatile Fresh Frozen Human Cadaver Circulation Model for Endovascular Training: A Trial of Face Validity. Ann Vasc Surg, 2018. 46: p. 345-350. Schreuder, H.W., et al., Face and construct validity of virtual reality simulation of laparoscopic gynecologic surgery. Am J Obstet Gynecol, 2009. 200(5): p. 540.e1-8. Piromchai, P., et al., The construct validity and reliability of an assessment tool for competency in cochlear implant surgery. Biomed Res Int, 2014. 2014: p. 192741. Awad, Z., et al., Construct validity of the ovine model in endoscopic sinus surgery training. Laryngoscope, 2015. 125(3): p. 539-43. Sharma, M., et al., Construct validity of fresh frozen human cadaver as a training model in minimal access surgery. Jsls, 2012. 16(3): p. 345-52. Martin, J.A., et al., Objective structured assessment of technical skill (OSATS) for surgical residents. Br J Surg, 1997. 84(2): p. 273-8. Hislop, S.J., et al., Simulator assessment of innate endovascular aptitude versus empirically correct performance. Journal of Vascular Surgery. 43(1): p. 47-55. Berger, P., et al., Validation of the Simulator for Testing and Rating Endovascular SkillS (STRESS)-machine in a setting of competence testing. J Cardiovasc Surg (Torino), 2010. 51(2): p. 253-6.
EP
230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262
Tables
Table 1 – Task Specific Checklist scoring tool
1. Selects Standard J-Tip Wire 2. Inserts J-Tip wire safely 3. Selects Pigtail catheter 4. Inserts Pigtail catheter safely 5. Removes J-Tip wire safely 6. Performs adequate aortic angiogram 7. Re-inserts J-Tip wire safely 8. Removes Pigtail catheter safely
Not done or incorrect 0 0 0 0 0 0 0 0
Done correctly 1 1 1 1 1 1 1 1
ACCEPTED MANUSCRIPT 0 0 0 0 0 0 0 0
1 1 1 1 1 1 1 1
266 Table 2 – Global Rating Score assessment tool
1
2
Respect for Tissue 3
4
Careful handling of tissues and/or lesion, but occasional potential for inadvertent tissue damage
1
2
M AN U
Frequently used unnecessary force on tissue and/or lesion, potential for tissue damage
SC
267
Time and Motion 3
Make unnecessary moves and/or excessive time
4
Efficient time and moves but some unnecessary moves and/or excessive time
1
2
Instrument Handling 3
4
Competent use of instruments, but occasionally appeared stiff or awkward
TE D
Repeated tentative, awkward, and/or inappropriate moves with instruments
2
Flow of Operation 3
1
2
AC C
Frequently stopped operating and seemed unsure of next move; demonstrated imprecise and/or wrong operative technique
EP
1
Very poor
1
2
Unacceptable quality
RI PT
9. Selects Cobra catheter 10. Inserts Cobra catheter safely 11. Cannulates left renal artery safely 12. Selects hydrophilic wire 13. Inserts hydrophilic wire safely 14. Advances Cobra catheter safely 15. Removes hydrophilic wire safely 16. Angiographic confirmation of left renal artery catheterisation
4
Demonstrated some forward planning with reasonable progression of procedure; careful operative technique with occasional errors
5
Consistently handled tissues and/or lesion appropriately with minimal tissue damage
5 Clear economy of moves and time with maximum efficiency
5 Fluid movements with instruments and no stiffness or awkwardness
5 Planned course of operation with effortless flow throughout; fluent, secure, and correct operative technique in all stages of procedure
Overall Performance 3
4
Competent
5 Clearly superior
Final Product 3
4
Average quality
5 Superior quality
268 269 270
Table 3 - Participant demographics. The absolute numbers of participants are given, and comparisons performed using the Fisher’s Exact Test. VR: virtual reality simulator.
Wears glasses Left handed
Novice (n=12) 1 0
Intermediate (n=4) 0 0
Expert (n=7) 0 1
Fisher’s Exact Test (p-value) 1.000 0.478
ACCEPTED MANUSCRIPT 6
2
4
1.000
7
1
1
0.169
0
3
4
0.002
0
0
0
N/A – zero participants
12
4
7
271
SC
Figures
N/A - all participants
EP
274 275
Figure 1 - Screenshot from an exemplar video of a participant performing cannulation of the left renal artery and confirmatory angiogram from access in the right femoral artery.
AC C
273
TE D
M AN U
272
RI PT
Plays a musical instrument Plays video games regularly Previous VR endovascular training Previous cadaver endovascular training Any previous cadaver training
M AN U
SC
RI PT
ACCEPTED MANUSCRIPT
276
TE D
EP
284
Figure 2 - Ability to perform the index procedure by participants with varying endovascular experience. Two independent endovascular experts marked anonymised video recordings of participants using the human cadaver model. A), B) and C) show box and whisker plot s demonstrating the task specific checklist score, global rating score, and overall procedure score respectively. ANOVA was used for statistical analysis and the results of post-hoc tests with Bonferroni correction are shown (NS: not significant; ***: P≤0.001; ****: P≤0.0001). D) Displays whether the examiners would be happy to supervise the participant performing the index procedure in theatre on a real patient (Pass = both examiners would be happy to supervise, Borderline = one examiner would be happy to supervise, Fail = neither examiner would be happy to supervise).
AC C
277 278 279 280 281 282 283