- Email: [email protected]
Communication Skills Assessment Using Human Avatars: Piloting a Virtual World Objective Structured Clinical Examination
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48
urologypracticejournal.com
Communication Skills Assessment Using Human Avatars: Piloting a Virtual World Objective Structured Clinical Examination Bruce R. Kava,* Allen D. Andrade, Robert Marcovich, Thaer Idress and Jorge G. Ruiz From the Department of Urology, University of Miami Miller School of Medicine (BRK, RM) and Laboratory of E-learning and Multimedia Research, Geriatric Research Education and Clinical Centers, Bruce W. Carter Veterans Affairs Medical Center (TI, JGR), Miami, Florida, and Geriatric Research Education and Clinical Centers, James J. Peters Veterans Affairs Medical Center, Bronx and Brookdale Department of Geriatrics and Palliative Medicine, Icahn School of Medicine at Mount Sinai, New York, New York (ADA)
Abstract
Abbreviations and Acronyms
Introduction: Proficiency in communication skills is a core competency of residency training. We evaluated the feasibility, acceptability and applicability of a virtual world objective structured clinical examination that enables practice based learning and assessment of resident communication skills. Methods: A virtual clinical encounter situated in 2 practice settings was developed that uses a human avatar physician and a standardized patient. Following an online tutorial house staff participated in 4 communication tasks, including shared decision making, delivering bad news, obtaining informed consent and disclosing a medical error. Validated instruments and semistructured interviews were done to assess house staff acceptability and applicability of the platform. Three faculty members used ACS (Affective Competency Scale) and communication specific assessment instruments to evaluate house staff performance. Results: A total of 12 urology house staff completed the simulation. Direct costs were approximately $1,000. The virtual world was easy to use and immersive. Applicability directly correlated with presence (Pearson r ¼ 0.67, p ¼ 0.01) and co-presence (Pearson r ¼ 0.8, p ¼ 0.002). House staff identified problems with 1) limited nonverbal cues, 2) too much information presented and 3) a lack of immediate feedback. The ICC (intraclass correlation) of faculty assessments was high for ACS at 0.53 (95% CI 0.36e0.69) for single measures, 0.77 (95% CI 0.63e0.86) for average measures and less for other assessment instruments.
4HCS = 4 Habits Coding Scheme BAS = Bad News Assessment Scale CSAS = Communication Skills Attitude Assessment Scale KEECC-A = Kalamazoo Essential Elements Communication Checklist-A OSCE = Objective Structured Clinical Encounter PSA = prostate specific antigen
Conclusions: A virtual world objective structured clinical examination is a feasible, acceptable and applicable method of communication skills assessment. Improving nonverbal cues, focusing on individual skill sets and providing immediate feedback are measures to be adopted in future iterations of this platform. Key Words: urology, computer simulation, clinical competence, social skills, physicians Submitted for publication October 14, 2015. No direct or indirect commercial incentive associated with publishing this article. The corresponding author certifies that, when applicable, a statement(s) has been included in the manuscript documenting institutional review board, ethics committee or ethical review board study approval; principles of Helsinki Declaration were followed in lieu of formal ethics committee approval; institutional animal care and use committee approval; all human subjects provided 2352-0779/17/41-1/0 UROLOGY PRACTICE Ó 2016 by AMERICAN UROLOGICAL ASSOCIATION EDUCATION
AND
written informed consent with guarantees of confidentiality; IRB approved protocol number; animal approved project number. Supported by a Society of Urologic Chairpersons and Program Directors grant. * Correspondence: Department of Urology, University of Miami Miller School of Medicine, 1600 Northwest 10th Ave., No. 1140, Miami, Florida 33101 (telephone: 305-243-4936; FAX: 305-243-4939; e-mail address: [email protected]).
RESEARCH, INC.
http://dx.doi.org/10.1016/j.urpr.2016.01.006 Vol. 4, 1-9, January 2017 Published by Elsevier
FLA 5.4.0 DTD URPR177_proof 16 September 2016 4:14 pm EO: UP-15-94
49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96
2
97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147
Communication Skills Assessment Using Human Avatars
Communications skills are designated as a core competency for residency training by ACGME (Accreditation Council for Graduate Medical Education). These skills are extremely difficult to teach and evaluate.1 Moreover, increasing time constraints and an expanding number of technical skills have drastically reduced the amount of time that can be committed toward fostering these humanistic aspects of medical care.2e5 As a result, skills such as delivering bad news or medical error disclosure are often left up to urology residents to learn through trial and error.2,5 The need for greater accountability to ensure patient safety requires urology program directors to find innovative ways for residents to improve these difficult communications skills. OSCEs provide opportunities for practice based learning and assessment of communication skills in a safe and reproducible environment. This method has proved to be a reliable and valid assessment tool6,7 but is expensive, requiring preparation, support staff, travel, equipment and location availability.8 Virtual worlds are computer generated platforms in which users interact via graphical character representatives called avatars. Individuals represented by specific avatars can communicate with each other via VoIP (Voice over Internet Protocol) communication. This portable, reusable, distance learning computer simulation can deploy multiple different avatar personas set in multiple practice settings. It allows for artificial advancement of time or disease progression in ways that are not possible in real life. Preliminary work incorporating immersive virtual world technology has demonstrated a positive impact on learning.9e11 Our objective was to evaluate the feasibility, acceptability and applicability of a virtual world OSCE in evaluating urology house staff proficiency with difficult communication skills. Feasibility was related to the technical challenge of creating an interactive platform in several practice settings and its attendant costs. For acceptability we characterized the participant sense of immersion in the virtual world using 2 parameters from the gaming and media industry, that is presence and co-presence. There is a paucity of educational data describing the knowledge, attitudes and self-perceived proficiency of urology house staff with various communication skills. Establishing these metrics and determining how applicable the house staff believed the platform to be were additional criteria that were critical to the aims of the project.
Methods
The study was institutional review board approved, participation was voluntary and all urology house staff at our
institution were eligible to participate. It was performed in 2 148 offices in different buildings of the medical campus, each 149 equipped with desktop computers capable of VoIP 150 communication. The virtual world was developed using 151 Second Life (Linden LabÒ) and it consisted of 2 virtual 152 world settings (see figure). After obtaining informed consent ½F1153 participants completed a demographic questionnaire and 154 CSAS.12 They watched a 40-minute communication skills 155 tutorial online developed from the AUA (American Uro- 156 157 logical Association) core curriculum.13 Four scripted encounters were developed (see Appendix), 158 each highlighting a specific communication skill. The “Wizard 159 of Oz” technique was deployed.14 In this technique an unseen 160 associate (one of the faculty members) played the role of a 161 standardized patient, processing participant responses in real 162 time and responding quickly enough to support an acceptable 163 interactive encounter. The de-identified audio recordings of 164 each encounter were archived on MPEG files and later 165 166 reviewed by the study team faculty. After each encounter participants and the standardized 167 patient completed ACS.15 Following the final encounter 168 house staff completed a post-encounter CSAS, and the 169 Presence, Co-presence and Applicability questionnaires.16e18 170 Participants were debriefed with a semistructured interview 171 172 performed by a study team member. The faculty consisted of 2 urologists and 1 geriatrics- 173 174 palliative care physician who concomitantly served as the avatar standardized patient. They independently evaluated 175 176 the archived audio files for each virtual OSCE using several 177 validated instruments. 178 179 Assessment Instruments 180 181 The Presence Questionnaire of Witmer et al evaluates the 182 perception of presence, defined as the subjective experience 183 of being in 1 place or environment even when one is 184 physically situated in another place.16 A 7-point Likert scale 185 was used with higher scores indicating a greater sense of 186 presence. 187 Co-presence is the subjective sensation of interacting 188 with another person. A 10-point Likert scale was used17 189 with higher scores indicating a greater sense of realism. 190 Applicability of the virtual world simulation was evalu191 ated using several validated criteria.18 A 10-point Likert scale 192 was used with higher scores indicating greater applicability. 193 ACS15 was the primary metric used to measure proficiency 194 for each communication skill. Using a 5-point Likert scale 6 195 global elements of the physician-patient interaction were 196 assessed. Scores above 3 for each component (total score 197 greater than 18) are generally associated with competency.15 198
FLA 5.4.0 DTD URPR177_proof 16 September 2016 4:14 pm EO: UP-15-94
Communication Skills Assessment Using Human Avatars
199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249
3
Figure. Desktop simulation was performed in cordoned area of communications laboratory. House staff member is seated in front of standard 23-inch computer monitor, interacting via microphone with avatar patient in office (a) and in hospital (b). Faculty member serving as designated standardized patient is seated in another locale and controls avatar patient movements and dialogue via “Wizard of Oz” technique (c).14 His screen depicts avatar physician and he follows script developed for particular encounter.
ACS is unique in that when completed by the house staff, it provides a valid measure of self-efficacy.11,15 Specific communication skills were also evaluated by items from several instruments. 1) KEECC-A19 was used to assess shared decision making and PSA screening. 2) We used 4HCS20 to assess providing informed consent for total prostatectomy. 3) BAS21 was used to evaluate how the residents delivered a diagnosis of prostate cancer. 4) The Medical Error Disclosure Rating Scale22 was used to assess proficiency in medical error disclosure. Data were entered directly into SPSSÒ. Nine positive items (ie “learning about communication skills will help me be a better physician”) and 7 negative items (ie “I can’t see the point in learning communication skills”) from CSAS were compared before and after the simulation using paired sample t-tests. Descriptive measures of house staff proficiency are provided as the mean of the 3 independent faculty scores. Based on the 5-point Likert scores individual items from KEECC-A, 4HCS and BAS were dichotomized into proficiency scores, including highdaverage faculty score greater than 3 or lowdaverage faculty score 3 or less.23,24 Medical error disclosure items were also dichotomized with an average score 5 or less for each global parameter considered low profiicency.22 Interrater reliability among the 3 faculty evaluations was assessed using a 2-way mixed effects model. The ICC is reported along with the 95% CI. Applicability was correlated with presence and co-presence scales using Pearson r.
Semistructured interviews were transcribed and reviewed by a study team member (BRK). Several themes emerged and were analyzed by open, axial and selective coding. Results Demographics and Attitudes
Eight junior residents (preliminary, and urology 1 and 2) and 4 senior residents (urology 3 and 4) participated in the study. All except 1 participant were male. Table 1 summarizes house staff experiences and attitudes toward difficult communication skills. Technical Feasibility
All participants completed the tutorial, OSCEs and the debriefing interview. No technical problems were encountered. Costs of development included leasing the virtual world space for 3 months ($50), virtual materials ($200) and time needed to develop the virtual world during 2 working days by a single experienced developer ($750). Presence, Co-Presence and House Staff Applicability
The virtual world experience was perceived as easy to use, realistic and immersive (table 2). Applicability scores directly correlated with presence (Pearson r ¼ 0.67, p ¼ 0.01) and co-presence (Pearson r ¼ 0.8, p ¼ 0.002).
FLA 5.4.0 DTD URPR177_proof 16 September 2016 4:14 pm EO: UP-15-94
250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 ½T1 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 ½T2 299 300
4
301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351
Communication Skills Assessment Using Human Avatars
Table 1. Select items instruments
from
Presence,
Co-presence
and
Applicability
Table 2. Select items instruments
from
Presence,
Co-presence
No. Participants (%) Previous experience with learning communication skills Delivering bad news: Medical school Postgraduate training None Providing informed consent: Medical school Postgraduate training None Medical error disclosure: Medical school Postgraduate training None
5 (42) 1 (8) 6 (50) 5 (42) 6 (50) 1 (8) 4 (33) 3 (25) 5 (42)
Previous experience with specialty specific communication skills Discussing prostate Ca screening: Medical school Postgraduate training None Disclosing prostate Ca diagnosis: Medical school Post-graduate training None Discussing prostate Ca treatment options: Yes No
2 (17) 5 (42) 5 (42) 2 (17) 5 (42) 5 (42) 6 (50) 6 (50)
Attitudes concerning communication skills Comfort level in delivering bad news to pt: Extremely or very comfortable Moderately comfortable Slightly comfortable Self-perceived proficiency in difficult communication skills: Very good Moderate Fair Effective communication skills cannot be taught: Disagree Agree Undecided Developing effective communication skills is as important as medical knowledge: Agree Disagree Undecided
9 (75) 2 (17) 1 (8)
Presence (scale 1e7): 1. Able to control events 2. Responsive to environment 3. Natural interaction with environment 4. How completely were your senses engaged 5. How much visual aspects involved you 6. How much auditory aspects involved you 7. How natural were movements 8. Your sense of objects/virtual subjects moving 9. How consistent was virtual world with real life 10. How quickly did you adjust to platform 11. How proficient at end 12. How involved were you in scenario Co-presence (scale 1e10): 1. Like face-to-face meeting 2. How real did interaction with pt seem 3. You can really get to know someone with this system Applicability (scale 1e10): 1. Interesting 2. Entertaining 3. Eager to use again 4. How helpful was learning experience 5. How much did it help you learn 6. Friendly to use 7. Difficult to complete 8. Effort required to learn material 9. Virtual world OSCE helpful 10. Recommend developing platform
and
Applicability
Mean SD
Median
4.7 4.3 4.3 3.9 5.8
5 5 4.5 4.5 6.5 e 4 e 6 4 5 6
4.1 4.8 4.25 4.6 5.75
(1.7) (1.3) (0.9) (1.6) (1.7) e (1.4) e (2) (1.9) (1.4) (1.1)
5.6 (2.8) 6.3 (2.4) 6.3 (2.8)
5.5 7 7.5
7.6 7.7 6.4 7.5 6.8 8.1 3.4 5 6.5 8.1
8 8 6.5 7.5 7 8.5 2.5 5 7 8
(1.8) (1.9) (1.6) (1.5) (1.7) (1.6) (2.3) (2.2) (2.6) (1.8)
Competency Assessment 3 (25) 5 (42) 4 (33) 8 (67) 3 (25) 1 (8)
8 (67) 2 (17) 2 (17)
Prestimulation and Post-Simulation Communication Skills Attitudes
No differences were noted between CSAS positive attitude items before vs after the virtual world simulation (presimulation and post-simulation mean SD score 32.8 6.4 and 34.2 7.6, respectively, p ¼ 0.317). There was a trend of less negative attitudes toward communication skills following the simulation but this did not reach statistical significance (mean presimulation and post-simulation negative attitude CSAS score 14.3 2.5 and 12.83 3.1, respectively, p ¼ 0.079)
Table 3 shows mean faculty and house staff ACS scores. Most of the house staff received average or just above average communication proficiency scores from the faculty. Residents were more critical of their own communication skills competency with more than 50% assessing themselves with low proficiency scores. The ICC of the ACS was 0.53 (95% CI 0.36e0.69) for single measures and 0.77 (95% CI 0.63e0 86) for average measures.
Specific Communications Skills Assessments
Table 4 shows faculty scoring from the communication specific instruments. There was wide variability in perceived house staff proficiency in the various skill sets. Particularly problematic were 1) understanding the patient perspective and bidirectional sharing of information for PSA screening, 2) breaking the bad news of a prostate cancer diagnosis, 3) gaining the patient perspective and demonstrating empathy when performing informed consent, and 4) being empathetic, apologizing and indicating how to prevent future medical errors at the time of an error disclosure. The ICC
FLA 5.4.0 DTD URPR177_proof 16 September 2016 4:14 pm EO: UP-15-94
352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 ½T3 382 383 384 385 386 387 388 389 390 391 392 393 ½T4 394 395 396 397 398 399 400 401 402
Communication Skills Assessment Using Human Avatars
403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453
5
Table 3. Encounter ACS scores based on faculty evaluations and house staff self-assessment scores, and number of residents with proficiency score less than 18 Simulation Scores*
Mean SD ACS: Confident Comfortable Empathetic/sensitive Respectful/professional Informative Comforting No. residents total score less than 18 (%) Mean SD self-assessment: Confident Comfortable Empathetic/sensitive Respectful/professional Informative Comforting No. residents total score less than 18 (%)
1. Shared Decision Making (PSA Screening)
2. Delivering Bad News (Prostate Ca Diagnosis)
3. Informed Consent (Radical Prostatectomy)
4. Disclosing Medical Error
20.8 4.0 3.7 0.8 3.7 0.9 3.2 0.8 3.7 0.7 3.4 0.9 3.2 0.9 3 (25)
20.3 6.1 3.6 0.9 3.6 0.8 3.3 1.2 3.7 0.8 3.4 1.1 3.3 1.2 3 (25)
21.2 4.1 3.7 0.8 3.6 0.8 3.3 1.0 3.6 0.8 3.6 0.8 3.3 1.1 3 (25)
19.2 3.4 3.1 3.1 3.5 3.2 2.9 4
(4.8) (0.6) (0.9) (1.2) (0.8) (0.9) (1.2) (33)
19.4 2.7 3.3 0.5 3.3 0.5 3.3 0.7 3.1 0.7 3.1 0.7 3.3 0.5 5 (42)
19.4 2.7 3.1 0.5 3.3 0.6 3.3 0.9 3.4 0.7 3.1 0.5 3.3 0.6 7 (58)
19.3 3.3 3.0 0.9 3.2 0.7 3.2 0.7 3.6 0.7 3.2 0.6 3.3 0.5 7 (58)
17.8 2.9 2.8 3.1 3.3 3.1 2.6 8
(2.8) (0.3) (0.3) (0.7) (0.8) (0.7) (0.9) (67)
*Score scale 1dunsatisfactory, 3daverage and 5dsuperior.
was 0.73 for BAS (95% CI 0.54e0.93), 0.27 for KEECC-A (95% CI 0.18e0.39), 0.26 for medical error disclosure (95% CI 0.20e0.36) and 0.25 for the 4 Habits model (95% CI 0.15e0.37).
suggested that immediate feedback from a faculty member be provided after each simulation.
Discussion Debriefing Interviews
The majority of residents felt immersed in the virtual world, commenting that “I forgot about my surroundings,” “I felt like I was in a room with a patient,” and “It felt as if I was working in the clinic without a physical exam.” They were critical about the paucity of nonverbal communication cues exhibited by the avatar, which were limited to looking down or away from the computer screen. This confused some of the house staff, who indicated that they felt “awkward” and “could not gauge whether [the avatar patient] was understanding what they were being told.” One resident indicated that the movements seemed “artificial [but] if I looked away from him, I was fine.” The house staff believed that too much information was presented in the tutorial. Several proposed that the skill sets should be presented as a series of modules. “Feeling awkward,” “uncomfortable” and “lacking confidence” were terms used to describe how residents felt about disclosing a medical error. Some thought that there were too many complications and several indicated that this changed the manner in which they approached informed consent for radical prostatectomy. They unanimously
There is growing recognition that effective communication skills translate to higher patient satisfaction and better quality medical care.23e30 While most medical schools offer students generic communication skills classes, during postgraduate training they are expected to transition to a practice based learning environment. For many residents a considerable amount of time has elapsed since classroom training, which may adversely impact recollection or retention of the topics that were discussed. Whether this is the case or whether more than 50% of our house staff were never taught the essential components of delivering bad news or disclosing a medical error is worth further evaluation. Our data identify glaring educational deficiencies in several essential communications skills in the postgraduate setting. This adds to the growing body of data on other subspecialties,4,5,25,29,30 which supports that there is a critical need to develop a structured communication skills curriculum in postgraduate medical training that includes shared and specialty specific communication skill sets. To our knowledge this is the first study to integrate a virtual world platform to assess communication skills proficiency for urology house staff. Using this platform the house staff were provided with an iterative learning
FLA 5.4.0 DTD URPR177_proof 16 September 2016 4:14 pm EO: UP-15-94
454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504
6
505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555
Communication Skills Assessment Using Human Avatars
Table 4. Proficiency scores of various items on specific communication skills instruments
PSA screening (KEECC-A*): 1. Builds relationship 2. Opens discussion 3. Gathers information 4. Understands pt perspective 5. Shares information 6. Reaches agreement 7. Provides closure Delivering prostate Ca diagnosis (BAS*): 1. Setting scene 2. Breaking news 3. Eliciting concerns 4. Information giving 5. General considerations Total prostatectomy informed consent (4HCS*): 1. Investing in beginning 2. Eliciting pt perspective 3. Demonstrating empathy 4. Investing in end Medical error disclosure (Medical Error Disclosure Rating Scale global scores):y 1. Explanation of medical facts regarding error 2. Honesty and truthfulness 3. Empathy 4. Prevention of future errors 5. General communication skills
No. Low Proficiency Scores (%)
Mean SD
Median (range)
3.2 3.3 3.4 3.0 3.2 3.4 3.4
(0.9) (0.9) (1.1) (1.1) (1.1) (1.0) (1.3)
3.3 3.7 4 3.2 3.5 3.8 3
(2.0e4.6) (1.7e4.3) (2.0e4.3) (1.0e4.3) (1.3e4.3) (1.7e4.3) (1.3e4.3)
6 5 4 7 6 5 4
3.7 3.0 3.0 3.5 3.2
(0.6) (1.0) (0.9) (0.9) (0.9)
3.8 3.1 3.2 3.8 3.3
(2.6e4.3) (1.1e4.5) (1.4e4.1) (2.3e4.7) (1.9e4.5)
1 (8) 5 (42) 3 (25) 4 (33) 4 (33)
3.5 3.4 3.2 3.5
(0.5) (0.8) (0.7) (0.6)
3.6 3.6 3.3 3.4
(2.9e4.5) (2.0e4.4) (1.7e4.3) (2.5e4.4)
2 4 4 3
(17) (33) (33) (25)
7.7 7.0 6.1 3.7 6.2
(1.5) (2.0) (2.6) (1.2) (1.9)
8 (4.0e9.3) 7 (3.3e10.0) 6.4 (2.0e10.0) 3.3 (2.0e6.7) 6 (3.3e8.7)
2 4 6 11 7
(17) (33) (50) (92) (58)
(50) (42) (33) (58) (50) (42) (33)
*Five 5-point Likert scale. y Ten-point Likert scale.
environment in which they could practice essential communication skills, identify their mistakes and test solutions that they devised without adversely impacting a patient. By enabling them to learn from their mistakes the platform provides opportunities for improved understanding, confidence and self-efficacy.30 One of the criteria that we believed to be essential to the feasibility and success of the platform was its acceptability to the house staff. Similar to the feeling that one gets when watching a motion picture, virtual world experiences trigger a subjective sense of immersion in the virtual world that lends authenticity to the experience and induces a willful suspension of disbelief.11 Presence and co-presence are often used to gauge this sense of immersion when evaluating video games and other virtual world technology. As we report, the greater the degree of this immersion in the virtual world, the greater the perceived applicability of the simulation. This is a critical finding in that it provides a rationale for exploring new 3dimensional immersive technologies that may strengthen the realism associated with the virtual world experience. We described how the virtual world OSCE can be archived and subsequently used as an assessment instrument. Specific skills associated with the lowest overall assessment scores included 1) gaining the patient perspective, 2) breaking bad news, 3) bidirectional communications and 4) failing to
apologize or discuss strategies to prevent future errors when discussing medical error prevention. These data provide a starting point with which to focus our educational efforts. The task of evaluating competency in communication skills is difficult due to the inherent complexities of grading communications.13 The high ICC associated with ACS corroborates that with little training the faculty could negotiate the instrument and evaluate the house staff reliably and consistently. In general, the progressively lower ICC associated with the other communication instruments was not surprising, given that each requires hours of dedicated training and standardization. Future studies focusing on specific communications skills must ensure that the faculty has adequate training and expertise with the specific communications objectives and instruments used to assess them. The debriefing interviews indicate the need for several modifications to virtual world OSCEs. Enhancing the nonverbal emotional cues expressed by the avatars will require technical upgrades. Future simulations must on 1 communication skill at a time and provide immediate faculty feedback. Finally, plans to establish a multidisciplinary communications curriculum to develop uniform criteria for teaching and assessing communication skills are under way. There are several limitations of the study. The small sample size limits the generalizability of our findings. While
FLA 5.4.0 DTD URPR177_proof 16 September 2016 4:14 pm EO: UP-15-94
556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606
Communication Skills Assessment Using Human Avatars
607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657
we limited the goals of this study to determining the technical feasibility, acceptability and applicability of a virtual world, a parallel comparison group in a real world OSCE could provide additional data on the comparative effectiveness of this virtual world assessment modality. Future studies will also need to focus on how the platform can be applied to self-assessment and expert assessment, mentorship and coaching, and individual or group learning.17,28e30
Conclusion
A virtual world OSCE provides a novel platform for teaching and evaluating communication skills proficiency during residency training in urology. The platform was technically feasible, acceptable and applicable. Acceptability was strongly associated with the perceived applicability of the platform. Faculty assessment was consistent using the primary assessment instrument (ACS) but less so with more complex communications instruments. Technical refinements that improve nonverbal cues are needed, as is providing immediate feedback to the house staff and focusing on 1 communication skill at a time. Future iterations of the virtual world OSCE should be developed in tandem with a comprehensive multidisciplinary communication skills curriculum. This will promote better communications between physicians and patients that may have many attendant benefits for urology and all medical specialties. Appendix. Four distinct scripted encounters were presented to each resident during desktop simulation and each virtual world OSCE highlights specific communication skill Encounter 1: Shared decision making in determining the need for prostate cancer screening Introduction: This is an initial visit for a patient who was referred to you to discuss prostate cancer screening. Instructions: You have 10 minutes to discuss prostate cancer screening with this patient. The patient is a 50 year old male who is a supermarket manager. He was specifically referred to you by his primary care doctor for “prostate cancer screening.” He has mild lower urinary tract symptoms (LUTS). His only other medical problem is hypertension that is managed with hydrochlorothiazide. He has no other past medical history, has no allergies, and his physical exam is relatively normal. Encounter 2: Delivering bad news that a patient has prostate cancer. Introduction: Following your initial visit, Mr. Thomas was found to have an elevated PSA level, that on repeat was once again elevated at 5.2 (free PSA 12%). After discussing the options with him, a TRUS-prostate biopsy was performed revealing Gleason 6 prostate cancer in 2/12 cores (right apex and left base), both involving 3 mm and less than 10% of each core. He has a 40 gram prostate and no obvious abnormalities were appreciated on ultrasound. Instructions: You have 15 minutes to deliver the bad news to Mr. Thomas that he has prostate cancer and to discuss the treatment options with him. He has been calling the office daily since the biopsy was performed last week and is extremely anxious. He told your secretary that his father had prostate cancer and that he was really worried about having it himself.
7
Encounter 3: Obtaining informed consent for a radical prostatectomy. Introduction: Mr. Thomas is a 50 year old male with clinical T1c prostate cancer, who has a PSA 5.2ng/ml and who has two cores of Gleason 6/10 cancer on biopsy. The patient has decided that he is going to have a radical prostatectomy. Instructions: You have 10 minutes to discuss radical prostatectomy with the patient and to provide an informed consent for the surgery. Encounter 4: Disclosing a medical error: complications during surgery including bleeding, inadvertent rectal injury, in addition to the need for resection of one of the nerve bundles. Introduction: Mr. Thomas had his radical prostatectomy yesterday. You are making hospital rounds. Surgery proved a lot more complicated than you thought it would be. The prostate was much larger than you had anticipated. This made it difficult to see well and there was a lot more blood loss than you expected (approximately 900 ml were lost). The patient did not get transfused in the operating room, but needs to have one now because his blood pressure was very low. Only one nerve bundle was preserved because of the difficulty with surgery. You also miscalculated where the prostate was and made a small laceration in the anterior rectal wall that needed a colorectal surgeon to repair it. Although the repair seemed to be fine, you need to discuss the potential need for a colostomy if the repair breaks down. Also, since the surgery was so complicated and there was a rectal injury, the patient’s catheter will have to be left in for a couple of weeks, he will need to be on no oral intake for today, and he will likely be in the hospital for at least 3e5 days.
References 1. Swing S: The ACGME outcome project: retrospective and prospective. Med Teach 2007; 29: 648. 2. Sise M, Sise CB, Sack DI et al: Surgeons’ attitudes about communicating with patients and their families. Curr Surg 2006; 63: 213. 3. Dosanijh S, Barnes J and Bhandari M: Barriers to breaking bad news among medical and surgical residents. Med Educ 2001; 35: 197. 4. Hebert H, Butera JN, Castillo J et al: Are we training our fellows adequately in delivering bad news to patients? A survey of hematology/oncology program directors. J Palliat Med 2009; 12: 1119. 5. Orgel E, McCarter R and Jacobs S: A failing medical educational model: a self- assessment by physicians at all levels of training of ability and comfort to deliver bad news. J Palliat Med 2010; 13: 677. 6. Sharp PC, Pearce KA, Konen JC et al: Using standardized patient instructors to teach health promotion interviewing skills. Fam Med 1996; 28: 103. 7. Brannick MT, Erol-Korkmaz HT and Prewett M: A systematic review of the reliability of objective structured clinical examination scores. Med Educ 2011; 45: 1181. 8. Patricio MF, Juliao M, Fareleira F et al: Is the OSCE a feasible tool to assess competencies in undergraduate medical education? Med Teach 2013; 35: 503. 9. Cook DA, Erwin PJ and Triola MM: Computerized virtual patients in health professions education: a systematic review and meta-analysis. Acad Med 2010; 85: 1589. 10. Kleinert R, Wahba R, Chang DH et al: 3D immersive patient simulators and their impact on learning success: a thematic review. J Med Internet Res 2015; 17: e91. 11. Andrade AD, Bagri A, Zaw K et al: Avatar-mediated training in the delivery of bad news in a virtual world. J Palliat Med 2010; 13: 1415.
FLA 5.4.0 DTD URPR177_proof 16 September 2016 4:14 pm EO: UP-15-94
658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708
8
709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759
Communication Skills Assessment Using Human Avatars
12. Rees C, Sheard C and Davies S: The development of a scale to measure medical students’ attitudes towards communication skills learning: the Communication Skills Attitude Scale (CSAS). Med Educ 2002; 36: 141. 13. Kava BR: Communication Skills AUA core curriculum. Available at https://www.auanet.org/university/core_topic.cfm?coreid¼172. Accessed October 14, 2015.
22. Stroud L, McIlroy J and Levinson W: Skills of internal medicine residents in disclosing medical errors: a study using standardized patients. Acad Med 2009; 84: 1803. 23. Ju M, Berman AT, Hwang WT et al: Assessing interpersonal and communication skills in radiation oncology residents: a pilot standardized patient program. Int J Radiat Oncol Biol Phys 2014; 88: 1129.
14. Detmer WM, Shiffman S, Wyatt JC et al: A continuous-speech interface to a decision support system: II. An evaluation using a Wizard-of-Oz experimental paradigm. J Am Med Inform Assoc 1995; 2: 46.
24. Eggly S, Afonso N, Rojas G et al: An assessment of residents’ competence in the delivery of bad news to patients. Acad Med 1997; 72: 397.
15. Quest TE, Ander DS and Ratcliff JJ: The validity and reliability of the affective competency score to evaluate death disclosure using standardized patients. J Palliat Med 2006; 9: 361.
25. Roberts CS, Cox CE, Reintgen DS et al: Influence of physician communication on newly diagnosed breast patients’ psychological adjustment and decision-making. Cancer 1994; 74: 336.
16. Witmer BG, Jerome CJ and Singer MJ: The factor structure of the presence questionnaire. Presence 2005; 14: 298. 17. Zhao S: Toward a taxonomy of copresence. Presence 2003; 12: 445.
26. Bredart A, Bouleuc C and Dolbeault S: Doctor-patient communication and satisfaction with care in oncology. Curr Opin Oncol 2005; 17: 351.
18. Kleinert R, Haiermann N, Wahba R et al: Design, realization, and first validation of an immersive web-based virtual world simulator for training clinical decisions in surgery. J Surg Educ 2015; 72: 1131.
27. Cousin G, Mast MS, Roter DL et al: Concordance between physician communication style and patient attitudes predicts patient satisfaction. Patient Educ Couns 2012; 87: 193.
19. Joyce BL, Steenbergh T and Scher E: Use of the Kalamazoo Essential Elements Communication Checklist (adapted) in an institutional interpersonal and communication skills curriculum. J Grad Med Educ 2010; 2: 165.
28. Chang JT, Hays RD, Shekelle PG et al: Patients’ global ratings of their health care are not associated with the technical quality of their care. Ann Intern Med 2006; 144: 665.
20. Krupat D, Frankel R, Stine T et al: Four Habits Coding Scheme: validation of an instrument to assess clinicians’ communication behavior. Patient Educ Couns 2006; 62: 38.
29. Alverson DC, Saiki SM, Kalishman S et al: Medical students learn over distance using virtual reality simulation. Sim Healthcare 2008; 3: 10.
21. Miller SJ, Hope T and Talbot DC: The development of a structured rating schedule (the BAS) to assess skills in breaking bad news. Br J Cancer 1999; 80: 792.
30. Joyce BL, Scher E, Steenbergh T et al: Development of an institutional resident curriculum in communication skills. J Grad Med Educ 2011; 3: 524.
Editorial Commentary
The current work is intriguing but there remain significant reservations about virtual world technology in assessing examinations of real human patients. What is the avatar exam measuring compared to the standard clinical exam? Is it the technical aspects of the questions, perhaps forgoing those skills required in the real world indirectness of most human communications?1 What is lost? Traditional measures of clinical competencies have found that face-to-face interactions are best reflective of professionalism.2 How does this compare? Does this assessment correlate with skill acquisition or degradations in clinical competency? Can learners best the test? There is an up side. While in its infancy, once honed, there is great promise in that it could lead to more of us to provide useful feedback to struggling learners since there are considerable current gaps in our evaluations.3 I look forward to more work in this area and I am pleased that urologists are helping lead the way.
Deborah Lightner Department of Urology Mayo Clinic Rochester, Minnesota
References 1. Suchman AL, Markakis K, Beckman HB et al: A model of empathic communications in the medical interview. JAMA 1997; 277: 678. 2. Hemmer PA, Hawkins R, Jackson JL et al: Assessing how well three evaluation methods detect deficiencies in medical students’ professionalism in two settings of an internal medicine clerkship. Acad Med 2000; 75: 167. 3. Yao DC and Wright SM: National survey of internal medicine residency program directors regarding problem residents. JAMA 2000; 284: 1099.
DIS 5.4.0 DTD URPR177_proof 16 September 2016 4:14 pm EO: UP-15-94
760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810
Communication Skills Assessment Using Human Avatars 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829 830 831 832 833 834 835 836 837 838 839 840 841 842 843 844 845 846 847 848 849 850 851 852 853 854 855 856 857 858 859 860 861 862 863 864 865 866 867 868 869 870 871 872
9
Reply by Authors
The goals of this study were strictly to determine the feasibility, acceptability and applicability of the virtual world OSCE. The ultimate goal is to create a virtual world platform that is portable and inexpensive, and replicates a traditional face-to-face OSCE. The next steps are to improve the nonverbal cues exhibited by the avatars, compare the virtual world OSCE with a traditional real world OSCE and quantify how feedback can improve house staff communications and clinical competency. Similar to the skill sets needed to perform surgery, communication skills are not innate and cannot be learned only from a lecture. They require a systematic approach consisting of deliberate practice based learning, close observation, feedback, graduated responsibility and mentorship.1 Physicians are often left to learn these skills on their own, leaving them with a limited repertoire. Simulation based training has been adopted by many high hazard institutions such as aviation, nuclear power plants and the military.2 Medicine has lagged behind in these
efforts but it is slowly adapting technology to create platforms in which students can face the consequences of different decisions without putting real patients at risk. Previous efforts have focused on developing technical skills. Our virtual world simulation is intended to ultimately teach physicians how to provide respectful, unbiased, empathetic and patient centered care that may be more receptive to the needs of the individual patient. Much work must be done to further these initiatives using this platform.
References 1. Levinson W, Lesser C and Epstein RM: Developing physician communication skills for patient-centered care. Health Affairs 2010; 29: 1310. 2. Ziv A, Wope PR, Small SD et al: Simulation-based medical education: an ethical imperative. Acad Med 2003; 78: 783.
DIS 5.4.0 DTD URPR177_proof 16 September 2016 4:14 pm EO: UP-15-94
873 874 875 876 877 878 879 880 881 882 883 884 885 886 887 888 889 890 891 892 893 894 895 896 897 898 899 900 901 902 903 904 905 906 907 908 909 910 911 912 913 914 915 916 917 918 919 920 921 922 923 924 925 926 927 928 929 930 931 932 933 934
urologypracticejournal.com
Communication Skills Assessment Using Human Avatars: Piloting a Virtual World Objective Structured Clinical Examination Bruce R. Kava,* Allen D. Andrade, Robert Marcovich, Thaer Idress and Jorge G. Ruiz From the Department of Urology, University of Miami Miller School of Medicine (BRK, RM) and Laboratory of E-learning and Multimedia Research, Geriatric Research Education and Clinical Centers, Bruce W. Carter Veterans Affairs Medical Center (TI, JGR), Miami, Florida, and Geriatric Research Education and Clinical Centers, James J. Peters Veterans Affairs Medical Center, Bronx and Brookdale Department of Geriatrics and Palliative Medicine, Icahn School of Medicine at Mount Sinai, New York, New York (ADA)
Abstract
Abbreviations and Acronyms
Introduction: Proficiency in communication skills is a core competency of residency training. We evaluated the feasibility, acceptability and applicability of a virtual world objective structured clinical examination that enables practice based learning and assessment of resident communication skills. Methods: A virtual clinical encounter situated in 2 practice settings was developed that uses a human avatar physician and a standardized patient. Following an online tutorial house staff participated in 4 communication tasks, including shared decision making, delivering bad news, obtaining informed consent and disclosing a medical error. Validated instruments and semistructured interviews were done to assess house staff acceptability and applicability of the platform. Three faculty members used ACS (Affective Competency Scale) and communication specific assessment instruments to evaluate house staff performance. Results: A total of 12 urology house staff completed the simulation. Direct costs were approximately $1,000. The virtual world was easy to use and immersive. Applicability directly correlated with presence (Pearson r ¼ 0.67, p ¼ 0.01) and co-presence (Pearson r ¼ 0.8, p ¼ 0.002). House staff identified problems with 1) limited nonverbal cues, 2) too much information presented and 3) a lack of immediate feedback. The ICC (intraclass correlation) of faculty assessments was high for ACS at 0.53 (95% CI 0.36e0.69) for single measures, 0.77 (95% CI 0.63e0.86) for average measures and less for other assessment instruments.
4HCS = 4 Habits Coding Scheme BAS = Bad News Assessment Scale CSAS = Communication Skills Attitude Assessment Scale KEECC-A = Kalamazoo Essential Elements Communication Checklist-A OSCE = Objective Structured Clinical Encounter PSA = prostate specific antigen
Conclusions: A virtual world objective structured clinical examination is a feasible, acceptable and applicable method of communication skills assessment. Improving nonverbal cues, focusing on individual skill sets and providing immediate feedback are measures to be adopted in future iterations of this platform. Key Words: urology, computer simulation, clinical competence, social skills, physicians Submitted for publication October 14, 2015. No direct or indirect commercial incentive associated with publishing this article. The corresponding author certifies that, when applicable, a statement(s) has been included in the manuscript documenting institutional review board, ethics committee or ethical review board study approval; principles of Helsinki Declaration were followed in lieu of formal ethics committee approval; institutional animal care and use committee approval; all human subjects provided 2352-0779/17/41-1/0 UROLOGY PRACTICE Ó 2016 by AMERICAN UROLOGICAL ASSOCIATION EDUCATION
AND
written informed consent with guarantees of confidentiality; IRB approved protocol number; animal approved project number. Supported by a Society of Urologic Chairpersons and Program Directors grant. * Correspondence: Department of Urology, University of Miami Miller School of Medicine, 1600 Northwest 10th Ave., No. 1140, Miami, Florida 33101 (telephone: 305-243-4936; FAX: 305-243-4939; e-mail address: [email protected]).
RESEARCH, INC.
http://dx.doi.org/10.1016/j.urpr.2016.01.006 Vol. 4, 1-9, January 2017 Published by Elsevier
FLA 5.4.0 DTD URPR177_proof 16 September 2016 4:14 pm EO: UP-15-94
49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96
2
97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147
Communication Skills Assessment Using Human Avatars
Communications skills are designated as a core competency for residency training by ACGME (Accreditation Council for Graduate Medical Education). These skills are extremely difficult to teach and evaluate.1 Moreover, increasing time constraints and an expanding number of technical skills have drastically reduced the amount of time that can be committed toward fostering these humanistic aspects of medical care.2e5 As a result, skills such as delivering bad news or medical error disclosure are often left up to urology residents to learn through trial and error.2,5 The need for greater accountability to ensure patient safety requires urology program directors to find innovative ways for residents to improve these difficult communications skills. OSCEs provide opportunities for practice based learning and assessment of communication skills in a safe and reproducible environment. This method has proved to be a reliable and valid assessment tool6,7 but is expensive, requiring preparation, support staff, travel, equipment and location availability.8 Virtual worlds are computer generated platforms in which users interact via graphical character representatives called avatars. Individuals represented by specific avatars can communicate with each other via VoIP (Voice over Internet Protocol) communication. This portable, reusable, distance learning computer simulation can deploy multiple different avatar personas set in multiple practice settings. It allows for artificial advancement of time or disease progression in ways that are not possible in real life. Preliminary work incorporating immersive virtual world technology has demonstrated a positive impact on learning.9e11 Our objective was to evaluate the feasibility, acceptability and applicability of a virtual world OSCE in evaluating urology house staff proficiency with difficult communication skills. Feasibility was related to the technical challenge of creating an interactive platform in several practice settings and its attendant costs. For acceptability we characterized the participant sense of immersion in the virtual world using 2 parameters from the gaming and media industry, that is presence and co-presence. There is a paucity of educational data describing the knowledge, attitudes and self-perceived proficiency of urology house staff with various communication skills. Establishing these metrics and determining how applicable the house staff believed the platform to be were additional criteria that were critical to the aims of the project.
Methods
The study was institutional review board approved, participation was voluntary and all urology house staff at our
institution were eligible to participate. It was performed in 2 148 offices in different buildings of the medical campus, each 149 equipped with desktop computers capable of VoIP 150 communication. The virtual world was developed using 151 Second Life (Linden LabÒ) and it consisted of 2 virtual 152 world settings (see figure). After obtaining informed consent ½F1153 participants completed a demographic questionnaire and 154 CSAS.12 They watched a 40-minute communication skills 155 tutorial online developed from the AUA (American Uro- 156 157 logical Association) core curriculum.13 Four scripted encounters were developed (see Appendix), 158 each highlighting a specific communication skill. The “Wizard 159 of Oz” technique was deployed.14 In this technique an unseen 160 associate (one of the faculty members) played the role of a 161 standardized patient, processing participant responses in real 162 time and responding quickly enough to support an acceptable 163 interactive encounter. The de-identified audio recordings of 164 each encounter were archived on MPEG files and later 165 166 reviewed by the study team faculty. After each encounter participants and the standardized 167 patient completed ACS.15 Following the final encounter 168 house staff completed a post-encounter CSAS, and the 169 Presence, Co-presence and Applicability questionnaires.16e18 170 Participants were debriefed with a semistructured interview 171 172 performed by a study team member. The faculty consisted of 2 urologists and 1 geriatrics- 173 174 palliative care physician who concomitantly served as the avatar standardized patient. They independently evaluated 175 176 the archived audio files for each virtual OSCE using several 177 validated instruments. 178 179 Assessment Instruments 180 181 The Presence Questionnaire of Witmer et al evaluates the 182 perception of presence, defined as the subjective experience 183 of being in 1 place or environment even when one is 184 physically situated in another place.16 A 7-point Likert scale 185 was used with higher scores indicating a greater sense of 186 presence. 187 Co-presence is the subjective sensation of interacting 188 with another person. A 10-point Likert scale was used17 189 with higher scores indicating a greater sense of realism. 190 Applicability of the virtual world simulation was evalu191 ated using several validated criteria.18 A 10-point Likert scale 192 was used with higher scores indicating greater applicability. 193 ACS15 was the primary metric used to measure proficiency 194 for each communication skill. Using a 5-point Likert scale 6 195 global elements of the physician-patient interaction were 196 assessed. Scores above 3 for each component (total score 197 greater than 18) are generally associated with competency.15 198
FLA 5.4.0 DTD URPR177_proof 16 September 2016 4:14 pm EO: UP-15-94
Communication Skills Assessment Using Human Avatars
199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249
3
Figure. Desktop simulation was performed in cordoned area of communications laboratory. House staff member is seated in front of standard 23-inch computer monitor, interacting via microphone with avatar patient in office (a) and in hospital (b). Faculty member serving as designated standardized patient is seated in another locale and controls avatar patient movements and dialogue via “Wizard of Oz” technique (c).14 His screen depicts avatar physician and he follows script developed for particular encounter.
ACS is unique in that when completed by the house staff, it provides a valid measure of self-efficacy.11,15 Specific communication skills were also evaluated by items from several instruments. 1) KEECC-A19 was used to assess shared decision making and PSA screening. 2) We used 4HCS20 to assess providing informed consent for total prostatectomy. 3) BAS21 was used to evaluate how the residents delivered a diagnosis of prostate cancer. 4) The Medical Error Disclosure Rating Scale22 was used to assess proficiency in medical error disclosure. Data were entered directly into SPSSÒ. Nine positive items (ie “learning about communication skills will help me be a better physician”) and 7 negative items (ie “I can’t see the point in learning communication skills”) from CSAS were compared before and after the simulation using paired sample t-tests. Descriptive measures of house staff proficiency are provided as the mean of the 3 independent faculty scores. Based on the 5-point Likert scores individual items from KEECC-A, 4HCS and BAS were dichotomized into proficiency scores, including highdaverage faculty score greater than 3 or lowdaverage faculty score 3 or less.23,24 Medical error disclosure items were also dichotomized with an average score 5 or less for each global parameter considered low profiicency.22 Interrater reliability among the 3 faculty evaluations was assessed using a 2-way mixed effects model. The ICC is reported along with the 95% CI. Applicability was correlated with presence and co-presence scales using Pearson r.
Semistructured interviews were transcribed and reviewed by a study team member (BRK). Several themes emerged and were analyzed by open, axial and selective coding. Results Demographics and Attitudes
Eight junior residents (preliminary, and urology 1 and 2) and 4 senior residents (urology 3 and 4) participated in the study. All except 1 participant were male. Table 1 summarizes house staff experiences and attitudes toward difficult communication skills. Technical Feasibility
All participants completed the tutorial, OSCEs and the debriefing interview. No technical problems were encountered. Costs of development included leasing the virtual world space for 3 months ($50), virtual materials ($200) and time needed to develop the virtual world during 2 working days by a single experienced developer ($750). Presence, Co-Presence and House Staff Applicability
The virtual world experience was perceived as easy to use, realistic and immersive (table 2). Applicability scores directly correlated with presence (Pearson r ¼ 0.67, p ¼ 0.01) and co-presence (Pearson r ¼ 0.8, p ¼ 0.002).
FLA 5.4.0 DTD URPR177_proof 16 September 2016 4:14 pm EO: UP-15-94
250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 ½T1 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 ½T2 299 300
4
301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351
Communication Skills Assessment Using Human Avatars
Table 1. Select items instruments
from
Presence,
Co-presence
and
Applicability
Table 2. Select items instruments
from
Presence,
Co-presence
No. Participants (%) Previous experience with learning communication skills Delivering bad news: Medical school Postgraduate training None Providing informed consent: Medical school Postgraduate training None Medical error disclosure: Medical school Postgraduate training None
5 (42) 1 (8) 6 (50) 5 (42) 6 (50) 1 (8) 4 (33) 3 (25) 5 (42)
Previous experience with specialty specific communication skills Discussing prostate Ca screening: Medical school Postgraduate training None Disclosing prostate Ca diagnosis: Medical school Post-graduate training None Discussing prostate Ca treatment options: Yes No
2 (17) 5 (42) 5 (42) 2 (17) 5 (42) 5 (42) 6 (50) 6 (50)
Attitudes concerning communication skills Comfort level in delivering bad news to pt: Extremely or very comfortable Moderately comfortable Slightly comfortable Self-perceived proficiency in difficult communication skills: Very good Moderate Fair Effective communication skills cannot be taught: Disagree Agree Undecided Developing effective communication skills is as important as medical knowledge: Agree Disagree Undecided
9 (75) 2 (17) 1 (8)
Presence (scale 1e7): 1. Able to control events 2. Responsive to environment 3. Natural interaction with environment 4. How completely were your senses engaged 5. How much visual aspects involved you 6. How much auditory aspects involved you 7. How natural were movements 8. Your sense of objects/virtual subjects moving 9. How consistent was virtual world with real life 10. How quickly did you adjust to platform 11. How proficient at end 12. How involved were you in scenario Co-presence (scale 1e10): 1. Like face-to-face meeting 2. How real did interaction with pt seem 3. You can really get to know someone with this system Applicability (scale 1e10): 1. Interesting 2. Entertaining 3. Eager to use again 4. How helpful was learning experience 5. How much did it help you learn 6. Friendly to use 7. Difficult to complete 8. Effort required to learn material 9. Virtual world OSCE helpful 10. Recommend developing platform
and
Applicability
Mean SD
Median
4.7 4.3 4.3 3.9 5.8
5 5 4.5 4.5 6.5 e 4 e 6 4 5 6
4.1 4.8 4.25 4.6 5.75
(1.7) (1.3) (0.9) (1.6) (1.7) e (1.4) e (2) (1.9) (1.4) (1.1)
5.6 (2.8) 6.3 (2.4) 6.3 (2.8)
5.5 7 7.5
7.6 7.7 6.4 7.5 6.8 8.1 3.4 5 6.5 8.1
8 8 6.5 7.5 7 8.5 2.5 5 7 8
(1.8) (1.9) (1.6) (1.5) (1.7) (1.6) (2.3) (2.2) (2.6) (1.8)
Competency Assessment 3 (25) 5 (42) 4 (33) 8 (67) 3 (25) 1 (8)
8 (67) 2 (17) 2 (17)
Prestimulation and Post-Simulation Communication Skills Attitudes
No differences were noted between CSAS positive attitude items before vs after the virtual world simulation (presimulation and post-simulation mean SD score 32.8 6.4 and 34.2 7.6, respectively, p ¼ 0.317). There was a trend of less negative attitudes toward communication skills following the simulation but this did not reach statistical significance (mean presimulation and post-simulation negative attitude CSAS score 14.3 2.5 and 12.83 3.1, respectively, p ¼ 0.079)
Table 3 shows mean faculty and house staff ACS scores. Most of the house staff received average or just above average communication proficiency scores from the faculty. Residents were more critical of their own communication skills competency with more than 50% assessing themselves with low proficiency scores. The ICC of the ACS was 0.53 (95% CI 0.36e0.69) for single measures and 0.77 (95% CI 0.63e0 86) for average measures.
Specific Communications Skills Assessments
Table 4 shows faculty scoring from the communication specific instruments. There was wide variability in perceived house staff proficiency in the various skill sets. Particularly problematic were 1) understanding the patient perspective and bidirectional sharing of information for PSA screening, 2) breaking the bad news of a prostate cancer diagnosis, 3) gaining the patient perspective and demonstrating empathy when performing informed consent, and 4) being empathetic, apologizing and indicating how to prevent future medical errors at the time of an error disclosure. The ICC
FLA 5.4.0 DTD URPR177_proof 16 September 2016 4:14 pm EO: UP-15-94
352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 ½T3 382 383 384 385 386 387 388 389 390 391 392 393 ½T4 394 395 396 397 398 399 400 401 402
Communication Skills Assessment Using Human Avatars
403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453
5
Table 3. Encounter ACS scores based on faculty evaluations and house staff self-assessment scores, and number of residents with proficiency score less than 18 Simulation Scores*
Mean SD ACS: Confident Comfortable Empathetic/sensitive Respectful/professional Informative Comforting No. residents total score less than 18 (%) Mean SD self-assessment: Confident Comfortable Empathetic/sensitive Respectful/professional Informative Comforting No. residents total score less than 18 (%)
1. Shared Decision Making (PSA Screening)
2. Delivering Bad News (Prostate Ca Diagnosis)
3. Informed Consent (Radical Prostatectomy)
4. Disclosing Medical Error
20.8 4.0 3.7 0.8 3.7 0.9 3.2 0.8 3.7 0.7 3.4 0.9 3.2 0.9 3 (25)
20.3 6.1 3.6 0.9 3.6 0.8 3.3 1.2 3.7 0.8 3.4 1.1 3.3 1.2 3 (25)
21.2 4.1 3.7 0.8 3.6 0.8 3.3 1.0 3.6 0.8 3.6 0.8 3.3 1.1 3 (25)
19.2 3.4 3.1 3.1 3.5 3.2 2.9 4
(4.8) (0.6) (0.9) (1.2) (0.8) (0.9) (1.2) (33)
19.4 2.7 3.3 0.5 3.3 0.5 3.3 0.7 3.1 0.7 3.1 0.7 3.3 0.5 5 (42)
19.4 2.7 3.1 0.5 3.3 0.6 3.3 0.9 3.4 0.7 3.1 0.5 3.3 0.6 7 (58)
19.3 3.3 3.0 0.9 3.2 0.7 3.2 0.7 3.6 0.7 3.2 0.6 3.3 0.5 7 (58)
17.8 2.9 2.8 3.1 3.3 3.1 2.6 8
(2.8) (0.3) (0.3) (0.7) (0.8) (0.7) (0.9) (67)
*Score scale 1dunsatisfactory, 3daverage and 5dsuperior.
was 0.73 for BAS (95% CI 0.54e0.93), 0.27 for KEECC-A (95% CI 0.18e0.39), 0.26 for medical error disclosure (95% CI 0.20e0.36) and 0.25 for the 4 Habits model (95% CI 0.15e0.37).
suggested that immediate feedback from a faculty member be provided after each simulation.
Discussion Debriefing Interviews
The majority of residents felt immersed in the virtual world, commenting that “I forgot about my surroundings,” “I felt like I was in a room with a patient,” and “It felt as if I was working in the clinic without a physical exam.” They were critical about the paucity of nonverbal communication cues exhibited by the avatar, which were limited to looking down or away from the computer screen. This confused some of the house staff, who indicated that they felt “awkward” and “could not gauge whether [the avatar patient] was understanding what they were being told.” One resident indicated that the movements seemed “artificial [but] if I looked away from him, I was fine.” The house staff believed that too much information was presented in the tutorial. Several proposed that the skill sets should be presented as a series of modules. “Feeling awkward,” “uncomfortable” and “lacking confidence” were terms used to describe how residents felt about disclosing a medical error. Some thought that there were too many complications and several indicated that this changed the manner in which they approached informed consent for radical prostatectomy. They unanimously
There is growing recognition that effective communication skills translate to higher patient satisfaction and better quality medical care.23e30 While most medical schools offer students generic communication skills classes, during postgraduate training they are expected to transition to a practice based learning environment. For many residents a considerable amount of time has elapsed since classroom training, which may adversely impact recollection or retention of the topics that were discussed. Whether this is the case or whether more than 50% of our house staff were never taught the essential components of delivering bad news or disclosing a medical error is worth further evaluation. Our data identify glaring educational deficiencies in several essential communications skills in the postgraduate setting. This adds to the growing body of data on other subspecialties,4,5,25,29,30 which supports that there is a critical need to develop a structured communication skills curriculum in postgraduate medical training that includes shared and specialty specific communication skill sets. To our knowledge this is the first study to integrate a virtual world platform to assess communication skills proficiency for urology house staff. Using this platform the house staff were provided with an iterative learning
FLA 5.4.0 DTD URPR177_proof 16 September 2016 4:14 pm EO: UP-15-94
454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504
6
505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555
Communication Skills Assessment Using Human Avatars
Table 4. Proficiency scores of various items on specific communication skills instruments
PSA screening (KEECC-A*): 1. Builds relationship 2. Opens discussion 3. Gathers information 4. Understands pt perspective 5. Shares information 6. Reaches agreement 7. Provides closure Delivering prostate Ca diagnosis (BAS*): 1. Setting scene 2. Breaking news 3. Eliciting concerns 4. Information giving 5. General considerations Total prostatectomy informed consent (4HCS*): 1. Investing in beginning 2. Eliciting pt perspective 3. Demonstrating empathy 4. Investing in end Medical error disclosure (Medical Error Disclosure Rating Scale global scores):y 1. Explanation of medical facts regarding error 2. Honesty and truthfulness 3. Empathy 4. Prevention of future errors 5. General communication skills
No. Low Proficiency Scores (%)
Mean SD
Median (range)
3.2 3.3 3.4 3.0 3.2 3.4 3.4
(0.9) (0.9) (1.1) (1.1) (1.1) (1.0) (1.3)
3.3 3.7 4 3.2 3.5 3.8 3
(2.0e4.6) (1.7e4.3) (2.0e4.3) (1.0e4.3) (1.3e4.3) (1.7e4.3) (1.3e4.3)
6 5 4 7 6 5 4
3.7 3.0 3.0 3.5 3.2
(0.6) (1.0) (0.9) (0.9) (0.9)
3.8 3.1 3.2 3.8 3.3
(2.6e4.3) (1.1e4.5) (1.4e4.1) (2.3e4.7) (1.9e4.5)
1 (8) 5 (42) 3 (25) 4 (33) 4 (33)
3.5 3.4 3.2 3.5
(0.5) (0.8) (0.7) (0.6)
3.6 3.6 3.3 3.4
(2.9e4.5) (2.0e4.4) (1.7e4.3) (2.5e4.4)
2 4 4 3
(17) (33) (33) (25)
7.7 7.0 6.1 3.7 6.2
(1.5) (2.0) (2.6) (1.2) (1.9)
8 (4.0e9.3) 7 (3.3e10.0) 6.4 (2.0e10.0) 3.3 (2.0e6.7) 6 (3.3e8.7)
2 4 6 11 7
(17) (33) (50) (92) (58)
(50) (42) (33) (58) (50) (42) (33)
*Five 5-point Likert scale. y Ten-point Likert scale.
environment in which they could practice essential communication skills, identify their mistakes and test solutions that they devised without adversely impacting a patient. By enabling them to learn from their mistakes the platform provides opportunities for improved understanding, confidence and self-efficacy.30 One of the criteria that we believed to be essential to the feasibility and success of the platform was its acceptability to the house staff. Similar to the feeling that one gets when watching a motion picture, virtual world experiences trigger a subjective sense of immersion in the virtual world that lends authenticity to the experience and induces a willful suspension of disbelief.11 Presence and co-presence are often used to gauge this sense of immersion when evaluating video games and other virtual world technology. As we report, the greater the degree of this immersion in the virtual world, the greater the perceived applicability of the simulation. This is a critical finding in that it provides a rationale for exploring new 3dimensional immersive technologies that may strengthen the realism associated with the virtual world experience. We described how the virtual world OSCE can be archived and subsequently used as an assessment instrument. Specific skills associated with the lowest overall assessment scores included 1) gaining the patient perspective, 2) breaking bad news, 3) bidirectional communications and 4) failing to
apologize or discuss strategies to prevent future errors when discussing medical error prevention. These data provide a starting point with which to focus our educational efforts. The task of evaluating competency in communication skills is difficult due to the inherent complexities of grading communications.13 The high ICC associated with ACS corroborates that with little training the faculty could negotiate the instrument and evaluate the house staff reliably and consistently. In general, the progressively lower ICC associated with the other communication instruments was not surprising, given that each requires hours of dedicated training and standardization. Future studies focusing on specific communications skills must ensure that the faculty has adequate training and expertise with the specific communications objectives and instruments used to assess them. The debriefing interviews indicate the need for several modifications to virtual world OSCEs. Enhancing the nonverbal emotional cues expressed by the avatars will require technical upgrades. Future simulations must on 1 communication skill at a time and provide immediate faculty feedback. Finally, plans to establish a multidisciplinary communications curriculum to develop uniform criteria for teaching and assessing communication skills are under way. There are several limitations of the study. The small sample size limits the generalizability of our findings. While
FLA 5.4.0 DTD URPR177_proof 16 September 2016 4:14 pm EO: UP-15-94
556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606
Communication Skills Assessment Using Human Avatars
607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657
we limited the goals of this study to determining the technical feasibility, acceptability and applicability of a virtual world, a parallel comparison group in a real world OSCE could provide additional data on the comparative effectiveness of this virtual world assessment modality. Future studies will also need to focus on how the platform can be applied to self-assessment and expert assessment, mentorship and coaching, and individual or group learning.17,28e30
Conclusion
A virtual world OSCE provides a novel platform for teaching and evaluating communication skills proficiency during residency training in urology. The platform was technically feasible, acceptable and applicable. Acceptability was strongly associated with the perceived applicability of the platform. Faculty assessment was consistent using the primary assessment instrument (ACS) but less so with more complex communications instruments. Technical refinements that improve nonverbal cues are needed, as is providing immediate feedback to the house staff and focusing on 1 communication skill at a time. Future iterations of the virtual world OSCE should be developed in tandem with a comprehensive multidisciplinary communication skills curriculum. This will promote better communications between physicians and patients that may have many attendant benefits for urology and all medical specialties. Appendix. Four distinct scripted encounters were presented to each resident during desktop simulation and each virtual world OSCE highlights specific communication skill Encounter 1: Shared decision making in determining the need for prostate cancer screening Introduction: This is an initial visit for a patient who was referred to you to discuss prostate cancer screening. Instructions: You have 10 minutes to discuss prostate cancer screening with this patient. The patient is a 50 year old male who is a supermarket manager. He was specifically referred to you by his primary care doctor for “prostate cancer screening.” He has mild lower urinary tract symptoms (LUTS). His only other medical problem is hypertension that is managed with hydrochlorothiazide. He has no other past medical history, has no allergies, and his physical exam is relatively normal. Encounter 2: Delivering bad news that a patient has prostate cancer. Introduction: Following your initial visit, Mr. Thomas was found to have an elevated PSA level, that on repeat was once again elevated at 5.2 (free PSA 12%). After discussing the options with him, a TRUS-prostate biopsy was performed revealing Gleason 6 prostate cancer in 2/12 cores (right apex and left base), both involving 3 mm and less than 10% of each core. He has a 40 gram prostate and no obvious abnormalities were appreciated on ultrasound. Instructions: You have 15 minutes to deliver the bad news to Mr. Thomas that he has prostate cancer and to discuss the treatment options with him. He has been calling the office daily since the biopsy was performed last week and is extremely anxious. He told your secretary that his father had prostate cancer and that he was really worried about having it himself.
7
Encounter 3: Obtaining informed consent for a radical prostatectomy. Introduction: Mr. Thomas is a 50 year old male with clinical T1c prostate cancer, who has a PSA 5.2ng/ml and who has two cores of Gleason 6/10 cancer on biopsy. The patient has decided that he is going to have a radical prostatectomy. Instructions: You have 10 minutes to discuss radical prostatectomy with the patient and to provide an informed consent for the surgery. Encounter 4: Disclosing a medical error: complications during surgery including bleeding, inadvertent rectal injury, in addition to the need for resection of one of the nerve bundles. Introduction: Mr. Thomas had his radical prostatectomy yesterday. You are making hospital rounds. Surgery proved a lot more complicated than you thought it would be. The prostate was much larger than you had anticipated. This made it difficult to see well and there was a lot more blood loss than you expected (approximately 900 ml were lost). The patient did not get transfused in the operating room, but needs to have one now because his blood pressure was very low. Only one nerve bundle was preserved because of the difficulty with surgery. You also miscalculated where the prostate was and made a small laceration in the anterior rectal wall that needed a colorectal surgeon to repair it. Although the repair seemed to be fine, you need to discuss the potential need for a colostomy if the repair breaks down. Also, since the surgery was so complicated and there was a rectal injury, the patient’s catheter will have to be left in for a couple of weeks, he will need to be on no oral intake for today, and he will likely be in the hospital for at least 3e5 days.
References 1. Swing S: The ACGME outcome project: retrospective and prospective. Med Teach 2007; 29: 648. 2. Sise M, Sise CB, Sack DI et al: Surgeons’ attitudes about communicating with patients and their families. Curr Surg 2006; 63: 213. 3. Dosanijh S, Barnes J and Bhandari M: Barriers to breaking bad news among medical and surgical residents. Med Educ 2001; 35: 197. 4. Hebert H, Butera JN, Castillo J et al: Are we training our fellows adequately in delivering bad news to patients? A survey of hematology/oncology program directors. J Palliat Med 2009; 12: 1119. 5. Orgel E, McCarter R and Jacobs S: A failing medical educational model: a self- assessment by physicians at all levels of training of ability and comfort to deliver bad news. J Palliat Med 2010; 13: 677. 6. Sharp PC, Pearce KA, Konen JC et al: Using standardized patient instructors to teach health promotion interviewing skills. Fam Med 1996; 28: 103. 7. Brannick MT, Erol-Korkmaz HT and Prewett M: A systematic review of the reliability of objective structured clinical examination scores. Med Educ 2011; 45: 1181. 8. Patricio MF, Juliao M, Fareleira F et al: Is the OSCE a feasible tool to assess competencies in undergraduate medical education? Med Teach 2013; 35: 503. 9. Cook DA, Erwin PJ and Triola MM: Computerized virtual patients in health professions education: a systematic review and meta-analysis. Acad Med 2010; 85: 1589. 10. Kleinert R, Wahba R, Chang DH et al: 3D immersive patient simulators and their impact on learning success: a thematic review. J Med Internet Res 2015; 17: e91. 11. Andrade AD, Bagri A, Zaw K et al: Avatar-mediated training in the delivery of bad news in a virtual world. J Palliat Med 2010; 13: 1415.
FLA 5.4.0 DTD URPR177_proof 16 September 2016 4:14 pm EO: UP-15-94
658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708
8
709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759
Communication Skills Assessment Using Human Avatars
12. Rees C, Sheard C and Davies S: The development of a scale to measure medical students’ attitudes towards communication skills learning: the Communication Skills Attitude Scale (CSAS). Med Educ 2002; 36: 141. 13. Kava BR: Communication Skills AUA core curriculum. Available at https://www.auanet.org/university/core_topic.cfm?coreid¼172. Accessed October 14, 2015.
22. Stroud L, McIlroy J and Levinson W: Skills of internal medicine residents in disclosing medical errors: a study using standardized patients. Acad Med 2009; 84: 1803. 23. Ju M, Berman AT, Hwang WT et al: Assessing interpersonal and communication skills in radiation oncology residents: a pilot standardized patient program. Int J Radiat Oncol Biol Phys 2014; 88: 1129.
14. Detmer WM, Shiffman S, Wyatt JC et al: A continuous-speech interface to a decision support system: II. An evaluation using a Wizard-of-Oz experimental paradigm. J Am Med Inform Assoc 1995; 2: 46.
24. Eggly S, Afonso N, Rojas G et al: An assessment of residents’ competence in the delivery of bad news to patients. Acad Med 1997; 72: 397.
15. Quest TE, Ander DS and Ratcliff JJ: The validity and reliability of the affective competency score to evaluate death disclosure using standardized patients. J Palliat Med 2006; 9: 361.
25. Roberts CS, Cox CE, Reintgen DS et al: Influence of physician communication on newly diagnosed breast patients’ psychological adjustment and decision-making. Cancer 1994; 74: 336.
16. Witmer BG, Jerome CJ and Singer MJ: The factor structure of the presence questionnaire. Presence 2005; 14: 298. 17. Zhao S: Toward a taxonomy of copresence. Presence 2003; 12: 445.
26. Bredart A, Bouleuc C and Dolbeault S: Doctor-patient communication and satisfaction with care in oncology. Curr Opin Oncol 2005; 17: 351.
18. Kleinert R, Haiermann N, Wahba R et al: Design, realization, and first validation of an immersive web-based virtual world simulator for training clinical decisions in surgery. J Surg Educ 2015; 72: 1131.
27. Cousin G, Mast MS, Roter DL et al: Concordance between physician communication style and patient attitudes predicts patient satisfaction. Patient Educ Couns 2012; 87: 193.
19. Joyce BL, Steenbergh T and Scher E: Use of the Kalamazoo Essential Elements Communication Checklist (adapted) in an institutional interpersonal and communication skills curriculum. J Grad Med Educ 2010; 2: 165.
28. Chang JT, Hays RD, Shekelle PG et al: Patients’ global ratings of their health care are not associated with the technical quality of their care. Ann Intern Med 2006; 144: 665.
20. Krupat D, Frankel R, Stine T et al: Four Habits Coding Scheme: validation of an instrument to assess clinicians’ communication behavior. Patient Educ Couns 2006; 62: 38.
29. Alverson DC, Saiki SM, Kalishman S et al: Medical students learn over distance using virtual reality simulation. Sim Healthcare 2008; 3: 10.
21. Miller SJ, Hope T and Talbot DC: The development of a structured rating schedule (the BAS) to assess skills in breaking bad news. Br J Cancer 1999; 80: 792.
30. Joyce BL, Scher E, Steenbergh T et al: Development of an institutional resident curriculum in communication skills. J Grad Med Educ 2011; 3: 524.
Editorial Commentary
The current work is intriguing but there remain significant reservations about virtual world technology in assessing examinations of real human patients. What is the avatar exam measuring compared to the standard clinical exam? Is it the technical aspects of the questions, perhaps forgoing those skills required in the real world indirectness of most human communications?1 What is lost? Traditional measures of clinical competencies have found that face-to-face interactions are best reflective of professionalism.2 How does this compare? Does this assessment correlate with skill acquisition or degradations in clinical competency? Can learners best the test? There is an up side. While in its infancy, once honed, there is great promise in that it could lead to more of us to provide useful feedback to struggling learners since there are considerable current gaps in our evaluations.3 I look forward to more work in this area and I am pleased that urologists are helping lead the way.
Deborah Lightner Department of Urology Mayo Clinic Rochester, Minnesota
References 1. Suchman AL, Markakis K, Beckman HB et al: A model of empathic communications in the medical interview. JAMA 1997; 277: 678. 2. Hemmer PA, Hawkins R, Jackson JL et al: Assessing how well three evaluation methods detect deficiencies in medical students’ professionalism in two settings of an internal medicine clerkship. Acad Med 2000; 75: 167. 3. Yao DC and Wright SM: National survey of internal medicine residency program directors regarding problem residents. JAMA 2000; 284: 1099.
DIS 5.4.0 DTD URPR177_proof 16 September 2016 4:14 pm EO: UP-15-94
760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810
Communication Skills Assessment Using Human Avatars 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829 830 831 832 833 834 835 836 837 838 839 840 841 842 843 844 845 846 847 848 849 850 851 852 853 854 855 856 857 858 859 860 861 862 863 864 865 866 867 868 869 870 871 872
9
Reply by Authors
The goals of this study were strictly to determine the feasibility, acceptability and applicability of the virtual world OSCE. The ultimate goal is to create a virtual world platform that is portable and inexpensive, and replicates a traditional face-to-face OSCE. The next steps are to improve the nonverbal cues exhibited by the avatars, compare the virtual world OSCE with a traditional real world OSCE and quantify how feedback can improve house staff communications and clinical competency. Similar to the skill sets needed to perform surgery, communication skills are not innate and cannot be learned only from a lecture. They require a systematic approach consisting of deliberate practice based learning, close observation, feedback, graduated responsibility and mentorship.1 Physicians are often left to learn these skills on their own, leaving them with a limited repertoire. Simulation based training has been adopted by many high hazard institutions such as aviation, nuclear power plants and the military.2 Medicine has lagged behind in these
efforts but it is slowly adapting technology to create platforms in which students can face the consequences of different decisions without putting real patients at risk. Previous efforts have focused on developing technical skills. Our virtual world simulation is intended to ultimately teach physicians how to provide respectful, unbiased, empathetic and patient centered care that may be more receptive to the needs of the individual patient. Much work must be done to further these initiatives using this platform.
References 1. Levinson W, Lesser C and Epstein RM: Developing physician communication skills for patient-centered care. Health Affairs 2010; 29: 1310. 2. Ziv A, Wope PR, Small SD et al: Simulation-based medical education: an ethical imperative. Acad Med 2003; 78: 783.
DIS 5.4.0 DTD URPR177_proof 16 September 2016 4:14 pm EO: UP-15-94
873 874 875 876 877 878 879 880 881 882 883 884 885 886 887 888 889 890 891 892 893 894 895 896 897 898 899 900 901 902 903 904 905 906 907 908 909 910 911 912 913 914 915 916 917 918 919 920 921 922 923 924 925 926 927 928 929 930 931 932 933 934