- Email: [email protected]
The use of a cognitive task analysis–based multimedia program to teach surgical decision making in flexor tendon repair
The American Journal of Surgery 195 (2008) 11–15
Association for Surgical Education
The use of a cognitive task analysis– based multimedia program to teach surgical decision making in flexor tendon repair Kali R. Luker, B.A., Maura E. Sullivan, Ph.D., M.S.N., Sarah E. Peyre, M.S.Ed., Randy Sherman, M.D., Tiffany Grunwald, M.D., M.S.Ed.* Department of Surgery, Division of Plastic & Reconstructive Surgery, Keck School of Medicine, University of Southern California, Santa Monica, CA, USA Manuscript received April 23, 2007; revised manuscript August 10, 2007 Presented at the Annual Meeting of the Association for Surgical Education, Washington, DC, April 11–14, 2007
Abstract Background: The aim of this study was to compare the surgical knowledge of residents before and after receiving a cognitive task analysis– based multimedia teaching module. Methods: Ten plastic surgery residents were evaluated performing flexor tendon repair on 3 occasions. Traditional learning occurred between the first and second trial and served as the control. A teaching module was introduced as an intervention between the second and third trial using cognitive task analysis to illustrate decision-making skills. Results: All residents showed improvement in their decision-making ability when performing flexor tendon repair after each surgical procedure. The group improved through traditional methods as well as exposure to our talk-aloud protocol (P ⬎ .01). After being trained using the cognitive task analysis curriculum the group displayed a statistically significant knowledge expansion (P ⬍ .01). Conclusions: Residents receiving cognitive task analysis– based multimedia surgical curriculum instruction achieved greater command of problem solving and are better equipped to make correct decisions in flexor tendon repair. © 2008 Excerpta Medica Inc. All rights reserved. Keywords: Cognitive task analysis; Knowledge elicitation; Procedural knowledge; Automated knowledge; Procedural skills; Technical skills
Two clear divisions become apparent when examining traditional surgical curriculum. One is the instruction of physiology, anatomy, and disease, and the second is the development of technical skills in an apprenticeship format. The first tier of learning traditionally has focused on the theory behind surgical procedures with minimal ability to illustrate the essential 3-dimensional visual-spatial relationships during these procedures. This classroom teaching has occurred in a logical, progressive manner whereas the 3-dimensional spatial relationships and technical skills have been taught in the operating room, dependent on the random order with which patients present themselves. There has been little coordination or overlap between these 2 tiers of teaching. Although these tiers of surgical education are not mutually exclusive and are obviously a simplification of the curricu-
* Corresponding author. Tel.: ⫹1-310 315-0215; fax: ⫹1-310 315-9392. E-mail address: [email protected]
lum, they are noticeable subdivisions that are difficult to synchronize. In contrast to the traditional surgical curriculum, current educational theory tells us that adult learners learn best when information is placed into a contextual format and learners need to be taught decision-making strategies during this time [1]. Before the computer age, there was no established method for accomplishing this other than to create paper scenarios or to teach using actual patients. This is no longer true in today’s world of advancing technologies. Simulation modules are being used to create computer-generated patients, providing a controlled subject for novice surgeons to learn from. By using these modules, patients and diseases can be generated in a thoughtful, logical manner, corresponding to existing curricula of surgical training programs. Instruction using many types of media has provided a sophisticated medium whereby we can incorporate traditional text and illustrations into a teaching program with interactive and 3-dimensional capabilities. According to the
0002-9610/08/$ – see front matter © 2008 Excerpta Medica Inc. All rights reserved. doi:10.1016/j.amjsurg.2007.08.052
12
K.R. Luker et al. / The American Journal of Surgery 195 (2008) 11–15
cognitive theory of multimedia learning, the act of building connections between verbal and pictorial mental models is an important step in conceptual understanding. A multimedia presentation with videotape, illustrations, animations, and text guides the learner to build a verbal mental model of the material, a pictorial mental model of the material, and connections between the two. Students who receive wellconstructed multimedia information perform better on tests designed to measure understanding and problem-solving ability, than do students who receive information presented only in words [2]. To build these multimedia tools, a method for effectively capturing an expert’s knowledge must be used. One specific method for obtaining sophisticated performance expertise in which many covert decisions are linked with complex overt actions is cognitive task analysis [3]. By applying cognitive task analysis to surgical procedures, we can focus instruction on the decision-making steps and component parts of a procedure that differentiate an expert from a novice surgeon. Cognitive task analysis is one of the more powerful design tools that has been used in the military and a number of highly technical fields to hasten the acquisition of expertise [4] and has been shown to improve learning of basic surgical skills [5,6]. By centering the teaching around decision-making points and available options in a specific simulation, cognitive task analysis prepares novices to transfer their knowledge when problem solving in a variety of related cases as opposed to the traditional Halstedian method in which learning is patient specific. In addition, using cognitive task analysis allows trained analysts to delve into expert surgeons’ decision-making techniques outside of the pressures of the operating room. Even without the time constraints and demands of the operating room, it has been shown that as surgeons develop expertise, their knowledge becomes automated. Automated knowledge operates outside of conscious awareness, making it difficult for experts to describe the discrete steps of the procedure [1,4,7]. For this reason, it often is difficult for experts to articulate the decision points they make during a procedure because they are not aware of many of their own decisions and mental analysis of tasks [4]. Incorporating cognitive task analysis into the design of a multimedia learning tool will give us the opportunity to transfer knowledge effectively from expert to novice, accelerate the acquisition of expertise, and enrich our domain of instruction. In addition, cognitive task analysis will provide a means for assessing decision-making skills and surgical judgment. This study evaluated the ability of a cognitive task analysis– based multimedia teaching module to instruct residents on flexor tendon repair. Methods Subjects Ten residents from the Division of Plastic and Reconstructive Surgery at the University of Southern California were recruited for this study. Ethics approval was obtained from the Institutional Review Board at the University of Southern California Health Sciences Campus. Informed consent was obtained from all participants and they were assured that the results from the study were confidential. Residents were in-
cluded from every year of the plastic surgery program and had varying levels of exposure to flexor tendon repair. There were 4 residents in the first year of the program, 3 in their second year, 2 in their third year, and 1 hand fellow. Three residents were involved in a clinical rotation dedicated to hand surgery at the time of the study and the others had unrelated clinical duties at the time of testing. Throughout the study, the residents served as their own control. Materials Three surgeons with expertise in flexor tendon repair participated in a cognitive task analysis that examined their decision-making processes and the necessary technical skills required to effectively perform a repair. The cognitive task analysis analysts were a faculty member and a graduate student from the School of Education. The cognitive task analysis consisted of a series of interviews. During the first interview, the cognitive task analysis analysts interviewed the expert surgeons with the intent to extract all relevant information describing flexor tendon repair. This information was drafted according to cognitive task analysis principles in a comprehensive document. The document described in a step-by-step fashion technical skill and emphasized critical details, decision-making points, and important maneuvers necessary to avoid potential pitfalls. In a second interview the analysts discussed revisions of the documents with the experts. In a third interview the analysts and surgeons came together to discuss and resolve potential conflicting points. The final document was reviewed and approved by all members of the group. This document included the technique of flexor tendon repair as well as the indications, contraindications, complications, and pitfalls related to flexor tendon repair. The development of the multimedia curriculum was based on the assembled cognitive task analysis document and was organized to emphasize the decision points and important details that are critical for mastery and completion of the overreaching surgical task. Illustrations, animations, and text guides were associated with videotaped clips to allow the learner to build a verbal and pictorial mental model of the task being described. These demonstrations also facilitated building connections between the 2 models and allowed the learner to witness the successful execution of the task. Production occurred in collaboration with experts from the Institute for Multimedia Literacy. A talk-aloud protocol also was created as a tool for assessment of the trainees’ understanding of the procedure, the salient decision points, and the factual information necessary to begin to view a case as an expert. It was formatted in a verbal examination format. The trainees were questioned at each decision point about their options and the advantages and disadvantages of these options. It also included questions regarding the performance of certain surgical techniques essential for a functional flexor tendon repair. When a trainee correctly identified an option at a decision point or an advantage or disadvantage of an option, they received a point. They also received points for correctly describing the surgical techniques that are critical to the procedure. The talk-aloud protocol was adapted from the cognitive task analysis and emphasized information directly
K.R. Luker et al. / The American Journal of Surgery 195 (2008) 11–15
relevant to the specific procedure. It omitted information that was deemed basic surgical knowledge or outside the realm of the protocol. Evaluation Ten plastic and reconstructive surgery residents of varying levels of training were asked to perform flexor tendon repairs on a cadaveric hand 3 separate times as if it were a real patient. Before each procedure, the residents completed a 10-minute questionnaire covering prior exposure to flexor tendon repair, how they prepared for this case, how they usually prepare for cases, and their history of didactic teaching of flexor tendon repair. Each surgery was observed by a trained evaluator who completed an objective checklist referring to the essential steps and decision points of the procedure. The checklist was designed to satisfy cognitive task analysis principles and mirror the task-specific checklists as described by Martin and Resnick [8]. Digital videotape was made for review with the trainee using a talk-aloud protocol to determine awareness of decision points, available options, advantages and disadvantages, and the reasons for their decisions. Each flexor tendon repair was reviewed immediately after completion of the task and a knowledge assessment using the talk-aloud protocol was performed. During the talk-aloud protocol, the residents watched a recording of the flexor tendon repair that they had just performed to refresh their memory of what they were thinking at the time. At each decision point the videotape was stopped and the resident was asked several questions that tested their knowledge of the decision-making steps of flexor tendon repair at that point as determined through cognitive task analysis as well as alternatives and rationale for each decision. This format allowed the resident to talk through their thought process and justifications for the decisions they made. Fig. 1 diagrams the layout of the study. Throughout the study the residents served as their own controls. The control group was considered to be all 10 residents from sessions 1 to 2. Any changes in scores between sessions 1 and 2 during the talk-aloud review of their videotape was considered to be the result of traditional learning as well as knowledge of the questions that were asked of them. The multimedia teaching module derived from the cognitive task analysis was given to all 10 residents after session 2. Any change in knowledge between sessions 2 and 3 was considered to be the result of the multimedia instruction. The residents who participated in the training and performed the final flexor tendon repair were considered to be the interventional group. Because 1 resident did not participate in the multimedia training, the final interventional group included 9 of the 10 total residents.
13
Training Nine residents underwent instruction using the multimedia program after their second flexor tendon repair. The material was cognitive task analysis– based and included theory and technique of flexor tendon repair, anatomy, physiology, pathology, indications and contraindications for surgical repair, and expert tips. The computer program included videotaped clips of a live procedure as well as illustrations, animations, and text guides. It emphasized the component steps and decision-making points as described through cognitive task analysis. Analysis The evaluation data were entered in a Microsoft Excel database (Microsoft Corp, Redmond, WA) and analyzed statistically using a nonparametric Wilcoxon test. Comparisons were performed between the change in scores from sessions 1 to 2 and sessions 2 to 3. We used the comparison between sessions 1 and 2 as our control to measure improvement through traditional methods as well as exposure to our talk-aloud protocol. Any change between sessions 2 and 3 was assumed to be the result of learning from the multimedia teaching module. We used statistical analysis to compare the total change in scores and knowledge of the advantages and disadvantages of options at each decision point. The results also were analyzed by subgrouping the residents according to their year in the Plastic and Reconstructive Surgery Program (year 1 vs year 2 and higher), exposure to flexor tendon repair (whether they had performed ⬍10 flexor tendon repairs vs performed ⬎10), scores after the first flexor tendon repair (⬍60 vs ⬎60), and improvement between sessions 1 and 2 (⬍10 vs ⬎10). These groupings attempted to evaluate whether level, experience, baseline knowledge, or extent of traditional learning between sessions 1 and 2 influenced training by the multimedia teaching module. We deemed a P value of less than .01 as statistically significant for this study. Results The difference of all 10 residents’ scores between sessions 1 and 2 was considered to be the study control, evaluating traditional learning and contamination via exposure to the talk-aloud review. The change in scores of the 9 residents who participated in the multimedia instruction between sessions 2 and 3 represented the interventional segment of the study. Fig. 2 diagrams the results of the scoring analysis including the mean, range, and 95% confidence intervals for total scores and knowledge of advantages and disadvantages of decision points. Although there
Fig. 1. Diagram of study layout and individual groups. TAP ⫽ talk-aloud protocol video review of flexor tendon repair.
14
K.R. Luker et al. / The American Journal of Surgery 195 (2008) 11–15
they were not significant. By introducing the teaching module between sessions 2 and 3, the residents were able to effectively build on their current understanding of the procedure and why certain decisions are made. This was verified with a statistical significance of a P value of less than .01. Although the results of this study were encouraging, there were some limitations. First, there was a small number of subjects enrolled in the study. Second, even though cognitive task analysis has face validity, no formal validation studies have been performed. Finally, the subjects in the study who were on their hand service at the time may have been contaminated because they were exposed to the questions of the talk-aloud protocol, as was their attending physician. This could have affected the way the residents and attending interacted with each other. These factors may have caused the residents to ask more probing questions and the attending to teach with a greater focus on decision points and details than had been automated for them previously. Fig. 2. Results of the scoring analysis including mean, range, and 95% confidence intervals for total scores and knowledge of advantages and disadvantages of decision points.
was an improvement trend from session 1 to session 2, this improvement was not statistically significant for either improvement in total score or knowledge of the advantages and disadvantages of each decision point. The mean increase in scores was 13.2 for total knowledge and 8.4 for understanding of advantages and disadvantages. In contrast, a statistically significant improvement (P ⬍ .01) was seen as a result of the intervention of the teaching videotape between sessions 2 and 3 for both total knowledge and understanding of advantages and disadvantages of decision points. No statistically significant difference in their increase in knowledge throughout the study was seen between any of the subgroups mentioned earlier. The average score increase for total knowledge was 34.0 and 19.4 for understanding of advantages and disadvantages, respectively. The actual performance during the flexor tendon repairs was not evaluated because the cadaveric arms were too dissimilar to an actual patient. Many options and situations that normally would present themselves in flexor tendon lacerations and were discussed in the videotape such as physical examination techniques, partial tendon laceration repair, and tourniquetting the arm could not be performed because of the limitations of the mock event. The residents’ knowledge of these situations was evaluated during questioning using the talk-aloud protocol. Comments We have shown that although a significant amount of learning occurs via traditional methods, augmenting a resident’s training with computer-based learning modules is an effective way to contextualize a procedure and create a bridge from the theory to technical skills. The first segment of our study controlled for learning that would be done based on traditional learning and a knowledge of the talkaloud questions. Although the results showed a marked increasing trend for both total knowledge and understanding of advantages and disadvantages of each decision point,
Conclusions Multimedia learning tools are rapidly becoming an integral component of surgical education curriculum. These tools allow residents to gain a basic understanding of the procedural tasks and the decision points that are encountered along the way. Traditional Halstedian learning in the operating room has several drawbacks in that there is time pressure and only one particular case is observed at a time so not all potential variants can be considered. In addition, it has been shown that an expert’s knowledge becomes automated. Therefore, an expert is unable to identify and describe the discrete decisions made throughout a procedure. Cognitive task analysis is a method for extracting and capturing this highly complex performance expertise in which many covert decisions are linked with difficult overt actions. It also provides a method for integrating the automated skills and knowledge of expert surgeons into a form more easily understood by novices. In addition, supplementing traditional surgical education with computerized instruction and simulation allows a resident to become proficient and knowledgeable about a procedure in a structured and cohesive manner. It grants a novice access to the underlying thought processes of expert surgeons and understanding of the decision-making processes motivating an expert surgeon’s actions. Before entering the operating room, residents are able to learn the basic principles and techniques of a procedure and the essential decision points. This allows them to spend their time in the operating room honing this knowledge and asking more probing, in-depth questions of the attending surgeon instead of struggling with the fundamentals. Moreover, because cognitive task analysis is based on several expert surgeons’ opinions and protocols, residents have access to a wider, more objective breadth of knowledge and all potential options than they would have if they had studied solely under one attending physician. Multimedia teaching creates a logical progression for understanding theory, decision making, and technical skills and allows the learner to advance through these objectives at their own pace. In further studies, we need to examine the effects that interactive multimedia teaching modules as ad-
K.R. Luker et al. / The American Journal of Surgery 195 (2008) 11–15
juncts to traditional “bedside teaching” have on the overall knowledge, critical thinking, and performance of a resident. Despite the limitations of this study, we believe that by applying cognitive task analysis to surgical procedures we are able to teach the decision-making steps and component parts of a procedure that differentiate an expert from a novice surgeon. It also enables the learner to transfer their knowledge from patient to patient and to prepare for a case more effectively. Tools that can be created from this cognitive task analysis include a multimedia teaching video with assessments and simulator-based learning modules. Simulations can be used to create computer-generated subjects and provide a standardized patient on which novice surgeons can practice and hone technical skills. Acknowledgment Supported by a grant from the Association for Surgical Education through the Center for Excellence in Surgical Education, Research and Training Program.
15
References [1] Ericsson KA, Charness N, Feltovich P, et al. The Cambridge Handbook of Expertise and Expert Performance. New York: Cambridge University Press; 2006. [2] Carroll AE, Schwartz MW. A comparison of a lecture and computer program to teach fundamentals of the draw-a-person test. Arch Pediatr Adolesc Med 2002;156:137– 40. [3] Schraagen JM, Chipman SF, Shalin VL. Cognitive Task Analysis. Mahwah, NJ: Lawrence Erlbaum Associates; 2000:531. [4] Clark RE, Estes F. Cognitive task analysis in training. Int J Educ Res 1996;25:403–17. [5] Velmahos GC, Toutouzas KG, Sillin LF, et al. Cognitive task analysis for teaching technical skills in an inanimate surgical skills laboratory. Am J Surg 2004;187:114 –9. [6] Sullivan ME, Brown CV, Peyre SE, et al. The use of cognitive task analysis to improve the learning of percutaneous tracheostomy placement. Am J Surg 2007;193:96 –9. [7] Gagne ED, Yekovich CW, Yekovich FR. Cognitive Psychology of School Learning. New York: Harper Collins; 1993. [8] Martin JA, Regehr G, Reznick R, et al. Objective structured assessment of technical skill (osats) for surgical residents. Br J Surg 1997; 84:273– 8.
Association for Surgical Education
The use of a cognitive task analysis– based multimedia program to teach surgical decision making in flexor tendon repair Kali R. Luker, B.A., Maura E. Sullivan, Ph.D., M.S.N., Sarah E. Peyre, M.S.Ed., Randy Sherman, M.D., Tiffany Grunwald, M.D., M.S.Ed.* Department of Surgery, Division of Plastic & Reconstructive Surgery, Keck School of Medicine, University of Southern California, Santa Monica, CA, USA Manuscript received April 23, 2007; revised manuscript August 10, 2007 Presented at the Annual Meeting of the Association for Surgical Education, Washington, DC, April 11–14, 2007
Abstract Background: The aim of this study was to compare the surgical knowledge of residents before and after receiving a cognitive task analysis– based multimedia teaching module. Methods: Ten plastic surgery residents were evaluated performing flexor tendon repair on 3 occasions. Traditional learning occurred between the first and second trial and served as the control. A teaching module was introduced as an intervention between the second and third trial using cognitive task analysis to illustrate decision-making skills. Results: All residents showed improvement in their decision-making ability when performing flexor tendon repair after each surgical procedure. The group improved through traditional methods as well as exposure to our talk-aloud protocol (P ⬎ .01). After being trained using the cognitive task analysis curriculum the group displayed a statistically significant knowledge expansion (P ⬍ .01). Conclusions: Residents receiving cognitive task analysis– based multimedia surgical curriculum instruction achieved greater command of problem solving and are better equipped to make correct decisions in flexor tendon repair. © 2008 Excerpta Medica Inc. All rights reserved. Keywords: Cognitive task analysis; Knowledge elicitation; Procedural knowledge; Automated knowledge; Procedural skills; Technical skills
Two clear divisions become apparent when examining traditional surgical curriculum. One is the instruction of physiology, anatomy, and disease, and the second is the development of technical skills in an apprenticeship format. The first tier of learning traditionally has focused on the theory behind surgical procedures with minimal ability to illustrate the essential 3-dimensional visual-spatial relationships during these procedures. This classroom teaching has occurred in a logical, progressive manner whereas the 3-dimensional spatial relationships and technical skills have been taught in the operating room, dependent on the random order with which patients present themselves. There has been little coordination or overlap between these 2 tiers of teaching. Although these tiers of surgical education are not mutually exclusive and are obviously a simplification of the curricu-
* Corresponding author. Tel.: ⫹1-310 315-0215; fax: ⫹1-310 315-9392. E-mail address: [email protected]
lum, they are noticeable subdivisions that are difficult to synchronize. In contrast to the traditional surgical curriculum, current educational theory tells us that adult learners learn best when information is placed into a contextual format and learners need to be taught decision-making strategies during this time [1]. Before the computer age, there was no established method for accomplishing this other than to create paper scenarios or to teach using actual patients. This is no longer true in today’s world of advancing technologies. Simulation modules are being used to create computer-generated patients, providing a controlled subject for novice surgeons to learn from. By using these modules, patients and diseases can be generated in a thoughtful, logical manner, corresponding to existing curricula of surgical training programs. Instruction using many types of media has provided a sophisticated medium whereby we can incorporate traditional text and illustrations into a teaching program with interactive and 3-dimensional capabilities. According to the
0002-9610/08/$ – see front matter © 2008 Excerpta Medica Inc. All rights reserved. doi:10.1016/j.amjsurg.2007.08.052
12
K.R. Luker et al. / The American Journal of Surgery 195 (2008) 11–15
cognitive theory of multimedia learning, the act of building connections between verbal and pictorial mental models is an important step in conceptual understanding. A multimedia presentation with videotape, illustrations, animations, and text guides the learner to build a verbal mental model of the material, a pictorial mental model of the material, and connections between the two. Students who receive wellconstructed multimedia information perform better on tests designed to measure understanding and problem-solving ability, than do students who receive information presented only in words [2]. To build these multimedia tools, a method for effectively capturing an expert’s knowledge must be used. One specific method for obtaining sophisticated performance expertise in which many covert decisions are linked with complex overt actions is cognitive task analysis [3]. By applying cognitive task analysis to surgical procedures, we can focus instruction on the decision-making steps and component parts of a procedure that differentiate an expert from a novice surgeon. Cognitive task analysis is one of the more powerful design tools that has been used in the military and a number of highly technical fields to hasten the acquisition of expertise [4] and has been shown to improve learning of basic surgical skills [5,6]. By centering the teaching around decision-making points and available options in a specific simulation, cognitive task analysis prepares novices to transfer their knowledge when problem solving in a variety of related cases as opposed to the traditional Halstedian method in which learning is patient specific. In addition, using cognitive task analysis allows trained analysts to delve into expert surgeons’ decision-making techniques outside of the pressures of the operating room. Even without the time constraints and demands of the operating room, it has been shown that as surgeons develop expertise, their knowledge becomes automated. Automated knowledge operates outside of conscious awareness, making it difficult for experts to describe the discrete steps of the procedure [1,4,7]. For this reason, it often is difficult for experts to articulate the decision points they make during a procedure because they are not aware of many of their own decisions and mental analysis of tasks [4]. Incorporating cognitive task analysis into the design of a multimedia learning tool will give us the opportunity to transfer knowledge effectively from expert to novice, accelerate the acquisition of expertise, and enrich our domain of instruction. In addition, cognitive task analysis will provide a means for assessing decision-making skills and surgical judgment. This study evaluated the ability of a cognitive task analysis– based multimedia teaching module to instruct residents on flexor tendon repair. Methods Subjects Ten residents from the Division of Plastic and Reconstructive Surgery at the University of Southern California were recruited for this study. Ethics approval was obtained from the Institutional Review Board at the University of Southern California Health Sciences Campus. Informed consent was obtained from all participants and they were assured that the results from the study were confidential. Residents were in-
cluded from every year of the plastic surgery program and had varying levels of exposure to flexor tendon repair. There were 4 residents in the first year of the program, 3 in their second year, 2 in their third year, and 1 hand fellow. Three residents were involved in a clinical rotation dedicated to hand surgery at the time of the study and the others had unrelated clinical duties at the time of testing. Throughout the study, the residents served as their own control. Materials Three surgeons with expertise in flexor tendon repair participated in a cognitive task analysis that examined their decision-making processes and the necessary technical skills required to effectively perform a repair. The cognitive task analysis analysts were a faculty member and a graduate student from the School of Education. The cognitive task analysis consisted of a series of interviews. During the first interview, the cognitive task analysis analysts interviewed the expert surgeons with the intent to extract all relevant information describing flexor tendon repair. This information was drafted according to cognitive task analysis principles in a comprehensive document. The document described in a step-by-step fashion technical skill and emphasized critical details, decision-making points, and important maneuvers necessary to avoid potential pitfalls. In a second interview the analysts discussed revisions of the documents with the experts. In a third interview the analysts and surgeons came together to discuss and resolve potential conflicting points. The final document was reviewed and approved by all members of the group. This document included the technique of flexor tendon repair as well as the indications, contraindications, complications, and pitfalls related to flexor tendon repair. The development of the multimedia curriculum was based on the assembled cognitive task analysis document and was organized to emphasize the decision points and important details that are critical for mastery and completion of the overreaching surgical task. Illustrations, animations, and text guides were associated with videotaped clips to allow the learner to build a verbal and pictorial mental model of the task being described. These demonstrations also facilitated building connections between the 2 models and allowed the learner to witness the successful execution of the task. Production occurred in collaboration with experts from the Institute for Multimedia Literacy. A talk-aloud protocol also was created as a tool for assessment of the trainees’ understanding of the procedure, the salient decision points, and the factual information necessary to begin to view a case as an expert. It was formatted in a verbal examination format. The trainees were questioned at each decision point about their options and the advantages and disadvantages of these options. It also included questions regarding the performance of certain surgical techniques essential for a functional flexor tendon repair. When a trainee correctly identified an option at a decision point or an advantage or disadvantage of an option, they received a point. They also received points for correctly describing the surgical techniques that are critical to the procedure. The talk-aloud protocol was adapted from the cognitive task analysis and emphasized information directly
K.R. Luker et al. / The American Journal of Surgery 195 (2008) 11–15
relevant to the specific procedure. It omitted information that was deemed basic surgical knowledge or outside the realm of the protocol. Evaluation Ten plastic and reconstructive surgery residents of varying levels of training were asked to perform flexor tendon repairs on a cadaveric hand 3 separate times as if it were a real patient. Before each procedure, the residents completed a 10-minute questionnaire covering prior exposure to flexor tendon repair, how they prepared for this case, how they usually prepare for cases, and their history of didactic teaching of flexor tendon repair. Each surgery was observed by a trained evaluator who completed an objective checklist referring to the essential steps and decision points of the procedure. The checklist was designed to satisfy cognitive task analysis principles and mirror the task-specific checklists as described by Martin and Resnick [8]. Digital videotape was made for review with the trainee using a talk-aloud protocol to determine awareness of decision points, available options, advantages and disadvantages, and the reasons for their decisions. Each flexor tendon repair was reviewed immediately after completion of the task and a knowledge assessment using the talk-aloud protocol was performed. During the talk-aloud protocol, the residents watched a recording of the flexor tendon repair that they had just performed to refresh their memory of what they were thinking at the time. At each decision point the videotape was stopped and the resident was asked several questions that tested their knowledge of the decision-making steps of flexor tendon repair at that point as determined through cognitive task analysis as well as alternatives and rationale for each decision. This format allowed the resident to talk through their thought process and justifications for the decisions they made. Fig. 1 diagrams the layout of the study. Throughout the study the residents served as their own controls. The control group was considered to be all 10 residents from sessions 1 to 2. Any changes in scores between sessions 1 and 2 during the talk-aloud review of their videotape was considered to be the result of traditional learning as well as knowledge of the questions that were asked of them. The multimedia teaching module derived from the cognitive task analysis was given to all 10 residents after session 2. Any change in knowledge between sessions 2 and 3 was considered to be the result of the multimedia instruction. The residents who participated in the training and performed the final flexor tendon repair were considered to be the interventional group. Because 1 resident did not participate in the multimedia training, the final interventional group included 9 of the 10 total residents.
13
Training Nine residents underwent instruction using the multimedia program after their second flexor tendon repair. The material was cognitive task analysis– based and included theory and technique of flexor tendon repair, anatomy, physiology, pathology, indications and contraindications for surgical repair, and expert tips. The computer program included videotaped clips of a live procedure as well as illustrations, animations, and text guides. It emphasized the component steps and decision-making points as described through cognitive task analysis. Analysis The evaluation data were entered in a Microsoft Excel database (Microsoft Corp, Redmond, WA) and analyzed statistically using a nonparametric Wilcoxon test. Comparisons were performed between the change in scores from sessions 1 to 2 and sessions 2 to 3. We used the comparison between sessions 1 and 2 as our control to measure improvement through traditional methods as well as exposure to our talk-aloud protocol. Any change between sessions 2 and 3 was assumed to be the result of learning from the multimedia teaching module. We used statistical analysis to compare the total change in scores and knowledge of the advantages and disadvantages of options at each decision point. The results also were analyzed by subgrouping the residents according to their year in the Plastic and Reconstructive Surgery Program (year 1 vs year 2 and higher), exposure to flexor tendon repair (whether they had performed ⬍10 flexor tendon repairs vs performed ⬎10), scores after the first flexor tendon repair (⬍60 vs ⬎60), and improvement between sessions 1 and 2 (⬍10 vs ⬎10). These groupings attempted to evaluate whether level, experience, baseline knowledge, or extent of traditional learning between sessions 1 and 2 influenced training by the multimedia teaching module. We deemed a P value of less than .01 as statistically significant for this study. Results The difference of all 10 residents’ scores between sessions 1 and 2 was considered to be the study control, evaluating traditional learning and contamination via exposure to the talk-aloud review. The change in scores of the 9 residents who participated in the multimedia instruction between sessions 2 and 3 represented the interventional segment of the study. Fig. 2 diagrams the results of the scoring analysis including the mean, range, and 95% confidence intervals for total scores and knowledge of advantages and disadvantages of decision points. Although there
Fig. 1. Diagram of study layout and individual groups. TAP ⫽ talk-aloud protocol video review of flexor tendon repair.
14
K.R. Luker et al. / The American Journal of Surgery 195 (2008) 11–15
they were not significant. By introducing the teaching module between sessions 2 and 3, the residents were able to effectively build on their current understanding of the procedure and why certain decisions are made. This was verified with a statistical significance of a P value of less than .01. Although the results of this study were encouraging, there were some limitations. First, there was a small number of subjects enrolled in the study. Second, even though cognitive task analysis has face validity, no formal validation studies have been performed. Finally, the subjects in the study who were on their hand service at the time may have been contaminated because they were exposed to the questions of the talk-aloud protocol, as was their attending physician. This could have affected the way the residents and attending interacted with each other. These factors may have caused the residents to ask more probing questions and the attending to teach with a greater focus on decision points and details than had been automated for them previously. Fig. 2. Results of the scoring analysis including mean, range, and 95% confidence intervals for total scores and knowledge of advantages and disadvantages of decision points.
was an improvement trend from session 1 to session 2, this improvement was not statistically significant for either improvement in total score or knowledge of the advantages and disadvantages of each decision point. The mean increase in scores was 13.2 for total knowledge and 8.4 for understanding of advantages and disadvantages. In contrast, a statistically significant improvement (P ⬍ .01) was seen as a result of the intervention of the teaching videotape between sessions 2 and 3 for both total knowledge and understanding of advantages and disadvantages of decision points. No statistically significant difference in their increase in knowledge throughout the study was seen between any of the subgroups mentioned earlier. The average score increase for total knowledge was 34.0 and 19.4 for understanding of advantages and disadvantages, respectively. The actual performance during the flexor tendon repairs was not evaluated because the cadaveric arms were too dissimilar to an actual patient. Many options and situations that normally would present themselves in flexor tendon lacerations and were discussed in the videotape such as physical examination techniques, partial tendon laceration repair, and tourniquetting the arm could not be performed because of the limitations of the mock event. The residents’ knowledge of these situations was evaluated during questioning using the talk-aloud protocol. Comments We have shown that although a significant amount of learning occurs via traditional methods, augmenting a resident’s training with computer-based learning modules is an effective way to contextualize a procedure and create a bridge from the theory to technical skills. The first segment of our study controlled for learning that would be done based on traditional learning and a knowledge of the talkaloud questions. Although the results showed a marked increasing trend for both total knowledge and understanding of advantages and disadvantages of each decision point,
Conclusions Multimedia learning tools are rapidly becoming an integral component of surgical education curriculum. These tools allow residents to gain a basic understanding of the procedural tasks and the decision points that are encountered along the way. Traditional Halstedian learning in the operating room has several drawbacks in that there is time pressure and only one particular case is observed at a time so not all potential variants can be considered. In addition, it has been shown that an expert’s knowledge becomes automated. Therefore, an expert is unable to identify and describe the discrete decisions made throughout a procedure. Cognitive task analysis is a method for extracting and capturing this highly complex performance expertise in which many covert decisions are linked with difficult overt actions. It also provides a method for integrating the automated skills and knowledge of expert surgeons into a form more easily understood by novices. In addition, supplementing traditional surgical education with computerized instruction and simulation allows a resident to become proficient and knowledgeable about a procedure in a structured and cohesive manner. It grants a novice access to the underlying thought processes of expert surgeons and understanding of the decision-making processes motivating an expert surgeon’s actions. Before entering the operating room, residents are able to learn the basic principles and techniques of a procedure and the essential decision points. This allows them to spend their time in the operating room honing this knowledge and asking more probing, in-depth questions of the attending surgeon instead of struggling with the fundamentals. Moreover, because cognitive task analysis is based on several expert surgeons’ opinions and protocols, residents have access to a wider, more objective breadth of knowledge and all potential options than they would have if they had studied solely under one attending physician. Multimedia teaching creates a logical progression for understanding theory, decision making, and technical skills and allows the learner to advance through these objectives at their own pace. In further studies, we need to examine the effects that interactive multimedia teaching modules as ad-
K.R. Luker et al. / The American Journal of Surgery 195 (2008) 11–15
juncts to traditional “bedside teaching” have on the overall knowledge, critical thinking, and performance of a resident. Despite the limitations of this study, we believe that by applying cognitive task analysis to surgical procedures we are able to teach the decision-making steps and component parts of a procedure that differentiate an expert from a novice surgeon. It also enables the learner to transfer their knowledge from patient to patient and to prepare for a case more effectively. Tools that can be created from this cognitive task analysis include a multimedia teaching video with assessments and simulator-based learning modules. Simulations can be used to create computer-generated subjects and provide a standardized patient on which novice surgeons can practice and hone technical skills. Acknowledgment Supported by a grant from the Association for Surgical Education through the Center for Excellence in Surgical Education, Research and Training Program.
15
References [1] Ericsson KA, Charness N, Feltovich P, et al. The Cambridge Handbook of Expertise and Expert Performance. New York: Cambridge University Press; 2006. [2] Carroll AE, Schwartz MW. A comparison of a lecture and computer program to teach fundamentals of the draw-a-person test. Arch Pediatr Adolesc Med 2002;156:137– 40. [3] Schraagen JM, Chipman SF, Shalin VL. Cognitive Task Analysis. Mahwah, NJ: Lawrence Erlbaum Associates; 2000:531. [4] Clark RE, Estes F. Cognitive task analysis in training. Int J Educ Res 1996;25:403–17. [5] Velmahos GC, Toutouzas KG, Sillin LF, et al. Cognitive task analysis for teaching technical skills in an inanimate surgical skills laboratory. Am J Surg 2004;187:114 –9. [6] Sullivan ME, Brown CV, Peyre SE, et al. The use of cognitive task analysis to improve the learning of percutaneous tracheostomy placement. Am J Surg 2007;193:96 –9. [7] Gagne ED, Yekovich CW, Yekovich FR. Cognitive Psychology of School Learning. New York: Harper Collins; 1993. [8] Martin JA, Regehr G, Reznick R, et al. Objective structured assessment of technical skill (osats) for surgical residents. Br J Surg 1997; 84:273– 8.