- Email: [email protected]
The impact of external feedback on computer-assisted learning for surgical technical skill training
The Impact of External Feedback on Computer-Assisted Learning for Surgical Technical Skill Training David A. Rogers, MD, Augusta, Georgia, Glenn Regehr, PhD, Toronto, Ontario, Canada, Thomas R. Howdieshell, MD, Karen A. Yeh, MD, Ellen Palm, MSMI, Augusta, Georgia
BACKGROUND: Computer-assisted learning (CAL) offers a number of potential advantages for surgical technical skills teaching. The purpose of this study was to evaluate the impact of individualized external feedback on surgical skill acquisition when a CAL package is used for instruction. METHODS: Freshman and sophomore students participated in a 1-hour CAL session designed to teach them how to tie a two-handed square knot. One group received individualized external feedback during the session and the other group did not. Subjects were videotaped performing the skill before and after the session. The tapes were independently analyzed, in blinded fashion, by three surgeons. Three measures were obtained: the total time for the task, whether or not the knot was square, and the general quality of the performance using a rating scale. RESULTS: Data from 105 subjects were available for final analysis. For both groups there were significant increases in the proportion of knots that were square when the posttest performance was compared with the pretest performance but there was no difference between groups on this measure. Comparison of the performance scores demonstrated that both groups had a significant improvement after the session but the performance scores were significantly better in the group that had received feedback. CONCLUSIONS: Novices in both groups using CAL showed improvement in two of the outcomes measured, suggesting that subjects in both groups attained some degree of competence with this skill. The higher posttest performance score for the group receiving feedback demonstrates that external feedback results in a higher level of mastery when CAL is used to teach sur-
From the Department of Surgery (DAR, TRH, KAY, EP), Medical College of Georgia, Augusta, Georgia, and the Department of Surgery (GR), University of Toronto, Toronto, Ontario, Canada. Requests for reprints should be addressed to David A. Rogers, MD, Section of Pediatric Surgery, Medical College of Georgia, BT 5729, Augusta, Georgia 30912-4070. Manuscript submitted September 23, 1999, and accepted in revised form March 6, 2000. Presented at the 19th Annual Meeting of the Association for Surgical Education, Boston, Massachusetts, April 7–10, 1999.
© 2000 by Excerpta Medica, Inc. All rights reserved.
gical technical skills. Am J Surg. 2000;179:341– 343. © 2000 by Excerpta Medica, Inc.
O
ur initial interest in CAL was that it might be a more efficient method to teach basic surgical skills.1 In that study, the students generally indicated that they preferred the more traditional lecture and feedback seminar because of the availability of the content expert to provide feedback as they learned the task. We concluded that this lack of expert feedback in the CAL group was the primary cause for the inferior level of mastery attained by this group. However, there were other differences between the groups and so we could not completely discount the possibility that confounding variables may have been, at least in part, responsible for our findings. Further, we had observed that many of the students who had used the CAL program alone had obtained a level of competence with this task and were interested in quantifying the degree to which this occurred. Our goals in this study were to rigorously evaluate the impact of external feedback on the acquisition of a basic surgical skill and to measure the degree to which learner’s using CAL alone had acquired a basic surgical skill.
METHODS Approval was obtained from the Institutional Review Board at the Medical College of Georgia. Freshman and sophomore medical students were enrolled in the study, and each was randomly assigned to either the CAL alone group or the CAL and feedback group. The subjects were not paid to participate in the study but were allowed to keep the knot-tying board, manual, and rope. Both groups used the CAL program that has been described previously.1 The subjects were placed in groups of seven or eight and each interacted with a single computer in a specially equipped classroom. At the beginning of the session, the subject was instructed to tie his or her “best and fastest pair of twohanded square knots,” and the performance was videotaped. The subject was then supplied with a knot-tying board and tying rope and participated in a 1-hour educational session. In the CAL with feedback group, a content expert was present to evaluate the subject’s performance and provide instruction. The feedback was provided intermittently throughout the session. In the CAL alone group, no feedback was given during the session. At the end of the session, the subject was given identical instructions and the posttest performance was videotaped. Finally, the subject was surveyed regarding his opinions about the session. Each videotaped performance was reviewed indepen0002-9610/00/$–see front matter PII S0002-9610(00)00341-X
341
IMPACT OF EXTERNAL FEEDBACK ON COMPUTER-ASSISTED LEARNING/ROGERS ET AL
TABLE I
TABLE II Average Performance Scores
CAL alone (n ⫽ 51) CAL plus feedback (n ⫽ 54) Between-group difference
Proportion of Knots Squared
Pretest
Posttest
Within-Group Difference
2.6 (1.94)
12.0 (4.87)
9.4*
3.1 (2.22)
15.8 (4.15)
12.7*
0.5
3.8†
Values shown are the average scores with the standard deviation shown in parentheses. * Significantly different by Student’s paired t test. † Significantly different by the independent sample Student’s t test. CAL ⫽ computer-assisted learning.
dently by three surgical faculty members. They recorded whether or not the knot was square, and in cases where all three did not agree, the majority opinion was recorded. Further, the quality of the knot tying was evaluated using a rating scale that explicitly identified all of the actions necessary for an optimal performance. The number generated using the rating scale was termed a performance score, with a maximum value of 24. Finally, the time for the task was recorded for each performance when a performance was attempted. Data were entered into the Paradox 8.0 database (Corel Corp., Ontario, Canada) and analyzed using Quattro Pro 8.0 (Corel Corp., Ontario, Canada) and SPSS 8.0 for Windows (SPSS, Inc., Chicago, IL).
RESULTS One hundred and eight subjects were enrolled in the study. Data from 3 of the subjects were excluded because it was not possible to evaluate one of the performances, and so data from 105 subjects were available for final analysis. Fifty-four subjects were in the CAL with feedback group and 51 subjects were in the CAL alone group. Average age, gender distribution, and handedness were similar for both groups. A two-way analysis of variance (ANOVA) evaluating the effect of time (pretest versus posttest) and group (CAL alone versus CAL with feedback) revealed a powerful main effect of time (F ⫽ 532.3, P ⬍0.001) and group (F ⫽ 19.7, P ⬍0.001) with a significant interaction between time and group (F ⫽ 12.12, P ⬍0.001). A subsequent comparison of the average performance scores for each of the two groups (Table I) using the paired Student’s t test showed that the scores increased significantly within both the CAL alone group (t ⫽ 13.4, P ⬍0.001) and the CAL with feedback group (t ⫽ 19.4, P ⬍0.001). The between-group average performance scores were compared using an independent samples Student’s t test showing that the posttest performance score was significantly higher in the CAL with feedback group when compared with the CAL alone group (t ⫽ 4.38, P ⬍0.001). The proportion of knots squared (Table II) increased significantly within both groups (McNemar P ⬍0.001) but there was no difference between groups in either the pretest (chi square ⫽ 1.36, P ⫽ 0.24) or posttest performance (chi square ⫽ 0.733, P ⫽ 0.39). No comparison of time was made for the pretest group because so few of the subjects 342
Pretest Posttest CAL alone (n ⫽ 51) CAL plus feedback (n ⫽ 54) Between-group difference
33 44 11†
87 92 5†
Within-Group Difference 54* 48*
Values shown are the percentage of subjects who tied a square knot. * Significant difference by the McNemar test. † No significant difference by chi square of proportions. CAL ⫽ computer-assisted learning.
could successfully complete the task, and comparison of the posttest times for the two groups demonstrated no significant difference between the CAL alone and the CAL with feedback groups (14.5 versus 16.1 seconds, t ⫽ 1.3, P ⫽ 0.19). Fifty-five percent of the subjects in the CAL with feedback group reported that having expert feedback available was a major positive factor in the acquisition of the skill.
COMMENTS The results of this study not only provide additional evidence for the importance of external feedback from a content expert in the instruction of basic surgical skills but they also provide some evidence for the effectiveness of CAL alone for this purpose. The acquisition of motor skills is subject to continuing research and several theories have evolved. These theories have become divergent enough to be considered a different “schools.”2,3 One unifying characteristic of these theories is the importance of feedback in skill mastery.2 The interaction of feedback and skill acquisition can be illustrated by examining one of the theories of skill acquisition that has received attention in the surgical literature.4,5 In this theory, psychomotor tasks are said to be learned in three phases: cognitive, associative, and autonomous.2 In the cognitive phase, the learner attains a degree of understanding of the task. During the associative phase, the learner practices the task and the performance is compared with that of the expert. The difference between the expert and learner performance is considered the error, and the learner attains mastery by minimizing this error. In the autonomous phase, the skill is performed without distinct cognitive awareness. It is during the associative phase that feedback is felt to play a major role. Feedback may be either internal or external. Internal feedback is that generated by the learner when the learner’s performance is compared with that of the experts. This comparison occurs through the sensory system, and in our study would occur when subjects compared their visual assessment of their own performance with that contrasted by the CAL representation of the expert’s performance. The fact that both the proportion of knots squared and the performance scores improved in the CAL alone group suggests that the CAL depiction was successful in providing learners with sufficient information about the task so that they were able to develop a cognitive picture of the task and use their internal feedback to correct some of the errors. Prior research in medical skills
THE AMERICAN JOURNAL OF SURGERY® VOLUME 179 APRIL 2000
IMPACT OF EXTERNAL FEEDBACK ON COMPUTER-ASSISTED LEARNING/ROGERS ET AL
instruction has also demonstrated the effectiveness of internal feedback for skill mastery.5,6 The other major type of feedback is external, as the source of the feedback is external to the learner. This source is typically a content expert and this type of feedback is generally divided into knowledge of results (KR) and knowledge of performance (KP). KR is defined as “information about the success of an action with respect to the environmental goal.”7 In contrast, KP is defined as “information about the pattern of the movement the learner has made.”7 In our study the desired result is a squared knot, and our results suggest that the content expert appeared to offer no advantage over the learner’s own internal feedback process when CAL was used to provide KR. In contrast, the fact that the CAL group that received feedback had a significantly higher performance score suggests that the external feedback provided a substantial advantage in this group of learners understanding of KP. We believe that this was due, in part, to the fact that the skill is complex and the learners were novices. The learner could recognize that the performance did not
match that of the expert but it required the expert to identify the error and outline the steps necessary to correct the learner’s performance.
REFERENCES 1. Rogers DA, Regehr G, Yeh KA, Howdieshell TR. Computerassisted learning versus a lecture and feedback seminar for teaching a basic surgical technical skill. Am J Surg. 1998;175:508 –510. 2. Kaufman HH, Wiegand RL, Tunick RH. Teaching surgeons to operate: principles of psychomotor skills training. Acta Neurochir. 1987;87:1–7. 3. DesCoteaux JG, Leclere H. Learning surgical technical skills. Can J Surg. 1995;38:33–38. 4. Kopta JA. An approach to the evaluation of operative skills. Surgery. 1971;70:297–303. 5. Nicks CM, Nelson DL, Lang NP. Use of the surgical skills laboratory for teaching medical students. Focus Surg Educ. 1986;3: 13–14. 6. Kardash K, Tessler MJ. Videotape feedback in teaching laryngoscopy. Can J Anaesth. 1997;44:54 –58. 7. Schmidt RA. Motor Learning and Performance. From Principles to Practice. Champaign, Ill: Human Kinetics Books; 1991.
THE AMERICAN JOURNAL OF SURGERY® VOLUME 179 APRIL 2000
343
BACKGROUND: Computer-assisted learning (CAL) offers a number of potential advantages for surgical technical skills teaching. The purpose of this study was to evaluate the impact of individualized external feedback on surgical skill acquisition when a CAL package is used for instruction. METHODS: Freshman and sophomore students participated in a 1-hour CAL session designed to teach them how to tie a two-handed square knot. One group received individualized external feedback during the session and the other group did not. Subjects were videotaped performing the skill before and after the session. The tapes were independently analyzed, in blinded fashion, by three surgeons. Three measures were obtained: the total time for the task, whether or not the knot was square, and the general quality of the performance using a rating scale. RESULTS: Data from 105 subjects were available for final analysis. For both groups there were significant increases in the proportion of knots that were square when the posttest performance was compared with the pretest performance but there was no difference between groups on this measure. Comparison of the performance scores demonstrated that both groups had a significant improvement after the session but the performance scores were significantly better in the group that had received feedback. CONCLUSIONS: Novices in both groups using CAL showed improvement in two of the outcomes measured, suggesting that subjects in both groups attained some degree of competence with this skill. The higher posttest performance score for the group receiving feedback demonstrates that external feedback results in a higher level of mastery when CAL is used to teach sur-
From the Department of Surgery (DAR, TRH, KAY, EP), Medical College of Georgia, Augusta, Georgia, and the Department of Surgery (GR), University of Toronto, Toronto, Ontario, Canada. Requests for reprints should be addressed to David A. Rogers, MD, Section of Pediatric Surgery, Medical College of Georgia, BT 5729, Augusta, Georgia 30912-4070. Manuscript submitted September 23, 1999, and accepted in revised form March 6, 2000. Presented at the 19th Annual Meeting of the Association for Surgical Education, Boston, Massachusetts, April 7–10, 1999.
© 2000 by Excerpta Medica, Inc. All rights reserved.
gical technical skills. Am J Surg. 2000;179:341– 343. © 2000 by Excerpta Medica, Inc.
O
ur initial interest in CAL was that it might be a more efficient method to teach basic surgical skills.1 In that study, the students generally indicated that they preferred the more traditional lecture and feedback seminar because of the availability of the content expert to provide feedback as they learned the task. We concluded that this lack of expert feedback in the CAL group was the primary cause for the inferior level of mastery attained by this group. However, there were other differences between the groups and so we could not completely discount the possibility that confounding variables may have been, at least in part, responsible for our findings. Further, we had observed that many of the students who had used the CAL program alone had obtained a level of competence with this task and were interested in quantifying the degree to which this occurred. Our goals in this study were to rigorously evaluate the impact of external feedback on the acquisition of a basic surgical skill and to measure the degree to which learner’s using CAL alone had acquired a basic surgical skill.
METHODS Approval was obtained from the Institutional Review Board at the Medical College of Georgia. Freshman and sophomore medical students were enrolled in the study, and each was randomly assigned to either the CAL alone group or the CAL and feedback group. The subjects were not paid to participate in the study but were allowed to keep the knot-tying board, manual, and rope. Both groups used the CAL program that has been described previously.1 The subjects were placed in groups of seven or eight and each interacted with a single computer in a specially equipped classroom. At the beginning of the session, the subject was instructed to tie his or her “best and fastest pair of twohanded square knots,” and the performance was videotaped. The subject was then supplied with a knot-tying board and tying rope and participated in a 1-hour educational session. In the CAL with feedback group, a content expert was present to evaluate the subject’s performance and provide instruction. The feedback was provided intermittently throughout the session. In the CAL alone group, no feedback was given during the session. At the end of the session, the subject was given identical instructions and the posttest performance was videotaped. Finally, the subject was surveyed regarding his opinions about the session. Each videotaped performance was reviewed indepen0002-9610/00/$–see front matter PII S0002-9610(00)00341-X
341
IMPACT OF EXTERNAL FEEDBACK ON COMPUTER-ASSISTED LEARNING/ROGERS ET AL
TABLE I
TABLE II Average Performance Scores
CAL alone (n ⫽ 51) CAL plus feedback (n ⫽ 54) Between-group difference
Proportion of Knots Squared
Pretest
Posttest
Within-Group Difference
2.6 (1.94)
12.0 (4.87)
9.4*
3.1 (2.22)
15.8 (4.15)
12.7*
0.5
3.8†
Values shown are the average scores with the standard deviation shown in parentheses. * Significantly different by Student’s paired t test. † Significantly different by the independent sample Student’s t test. CAL ⫽ computer-assisted learning.
dently by three surgical faculty members. They recorded whether or not the knot was square, and in cases where all three did not agree, the majority opinion was recorded. Further, the quality of the knot tying was evaluated using a rating scale that explicitly identified all of the actions necessary for an optimal performance. The number generated using the rating scale was termed a performance score, with a maximum value of 24. Finally, the time for the task was recorded for each performance when a performance was attempted. Data were entered into the Paradox 8.0 database (Corel Corp., Ontario, Canada) and analyzed using Quattro Pro 8.0 (Corel Corp., Ontario, Canada) and SPSS 8.0 for Windows (SPSS, Inc., Chicago, IL).
RESULTS One hundred and eight subjects were enrolled in the study. Data from 3 of the subjects were excluded because it was not possible to evaluate one of the performances, and so data from 105 subjects were available for final analysis. Fifty-four subjects were in the CAL with feedback group and 51 subjects were in the CAL alone group. Average age, gender distribution, and handedness were similar for both groups. A two-way analysis of variance (ANOVA) evaluating the effect of time (pretest versus posttest) and group (CAL alone versus CAL with feedback) revealed a powerful main effect of time (F ⫽ 532.3, P ⬍0.001) and group (F ⫽ 19.7, P ⬍0.001) with a significant interaction between time and group (F ⫽ 12.12, P ⬍0.001). A subsequent comparison of the average performance scores for each of the two groups (Table I) using the paired Student’s t test showed that the scores increased significantly within both the CAL alone group (t ⫽ 13.4, P ⬍0.001) and the CAL with feedback group (t ⫽ 19.4, P ⬍0.001). The between-group average performance scores were compared using an independent samples Student’s t test showing that the posttest performance score was significantly higher in the CAL with feedback group when compared with the CAL alone group (t ⫽ 4.38, P ⬍0.001). The proportion of knots squared (Table II) increased significantly within both groups (McNemar P ⬍0.001) but there was no difference between groups in either the pretest (chi square ⫽ 1.36, P ⫽ 0.24) or posttest performance (chi square ⫽ 0.733, P ⫽ 0.39). No comparison of time was made for the pretest group because so few of the subjects 342
Pretest Posttest CAL alone (n ⫽ 51) CAL plus feedback (n ⫽ 54) Between-group difference
33 44 11†
87 92 5†
Within-Group Difference 54* 48*
Values shown are the percentage of subjects who tied a square knot. * Significant difference by the McNemar test. † No significant difference by chi square of proportions. CAL ⫽ computer-assisted learning.
could successfully complete the task, and comparison of the posttest times for the two groups demonstrated no significant difference between the CAL alone and the CAL with feedback groups (14.5 versus 16.1 seconds, t ⫽ 1.3, P ⫽ 0.19). Fifty-five percent of the subjects in the CAL with feedback group reported that having expert feedback available was a major positive factor in the acquisition of the skill.
COMMENTS The results of this study not only provide additional evidence for the importance of external feedback from a content expert in the instruction of basic surgical skills but they also provide some evidence for the effectiveness of CAL alone for this purpose. The acquisition of motor skills is subject to continuing research and several theories have evolved. These theories have become divergent enough to be considered a different “schools.”2,3 One unifying characteristic of these theories is the importance of feedback in skill mastery.2 The interaction of feedback and skill acquisition can be illustrated by examining one of the theories of skill acquisition that has received attention in the surgical literature.4,5 In this theory, psychomotor tasks are said to be learned in three phases: cognitive, associative, and autonomous.2 In the cognitive phase, the learner attains a degree of understanding of the task. During the associative phase, the learner practices the task and the performance is compared with that of the expert. The difference between the expert and learner performance is considered the error, and the learner attains mastery by minimizing this error. In the autonomous phase, the skill is performed without distinct cognitive awareness. It is during the associative phase that feedback is felt to play a major role. Feedback may be either internal or external. Internal feedback is that generated by the learner when the learner’s performance is compared with that of the experts. This comparison occurs through the sensory system, and in our study would occur when subjects compared their visual assessment of their own performance with that contrasted by the CAL representation of the expert’s performance. The fact that both the proportion of knots squared and the performance scores improved in the CAL alone group suggests that the CAL depiction was successful in providing learners with sufficient information about the task so that they were able to develop a cognitive picture of the task and use their internal feedback to correct some of the errors. Prior research in medical skills
THE AMERICAN JOURNAL OF SURGERY® VOLUME 179 APRIL 2000
IMPACT OF EXTERNAL FEEDBACK ON COMPUTER-ASSISTED LEARNING/ROGERS ET AL
instruction has also demonstrated the effectiveness of internal feedback for skill mastery.5,6 The other major type of feedback is external, as the source of the feedback is external to the learner. This source is typically a content expert and this type of feedback is generally divided into knowledge of results (KR) and knowledge of performance (KP). KR is defined as “information about the success of an action with respect to the environmental goal.”7 In contrast, KP is defined as “information about the pattern of the movement the learner has made.”7 In our study the desired result is a squared knot, and our results suggest that the content expert appeared to offer no advantage over the learner’s own internal feedback process when CAL was used to provide KR. In contrast, the fact that the CAL group that received feedback had a significantly higher performance score suggests that the external feedback provided a substantial advantage in this group of learners understanding of KP. We believe that this was due, in part, to the fact that the skill is complex and the learners were novices. The learner could recognize that the performance did not
match that of the expert but it required the expert to identify the error and outline the steps necessary to correct the learner’s performance.
REFERENCES 1. Rogers DA, Regehr G, Yeh KA, Howdieshell TR. Computerassisted learning versus a lecture and feedback seminar for teaching a basic surgical technical skill. Am J Surg. 1998;175:508 –510. 2. Kaufman HH, Wiegand RL, Tunick RH. Teaching surgeons to operate: principles of psychomotor skills training. Acta Neurochir. 1987;87:1–7. 3. DesCoteaux JG, Leclere H. Learning surgical technical skills. Can J Surg. 1995;38:33–38. 4. Kopta JA. An approach to the evaluation of operative skills. Surgery. 1971;70:297–303. 5. Nicks CM, Nelson DL, Lang NP. Use of the surgical skills laboratory for teaching medical students. Focus Surg Educ. 1986;3: 13–14. 6. Kardash K, Tessler MJ. Videotape feedback in teaching laryngoscopy. Can J Anaesth. 1997;44:54 –58. 7. Schmidt RA. Motor Learning and Performance. From Principles to Practice. Champaign, Ill: Human Kinetics Books; 1991.
THE AMERICAN JOURNAL OF SURGERY® VOLUME 179 APRIL 2000
343