- Email: [email protected]
Development and Impact Evaluation of an E-Learning Radiation Oncology Module
International Journal of
Radiation Oncology biology
physics
www.redjournal.org
Radiation Oncology Education
Development and Impact Evaluation of an E-Learning Radiation Oncology Module Joanne Alfieri, M.D., C.M.,* Lorraine Portelance, M.D.,* Luis Souhami, M.D.,* Yvonne Steinert, Ph.D.,y Peter McLeod, M.D.,y Fleure Gallant, M.D., C.M.,* and Giovanni Artho, M.D.z Departments of *Radiation Oncology and zRadiology, McGill University Health Centre, and yCentre for Medical Education, McGill University, Montreal, QC, Canada Received Aug 11, 2010, and in revised form Jun 30, 2011. Accepted for publication Jul 6, 2011
Summary Radiographic anatomy is key to radiation treatment planning. An interactive webbased module was developed to improve residents’ knowledge in this domain and was tested in a multicenter, randomized controlled study. Those residents having access to the module showed a significant improvement in test scores when compared to the group of residents without. A retrospective performance survey showed long-term retention and self- perceived changes in residents’ knowledge. Interactive e-learning teaching modules may,
Purpose: Radiation oncologists are faced with the challenge of irradiating tumors to a curative dose while limiting toxicity to healthy surrounding tissues. This can be achieved only with superior knowledge of radiologic anatomy and treatment planning. Educational resources designed to meet these specific needs are lacking. A web-based interactive module designed to improve residents’ knowledge and application of key anatomy concepts pertinent to radiotherapy treatment planning was developed, and its effectiveness was assessed. Methods and Materials: The module, based on gynecologic malignancies, was developed in collaboration with a multidisciplinary team of subject matter experts. Subsequently, a multicentre randomized controlled study was conducted to test the module’s effectiveness. Thirtysix radiation oncology residents participated in the study; 1920 were granted access to the module (intervention group), and 17 in the control group relied on traditional methods to acquire their knowledge. Pretests and posttests were administered to all participants. Statistical analysis was carried out using paired t test, analysis of variance, and post hoc tests. Results: The randomized control study revealed that the intervention group’s pretest and posttest mean scores were 35% and 52%, respectively, and those of the control group were 37% and 42%, respectively. The mean improvement in test scores was 17% (p < 0.05) for the intervention group and 5% (p Z not significant) for the control group. Retrospective pretest and posttest surveys showed a statistically significant change on all measured module objectives. Conclusions: The use of an interactive e-learning teaching module for radiation oncology is an effective method to improve the radiologic anatomy knowledge and treatment planning skills of radiation oncology residents. Ó 2012 Elsevier Inc. Keywords: E-learning, Medical education, Gynecology, Radiologic anatomy, Treatment planning
Reprint requests to: Joanne Alfieri, M.D., C.M., Montreal General Hospital, 1650 Cedar Avenue, D5-400, Montreal, QC, Canada H3G 1A4 Tel: (514) 934-8040; Fax: (514) 934-8392; E-mail: [email protected] Parts of this work were presented at CARO 2005 (Victoria, British Columbia, Canada), ASTRO 2005 (Denver, Colorado, USA), CARO 2009 (Quebec City, Quebec, Canada), ASTRO 2009 (Chicago, Illinois, USA), Canadian Conference on Medical Education (CCME) 2009 (Edmonton, Alberta, Canada), and An International Association for Medical Education (AMEE) 2009 (Malaga, Spain). Int J Radiation Oncol Biol Phys, Vol. 82, No. 3, pp. e573ee580, 2012 0360-3016/$ - see front matter Ó 2012 Elsevier Inc. All rights reserved. doi:10.1016/j.ijrobp.2011.07.002
The development phase of this study was supported by an unrestricted grant from Philips Medical Systems Canada. The evaluation phase was funded jointly by the Department of Radiation Oncology of the McGill University Health Centre and by the Centre for Medical Education, Faculty of Medicine, McGill University. Conflict of interest: none. AcknowledgmentdThe authors thank the participating residents and expert physicians who reviewed the module.
e574 Alfieri et al.
International Journal of Radiation Oncology Biology Physics
therefore, be an effective method to improve radiation oncology residents’ anatomic knowledge and treatment planning skills.
Introduction Radiation oncology residents are regularly faced with the challenge of designing radiation treatment plans that will adequately treat tumors and areas at risk of microscopic involvement while limiting toxicity to healthy surrounding tissues. To achieve this, superior knowledge of radiologic anatomy is crucial (1). Learning important and pertinent anatomy principles of radiotherapy requires the best available learning methods. Given the nature of the practice of radiation oncology, it is evident that electronically based learning tools should play a significant role in any curriculum for trainees. A literature search revealed that web-based teaching materials on this topic are limited (2, 3). Advancements in computer technology have resulted in major changes in health care and medical education (4). Multimedia is an exciting avenue that can be exploited for teaching purposes because it is both dynamic and interactive. The general trend toward exploring the Internet as a platform for instructional tools has been fueled by student facility with technologic devices combined with a broad range of user-friendly Internetebased education sites (5e8). Webbased learning tools are as effective as and complement traditional forms of instruction but are not a replacement for them (5, 9e12). Web-based learning tools are widely used by residents. Not only do they allow flexibility with respect to place, timing, and pace of learning, but also they are simple to modify and update (13, 14). Surprisingly little attention has been paid to the use of web-based instructional tools for the education of radiation oncology trainees. Recognizing the likely benefits of an electronically based learning module for radiation oncology residents, we designed and developed an interactive web-based site to facilitate learning of key radiologic anatomy concepts pertinent to radiotherapy treatment planning. We arbitrarily chose gynecologic malignancies as the subject content area. The main goals of the project were (1) to ensure that the module was pertinent to radiation oncology practice, thereby avoiding the creation of yet another anatomy atlas, and (2) to establish whether such a module would significantly enhance the knowledge of radiation oncology residents. In this article we report on the two phases of the project. The initial development phase of the project was dedicated to the creation of a module based on state-of-the-art information technology and to the collection of user feedback with regard to its content, pertinence, and user friendliness. The second, or evaluation, phase was designed to determine whether the web-based learning module, when used as an adjunct to the conventional learning tools available, enhances the knowledge and technical skills of radiation oncology residents in the realm of gynecologic malignancies.
Methods and Materials Development phase The development of the web-based module, which targeted both junior and senior radiation oncology residents, included the
collaboration of a multidisciplinary team of subject matter experts in anatomy, radiation oncology, radiology, instructional design, and multimedia programming. The module’s content was determined by three radiation oncologists with expertise in gynecologic malignancies and a radiation oncology resident. All of the expert radiation oncologists were experienced national certification examiners. The content of the gynecologic malignancies module was divided into four chapters, each representing different tumor sites in the female genital system: uterus, uterine cervix, vagina, and vulva. Learning objectives were established for each chapter and were subdivided into two sections: (1) normal anatomy, including blood supply, nerve supply, and lymphatic drainage, and (2) staging and practice cases with exercises. We compiled a catalog of original, normal, and pathologic radiologic images and clinically relevant, authentic, custom-designed graphics. To enhance learning, each section included a paragraph of written content combined with a visual image. All images were rendered interactive by hyperlinking action key words in the text. When clicked, the corresponding structures within the matched image would be highlighted. To ensure accuracy, the images and action words were reviewed by a collaborating radiologist who was not involved in developing the catalog. We collected relevant case histories and incorporated them into practice exercises appended to the conclusion of each section with a view to reinforcing learned concepts and enhancing clinical relevance. The practice exercises were devised to emulate the reality of radiation treatment planning in a clinical setting, including contouring gross tumor volumes (GTV), clinical target volumes (CTV), and organs at risk (OAR) on computed tomography scan slices and estimating treatment field boundaries on two-dimensional orthogonal radiologic images. Assistance with the technologic aspects of web-based instruction was provided by Instructional Multimedia Services of McGill University. Specialized consultants, including instructional designers and computer programmers, developed the web-based module using a rapid prototyping design, an adaptation to the Analysis, Design, Development, Implementation, Evaluation model (15, 16). Macromedia Director software was used for final authoring. The software is a system requirement to view the module; however, it is easily obtainable for free on the Internet. The programming was designed with built-in flexibility to support updates. The module’s design was based on theory-derived principles of teaching practice. It followed closely the recommendations put forth by Cook and Dupras on the design of effective web-based learning (17). These recommendations state that effective webbased teaching must adhere to established teaching principles such as using active learning, solving real-life problems, using current knowledge, and providing feedback. To ensure that the learner is an active contributor to the learning process, the module was created to be as interactive as possible, and the learning is self directed. To address the need for real-life problems, case-based scenarios were incorporated into the module to ensure that the learning activities would be relevant for radiation oncology residents. With respect to using current knowledge to build new learning, authentic tasks were created in the module that required
Volume 82 Number 3 2012 the resident to contour on radiologic images, as is required in the clinical setting. Finally, self assessment, immediate feedback, and reflection on the learner’s performance were built into the module. Scaffolding was achieved through the modeling of expert performance without the need of having the expert present. During the initial development phase, 10 McGill University radiation oncology residents volunteered to participate in a trial run of the web-based program for a first-impression survey on the pertinence and user friendliness of the module. Their valuable feedback was used to further refine the module. As a final step, content validity was established by the solicitation of 10 experts in the fields of radiation oncology, radiology, and gynecology oncology, from across Canada, to review the module for content inaccuracies, quality, and completeness.
Evaluation phase The second phase of our study involved formal evaluation of the module by use of three of the four outcome levels of Kirkpatrick’s framework (18) as described below:
Level one: assessment of learner satisfaction with the educational intervention This involved a courseware evaluation survey (CES) developed to evaluate participants’ satisfaction with five variables: content, user friendliness, interactivity, learner control, and graphic design. The CES was a 20-item questionnaire divided into four sections: content; graphics and media; navigation, organization, and instructional design; and overall impressions. The CES was administered to the residents in the intervention group only (n Z 1819).
Evaluation of E-Learning in radiation oncology e575 residency examinations. The pretest was divided into four distinct anatomic sections: uterus, cervix, vagina, and vulva. Each section was composed of eight questions. The first five questions were based on pathology, basic anatomy such as nerve supply and lymphatic drainage, and radiologic anatomy. The remaining three questions were case based and consisted of questions on staging; treatment planning, including contouring CTVs and drawing standard fields; and physics. The test and answer keys were validated by experts in the field of gynecologic radiation oncology. To ensure that each group had questions of equal difficulty, the pretest and posttest were the same, but the ordering of the questions was different. The tests were coded by an administrator and were scored anonymously by an independent third party using the detailed answer key. After the scores had been recorded, all tests were also scored anonymously by the principal investigator (J.A.) to ensure reliability. There was 100% concordance between the two scorers. Both test groups carried out their normal residency training over the course of the next 2 months. After this time had elapsed, both groups wrote the posttest. The intervention group was also asked to complete a CES. Statistical analysis was carried out using analysis of variance and post hoc tests, including trend analysis for the effect of residency year. The independent variables used were test time (repeated measure: pre and post), group (between groups: control vs. intervention), and residency year (between groups: 2, 3, 4, and 5). All analyses were conducted using an alpha level of 0.05. In addition, the following variables, which may have affected learning from the module, were examined: number of rotations completed in gynecologic malignancies, time spent on the module, and English as a mother tongue.
Level three: assessment of transfer of training Level two: student learning This involved a multicentre randomized controlled study using pre/post knowledge testing before and after residents used the module. Radiation oncology residents from the three postgraduate radiation oncology programs in Quebec were recruited by extending an invitation to participate in the study during their academic half day. All PGY-2 to PGY-5 radiation oncology residents were eligible for the study. The 10 residents who participated in the trial run conducted in the development phase, PGY-1 residents, and fellows were excluded. It was decided that a difference in mean test scores of 15% or more between the intervention and control groups would be a meaningful effect. To detect this difference in means at a significance level of 95% and with a power of 0.9, we estimated that a total sample size of 44 residents (22 per group) would be necessary. Alternatively, by use of a power of 0.8, a total sample size of 34 residents (17 per group) would be necessary. Therefore, our goal was to recruit between 34 and 44 residents in total. After institutional review board approval and participants’ informed consent were obtained, 36 residents were stratified by residency level and randomized to either the control group or the intervention group. The intervention group members completed the demographic profile survey and the pretest and then gained access to the module. Each received an e-mail containing an individualized, anonymous username and password. Logins and time spent on the module were tracked. The control group completed the demographic profile survey and pretest, but deleted for clarity were not given access to the module. The pretest, which was designed in conjunction with an expert involved with the national certification examinations, was based on questions typical of in-training
This involved a retrospective performance survey. Retrospective pre/post questionnaires are commonly used for quantitative analysis in medical education research (19). This survey was designed to detect changes in behavior and long-term retention of skills learned from an intervention. All participants completed self ratings of perceived changes of their knowledge and planning skills as a result of the intervention (20). The retrospective performance survey was sent out by e-mail 1 month after the end of the study to the residents in the intervention group. It addressed the various learning objectives that are stated in the module, for example, “Knowledge about normal female pelvic anatomy” and “Skill in outlining CTV structures on computed tomography or magnetic resonance images.” Residents were asked to self assess their knowledge and skills before and after having access to the module.
Level four: outcomes This would have examined the effects on the environment resulting from the application of training. In our case, such an assessment would have required a very large sample size with extensive review of patient records to assess the results at the level of a change in resident practices and improvement in patient outcomes. Such an intervention was beyond the scope of this project.
Results Development phase We successfully produced an interactive, web-based, radiologic anatomy learning module tailored to the needs of radiation
e576 Alfieri et al.
International Journal of Radiation Oncology Biology Physics
Fig. 1. Screen capture from the web-based module. Anatomic concepts pertinent to radiation oncology residents, such as lymphatic drainage, are demonstrated on different radiologic studies, including radiographs, computed tomography, and magnetic resonance imaging (MRI), to make the module as relevant as possible.
oncology residents. This proved to be feasible with the current available techniques. Sample screen captures from the different sections and chapters can be seen in Figs. 1 and 2. Early feedback from the 10 radiation oncology residents who reviewed the module in the trial run of the development phase was largely positive. Nine of the 10 thought that it provided a high degree of learner control and described it as being interactive and easily navigable. These comments were instrumental in the identification of a flaw in the manner in which GTV, CTV, and OAR volumes were being identified. Initially a drag-and-drop label was planned, but many residents found this unrealistic. In response, a pen tool was created to mimic the manner in which these volumes are contoured by radiation oncologists on current treatment planning software. Overall, the early feedback demonstrated enthusiasm for the learning module.
Evaluation phase The results of the demographic survey from the evaluation phase of the module revealed that the two groups were well balanced as a result of the randomization (Table 1).
Also evident from the demographic profile survey was that the majority of participants are already using the internet to learn about anatomy (78%) and treatment planning (86%). An even higher proportion of participants stated that they had a preference for an online learning approach for anatomy (94%) and treatment planning (92%), as opposed to traditional textbook resources. Login tracking revealed that the mean number of logins per resident was 4.3 (range, 2e10) and the mean time spent on the module was 2.02 hours (range, 1.25e3.00 hours). With respect to the pretest and posttest scores, the difference in mean scores between pretest and posttest for the control group was an improvement of 5%. The difference in mean scores between pretest and posttest for the intervention group was an improvement of 17%. There was an absolute difference between the two groups of 12% (Fig. 3). Analysis of posttest vs. pretest scores run independently for each group showed the difference to be highly statistically significant in the intervention group (p < 0.0001) compared with the control group (p Z 0.0523) according to a paired two-tailed t test at a 0.05 significance level. A key finding from the analysis of variance was an interaction of time (pretest vs. posttest) and group (control vs. intervention) such that the intervention group’s increase in test scores from pretest to posttest
Volume 82 Number 3 2012
Evaluation of E-Learning in radiation oncology e577
Fig. 2. Screen capture from the web-based module. Practice exercises strengthen concepts learned in the previous sections and target crucial skills such as, in this case, contouring the gross tumor volume (GTV) on a computed tomography (CT) image from a simulation scan for a patient with cervical cancer. was statistically significant (p Z 0.007), but the control group’s increase in test scores was not. Also, there was a trend whereby increasing residency year generally meant higher scores regardless of the group or test time. The median differences in scores between pretest and posttest for the different demographic variables is shown in Table 2. The feedback obtained from the CES was overwhelmingly positive, with a mean of 3.3 out of 4 for each of the four variables. With respect to the perceived difficulty level of the web-based module, the survey revealed that 0%, 18%, 64%, and 18% of participants found the material very difficult, difficult, average, and easy, respectively. Interestingly, in the overall impressions category, questions such as “I would use a similar module for different anatomic sites” and “I would recommend this module to other residents” had the largest number of responses in the strongly agree category, 59% and 47%, respectively. When the agree and strongly agree categories are grouped together, the majority of responders (88%) can be found. With respect to the retrospective pretest vs. posttest performance survey, the differences in scores for all objectives reached statistical significance (Table 3).
Discussion Development phase The feedback we received during the design and development of the web-based module was valuable; the residents highlighted that interactivity and pertinence were critical to the learning process. Adult learning principles indicate that there is a higher chance of retention of learned material if learners perceive that material to be directly related to their perceived future needs (21).
Evaluation phase In this information age, it is not surprising that residents expressed a preference for online materials with respect to anatomy and treatment planning. We adopted an electronic approach mostly for its flexibility and the fact that, as opposed to textbooks, the content could be easily updated. The significance of a mean pretest score of 36% across all residents likely indicates that it was a test with a high difficulty
International Journal of Radiation Oncology Biology Physics
e578 Alfieri et al. Table 1
Table 2
Demographic profile survey results Characteristic
Control group
Intervention group
n Male Female English mother tongue PGY2 PGY3 PGY4 PGY5 Gynecologic experience (mo)
17 6 11 5 3 4 5 5 3.1
19 9 10 4 3 6 4 6 2.4
Median difference from pretest to posttest
Characteristic
Control group (%)
Intervention group (%)
Male Female English mother tongue PGY2 PGY3 PGY4 PGY5
12 6 6 3 12 3 8
9 16 20 19 9 26 11
Abbreviation: 1PGY Z Post-graduate year.
Abbreviation: 1PGY Z Post-graduate year.
level; this is supported by the fact that the mean score by experts in the field was only 50%. Seeing that the pretest and posttest contained the same questions in a different order, the test was intentionally designed to be difficult in an attempt to avoid the ceiling effectda measurement problem that occurs when test items are not challenging enough and thus, high scoring individuals’ improvement cannot be captured on subsequent testing. The test questions were based on typical residency examinations and were written with an expert involved in national certification examinations. Given that the pretest and posttest questions were based on current evaluation methods for radiation oncology residents, the low expert test scores may indicate misalignment across the curriculum, feedback, and evaluation triad (22). In other words, the issue may be whether the knowledge that is evaluated in national certification examinations is relevant to clinical practice. The results do show that residents’ knowledge increased to approximately the level of expert knowledge with the intervention. However, the low expert scores may have identified a weakness in radiation oncology education as a whole. It seems that as educators, we should perhaps be focusing our efforts on tapping into what “expert” clinically relevant knowledge actually is. Only when this is more accurately understood can we subsequently devise better ways to evaluate this type of knowledge in our trainees. The significant increase in scores from pretest to posttest in the intervention group clearly indicates that use of the module increased the knowledge and skills of residents who had access. Also, the CES survey results revealed an overall enthusiasm for the development of additional modules. These data support the implementation of webbased instruction as an adjunct to the current traditional curriculum
Fig. 3. Pretest and posttest mean score results for control vs. intervention groups.
(e.g., textbooks, lectures) for radiation oncology residents, justifying the creation of additional modules for each anatomic site. In fact, the success of the gynecologic module has allowed us to secure funding for the creation of a head-and-neck module, which is currently under way. Although not statistically significant, there was a trend indicating that the improvement in test scores was greater for junior residents than for senior residents (data not shown). With this taken into consideration, it may be advantageous for residents to complete these modules as an introduction to each anatomic site before commencing their clinical rotation focusing on that site. Web-based instruction is also an efficient way to enhance the postgraduate curriculum, inasmuch as cognitive scaffolding is achieved through the modeling of expert performance without the need for the expert to be present (22). Retrospective performance survey results demonstrated longterm retention and self-perceived changes in residents’ knowledge. This survey was administered 1 month after the posttest, and although follow-up surveys are needed to demonstrate even longer-term changes, the fact that all objectives showed statistically significant changes suggests that the module’s content and design were thought to be pertinent and beneficial by the residents who had access to it. The limitations of this study include confounders common to all pretest and posttest designs, which were likely reduced by randomization, including (1) history effects (i.e., one group having richer learning experiences than the other), (2) maturation effects (i.e., self-directed learners will read around the subject after a difficult test), (3) testing effects (i.e., whether testing affects both groups differently), and (4) repeated measurement (i.e., the pretest affects the posttest result). The last effect could have been reduced further by using a Solomon four-group design; however, there were not enough residents across the three Quebec radiation oncology residency programs to enable us to do so. Perhaps, if the study were to be repeated on a larger scale, this design would be beneficial. The small sample size can be considered a limitation, but it is comparable to other evaluation studies of educational interventions found in the literature (4, 10, 23). We also cannot rule out the Hawthorne effect, a reaction whereby subjects improve the performance that is being experimentally measured simply because they are being studied and not in response to the experimental intervention (24). Although it may be tempting to compare the present intervention with an existing and comparable intervention from a different mediumda textbook chapter on gynecologic malignancies, for exampledCook advises against such mediacomparative research, stating that it “is logically impossible because there are no valid comparison groups” (25).
Volume 82 Number 3 2012 Table 3
Evaluation of E-Learning in radiation oncology e579
Retrospective performance survey (n Z 12) Objective
Before
After
Change (p < 0.05)
Knowledge about normal female pelvic anatomy Knowledge about blood supply Knowledge about nerve supply Knowledge about lymphatic drainage Knowledge about normal radiology Knowledge about the organs at risk in treatment planning Skill in outlining CTV structures on CT or MRI images Skill in outlining gynecological tumors on imaging Skill in drawing treatment fields according to bony landmarks in traditional planning at the standard simulator Knowledge of FIGO staging Skill in designing optimal treatment plans for typical cases
2.5 2.3 1.9 3.1 2.8 3.3 2.8 2.8 3.0
3.8 3.3 3.0 4.1 3.9 4.3 3.8 3.8 3.9
1.3 1.0 1.1 1.0 1.1 1.0 1.0 1.0 0.9
3.0 2.6
3.8 3.7
0.8 1.1
Results indicate a retrospectively self-rated score representing perception of knowledge and skills before and after access to the module and also the absolute change in scores. Abbreviations: CTV Z clinical target volume; CT Z computed tomography; MRI Z magnetic resonance imaging; FIGO Z International Federation of Gynecology and Obstetrics; 1 Z very poor; 2 Z below average; 3 Z average; 4 Z above average; 5 Z excellent. 12 of the 19 residents (63.2%) from the intervention group responded to the survey
Most importantly, this study simply measured outcomes. Without a qualitative study, we cannot unequivocally make conclusions about what aspects of the software contributed most to resident learning. A variety of approaches, of which protocol analysis is one, could have been used to uncover how residents learn from this educational intervention (26). As stated by Cook, we need to move from doing research to learn whether web-based learning should be used, to doing research to learn when and how it should be used (27).
Conclusions The use of multimedia provides a modern and dynamic approach to medical education. The creation of this learning module proves that the development of an interactive, web-based teaching tool is feasible with current techniques. It is currently being used as a model on which to build future modules of other disease sites. The formal, interinstitutional evaluation of the learning module confirmed the hypothesis that the interactive nature and the high degree of learner control would promote self-directed learning. It was shown to have high learner satisfaction rates and to positively affect learning outcomes by enhancing the radiologic anatomy knowledge and technical skills of residents in the domain of gynecologic malignancies. This provides further evidence of the importance of e-learning and will set higher standards for electronic medical educational material in the future. Long-term changes in behavior were difficult to assess from the measures used in this study. Further evaluative research is required to examine the characteristics of effective web-based learning environments that foster knowledge transfer to resident practices and improve patient outcomes.
References 1. Pelizzari CA, Grzeszczuk R, Chen G, et al. Volumetric visualization of anatomy for treatment planning. Int J Radiat Oncol Biol Phys 1996; 34:205e211. 2. Campeau M, Lasalle S, Lambert C, et al. Self-directed learning module for radiation therapy treatment planning [Abstract]. J Clin Oncol 2007;25(Suppl):18S.
3. Valentini V, Dinapoli N, Nori S, et al. An application of visible human database in radiotherapy: Tutorial for image guided external radiotherapy (TIGER). Radiother Oncol 2004;70:165e169. 4. Nicholson D, Chalk C, Funnell RJ, et al. Can virtual reality improve anatomy education? A randomized controlled study of a computergenerated three-dimensional anatomical ear model. Med Educ 2006; 40:1081e1087. 5. Nattestad A, Attstrom R, Mattheos N, et al. Web-based interactive learning programmes. Eur J Dent Educ 2002;6(Suppl.3):127e137. 6. Cook DA, Garside S, Levinson AJ, et al. What do we mean by webbased learning? A systematic review of the variability of interventions. Med Educ 2010;44:765e774. 7. Merk M, Knuechel R, Perez-Bouza A. Web-based virtual microscopy at the RWTH Aachen University: Didactic concept, methods and analysis of acceptance by the students. Ann Anat 2010;192:383e387. 8. Crossingham JL, Jenkinson J, Woolridge N, et al. Interpreting threedimensional structures from two-dimensional images: A web-based interactive 3D teaching model of surgical liver anatomy. HPB (Oxford) 2009;11:523e528. 9. Cohen HB, Walker SR, Tenenbaum HC. Interdisciplinary, web-based, self-study, interactive programs in the dental undergraduate program: A pilot. J Dent Educ 2003;67:661e667. 10. Fukuchi SG, Offutt LA, Sacks J, et al. Teaching a multidisciplinary approach to cancer treatment during surgical clerkship via an interactive board game. Am J Surg 2000;179:337e340. 11. Nattestad A, Attstrom R. Introduction to theme 4: The virtual potential. Eur J Dent Educ 2002;6(Suppl3):125e126. 12. Chumley-Jones HS, Dobbie A, Alford CL. Web-based learning: Sound educational method or hype? A review of the evaluation literature. Acad Med 2002;77(10Suppl):S86eS93. 13. Shaffer K, Small JE. Blended learning in medical education: Use of an integrated approach with web-based small group modules and didactic instruction for teaching radiologic anatomy. Acad Radiol 2004;11: 1059e1070. 14. Reid JR, Goske MJ, Hewson MG, et al. Computers in radiology: Creating an international comprehensive web-based curriculum in pediatric radiology. AJR Am J Roentgenol 2004;182:797e801. 15. Weston C, Gandell T, McAlpine L, et al. Designing instruction for the context of on-line learning. The Internet and Higher Education 1999;2: 35e44. 16. Dick W, Carey L. The systematic design of instruction. 4th ed. New York: Harper Collins College Publishers; 1996. 17. Cook D, Dupras D. A practical guide to developing effective webbased learning. J Gen Intern Med 2004;19:698e707.
e580 Alfieri et al. 18. Kirkpatrick DL. Evaluating training programs: The four levels. San Francisco: Berrett-Koehler; 2004. 19. Skeff KM, Stratos GA, Bergen MR. Evaluation of a medical faculty development program: A comparison of traditional pre/post and retrospective pre/post self-assessment ratings. Eval Health Prof 1992;15: 350e366. 20. Curran V. An eclectic model for evaluating web-based continuing medical education courseware systems. Eval Health Prof 2000;23:318e347. 21. Letterie GS. Medical education as a science: The quality of evidence for computer-assisted instruction. Am J Obstet Gynecol 2003;188: 849e853. 22. Pellegrino J. Understanding how students learn and inferring what they know: Implications for the design of curriculum, instruction and assessment. In: Smith MJ, editor. NSF K-12 Mathematics and science curriculum and implementation centers conference proceedings.
International Journal of Radiation Oncology Biology Physics
23. 24. 25. 26.
27.
Washington, DC: National Science Foundation and American Geological Institute; 2002. p. 76e92. Hariri S, Rawn C, Srivastava S, et al. Evaluation of a surgical simulator for learning clinical anatomy. Med Educ 2004;38:896e902. McCarney R, Warner J, Iliffe S, et al. The Hawthorne Effect: A randomised, controlled trial. BMC Med Res Methodol 2007;7:30e38. Cook D. The research we still are not doing: An agenda for the study of computer-based learning. Acad Med 2005;80:541e548. National Research Council (NRC). Implications of the new foundations for assessment design. In: Pellegrino JW, Chudowsky N, Glaser R, editors. Knowing what students know: The science and design of educational assessment. Washington, DC: National Academy Press; 2001. p. 177e219. Cook D. Where are we with web-based learning in medical education? Med Teach 2006;28:594e598.
Radiation Oncology biology
physics
www.redjournal.org
Radiation Oncology Education
Development and Impact Evaluation of an E-Learning Radiation Oncology Module Joanne Alfieri, M.D., C.M.,* Lorraine Portelance, M.D.,* Luis Souhami, M.D.,* Yvonne Steinert, Ph.D.,y Peter McLeod, M.D.,y Fleure Gallant, M.D., C.M.,* and Giovanni Artho, M.D.z Departments of *Radiation Oncology and zRadiology, McGill University Health Centre, and yCentre for Medical Education, McGill University, Montreal, QC, Canada Received Aug 11, 2010, and in revised form Jun 30, 2011. Accepted for publication Jul 6, 2011
Summary Radiographic anatomy is key to radiation treatment planning. An interactive webbased module was developed to improve residents’ knowledge in this domain and was tested in a multicenter, randomized controlled study. Those residents having access to the module showed a significant improvement in test scores when compared to the group of residents without. A retrospective performance survey showed long-term retention and self- perceived changes in residents’ knowledge. Interactive e-learning teaching modules may,
Purpose: Radiation oncologists are faced with the challenge of irradiating tumors to a curative dose while limiting toxicity to healthy surrounding tissues. This can be achieved only with superior knowledge of radiologic anatomy and treatment planning. Educational resources designed to meet these specific needs are lacking. A web-based interactive module designed to improve residents’ knowledge and application of key anatomy concepts pertinent to radiotherapy treatment planning was developed, and its effectiveness was assessed. Methods and Materials: The module, based on gynecologic malignancies, was developed in collaboration with a multidisciplinary team of subject matter experts. Subsequently, a multicentre randomized controlled study was conducted to test the module’s effectiveness. Thirtysix radiation oncology residents participated in the study; 1920 were granted access to the module (intervention group), and 17 in the control group relied on traditional methods to acquire their knowledge. Pretests and posttests were administered to all participants. Statistical analysis was carried out using paired t test, analysis of variance, and post hoc tests. Results: The randomized control study revealed that the intervention group’s pretest and posttest mean scores were 35% and 52%, respectively, and those of the control group were 37% and 42%, respectively. The mean improvement in test scores was 17% (p < 0.05) for the intervention group and 5% (p Z not significant) for the control group. Retrospective pretest and posttest surveys showed a statistically significant change on all measured module objectives. Conclusions: The use of an interactive e-learning teaching module for radiation oncology is an effective method to improve the radiologic anatomy knowledge and treatment planning skills of radiation oncology residents. Ó 2012 Elsevier Inc. Keywords: E-learning, Medical education, Gynecology, Radiologic anatomy, Treatment planning
Reprint requests to: Joanne Alfieri, M.D., C.M., Montreal General Hospital, 1650 Cedar Avenue, D5-400, Montreal, QC, Canada H3G 1A4 Tel: (514) 934-8040; Fax: (514) 934-8392; E-mail: [email protected] Parts of this work were presented at CARO 2005 (Victoria, British Columbia, Canada), ASTRO 2005 (Denver, Colorado, USA), CARO 2009 (Quebec City, Quebec, Canada), ASTRO 2009 (Chicago, Illinois, USA), Canadian Conference on Medical Education (CCME) 2009 (Edmonton, Alberta, Canada), and An International Association for Medical Education (AMEE) 2009 (Malaga, Spain). Int J Radiation Oncol Biol Phys, Vol. 82, No. 3, pp. e573ee580, 2012 0360-3016/$ - see front matter Ó 2012 Elsevier Inc. All rights reserved. doi:10.1016/j.ijrobp.2011.07.002
The development phase of this study was supported by an unrestricted grant from Philips Medical Systems Canada. The evaluation phase was funded jointly by the Department of Radiation Oncology of the McGill University Health Centre and by the Centre for Medical Education, Faculty of Medicine, McGill University. Conflict of interest: none. AcknowledgmentdThe authors thank the participating residents and expert physicians who reviewed the module.
e574 Alfieri et al.
International Journal of Radiation Oncology Biology Physics
therefore, be an effective method to improve radiation oncology residents’ anatomic knowledge and treatment planning skills.
Introduction Radiation oncology residents are regularly faced with the challenge of designing radiation treatment plans that will adequately treat tumors and areas at risk of microscopic involvement while limiting toxicity to healthy surrounding tissues. To achieve this, superior knowledge of radiologic anatomy is crucial (1). Learning important and pertinent anatomy principles of radiotherapy requires the best available learning methods. Given the nature of the practice of radiation oncology, it is evident that electronically based learning tools should play a significant role in any curriculum for trainees. A literature search revealed that web-based teaching materials on this topic are limited (2, 3). Advancements in computer technology have resulted in major changes in health care and medical education (4). Multimedia is an exciting avenue that can be exploited for teaching purposes because it is both dynamic and interactive. The general trend toward exploring the Internet as a platform for instructional tools has been fueled by student facility with technologic devices combined with a broad range of user-friendly Internetebased education sites (5e8). Webbased learning tools are as effective as and complement traditional forms of instruction but are not a replacement for them (5, 9e12). Web-based learning tools are widely used by residents. Not only do they allow flexibility with respect to place, timing, and pace of learning, but also they are simple to modify and update (13, 14). Surprisingly little attention has been paid to the use of web-based instructional tools for the education of radiation oncology trainees. Recognizing the likely benefits of an electronically based learning module for radiation oncology residents, we designed and developed an interactive web-based site to facilitate learning of key radiologic anatomy concepts pertinent to radiotherapy treatment planning. We arbitrarily chose gynecologic malignancies as the subject content area. The main goals of the project were (1) to ensure that the module was pertinent to radiation oncology practice, thereby avoiding the creation of yet another anatomy atlas, and (2) to establish whether such a module would significantly enhance the knowledge of radiation oncology residents. In this article we report on the two phases of the project. The initial development phase of the project was dedicated to the creation of a module based on state-of-the-art information technology and to the collection of user feedback with regard to its content, pertinence, and user friendliness. The second, or evaluation, phase was designed to determine whether the web-based learning module, when used as an adjunct to the conventional learning tools available, enhances the knowledge and technical skills of radiation oncology residents in the realm of gynecologic malignancies.
Methods and Materials Development phase The development of the web-based module, which targeted both junior and senior radiation oncology residents, included the
collaboration of a multidisciplinary team of subject matter experts in anatomy, radiation oncology, radiology, instructional design, and multimedia programming. The module’s content was determined by three radiation oncologists with expertise in gynecologic malignancies and a radiation oncology resident. All of the expert radiation oncologists were experienced national certification examiners. The content of the gynecologic malignancies module was divided into four chapters, each representing different tumor sites in the female genital system: uterus, uterine cervix, vagina, and vulva. Learning objectives were established for each chapter and were subdivided into two sections: (1) normal anatomy, including blood supply, nerve supply, and lymphatic drainage, and (2) staging and practice cases with exercises. We compiled a catalog of original, normal, and pathologic radiologic images and clinically relevant, authentic, custom-designed graphics. To enhance learning, each section included a paragraph of written content combined with a visual image. All images were rendered interactive by hyperlinking action key words in the text. When clicked, the corresponding structures within the matched image would be highlighted. To ensure accuracy, the images and action words were reviewed by a collaborating radiologist who was not involved in developing the catalog. We collected relevant case histories and incorporated them into practice exercises appended to the conclusion of each section with a view to reinforcing learned concepts and enhancing clinical relevance. The practice exercises were devised to emulate the reality of radiation treatment planning in a clinical setting, including contouring gross tumor volumes (GTV), clinical target volumes (CTV), and organs at risk (OAR) on computed tomography scan slices and estimating treatment field boundaries on two-dimensional orthogonal radiologic images. Assistance with the technologic aspects of web-based instruction was provided by Instructional Multimedia Services of McGill University. Specialized consultants, including instructional designers and computer programmers, developed the web-based module using a rapid prototyping design, an adaptation to the Analysis, Design, Development, Implementation, Evaluation model (15, 16). Macromedia Director software was used for final authoring. The software is a system requirement to view the module; however, it is easily obtainable for free on the Internet. The programming was designed with built-in flexibility to support updates. The module’s design was based on theory-derived principles of teaching practice. It followed closely the recommendations put forth by Cook and Dupras on the design of effective web-based learning (17). These recommendations state that effective webbased teaching must adhere to established teaching principles such as using active learning, solving real-life problems, using current knowledge, and providing feedback. To ensure that the learner is an active contributor to the learning process, the module was created to be as interactive as possible, and the learning is self directed. To address the need for real-life problems, case-based scenarios were incorporated into the module to ensure that the learning activities would be relevant for radiation oncology residents. With respect to using current knowledge to build new learning, authentic tasks were created in the module that required
Volume 82 Number 3 2012 the resident to contour on radiologic images, as is required in the clinical setting. Finally, self assessment, immediate feedback, and reflection on the learner’s performance were built into the module. Scaffolding was achieved through the modeling of expert performance without the need of having the expert present. During the initial development phase, 10 McGill University radiation oncology residents volunteered to participate in a trial run of the web-based program for a first-impression survey on the pertinence and user friendliness of the module. Their valuable feedback was used to further refine the module. As a final step, content validity was established by the solicitation of 10 experts in the fields of radiation oncology, radiology, and gynecology oncology, from across Canada, to review the module for content inaccuracies, quality, and completeness.
Evaluation phase The second phase of our study involved formal evaluation of the module by use of three of the four outcome levels of Kirkpatrick’s framework (18) as described below:
Level one: assessment of learner satisfaction with the educational intervention This involved a courseware evaluation survey (CES) developed to evaluate participants’ satisfaction with five variables: content, user friendliness, interactivity, learner control, and graphic design. The CES was a 20-item questionnaire divided into four sections: content; graphics and media; navigation, organization, and instructional design; and overall impressions. The CES was administered to the residents in the intervention group only (n Z 1819).
Evaluation of E-Learning in radiation oncology e575 residency examinations. The pretest was divided into four distinct anatomic sections: uterus, cervix, vagina, and vulva. Each section was composed of eight questions. The first five questions were based on pathology, basic anatomy such as nerve supply and lymphatic drainage, and radiologic anatomy. The remaining three questions were case based and consisted of questions on staging; treatment planning, including contouring CTVs and drawing standard fields; and physics. The test and answer keys were validated by experts in the field of gynecologic radiation oncology. To ensure that each group had questions of equal difficulty, the pretest and posttest were the same, but the ordering of the questions was different. The tests were coded by an administrator and were scored anonymously by an independent third party using the detailed answer key. After the scores had been recorded, all tests were also scored anonymously by the principal investigator (J.A.) to ensure reliability. There was 100% concordance between the two scorers. Both test groups carried out their normal residency training over the course of the next 2 months. After this time had elapsed, both groups wrote the posttest. The intervention group was also asked to complete a CES. Statistical analysis was carried out using analysis of variance and post hoc tests, including trend analysis for the effect of residency year. The independent variables used were test time (repeated measure: pre and post), group (between groups: control vs. intervention), and residency year (between groups: 2, 3, 4, and 5). All analyses were conducted using an alpha level of 0.05. In addition, the following variables, which may have affected learning from the module, were examined: number of rotations completed in gynecologic malignancies, time spent on the module, and English as a mother tongue.
Level three: assessment of transfer of training Level two: student learning This involved a multicentre randomized controlled study using pre/post knowledge testing before and after residents used the module. Radiation oncology residents from the three postgraduate radiation oncology programs in Quebec were recruited by extending an invitation to participate in the study during their academic half day. All PGY-2 to PGY-5 radiation oncology residents were eligible for the study. The 10 residents who participated in the trial run conducted in the development phase, PGY-1 residents, and fellows were excluded. It was decided that a difference in mean test scores of 15% or more between the intervention and control groups would be a meaningful effect. To detect this difference in means at a significance level of 95% and with a power of 0.9, we estimated that a total sample size of 44 residents (22 per group) would be necessary. Alternatively, by use of a power of 0.8, a total sample size of 34 residents (17 per group) would be necessary. Therefore, our goal was to recruit between 34 and 44 residents in total. After institutional review board approval and participants’ informed consent were obtained, 36 residents were stratified by residency level and randomized to either the control group or the intervention group. The intervention group members completed the demographic profile survey and the pretest and then gained access to the module. Each received an e-mail containing an individualized, anonymous username and password. Logins and time spent on the module were tracked. The control group completed the demographic profile survey and pretest, but deleted for clarity were not given access to the module. The pretest, which was designed in conjunction with an expert involved with the national certification examinations, was based on questions typical of in-training
This involved a retrospective performance survey. Retrospective pre/post questionnaires are commonly used for quantitative analysis in medical education research (19). This survey was designed to detect changes in behavior and long-term retention of skills learned from an intervention. All participants completed self ratings of perceived changes of their knowledge and planning skills as a result of the intervention (20). The retrospective performance survey was sent out by e-mail 1 month after the end of the study to the residents in the intervention group. It addressed the various learning objectives that are stated in the module, for example, “Knowledge about normal female pelvic anatomy” and “Skill in outlining CTV structures on computed tomography or magnetic resonance images.” Residents were asked to self assess their knowledge and skills before and after having access to the module.
Level four: outcomes This would have examined the effects on the environment resulting from the application of training. In our case, such an assessment would have required a very large sample size with extensive review of patient records to assess the results at the level of a change in resident practices and improvement in patient outcomes. Such an intervention was beyond the scope of this project.
Results Development phase We successfully produced an interactive, web-based, radiologic anatomy learning module tailored to the needs of radiation
e576 Alfieri et al.
International Journal of Radiation Oncology Biology Physics
Fig. 1. Screen capture from the web-based module. Anatomic concepts pertinent to radiation oncology residents, such as lymphatic drainage, are demonstrated on different radiologic studies, including radiographs, computed tomography, and magnetic resonance imaging (MRI), to make the module as relevant as possible.
oncology residents. This proved to be feasible with the current available techniques. Sample screen captures from the different sections and chapters can be seen in Figs. 1 and 2. Early feedback from the 10 radiation oncology residents who reviewed the module in the trial run of the development phase was largely positive. Nine of the 10 thought that it provided a high degree of learner control and described it as being interactive and easily navigable. These comments were instrumental in the identification of a flaw in the manner in which GTV, CTV, and OAR volumes were being identified. Initially a drag-and-drop label was planned, but many residents found this unrealistic. In response, a pen tool was created to mimic the manner in which these volumes are contoured by radiation oncologists on current treatment planning software. Overall, the early feedback demonstrated enthusiasm for the learning module.
Evaluation phase The results of the demographic survey from the evaluation phase of the module revealed that the two groups were well balanced as a result of the randomization (Table 1).
Also evident from the demographic profile survey was that the majority of participants are already using the internet to learn about anatomy (78%) and treatment planning (86%). An even higher proportion of participants stated that they had a preference for an online learning approach for anatomy (94%) and treatment planning (92%), as opposed to traditional textbook resources. Login tracking revealed that the mean number of logins per resident was 4.3 (range, 2e10) and the mean time spent on the module was 2.02 hours (range, 1.25e3.00 hours). With respect to the pretest and posttest scores, the difference in mean scores between pretest and posttest for the control group was an improvement of 5%. The difference in mean scores between pretest and posttest for the intervention group was an improvement of 17%. There was an absolute difference between the two groups of 12% (Fig. 3). Analysis of posttest vs. pretest scores run independently for each group showed the difference to be highly statistically significant in the intervention group (p < 0.0001) compared with the control group (p Z 0.0523) according to a paired two-tailed t test at a 0.05 significance level. A key finding from the analysis of variance was an interaction of time (pretest vs. posttest) and group (control vs. intervention) such that the intervention group’s increase in test scores from pretest to posttest
Volume 82 Number 3 2012
Evaluation of E-Learning in radiation oncology e577
Fig. 2. Screen capture from the web-based module. Practice exercises strengthen concepts learned in the previous sections and target crucial skills such as, in this case, contouring the gross tumor volume (GTV) on a computed tomography (CT) image from a simulation scan for a patient with cervical cancer. was statistically significant (p Z 0.007), but the control group’s increase in test scores was not. Also, there was a trend whereby increasing residency year generally meant higher scores regardless of the group or test time. The median differences in scores between pretest and posttest for the different demographic variables is shown in Table 2. The feedback obtained from the CES was overwhelmingly positive, with a mean of 3.3 out of 4 for each of the four variables. With respect to the perceived difficulty level of the web-based module, the survey revealed that 0%, 18%, 64%, and 18% of participants found the material very difficult, difficult, average, and easy, respectively. Interestingly, in the overall impressions category, questions such as “I would use a similar module for different anatomic sites” and “I would recommend this module to other residents” had the largest number of responses in the strongly agree category, 59% and 47%, respectively. When the agree and strongly agree categories are grouped together, the majority of responders (88%) can be found. With respect to the retrospective pretest vs. posttest performance survey, the differences in scores for all objectives reached statistical significance (Table 3).
Discussion Development phase The feedback we received during the design and development of the web-based module was valuable; the residents highlighted that interactivity and pertinence were critical to the learning process. Adult learning principles indicate that there is a higher chance of retention of learned material if learners perceive that material to be directly related to their perceived future needs (21).
Evaluation phase In this information age, it is not surprising that residents expressed a preference for online materials with respect to anatomy and treatment planning. We adopted an electronic approach mostly for its flexibility and the fact that, as opposed to textbooks, the content could be easily updated. The significance of a mean pretest score of 36% across all residents likely indicates that it was a test with a high difficulty
International Journal of Radiation Oncology Biology Physics
e578 Alfieri et al. Table 1
Table 2
Demographic profile survey results Characteristic
Control group
Intervention group
n Male Female English mother tongue PGY2 PGY3 PGY4 PGY5 Gynecologic experience (mo)
17 6 11 5 3 4 5 5 3.1
19 9 10 4 3 6 4 6 2.4
Median difference from pretest to posttest
Characteristic
Control group (%)
Intervention group (%)
Male Female English mother tongue PGY2 PGY3 PGY4 PGY5
12 6 6 3 12 3 8
9 16 20 19 9 26 11
Abbreviation: 1PGY Z Post-graduate year.
Abbreviation: 1PGY Z Post-graduate year.
level; this is supported by the fact that the mean score by experts in the field was only 50%. Seeing that the pretest and posttest contained the same questions in a different order, the test was intentionally designed to be difficult in an attempt to avoid the ceiling effectda measurement problem that occurs when test items are not challenging enough and thus, high scoring individuals’ improvement cannot be captured on subsequent testing. The test questions were based on typical residency examinations and were written with an expert involved in national certification examinations. Given that the pretest and posttest questions were based on current evaluation methods for radiation oncology residents, the low expert test scores may indicate misalignment across the curriculum, feedback, and evaluation triad (22). In other words, the issue may be whether the knowledge that is evaluated in national certification examinations is relevant to clinical practice. The results do show that residents’ knowledge increased to approximately the level of expert knowledge with the intervention. However, the low expert scores may have identified a weakness in radiation oncology education as a whole. It seems that as educators, we should perhaps be focusing our efforts on tapping into what “expert” clinically relevant knowledge actually is. Only when this is more accurately understood can we subsequently devise better ways to evaluate this type of knowledge in our trainees. The significant increase in scores from pretest to posttest in the intervention group clearly indicates that use of the module increased the knowledge and skills of residents who had access. Also, the CES survey results revealed an overall enthusiasm for the development of additional modules. These data support the implementation of webbased instruction as an adjunct to the current traditional curriculum
Fig. 3. Pretest and posttest mean score results for control vs. intervention groups.
(e.g., textbooks, lectures) for radiation oncology residents, justifying the creation of additional modules for each anatomic site. In fact, the success of the gynecologic module has allowed us to secure funding for the creation of a head-and-neck module, which is currently under way. Although not statistically significant, there was a trend indicating that the improvement in test scores was greater for junior residents than for senior residents (data not shown). With this taken into consideration, it may be advantageous for residents to complete these modules as an introduction to each anatomic site before commencing their clinical rotation focusing on that site. Web-based instruction is also an efficient way to enhance the postgraduate curriculum, inasmuch as cognitive scaffolding is achieved through the modeling of expert performance without the need for the expert to be present (22). Retrospective performance survey results demonstrated longterm retention and self-perceived changes in residents’ knowledge. This survey was administered 1 month after the posttest, and although follow-up surveys are needed to demonstrate even longer-term changes, the fact that all objectives showed statistically significant changes suggests that the module’s content and design were thought to be pertinent and beneficial by the residents who had access to it. The limitations of this study include confounders common to all pretest and posttest designs, which were likely reduced by randomization, including (1) history effects (i.e., one group having richer learning experiences than the other), (2) maturation effects (i.e., self-directed learners will read around the subject after a difficult test), (3) testing effects (i.e., whether testing affects both groups differently), and (4) repeated measurement (i.e., the pretest affects the posttest result). The last effect could have been reduced further by using a Solomon four-group design; however, there were not enough residents across the three Quebec radiation oncology residency programs to enable us to do so. Perhaps, if the study were to be repeated on a larger scale, this design would be beneficial. The small sample size can be considered a limitation, but it is comparable to other evaluation studies of educational interventions found in the literature (4, 10, 23). We also cannot rule out the Hawthorne effect, a reaction whereby subjects improve the performance that is being experimentally measured simply because they are being studied and not in response to the experimental intervention (24). Although it may be tempting to compare the present intervention with an existing and comparable intervention from a different mediumda textbook chapter on gynecologic malignancies, for exampledCook advises against such mediacomparative research, stating that it “is logically impossible because there are no valid comparison groups” (25).
Volume 82 Number 3 2012 Table 3
Evaluation of E-Learning in radiation oncology e579
Retrospective performance survey (n Z 12) Objective
Before
After
Change (p < 0.05)
Knowledge about normal female pelvic anatomy Knowledge about blood supply Knowledge about nerve supply Knowledge about lymphatic drainage Knowledge about normal radiology Knowledge about the organs at risk in treatment planning Skill in outlining CTV structures on CT or MRI images Skill in outlining gynecological tumors on imaging Skill in drawing treatment fields according to bony landmarks in traditional planning at the standard simulator Knowledge of FIGO staging Skill in designing optimal treatment plans for typical cases
2.5 2.3 1.9 3.1 2.8 3.3 2.8 2.8 3.0
3.8 3.3 3.0 4.1 3.9 4.3 3.8 3.8 3.9
1.3 1.0 1.1 1.0 1.1 1.0 1.0 1.0 0.9
3.0 2.6
3.8 3.7
0.8 1.1
Results indicate a retrospectively self-rated score representing perception of knowledge and skills before and after access to the module and also the absolute change in scores. Abbreviations: CTV Z clinical target volume; CT Z computed tomography; MRI Z magnetic resonance imaging; FIGO Z International Federation of Gynecology and Obstetrics; 1 Z very poor; 2 Z below average; 3 Z average; 4 Z above average; 5 Z excellent. 12 of the 19 residents (63.2%) from the intervention group responded to the survey
Most importantly, this study simply measured outcomes. Without a qualitative study, we cannot unequivocally make conclusions about what aspects of the software contributed most to resident learning. A variety of approaches, of which protocol analysis is one, could have been used to uncover how residents learn from this educational intervention (26). As stated by Cook, we need to move from doing research to learn whether web-based learning should be used, to doing research to learn when and how it should be used (27).
Conclusions The use of multimedia provides a modern and dynamic approach to medical education. The creation of this learning module proves that the development of an interactive, web-based teaching tool is feasible with current techniques. It is currently being used as a model on which to build future modules of other disease sites. The formal, interinstitutional evaluation of the learning module confirmed the hypothesis that the interactive nature and the high degree of learner control would promote self-directed learning. It was shown to have high learner satisfaction rates and to positively affect learning outcomes by enhancing the radiologic anatomy knowledge and technical skills of residents in the domain of gynecologic malignancies. This provides further evidence of the importance of e-learning and will set higher standards for electronic medical educational material in the future. Long-term changes in behavior were difficult to assess from the measures used in this study. Further evaluative research is required to examine the characteristics of effective web-based learning environments that foster knowledge transfer to resident practices and improve patient outcomes.
References 1. Pelizzari CA, Grzeszczuk R, Chen G, et al. Volumetric visualization of anatomy for treatment planning. Int J Radiat Oncol Biol Phys 1996; 34:205e211. 2. Campeau M, Lasalle S, Lambert C, et al. Self-directed learning module for radiation therapy treatment planning [Abstract]. J Clin Oncol 2007;25(Suppl):18S.
3. Valentini V, Dinapoli N, Nori S, et al. An application of visible human database in radiotherapy: Tutorial for image guided external radiotherapy (TIGER). Radiother Oncol 2004;70:165e169. 4. Nicholson D, Chalk C, Funnell RJ, et al. Can virtual reality improve anatomy education? A randomized controlled study of a computergenerated three-dimensional anatomical ear model. Med Educ 2006; 40:1081e1087. 5. Nattestad A, Attstrom R, Mattheos N, et al. Web-based interactive learning programmes. Eur J Dent Educ 2002;6(Suppl.3):127e137. 6. Cook DA, Garside S, Levinson AJ, et al. What do we mean by webbased learning? A systematic review of the variability of interventions. Med Educ 2010;44:765e774. 7. Merk M, Knuechel R, Perez-Bouza A. Web-based virtual microscopy at the RWTH Aachen University: Didactic concept, methods and analysis of acceptance by the students. Ann Anat 2010;192:383e387. 8. Crossingham JL, Jenkinson J, Woolridge N, et al. Interpreting threedimensional structures from two-dimensional images: A web-based interactive 3D teaching model of surgical liver anatomy. HPB (Oxford) 2009;11:523e528. 9. Cohen HB, Walker SR, Tenenbaum HC. Interdisciplinary, web-based, self-study, interactive programs in the dental undergraduate program: A pilot. J Dent Educ 2003;67:661e667. 10. Fukuchi SG, Offutt LA, Sacks J, et al. Teaching a multidisciplinary approach to cancer treatment during surgical clerkship via an interactive board game. Am J Surg 2000;179:337e340. 11. Nattestad A, Attstrom R. Introduction to theme 4: The virtual potential. Eur J Dent Educ 2002;6(Suppl3):125e126. 12. Chumley-Jones HS, Dobbie A, Alford CL. Web-based learning: Sound educational method or hype? A review of the evaluation literature. Acad Med 2002;77(10Suppl):S86eS93. 13. Shaffer K, Small JE. Blended learning in medical education: Use of an integrated approach with web-based small group modules and didactic instruction for teaching radiologic anatomy. Acad Radiol 2004;11: 1059e1070. 14. Reid JR, Goske MJ, Hewson MG, et al. Computers in radiology: Creating an international comprehensive web-based curriculum in pediatric radiology. AJR Am J Roentgenol 2004;182:797e801. 15. Weston C, Gandell T, McAlpine L, et al. Designing instruction for the context of on-line learning. The Internet and Higher Education 1999;2: 35e44. 16. Dick W, Carey L. The systematic design of instruction. 4th ed. New York: Harper Collins College Publishers; 1996. 17. Cook D, Dupras D. A practical guide to developing effective webbased learning. J Gen Intern Med 2004;19:698e707.
e580 Alfieri et al. 18. Kirkpatrick DL. Evaluating training programs: The four levels. San Francisco: Berrett-Koehler; 2004. 19. Skeff KM, Stratos GA, Bergen MR. Evaluation of a medical faculty development program: A comparison of traditional pre/post and retrospective pre/post self-assessment ratings. Eval Health Prof 1992;15: 350e366. 20. Curran V. An eclectic model for evaluating web-based continuing medical education courseware systems. Eval Health Prof 2000;23:318e347. 21. Letterie GS. Medical education as a science: The quality of evidence for computer-assisted instruction. Am J Obstet Gynecol 2003;188: 849e853. 22. Pellegrino J. Understanding how students learn and inferring what they know: Implications for the design of curriculum, instruction and assessment. In: Smith MJ, editor. NSF K-12 Mathematics and science curriculum and implementation centers conference proceedings.
International Journal of Radiation Oncology Biology Physics
23. 24. 25. 26.
27.
Washington, DC: National Science Foundation and American Geological Institute; 2002. p. 76e92. Hariri S, Rawn C, Srivastava S, et al. Evaluation of a surgical simulator for learning clinical anatomy. Med Educ 2004;38:896e902. McCarney R, Warner J, Iliffe S, et al. The Hawthorne Effect: A randomised, controlled trial. BMC Med Res Methodol 2007;7:30e38. Cook D. The research we still are not doing: An agenda for the study of computer-based learning. Acad Med 2005;80:541e548. National Research Council (NRC). Implications of the new foundations for assessment design. In: Pellegrino JW, Chudowsky N, Glaser R, editors. Knowing what students know: The science and design of educational assessment. Washington, DC: National Academy Press; 2001. p. 177e219. Cook D. Where are we with web-based learning in medical education? Med Teach 2006;28:594e598.