- Email: [email protected]
Piloting a web-based continuing professional development program for asthma education
i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 7 5 ( 2 0 0 6 ) 708–713
journal homepage: www.intl.elsevierhealth.com/journals/ijmi
Piloting a web-based continuing professional development program for asthma education夽 J. Leith Sly a , Enrico Lombardi b , Merci Kusel c , Peter D. Sly c,∗ a
Educational & Online Technologies, Curtin University of Technology, Perth, Australia Pediatric Allergy and Pulmonology Centre, Anna Meyer Children’s Hospital, Florence, Italy c Division of Clinical Sciences, Telethon Institute for Child Health Research & Centre for Child Health Research, University of Western Australia, P.O. Box 855, W. Perth 6872, Australia b
a r t i c l e
i n f o
a b s t r a c t
Article history:
Purpose: Continuing professional development is an integral component of modern medical
Received 17 May 2005
practice, yet traditional educational methods are impractical for many Primary Care Physi-
Received in revised form
cians. Web-based programs may fulfill the requirements of busy practitioners who have
20 September 2005
difficulty attending formal education sessions.
Accepted 27 September 2005
Methods: We piloted the use of a learning management system to deliver asthma education materials to Primary Care Physicians in both Australia and Italy in their native languages. Each group of Physicians accessed an education module which contained content pages,
Keywords:
self-tests, a quiz and a survey. Details of how the Physicians used the system, their prefer-
Continuing medical education
ences and performance on the assessment were monitored.
Computer-assisted instruction
Results: The learning management system was well received by both Italian and Australian
Internet
Physicians. Thirty-eight (18 Australian, 20 Italian) Physicians used the system. Participants
World wide web
visited an average of 8.8 pages, with a mean time per hit of 2.9 min. Formative assessment was undertaken by 63.2% and summative assessment by 68.4% of participants. There were no substantial differences in performance between Physicians from both countries. Italian physicians tended to use the system after hours whereas Australian Physicians appear to do so between patient visits. Conclusions: Simple web-based systems are suitable for delivering educational materials to Primary Care Physicians in a manner likely to be used. © 2005 Elsevier Ireland Ltd. All rights reserved.
1.
Introduction
Continuing professional development (CPD) is important for primary care Physicians however it is becoming increasingly difficult for Physicians to fit this into busy schedules. Keeping up to date with all the latest information, diagnostic procedures, and treatment options for the vast array of medical conditions that they are presented with on a day-to-day basis
夽 ∗
is challenging. The framework for continuing education traditionally has involved a lecture/presentation type situation scheduled for a particular time and place. It is even more difficult for primary care Physicians working in rural communities as most of these meetings are held in city locations. Flexible delivery of educational materials is being increasingly investigated as an alternative approach to encourage maximal Physician participation.
This project was supported by a grant from the European School of Respiratory Medicine. Corresponding author. Tel.: +61 8 9489 7810; fax: +61 8 9489 7706. E-mail address: [email protected] (P.D. Sly).
1386-5056/$ – see front matter © 2005 Elsevier Ireland Ltd. All rights reserved. doi:10.1016/j.ijmedinf.2005.09.004
i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 7 5 ( 2 0 0 6 ) 708–713
Assessment is a powerful catalyst for educational change and current educational theory extols the benefits of formative assessment; especially the feedback it provides to the learner. Provision of appropriate feedback on learning performance is considered to be an essential component of effective learning. Feedback is consistently part of any definition of formative assessment. Black and Wiliam [1] and Sadler [2] specifically refer to the role of feedback in modifying or improving the learning process. Furthermore Rolfe and McPherson [3] see it as allowing students to take responsibility for their own learning. Formative assessment and the feedback it provides give students both the opportunities to assess and change their present skill or knowledge as well as the ability to improve a process before a finished product is reached. For formative assessment to be effective, learners need to perceive a gap between their present knowledge position and their desired position of skill or knowledge and take some action in an attempt to close the gap [4,5]. As personal factors and beliefs about learning can affect both reception of the feedback and the decision on how to respond, the effectiveness of formative assessment will depend not only on the context of the feedback and the associated learning opportunities but also on assumptions about student motivation and self perceptions [1]. By providing the flexibility that is inherent in a web-based learning management system primary care Physicians and medical undergraduates will be provided with an environment that will be both convenient and conducive to ongoing professional development. The rapid pace of technological innovation has resulted in more units and courses being developed in the online environment. Within this framework of a flexible delivery system offering small modules of content accompanied by formative assessment in the form of self tests, we developed a module to provide continuing education on “Age-appropriate aerosol delivery devices” to primary care Physicians in Australia and primary care Paediatricians in Italy. This paper presents results of a pilot study that examines the use of an online module in continuing medical education. The program recognised that medical practitioners may use different strategies but the flexibility of the approach allowed participants to direct their own learning both in content order, assessment approach and flexibility of time of access. As a prelude to developing a course on Asthma and Allergies in children one module was developed and tested. This module is the subject of this report. The WebCT (Web Course Tools) learning management system was used to provide the Module which consisted of an overview of the module, a section on content, a self quiz and a multiple-choice computer generated test with an explanation to accompany each of the possible answers.
2.
Methods
2.1.
Learning management system–WebCT
WebCT is one of the most widely used learning management systems in higher education enabling the delivery of online education around the world. Use of WebCT enables the provision of alternative and flexible modes of delivery and
709
provides a coordinated approach to the development and implementation of a World Wide Web based teaching and learning environment. Features of the WebCT system include the ability to publish lecture notes, administer online assessment, implement bulletin boards and chat rooms and provide other features in a secure web environment with full user authentication, grading and tracking. In this project we did not use either the bulletin boards or chat rooms. Content, self tests, and an assessed test and survey for the Module were placed within the WebCT environment and hence was made available via the web. As all user and instructor interfaces using WebCT can be accessed using any standard web browser on any internet-connected computer all sections were readily available at a time convenient to the particular user.
2.2.
Asthma education module
The asthma education module was developed in English by an asthma specialist [PDS] and then re-written in Italian by a native speaker who is also a Respiratory Physician [EL]. All questions and accompanying feedback were also provided in Italian. WebCT system instructions were translated where possible and Physicians were required to authenticate with a username and password in order to access the asthma education module within the WebCT system. The module consisted of an overview page, a content module that contained six individual pages, each accompanied by a self-test, a quiz and a survey. Each individual content page provided instructions to alert Physicians to the various options available—progressing to new content, revisiting previous content, taking a self-test, etc. Self tests and the quiz could be attempted multiple times.
2.3.
Initial pilot
An initial pilot, conducted with 10 primary care Physicians, identified ambiguity with some questions that required Physicians to select more that one correct answer and with some survey questions (data not shown). These sections were reworked prior to the present study.
2.4.
Implementation
Two investigators, one in Australia [MK] and one in Italy [EL], recruited Physicians for the study. These Physicians were given identification numbers and passwords that would allow them access to the WebCT system. Only the WebCT manager [JLS] was able to view student progress and results. The content module consisted of a one-page overview that alerted users to the various components of the online module and six topic-specific pages. Each page was accompanied by a short self-test that provided feedback on answers (both correct and incorrect). The self-test could be attempted multiple times and the results were not graded (formative assessment). The summative quiz contained 15 questions and had a 30 min time limit. There was no limit on the number of times the quiz could be attempted. On completion of the quiz participants submitted it for automatic marking by the learning management system. They were then able to view the quiz with feedback on responses to each question.
710
i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 7 5 ( 2 0 0 6 ) 708–713
Table 1 – Tracking usage of content pages within the Module Topic
Australian Physicians
Age-appropriate drug delivery in children with asthma Delivery device Spacers Dry-powder inhalers Nebulizers Age-appropriate delivery devices
3.
Italian Physicians
Hits
Time per hit (min:s)
Hits
Time per hit (min:s)
26
1:33
35
1:31
25 22 21 20 18
1:54 1:19 2:59 2:52 5:26
49 42 31 27 26
1:44 2:26 2:10 2:27 10:34
Results
Two groups of primary care Physicians were given access to the education module. The results are reported for the entire group and then divided according to nationality. The module included 6 content pages in each language, a self-test on each content page, 15 quiz questions (30 min time limit) and a survey. Access to the program was available over a 6-month period. Data were collected on the time of day of access, page hits, time spent per content page, number of participants using self-tests, hits on self-tests, number of participants attempting quiz, time spent on quiz, and quiz results. In total 38 Physicians interacted with the WebCT system. Three-hundred and forty-two hits were recorded on the content pages. The average time per hit was 2.9 min (range
1.3–10.6 min). Participants visited an average of 8.8 pages. Details of page usage are shown in Table 1. The formative assessment self-tests were used by 63.2% of participants with the average number of self-tests taken per participant being 3.2. The summative assessment was attempted by 68.4% of participants with a mean score of 72.8% (S.D. = 14.3). Twenty-six participants submitted the survey. The webbased system was seen as meeting the needs of family Physicians (Table 2). The feedback on incorrect responses was seen as valuable and the content was seen as enhancing knowledge and clinical decision making. All requested further modules while some requested involvement in the development process. There were no systemic differences between the comments received from Australian and Italian Physicians. The total group of 38 was made up of 18 Australian and 20 Italian Physicians. A summary of the data by group is
Table 2 – Summary data showing survey questions and Physicians responses Question
The content in this module is relevant to my clinical practice The content in this module will help me make management decisions in my clinical practice The content in this module was too simple to be useful The testing facility increased the educational value of this module The testing facility was a waste of time The questions on the clinical scenarios were a waste of time The feedback provided on incorrect answers increased the educational value of this module The feedback provided on incorrect answers was a waste of time as it only repeated the content of the module The feedback provided on incorrect answers was valuable because it repeated important information from the content of the module The feedback provided on incorrect answers alerted me to gaps in my knowledge The feedback provided on incorrect answers was unnecessary, simply knowing which questions I got wrong would have been sufficient I would be keen to work through additional modules on other topics I would recommend this educational program to my colleagues The web-based delivery of this module is well suited to the educational needs of General Practitioners I prefer attending educational sessions rather than doing web-based education programs
Australian Mean
S.D.
1.38 1.69
0.50 0.48
3.94 1.88 4.19 4.50 1.69
Italian Mean
S.D.
Mode
1 2
1.30 1.50
0.48 0.53
1 1, 2
0.44 0.81 0.54 0.52 0.70
4 2 4 4, 5 1, 2
4.00 1.60 4.50 4.50 1.50
0.47 0.52 0.53 0.71 0.53
4 2 4, 5 5 1, 2
4.19
0.66
4
4.30
0.48
4
1.88
0.50
2
1.70
0.48
2
2.56
0.73
2
1.90
0.57
2
4.19
0.66
4
4.50
0.53
4, 5
2.06
0.77
2
1.50
0.53
1, 2
2.00 2.00
0.52 0.73
2 2
1.80 1.80
0.42 0.42
2 2
3.44
0.63
3
3.20
0.92
3
1: Strongly agree, 2: Agree, 3: Neither agree or disagree, 4: Disagree, 5: Strongly disagree.
Mode
i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 7 5 ( 2 0 0 6 ) 708–713
711
Table 3 – Summary data showing use of WebCT learning management system by Australian and Italian Physicians Australians, n = 18 Number using self-tests Hits on self-tests (mean per user) Pages hits (mean) Time per content page (mean) min Number attempting quiz Quiz results (mean %)
Italians, n = 20
7 3.4 13.6 2.7 14 78.1
17 2.7 19.9 2.6 12 73.3
p 0.006 0.44 0.07 0.97 0.9 0.40
system. Fig. 2 shows that Italian participants accessed the web-based module mainly at night while the Australian participants were more likely to log on midmorning and in late afternoon.
4.
Fig. 1 – Time taken (min) by Australian (filled bars) and Italian (open bars) Physicians to do the summative assessment quiz available at the end of the module.
presented in Table 3. The self test was used by 17 Italian Physicians and 7 Australian Physicians (p = 0.006) with the mean per user being 2.7 and 3.4, respectively (p = 0.44). Twelve Italian Physicians and 14 Australian Physicians attempted the summative assessment test with average percentage scores of 66.7% (S.D. = 16.8) and 78.1% (S.D. = 16.8), respectively (p = 0.04). The majority of the participants took 10–12 min to complete the test (Fig. 1). The most obvious difference between the Italian and Australian Physicians was the time of day they accessed the
Fig. 2 – Time of the day that Australian (filled bars) and Italian (open bars) Physicians accessed the online education module.
Discussion
The program we set up offered medical information and CPD via the web. This required Physicians to authenticate with a username and password but allowed them to access information at a time most convenient to them. The learning management system was simple to use and is widely used in the tertiary education sector with little or no instruction for students. We endeavoured to provide a flexible method for continuing medical education, acceptable to Physicians in both countries. The Physicians were able to visit content pages in any order they chose. They were also able to attempt self tests and the quiz in an order that suited their needs, that is, the learning management system did not impose a linear progression through the materials but rather allowed individual tailoring by participating Physicians. To maximize efficiency teaching and learning activities must be interactive. Primary care Physicians have an ongoing need to keep their knowledge and skills up to date. The first step is for them to understand how current their knowledge actually is. Feedback on assessment performance provides the primary care Physician with a “report card”, highlighting areas in which their knowledge is weak. Feedback as part of formative assessment is specifically intended to improve and accelerate learning [2]. Higher education research suggests that students use the feedback supplied on incorrect answers, as triggers to help them identify areas to study. Research [6–9] has strongly pointed to the benefits of this type of assessment in improving student learning. While the WebCT learning management system can use a variety of question types to assess knowledge, the most commonly used are multiple-choice tests. Multiple-choice tests, computer-generated or otherwise, have a number of obvious advantages, especially as part of an overall assessment strategy. Brown and Knight [10] cite the range of content which can be covered by the items, the ease of marking and the ease of identification of major content areas that students find difficult, as key advantages for multiple-choice tests. If properly designed, a multiple-choice test can identify and test a range of skills as well as reducing the possibility of success by chance [11]. While it is acknowledged that questions still remain as to how online courses support quality and success in continu-
712
i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 7 5 ( 2 0 0 6 ) 708–713
ing medical professional development this course provided Physicians with content and feedback while giving them the flexibility to interact with the system in the way they wished and at the time they chose.
4.1.
the reasons why a multitude of different approaches is necessary. Also, in this pilot study we did not aim to determine how well Physicians learnt the contents of the module but rather to investigate the potential usefulness of learning management systems. Systems that are not used will not improve learning!
Flexible delivery to meet professional needs 4.2.
Online learning environments meet the need for increased flexibility, allowing access at a time convenient to the user. This project has highlighted the ability that a web-based system has in allowing users to access materials at times suitable for them. Our results show that Physicians access materials during the working day, late at night and even into the early hours of the morning. This type of flexibility is likely to encourage maximal participation in CPD activities. As it also required Physicians to identify themselves with an allocated username and password it provided a flexible method of delivery while also providing a mechanism that could be used to monitor or collect information for CPD. Other reports have found that Physicians favour online learning. Letterie [12] examined 210 reports, published between 1988 and 2000, describing Physician use of computer-assisted instruction in medical education. Ninety-six percent of the articles described a favourable impact of computer-assisted instruction. The responses to the survey questions showed that Physicians found the module content both relevant to their clinical practice and useful for making management decisions in their clinical practice. The clinical scenarios were valuable and Physicians found the testing facility and the feedback increased the educational value of the module. They were keen for additional modules and would recommend the program to their colleagues. Such positive responses lend support to using programs such as this one to improve GP education. The key strength of this system was the flexibility it provided users. It allowed choice of access time, method of interaction (linear or otherwise), and it did not put constraints on the number of times content could be accessed or questions answered. It provided feedback on all questions and users could take as much time as they required to review the feedback. In essence it allowed users to direct their own learning and to manage this at a pace that suited their particular situation. CPD is now required of Physicians in the UK and is generally regarded as more relevant to daily work than the narrower Continuing Medical Education (CME) [13]. CPD is seen as broader as it recognises that a Physician is involved in more than clinical work, e.g. management, teaching, research and other activities. CPD uses a structured and planned approach in contrast to the informal approach to CME. Instead of Physicians going to lectures and meeting as they occurred, Physicians are now required to consider their individual professional development needs and direct their own planned schedule. Key features of the CPD approach are reflection, self-directed learning and planning. The WebCT module we developed fulfils these requirements. In this particular pilot study Physicians were recruited through local Primary Care Physician groups. While this method of recruitment may have introduced selection bias, in that only more interested Physicians agreed to participate, this does not invalidate the results. The problems of engaging all Physicians in CPD activities are well recognised and one of
Recommendations for the future
Online interaction with individual Physicians would help address issues related to content queries, online access difficulties and any other issues that arose during the project. The WebCT manager in this project was always in a position to monitor progress of the individual Physicians. However, as the identification process we used essentially made the Physicians anonymous to all bar the recruiting investigators there was not the ability to interact with individuals. The manager was in a position to see how Physicians were interacting with the system and could have contacted Physicians either via email or by using the bulletin board facility that was not implemented in this project. Future projects should train one or two Physicians as assistants. These assistants will be able to interact with the participating Physicians either via email or by monitoring of the bulletin board facility. In conclusion, we believe this study has highlighted that Australian and Italian Physicians used similar learning strategies. This supports the use of web-based asthma education materials developed in one centre and distributed widely, provided the materials are translated and/or re-written by local specialists.
Summary points What was known before the study? • Continuing medical education is important for physicians. • Traditional lectures and small group activities only attract those interested. • While web-based learning management systems are theoretically attractive little evidence of their fitness for this purpose has been documented. What the study has added to the body of knowledge? • A web-based system is acceptable to doctors in different parts of the world. • Highlighted differences in access times between Italian and Australian physicians. • Highlighted the importance of providing a flexible information delivery system for physicians. • Self tests accepted as a good way of assessing knowledge and identifying knowledge “gaps”. • Having the module available online with no constraints on the number of access times gives physicians a convenient online repository of medical information.
i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 7 5 ( 2 0 0 6 ) 708–713
references [8] [1] P. Black, D. Wiliam, Assessment and classroom learning, Assess. Educ. 5 (1998) 7–74. [2] D.R. Sadler, Formative assessment: revisiting the territory, Assess. Educ. 5 (1998) 77–83. [3] I. Rolfe, J. McPherson, Formative assessment: how am I doing? Lancet 345 (1995) 837–839. [4] A. Ramaprasad, On the definition of feedback, Behav. Sci. 28 (1983) 4–13. [5] D.R. Sadler, Formative assessment and design of instruction systems, Instruct. Sci. 18 (1989) 119–144. [6] L. Sly, Practice tests as formative assessment improve student performance on computer managed learning assessments, Assess. Eval. High. Educ. 24 (1999) 339–343. [7] L. Sly, L.J. Rennie, Computer managed learning: its use in formative as well as summative assessment, in: M. Danson, R. Sherrat (Eds.), Proceedings of the Flexible Learning: Third
[9]
[10] [11] [12]
[13]
713
Annual Computer Assisted Assessment Conference, Loughborough University, 1999, pp. 179–189. L. Sly, L.J. Rennie, Computer managed learning as an aid to formative assessment in higher education., in: S. Brown, J. Bull, P. Race (Eds.), Computer Assisted Assessment of Students, Kogan Page, London, 1999, pp. 113–120. J.L. Sly, Computer Managed Learning Assessment in Higher Education: The Effect of a Practice Test, Ph.D. Thesis, Curtin University, Australia, 2001. S. Brown, P. Knight, Assessing learners in higher education, Kogan Page, London, 1994. N.F. Gronlund, Constructing Achievement Tests, Prentice Hall Inc., Englewood Cliffs, NJ, 1982. G.S. Letterie, Medical education as science: the quality of evidence for computer-assisted instruction, Am. J. Obstetr. Gynecol. (2003) 849–853. A.M. Weindling, Education and training: continuing professional development, Curr. Paediatr. 11 (2001) 369–374.
journal homepage: www.intl.elsevierhealth.com/journals/ijmi
Piloting a web-based continuing professional development program for asthma education夽 J. Leith Sly a , Enrico Lombardi b , Merci Kusel c , Peter D. Sly c,∗ a
Educational & Online Technologies, Curtin University of Technology, Perth, Australia Pediatric Allergy and Pulmonology Centre, Anna Meyer Children’s Hospital, Florence, Italy c Division of Clinical Sciences, Telethon Institute for Child Health Research & Centre for Child Health Research, University of Western Australia, P.O. Box 855, W. Perth 6872, Australia b
a r t i c l e
i n f o
a b s t r a c t
Article history:
Purpose: Continuing professional development is an integral component of modern medical
Received 17 May 2005
practice, yet traditional educational methods are impractical for many Primary Care Physi-
Received in revised form
cians. Web-based programs may fulfill the requirements of busy practitioners who have
20 September 2005
difficulty attending formal education sessions.
Accepted 27 September 2005
Methods: We piloted the use of a learning management system to deliver asthma education materials to Primary Care Physicians in both Australia and Italy in their native languages. Each group of Physicians accessed an education module which contained content pages,
Keywords:
self-tests, a quiz and a survey. Details of how the Physicians used the system, their prefer-
Continuing medical education
ences and performance on the assessment were monitored.
Computer-assisted instruction
Results: The learning management system was well received by both Italian and Australian
Internet
Physicians. Thirty-eight (18 Australian, 20 Italian) Physicians used the system. Participants
World wide web
visited an average of 8.8 pages, with a mean time per hit of 2.9 min. Formative assessment was undertaken by 63.2% and summative assessment by 68.4% of participants. There were no substantial differences in performance between Physicians from both countries. Italian physicians tended to use the system after hours whereas Australian Physicians appear to do so between patient visits. Conclusions: Simple web-based systems are suitable for delivering educational materials to Primary Care Physicians in a manner likely to be used. © 2005 Elsevier Ireland Ltd. All rights reserved.
1.
Introduction
Continuing professional development (CPD) is important for primary care Physicians however it is becoming increasingly difficult for Physicians to fit this into busy schedules. Keeping up to date with all the latest information, diagnostic procedures, and treatment options for the vast array of medical conditions that they are presented with on a day-to-day basis
夽 ∗
is challenging. The framework for continuing education traditionally has involved a lecture/presentation type situation scheduled for a particular time and place. It is even more difficult for primary care Physicians working in rural communities as most of these meetings are held in city locations. Flexible delivery of educational materials is being increasingly investigated as an alternative approach to encourage maximal Physician participation.
This project was supported by a grant from the European School of Respiratory Medicine. Corresponding author. Tel.: +61 8 9489 7810; fax: +61 8 9489 7706. E-mail address: [email protected] (P.D. Sly).
1386-5056/$ – see front matter © 2005 Elsevier Ireland Ltd. All rights reserved. doi:10.1016/j.ijmedinf.2005.09.004
i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 7 5 ( 2 0 0 6 ) 708–713
Assessment is a powerful catalyst for educational change and current educational theory extols the benefits of formative assessment; especially the feedback it provides to the learner. Provision of appropriate feedback on learning performance is considered to be an essential component of effective learning. Feedback is consistently part of any definition of formative assessment. Black and Wiliam [1] and Sadler [2] specifically refer to the role of feedback in modifying or improving the learning process. Furthermore Rolfe and McPherson [3] see it as allowing students to take responsibility for their own learning. Formative assessment and the feedback it provides give students both the opportunities to assess and change their present skill or knowledge as well as the ability to improve a process before a finished product is reached. For formative assessment to be effective, learners need to perceive a gap between their present knowledge position and their desired position of skill or knowledge and take some action in an attempt to close the gap [4,5]. As personal factors and beliefs about learning can affect both reception of the feedback and the decision on how to respond, the effectiveness of formative assessment will depend not only on the context of the feedback and the associated learning opportunities but also on assumptions about student motivation and self perceptions [1]. By providing the flexibility that is inherent in a web-based learning management system primary care Physicians and medical undergraduates will be provided with an environment that will be both convenient and conducive to ongoing professional development. The rapid pace of technological innovation has resulted in more units and courses being developed in the online environment. Within this framework of a flexible delivery system offering small modules of content accompanied by formative assessment in the form of self tests, we developed a module to provide continuing education on “Age-appropriate aerosol delivery devices” to primary care Physicians in Australia and primary care Paediatricians in Italy. This paper presents results of a pilot study that examines the use of an online module in continuing medical education. The program recognised that medical practitioners may use different strategies but the flexibility of the approach allowed participants to direct their own learning both in content order, assessment approach and flexibility of time of access. As a prelude to developing a course on Asthma and Allergies in children one module was developed and tested. This module is the subject of this report. The WebCT (Web Course Tools) learning management system was used to provide the Module which consisted of an overview of the module, a section on content, a self quiz and a multiple-choice computer generated test with an explanation to accompany each of the possible answers.
2.
Methods
2.1.
Learning management system–WebCT
WebCT is one of the most widely used learning management systems in higher education enabling the delivery of online education around the world. Use of WebCT enables the provision of alternative and flexible modes of delivery and
709
provides a coordinated approach to the development and implementation of a World Wide Web based teaching and learning environment. Features of the WebCT system include the ability to publish lecture notes, administer online assessment, implement bulletin boards and chat rooms and provide other features in a secure web environment with full user authentication, grading and tracking. In this project we did not use either the bulletin boards or chat rooms. Content, self tests, and an assessed test and survey for the Module were placed within the WebCT environment and hence was made available via the web. As all user and instructor interfaces using WebCT can be accessed using any standard web browser on any internet-connected computer all sections were readily available at a time convenient to the particular user.
2.2.
Asthma education module
The asthma education module was developed in English by an asthma specialist [PDS] and then re-written in Italian by a native speaker who is also a Respiratory Physician [EL]. All questions and accompanying feedback were also provided in Italian. WebCT system instructions were translated where possible and Physicians were required to authenticate with a username and password in order to access the asthma education module within the WebCT system. The module consisted of an overview page, a content module that contained six individual pages, each accompanied by a self-test, a quiz and a survey. Each individual content page provided instructions to alert Physicians to the various options available—progressing to new content, revisiting previous content, taking a self-test, etc. Self tests and the quiz could be attempted multiple times.
2.3.
Initial pilot
An initial pilot, conducted with 10 primary care Physicians, identified ambiguity with some questions that required Physicians to select more that one correct answer and with some survey questions (data not shown). These sections were reworked prior to the present study.
2.4.
Implementation
Two investigators, one in Australia [MK] and one in Italy [EL], recruited Physicians for the study. These Physicians were given identification numbers and passwords that would allow them access to the WebCT system. Only the WebCT manager [JLS] was able to view student progress and results. The content module consisted of a one-page overview that alerted users to the various components of the online module and six topic-specific pages. Each page was accompanied by a short self-test that provided feedback on answers (both correct and incorrect). The self-test could be attempted multiple times and the results were not graded (formative assessment). The summative quiz contained 15 questions and had a 30 min time limit. There was no limit on the number of times the quiz could be attempted. On completion of the quiz participants submitted it for automatic marking by the learning management system. They were then able to view the quiz with feedback on responses to each question.
710
i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 7 5 ( 2 0 0 6 ) 708–713
Table 1 – Tracking usage of content pages within the Module Topic
Australian Physicians
Age-appropriate drug delivery in children with asthma Delivery device Spacers Dry-powder inhalers Nebulizers Age-appropriate delivery devices
3.
Italian Physicians
Hits
Time per hit (min:s)
Hits
Time per hit (min:s)
26
1:33
35
1:31
25 22 21 20 18
1:54 1:19 2:59 2:52 5:26
49 42 31 27 26
1:44 2:26 2:10 2:27 10:34
Results
Two groups of primary care Physicians were given access to the education module. The results are reported for the entire group and then divided according to nationality. The module included 6 content pages in each language, a self-test on each content page, 15 quiz questions (30 min time limit) and a survey. Access to the program was available over a 6-month period. Data were collected on the time of day of access, page hits, time spent per content page, number of participants using self-tests, hits on self-tests, number of participants attempting quiz, time spent on quiz, and quiz results. In total 38 Physicians interacted with the WebCT system. Three-hundred and forty-two hits were recorded on the content pages. The average time per hit was 2.9 min (range
1.3–10.6 min). Participants visited an average of 8.8 pages. Details of page usage are shown in Table 1. The formative assessment self-tests were used by 63.2% of participants with the average number of self-tests taken per participant being 3.2. The summative assessment was attempted by 68.4% of participants with a mean score of 72.8% (S.D. = 14.3). Twenty-six participants submitted the survey. The webbased system was seen as meeting the needs of family Physicians (Table 2). The feedback on incorrect responses was seen as valuable and the content was seen as enhancing knowledge and clinical decision making. All requested further modules while some requested involvement in the development process. There were no systemic differences between the comments received from Australian and Italian Physicians. The total group of 38 was made up of 18 Australian and 20 Italian Physicians. A summary of the data by group is
Table 2 – Summary data showing survey questions and Physicians responses Question
The content in this module is relevant to my clinical practice The content in this module will help me make management decisions in my clinical practice The content in this module was too simple to be useful The testing facility increased the educational value of this module The testing facility was a waste of time The questions on the clinical scenarios were a waste of time The feedback provided on incorrect answers increased the educational value of this module The feedback provided on incorrect answers was a waste of time as it only repeated the content of the module The feedback provided on incorrect answers was valuable because it repeated important information from the content of the module The feedback provided on incorrect answers alerted me to gaps in my knowledge The feedback provided on incorrect answers was unnecessary, simply knowing which questions I got wrong would have been sufficient I would be keen to work through additional modules on other topics I would recommend this educational program to my colleagues The web-based delivery of this module is well suited to the educational needs of General Practitioners I prefer attending educational sessions rather than doing web-based education programs
Australian Mean
S.D.
1.38 1.69
0.50 0.48
3.94 1.88 4.19 4.50 1.69
Italian Mean
S.D.
Mode
1 2
1.30 1.50
0.48 0.53
1 1, 2
0.44 0.81 0.54 0.52 0.70
4 2 4 4, 5 1, 2
4.00 1.60 4.50 4.50 1.50
0.47 0.52 0.53 0.71 0.53
4 2 4, 5 5 1, 2
4.19
0.66
4
4.30
0.48
4
1.88
0.50
2
1.70
0.48
2
2.56
0.73
2
1.90
0.57
2
4.19
0.66
4
4.50
0.53
4, 5
2.06
0.77
2
1.50
0.53
1, 2
2.00 2.00
0.52 0.73
2 2
1.80 1.80
0.42 0.42
2 2
3.44
0.63
3
3.20
0.92
3
1: Strongly agree, 2: Agree, 3: Neither agree or disagree, 4: Disagree, 5: Strongly disagree.
Mode
i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 7 5 ( 2 0 0 6 ) 708–713
711
Table 3 – Summary data showing use of WebCT learning management system by Australian and Italian Physicians Australians, n = 18 Number using self-tests Hits on self-tests (mean per user) Pages hits (mean) Time per content page (mean) min Number attempting quiz Quiz results (mean %)
Italians, n = 20
7 3.4 13.6 2.7 14 78.1
17 2.7 19.9 2.6 12 73.3
p 0.006 0.44 0.07 0.97 0.9 0.40
system. Fig. 2 shows that Italian participants accessed the web-based module mainly at night while the Australian participants were more likely to log on midmorning and in late afternoon.
4.
Fig. 1 – Time taken (min) by Australian (filled bars) and Italian (open bars) Physicians to do the summative assessment quiz available at the end of the module.
presented in Table 3. The self test was used by 17 Italian Physicians and 7 Australian Physicians (p = 0.006) with the mean per user being 2.7 and 3.4, respectively (p = 0.44). Twelve Italian Physicians and 14 Australian Physicians attempted the summative assessment test with average percentage scores of 66.7% (S.D. = 16.8) and 78.1% (S.D. = 16.8), respectively (p = 0.04). The majority of the participants took 10–12 min to complete the test (Fig. 1). The most obvious difference between the Italian and Australian Physicians was the time of day they accessed the
Fig. 2 – Time of the day that Australian (filled bars) and Italian (open bars) Physicians accessed the online education module.
Discussion
The program we set up offered medical information and CPD via the web. This required Physicians to authenticate with a username and password but allowed them to access information at a time most convenient to them. The learning management system was simple to use and is widely used in the tertiary education sector with little or no instruction for students. We endeavoured to provide a flexible method for continuing medical education, acceptable to Physicians in both countries. The Physicians were able to visit content pages in any order they chose. They were also able to attempt self tests and the quiz in an order that suited their needs, that is, the learning management system did not impose a linear progression through the materials but rather allowed individual tailoring by participating Physicians. To maximize efficiency teaching and learning activities must be interactive. Primary care Physicians have an ongoing need to keep their knowledge and skills up to date. The first step is for them to understand how current their knowledge actually is. Feedback on assessment performance provides the primary care Physician with a “report card”, highlighting areas in which their knowledge is weak. Feedback as part of formative assessment is specifically intended to improve and accelerate learning [2]. Higher education research suggests that students use the feedback supplied on incorrect answers, as triggers to help them identify areas to study. Research [6–9] has strongly pointed to the benefits of this type of assessment in improving student learning. While the WebCT learning management system can use a variety of question types to assess knowledge, the most commonly used are multiple-choice tests. Multiple-choice tests, computer-generated or otherwise, have a number of obvious advantages, especially as part of an overall assessment strategy. Brown and Knight [10] cite the range of content which can be covered by the items, the ease of marking and the ease of identification of major content areas that students find difficult, as key advantages for multiple-choice tests. If properly designed, a multiple-choice test can identify and test a range of skills as well as reducing the possibility of success by chance [11]. While it is acknowledged that questions still remain as to how online courses support quality and success in continu-
712
i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 7 5 ( 2 0 0 6 ) 708–713
ing medical professional development this course provided Physicians with content and feedback while giving them the flexibility to interact with the system in the way they wished and at the time they chose.
4.1.
the reasons why a multitude of different approaches is necessary. Also, in this pilot study we did not aim to determine how well Physicians learnt the contents of the module but rather to investigate the potential usefulness of learning management systems. Systems that are not used will not improve learning!
Flexible delivery to meet professional needs 4.2.
Online learning environments meet the need for increased flexibility, allowing access at a time convenient to the user. This project has highlighted the ability that a web-based system has in allowing users to access materials at times suitable for them. Our results show that Physicians access materials during the working day, late at night and even into the early hours of the morning. This type of flexibility is likely to encourage maximal participation in CPD activities. As it also required Physicians to identify themselves with an allocated username and password it provided a flexible method of delivery while also providing a mechanism that could be used to monitor or collect information for CPD. Other reports have found that Physicians favour online learning. Letterie [12] examined 210 reports, published between 1988 and 2000, describing Physician use of computer-assisted instruction in medical education. Ninety-six percent of the articles described a favourable impact of computer-assisted instruction. The responses to the survey questions showed that Physicians found the module content both relevant to their clinical practice and useful for making management decisions in their clinical practice. The clinical scenarios were valuable and Physicians found the testing facility and the feedback increased the educational value of the module. They were keen for additional modules and would recommend the program to their colleagues. Such positive responses lend support to using programs such as this one to improve GP education. The key strength of this system was the flexibility it provided users. It allowed choice of access time, method of interaction (linear or otherwise), and it did not put constraints on the number of times content could be accessed or questions answered. It provided feedback on all questions and users could take as much time as they required to review the feedback. In essence it allowed users to direct their own learning and to manage this at a pace that suited their particular situation. CPD is now required of Physicians in the UK and is generally regarded as more relevant to daily work than the narrower Continuing Medical Education (CME) [13]. CPD is seen as broader as it recognises that a Physician is involved in more than clinical work, e.g. management, teaching, research and other activities. CPD uses a structured and planned approach in contrast to the informal approach to CME. Instead of Physicians going to lectures and meeting as they occurred, Physicians are now required to consider their individual professional development needs and direct their own planned schedule. Key features of the CPD approach are reflection, self-directed learning and planning. The WebCT module we developed fulfils these requirements. In this particular pilot study Physicians were recruited through local Primary Care Physician groups. While this method of recruitment may have introduced selection bias, in that only more interested Physicians agreed to participate, this does not invalidate the results. The problems of engaging all Physicians in CPD activities are well recognised and one of
Recommendations for the future
Online interaction with individual Physicians would help address issues related to content queries, online access difficulties and any other issues that arose during the project. The WebCT manager in this project was always in a position to monitor progress of the individual Physicians. However, as the identification process we used essentially made the Physicians anonymous to all bar the recruiting investigators there was not the ability to interact with individuals. The manager was in a position to see how Physicians were interacting with the system and could have contacted Physicians either via email or by using the bulletin board facility that was not implemented in this project. Future projects should train one or two Physicians as assistants. These assistants will be able to interact with the participating Physicians either via email or by monitoring of the bulletin board facility. In conclusion, we believe this study has highlighted that Australian and Italian Physicians used similar learning strategies. This supports the use of web-based asthma education materials developed in one centre and distributed widely, provided the materials are translated and/or re-written by local specialists.
Summary points What was known before the study? • Continuing medical education is important for physicians. • Traditional lectures and small group activities only attract those interested. • While web-based learning management systems are theoretically attractive little evidence of their fitness for this purpose has been documented. What the study has added to the body of knowledge? • A web-based system is acceptable to doctors in different parts of the world. • Highlighted differences in access times between Italian and Australian physicians. • Highlighted the importance of providing a flexible information delivery system for physicians. • Self tests accepted as a good way of assessing knowledge and identifying knowledge “gaps”. • Having the module available online with no constraints on the number of access times gives physicians a convenient online repository of medical information.
i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 7 5 ( 2 0 0 6 ) 708–713
references [8] [1] P. Black, D. Wiliam, Assessment and classroom learning, Assess. Educ. 5 (1998) 7–74. [2] D.R. Sadler, Formative assessment: revisiting the territory, Assess. Educ. 5 (1998) 77–83. [3] I. Rolfe, J. McPherson, Formative assessment: how am I doing? Lancet 345 (1995) 837–839. [4] A. Ramaprasad, On the definition of feedback, Behav. Sci. 28 (1983) 4–13. [5] D.R. Sadler, Formative assessment and design of instruction systems, Instruct. Sci. 18 (1989) 119–144. [6] L. Sly, Practice tests as formative assessment improve student performance on computer managed learning assessments, Assess. Eval. High. Educ. 24 (1999) 339–343. [7] L. Sly, L.J. Rennie, Computer managed learning: its use in formative as well as summative assessment, in: M. Danson, R. Sherrat (Eds.), Proceedings of the Flexible Learning: Third
[9]
[10] [11] [12]
[13]
713
Annual Computer Assisted Assessment Conference, Loughborough University, 1999, pp. 179–189. L. Sly, L.J. Rennie, Computer managed learning as an aid to formative assessment in higher education., in: S. Brown, J. Bull, P. Race (Eds.), Computer Assisted Assessment of Students, Kogan Page, London, 1999, pp. 113–120. J.L. Sly, Computer Managed Learning Assessment in Higher Education: The Effect of a Practice Test, Ph.D. Thesis, Curtin University, Australia, 2001. S. Brown, P. Knight, Assessing learners in higher education, Kogan Page, London, 1994. N.F. Gronlund, Constructing Achievement Tests, Prentice Hall Inc., Englewood Cliffs, NJ, 1982. G.S. Letterie, Medical education as science: the quality of evidence for computer-assisted instruction, Am. J. Obstetr. Gynecol. (2003) 849–853. A.M. Weindling, Education and training: continuing professional development, Curr. Paediatr. 11 (2001) 369–374.