- Email: [email protected]
Developing and providing an online (web-based) clinical research design course in Japan: Lessons learned
Available online at www.sciencedirect.com
Journal of Prosthodontic Research 55 (2011) 61–68 www.elsevier.com/locate/jpor
Review
Developing and providing an online (web-based) clinical research design course in Japan: Lessons learned Glenn T. Clark DDS, MSa, Roseann Mulligan DDS, MSb, Kazuyoshi Baba DDS, PhDc,* b
a Orofacial Pain and Oral Medicine Center, Herman Ostrow School of Dentistry, University of Southern California, Los Angeles, CA, USA Community Dentistry Programs and Hospital Affairs, Herman Ostrow School of Dentistry, University of Southern California, Los Angeles, CA, USA c Department of Prosthodontics, School of Dentistry, Showa University, 2-1-1 Kitasenzoku, Ohta-ku, Tokyo 145-8515, Japan
Received 17 January 2011; accepted 30 January 2011
Abstract This article reports on the lessons learned while teaching an 8-week-long online course about the principles of clinical research design in Japan. Student activity data and how it relates to performance in the course are presented. As prolog, this article focuses on the barriers and solutions to creating and delivering a web-based course and it lists and discusses the most common concerns that educators often have about this process, namely, cost of the system and time requirement of the faculty. Options that must be considered when selecting the support software and hardware needed to conduct live streaming lecture, online video-based conference course are presented. The ancillary role of e-mail based distribution lists as an essential instruction tool within an interactive, instructor-supervised online course is discussed. This article then discusses the inclusion of active learning elements within an online course as well as the pros and cons regarding open-book versus closed book, proctored testing. Lastly, copyright issues the online instructor should know about are discussed. The student tracking data show that as the course progresses, students will reduce the number for page viewings. We speculate that this reduction is due to a combination of conflicting priorities plus increasing efficiency of the students at extracting the critical information. The article also concludes that software and hardware costs to deliver an online course are relatively minor but the faculty’s time requirement is initially substantially higher than teaching in a conventional face-to-face course. # 2011 Japan Prosthodontic Society. Published by Elsevier Ireland. All rights reserved. Keywords: Clinical research design; Online course; Web-based course; Video conference
Contents 1.
2.
Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.1. Why create and deliver a web-based course? . . . . . . . . . . . . . . . . . . . . . . . 1.2. What is active learning and why is it important? . . . . . . . . . . . . . . . . . . . . 1.3. The expense involved . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.4. Faculty time engaged in online education . . . . . . . . . . . . . . . . . . . . . . . . . 1.5. Software platforms for web-based teaching . . . . . . . . . . . . . . . . . . . . . . . . 1.6. Servers—jointly managed or independent . . . . . . . . . . . . . . . . . . . . . . . . . 1.7. Evidence about the relative efficacy of web-based teaching . . . . . . . . . . . . . 1.8. Moving a live-lecture presentation to a streaming online lecture presentation 1.9. The role for E-mail. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.10. How to achieve active learning with online homework assignments? . . . . . . 1.11. Testing types: open or closed book and proctored . . . . . . . . . . . . . . . . . . . 1.12. Avoiding copyright violations in an online course . . . . . . . . . . . . . . . . . . . Case report: design of an online clinical research design course . . . . . . . . . . . . . . 2.1. Structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2. Students . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.3. Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . .
* Corresponding author. Tel.: +81 3 3787 1151; fax: +81 3 9784 6468. E-mail address: [email protected] (K. Baba). 1883-1958/$ – see front matter # 2011 Japan Prosthodontic Society. Published by Elsevier Ireland. All rights reserved. doi:10.1016/j.jpor.2011.01.004
. . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . .
62 62 62 62 63 63 63 63 64 64 64 65 65 65 65 65 66
62
3.
4.
G.T. Clark et al. / Journal of Prosthodontic Research 55 (2011) 61–68
2.4. Learning management system . . . . . . . . . . . . . 2.5. Weekly video conferencing . . . . . . . . . . . . . . . 2.6. Desktop lecture recording system . . . . . . . . . . . 2.7. Textbook . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.8. Student opinion questionnaire . . . . . . . . . . . . . Results from an online clinical research design course . 3.1. Student drop-outs . . . . . . . . . . . . . . . . . . . . . . 3.2. Student online activity data . . . . . . . . . . . . . . . 3.3. Student opinion data . . . . . . . . . . . . . . . . . . . . Summary and conclusions. . . . . . . . . . . . . . . . . . . . . Acknowledgement . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
1. Introduction 1.1. Why create and deliver a web-based course? In December of 2000, a bipartisan, US Congressional committee published a comprehensive report on the issue of Internet usage and education [1] concluding that the promise of web-based education lay in its unique ability to: allow studentcentered rather than classroom-centered learning; focus on the strengths and needs of individual learners; and make life-long learning a reality. They summarized their findings with a call for substantial continuous and relevant training and support for educators in how to develop and deliver web-based content to students. In 2004, the Japanese Ministry of Education, Culture, Sports, Science and Technology established a ‘‘media education development center’’ in order to promote e-Learning in university education [2]. In 2007 this same Ministry published an update on the current status and future prospective of e-Learning in Japan which expressed support for this form of education [3]. In contrast with these actions in 2002 the State University of New York (SUNY) at Buffalo dropped its online M.B.A. program [4] and in 2004 the United Kingdom’s eUniversity (UkeU) shut down as well. Purportedly the SUNY program was ended as online programs weren’t ‘‘worth the expense and hassle’’ whereas the UKeU program was doomed due to multiple factors including substantial up-front investment of public monies, lack of private sector funding, low enrollments that missed targeted projections, brand confusion and an incomplete in-house platform on which to operate the system [5]. In the opinion of one author had impatience for results not taken hold, this elearning venture would probably have covered its costs in another five years and become a major online brand. There are also many success stories in online higher education and herein we will discuss only how to build one good course. One principle that applies is ‘‘a bad course which is put online is still a bad course, only now more expensive.’’ 1.2. What is active learning and why is it important? Probably the most important concept for creating a good course, whether online or face-to-face, is the promotion of an environment of ‘‘active student learning’’ rather than passive listening. Active learning is synonymous with increased student
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
66 66 66 66 66 66 66 67 67 68 68 68
involvement in his/her education. Active learning is not a concept exclusive to web-based delivery of educational content and can occur in any classroom. Presumably interactivity and student engagement is the expectation for on-line educational programs. If such a strategy can be achieved the student will acquire knowledge substantially faster with increased depth and diversity of content. Furthermore the knowledge pathway that the students pursue can focus on the individual interests of the student (thus the term student-centered learning was coined). A search of the literature reveals that several medical and dental educators have described their efforts in this arena and generally all are enthusiastic about the method [6–12]. For the educators interested in becoming involved in this type of content delivery, we suggest their enrolling in 2 or 3 different types of online courses to assess from the student perception what works and what does not before launching their own online course. 1.3. The expense involved There is currently very little evidence to suggest that online teaching will eventually prove to be cheaper than traditional methods as the costs are varied. Although a 1998 report describes both faculty and administrators’ resistance to online instruction in part due to costs [13], in general data is lacking on this topic. Online methodologies are relatively immature and subject to much change in the marketplace; therefore it is premature to attempt an assessment of future real costs. Educators currently developing web-based content in today’s climate would agree that compared to a traditional in-class lecture presentation, creating a computer based, web-accessible learning module is neither faster to develop nor cheaper to provide. The necessity to be cheaper-to-develop and/or deliver has not always been an absolute requirement for educational innovation. Effectiveness is typically used as the metric. For example when slides became readily available for use in teaching, they quickly assumed a prominent role due to studies demonstrating their effectiveness over conventional lecturing [14]. While this added a cost element, it was obviously not an impediment to the widespread, rapid adaptation of this method in education. Perhaps the same will also be said in the future of web-based education. Conversion of a standard PowerPoint lecture to an online lecture can be as simple as buying a
G.T. Clark et al. / Journal of Prosthodontic Research 55 (2011) 61–68
microphone, recording and attaching an audio explanation to each digital slide. Many faculties are in transition, providing hybrid courses where they provide live lectures supplemented with a web site posting of part or all of the lecture. With modern desktop recording systems this process is actually quite easy. Of course, PowerPoint slides, whether delivered live or with an audio track, do not change the fact that both are passive listening methods.
63
Table 1 Programs/platforms for Web-based course development and delivery. Program title Blackboard/WebCT eCollege.com e-Education Moodle (open source) PREP Online SumTotal TopClass
URL www.blackboard.net http://www.ecollege.com www.e-education.com www.moodle.com www.computerprep.com http://www.sumtotalsystems.com/ www.wbtsystems.com
1.4. Faculty time engaged in online education Hidden costs that are uniquely attributable to an ‘‘online course’’ involve the multiple hours it takes to: record an audio note associated with each slide; set up a course web-page inside a learning management system that allows student access to course materials; develop a system for passwords and usernames for each student in the course; and, create (and respond to) an e-mail based distribution list for the students enrolled in the course. Typically these tasks are not needed for a live face-to-face style course. However there are some commercial software programs that attempt to streamline many of the administrative duties just enumerated. 1.5. Software platforms for web-based teaching There are many e-learning software packages that tout their ability to be able to assist faculty and private companies with the development and delivery of online educational modules. The larger ones are essentially designed as an overall learning management system (LMS) that aid the University in student registration, fee collection, grading, and content delivery. The demand for such programs is likely to grow, however presently these platforms are expensive to purchase and set up at the early stages of program development. The larger LMSs often require long learning curves and substantial technical assistance before any educational content is ever delivered online. Moreover, sometimes the individuals that hold the purse strings of a School or University cannot justify the expense since only a few faculties are currently engaged in this type of activity and it is the worry of every administrator that they will spend a large amount of money for a system that sits idle. This expense can be substantial if one considers the annual site license fees for the software programs, the technical support and training staff and the increased capacity of web-server hardware needed. These platforms usually have a large database built-in to the program that allows the program to extensively track a student’s activity with the University (i.e. registration, etc.) as well as their performance in the course (i.e. time online, page hits and student performances on quizzes and tests). These online LMS programs have multiple other features including upgraded security, class bulletin boards, individual student web pages, shared whiteboards and various chat rooms. These features are truly impressive when such a system is fully integrated into the curriculum activity of a University and adequate training and technical support is available. Transformative teaching activities can result. Some examples of the platforms and programs that are available are listed alphabetically in Table 1.
For initial efforts in e-learning, a less expensive program is an open-source system called Moodle. It is not the intent of this article to critique these programs, although others have performed comparative reviews [15–17]. 1.6. Servers—jointly managed or independent One essential fact for a faculty member to establish early on is to determine if his/her school has a server and if so will the faculty member be allowed to host online courses using that resource. If the answer is yes, then move ahead and follow the steps outlined below. If the answer is no, it is still possible to move forward but it will be necessary for the faculty member to either purchase and set up a separate server in the host department or rent server space from a commercial Internet service provider on which to host the online course [18]. Servers are fairly easy to purchase and set up. Once the server availability problem is overcome, then it is logical to move forward with the process of designing and creating educational content for delivery on the Internet. One commonly alleged barrier is that sometimes faculty have the perception that they must wait for their administrators to purchase and implement a School- or University-wide e-learning platform before they can get into online education. This is not so, and online teaching can be achieved once downloading of an program such as an open source software program is completed. Of course, help from a competent Internet/server expert to get started is likely to help solve problems such as low bandwidths and formidable and impenetrable University firewalls. Although these systems are put in place to protect important data from outside hacking, they can be limiting factors when students away from campus want to get access to an online course. The university or school server access restrictions may make it necessary to go through a committee before any changes to security protocols can be made. If these restrictions are able to be negotiated and the server or web-master is astute, then we suggested to the faculty use the University server even if the University has not selected a commercial platform for elearning for there are solutions that can be put in place for a single e-course. 1.7. Evidence about the relative efficacy of web-based teaching Again it is too early to answer this question definitively. There are only a few published articles that directly compare web-based delivery of educational content to traditional lecture
64
G.T. Clark et al. / Journal of Prosthodontic Research 55 (2011) 61–68
based delivery. Some demonstrate no difference in outcome measures [19] or test scores and overall student satisfaction scores for the two methods [20] while other studies showed improvement in test scores for those engaged in a virtual learning environment compared to those receiving the traditional on-site lecture-based education [21]. In contrast with these articles is a recent non-randomized study of students who self-selected for a fully live, hybrid or fully virtual online course in economics [22]. The final scores on their multiplechoice examinations showed a 5–6% point difference between the fully virtual course (lowest) and the other two course types (higher) with no significant differences between the fully live course and the hybrid course outcomes. In our opinion the author conclusion, that choosing a completely on-line course (fully virtual) carries a penalty that would need to be offset by significant advantages in convenience or other factors important to the student is premature until a randomized study is conducted to compare these educational methodologies. Clearly, more assessment is needed along with cost-comparison studies, especially as the methodologies underlying this teaching strategy become more mature. We suspect that the real barriers to online education are the initial difficulty with implementation and reliability of student access to a fast Internet connection, not the relative cost of producing an online lecture versus a real-time lecture or any difference in efficacy between the methods. Moreover, progressively more students have broadband online connections in their homes or dormitories. We also expect that the difficulty of implementation barrier will diminish as the software programs become increasingly user-friendly and the comfort of faculty increases in the domain of digital data manipulation and web-page and website management. 1.8. Moving a live-lecture presentation to a streaming online lecture presentation The methods of providing educational content at a distance are many and most of the time it will be factors such as the target audience and budgetary constraints that will dictate the presentation methodology to be employed. It is our opinion that non-computer based distance education methodologies, which include mailed video and audiotape or CD based programs and mailed self-paced text and image-based student workbook programs solve the problem of the geographically and schedule challenged student. These methods are generally the least expensive way to deliver distance education but cannot be classified as an instructor-lead, interactive course. Without an instructor, such courses are not much better than buying a textbook and reading it. A much more engaging method of online teaching is to use desktop recording software to record a live lecture that is combined with a web-conferencing system so that students can log on while the recording is being made and post-questions, usually via a chat bar. Desktop recording requires a microphone and capture software (e.g. www.camtasia.com) so that the lecturer can provide a voice over as slides (e.g. PowerPoint by Microsoft) are being displayed. If the lecturer has a fast Internet connection, then
Table 2 Programs/platforms for creating streaming video-based lectures. Program title ViVu GoToMeeting WebEx ooVoo Adobe Connect Now Camtasia
URL www.ViVu.tv www.GoToMeeting.com www.webex.com www.oovoo.com www.adobeconnect.com www.techsmith.com/camtasia
adding a webcam or setting up a camcorder allows the lecturer to use web-conferencing software (e.g. www.vivu.tv) (Table 2). This allows the lecturer the option of showing him/herself in a headview or while at the white board if drawing on the board is part of the lecture process. The instructor can share his/her desktop screen or show slides as well as have student interactions via the chatbar while at the same time recording the entire interaction which is then prepared for streaming. In fact, the link to the video feed is often ready for viewing by students who want to review the material again in less than a few minutes. During such reviews there would be no live chatbar feature for the students to ask or answer questions thus resulting in a static model with no interactivity. This model of creating audio/video lessons that can be streamed and delivered over the Internet solves the time zone issues, but unless it is viewed in real time but all students, there will be little interactivity. 1.9. The role for E-mail In the typical course prior to the Internet, student-to-faculty interactivity occurs both during and following the live-lecture via questions asked and answered or during the designated ‘‘office hours’’ of the faculty. This interactivity can be built into an Internet course using Skype or instant messaging. In these sessions students can generate private or semi-public (distributed to the entire class via a distribution list of the students in the course) as part of a standard e-mail program (e.g. Microsoft Outlook). This distribution list is typically assigned a nickname that when entered in the e-mail’s ‘‘Send To’’ field, allows the instructor’s response to the student question (contained in the body of the e-mail communique´) to be automatically sent to everyone listed. Regardless of the method, it must be emphasized that gaining some degree of interactivity between the student and faculty is a necessity in order to help the student internalize the material. 1.10. How to achieve active learning with online homework assignments? Engaging the student in a work project that actually utilizes the concepts being taught promotes ‘‘active learning’’ and helps to achieve the maximum development of a student’s talent [23,24]. An example of an active learning project requiring indepth thinking rather than short term memorization would be a weekly homework activity that involves a task using the concepts taught in the course. The additional advantages for
G.T. Clark et al. / Journal of Prosthodontic Research 55 (2011) 61–68
this type of homework include: one-on-one faculty-to-student interactions; the student must become an active participant; and a record of the degree and nature of the student–faculty interactivity is generated (assuming e-mail is used to receive and respond to the students). 1.11. Testing types: open or closed book and proctored Most educators agree that some student assessment is required regardless of whether a course is live or on the Internet and there are many different methods available to evaluate a student’s mastery of course content. The general rule is that, when there is a need to carefully distinguish and rank student performance, proctored and closed book testing is required. A contrasting method is non-proctored and open-book testing which if done online has significant cost savings and provides immediate feedback to the student. No attempt will be made to compare or contrast these different methods in this article except to cite one study in which student achievement on openbook versus closed-book examinations in a higher education setting found no significant differences in total examination scores between the two types [25]. If the instructor deems that a proctored, closed-book test is not needed or impractical to provide a reasonable alternative online model would be an open-book, non-proctored test, which most LMS’s have the ability to deliver and automatically grade. Timed delivery of the exam is possible as is the automatic recording of the user’s IP address, time spent and time of entry into the exam site. The course instructor can set the quiz for multiple or single attempts and provide individual question feedback as well that can be simple (e.g. correct or incorrect) or more complex (e.g. the rationale for why an answer is correct or incorrect). With this feedback an additional learning opportunity is presented. Nonproctored, open-book online testing is a reasonable strategy for assessment especially if the instructor uses multiple small tests or periodic quizzes as a way to encourage the student to review the virtual-lecture presentations and not as a method to determine the final grade. There are however disadvantages to this exam method (e.g. another unauthorized person could take the test in place of the student or another student could take the test and give out the answers). Such deceitfulness is not preventable with online non-proctored testing however such potential for misuse should be weighed against the advantage of allowing the student to take the test whenever and wherever they want and receive immediate feedback. 1.12. Avoiding copyright violations in an online course While it is not a violation of U.S. copyright law, Title 17 of the United States Code, for an educator to use a small portion of copyrighted material for educational purposes, it is always wise to ask for and receive permission to use this material. This exception to copyright law giving limited permission to educators is called the ‘‘fair use’’ clause. While some may interpret this differently, the generally accepted interpretation is that educators, under most circumstances, may copy a single chapter from a book; an excerpt from a work combining language
65
and illustrations not exceeding two pages or 10% of the work, an article, short story, or essay of 2500 words or less, or excerpts of up to 1000 words or 10% of a longer work, whichever is less; or a single chart, graph, diagram, drawing, cartoon, or picture from a book, periodical, or newspaper. However, once an educator moves from distributing paper copies in a classroom to copying, altering, and/or distributing information collected from the Internet, the ‘‘fair use’’ clause may not apply. There are currently several potential actions being considered by the U.S. Congress which will update this law as it applies to distance education over the Internet for the use of the Internet for teaching has engendered a variety of scenarios that make the interpretation of the ‘‘fair use’’ clause unclear. For example if an educator has a size-limited class who have unique passwords for the Internet course materials and a clear understanding by all students that they cannot recopy or redistribute the material, then this situation is analogous to live-class room activity and the ‘‘fair use’’ may apply. Conversely posting material that is copyrighted without permission on a website where open access can occur by anyone, clearly does not fit within the ‘‘fair use’’ clause. Copyright means that any original work of authorship that is a tangible expression is copyrighted immediately upon creation. An author does not have to register the work, announce that the work is copyright protected, or display the copyright symbol to enjoy copyright protection. When in doubt, assume a work, picture or photo is copyrighted and ask permission to use it. Instructors should not make the mistake of believing that including quotation marks or a line of attribution satisfies copyright requirements. If you properly quote and/or credit a work’s author, experts say, you cannot be accused of plagiarism, but you may still be accused of copyright infringement. 2. Case report: design of an online clinical research design course 2.1. Structure A 10-week course was set up and conducted with the stated purpose of teaching young faculty and residents in the discipline of Prosthodontics about clinical research design. 2.2. Students There were 19 students (5 female and 14 males) who were identified via an email announcement that went to the all members of the Japanese Society of Prosthodontics. Exclusion criteria include: the inability to log in to the website and participate in the video conference because the student did not purchase or have access to a web-camera and a microphoneheadset. Inclusion criteria were being: a member of Japan Prosthodontic Society or Showa Dental Association; under the age of 45 years old; and able to access the Internet using Skype. Potential students submitted an application form and final candidates were selected. From the 19 applicants 3 were excluded because they could not log-on successfully from within their University (firewall issues) or did not have adequate bandwidth at their homes to participate in the video conference.
66
G.T. Clark et al. / Journal of Prosthodontic Research 55 (2011) 61–68
Table 3 Student questionnaire results (n = 13). Statements
Slightly, moderately or strongly disagree
Neutral opinion
Slightly, moderately or strongly agree
I very much enjoyed this form of teaching! I believe online teaching is as effective as live face-to-face teaching! I would have liked this course better if it were offered in Japanese language! I would like to try additional online courses in my field! I would recommend online courses to my colleagues! I found it easy to use the course web-site! I found the textbook very helpful in this course! I found the live video conferences very helpful! I found the videos of the lectures very helpful! I found the PDF handouts of the lectures very helpful! I found the weekly quizzes very helpful! I found the article reviews I had to write very helpful! I found the research protocol I had to write very helpful! The biggest problem I had with this course was my spoken English language ability! The biggest problem with this course was the lack of time necessary for me to complete the work!
0.0% 46.2% 15.4% 0.0% 0.0% 7.7% 0.0% 15.4% 7.7% 7.7% 0.0% 0.0% 0.0% 30.8% 15.4%
15.4% 7.7% 7.7% 0.0% 7.7% 30.8% 7.7% 15.4% 7.7% 7.7% 7.7% 15.4% 7.7% 0.0% 15.4%
84.6% 46.2% 76.9% 100% 92.3% 61.5% 92.3% 69.2% 84.6% 84.6% 92.3% 84.6% 92.3% 69.2% 69.2%
The 16 enrolled students were 4 females and 12 males (mean age 1 SD = 30.2 3.2 years). 2.3. Server All activity associated with the course was hosted on a windows based server running the PHP and Server 2005 operating systems. 2.4. Learning management system The open-source learning management system [www.moodle.com], allowed us to create a course website with a unique log-in password for each student. We placed on this website course information including an introductory statement, a detailed description, a policy on testing and remediation and a recommended textbook as well as weekly activity links. For example, each week contained links for downloading a PDF of the PowerPoint slides used in the lecture, a link to a streaming lecture recording, a multiple choice quiz based on the lecture, a description and sample write up of the homework assignment (total of 9 assignments) and a link for each student to upload their completed homework. 2.5. Weekly video conferencing A video conferencing system using a commercial video conferencing software (VCS) program [http://vivu.tv/vivuweb/ ] was utilized to host a question and answer session on a weekly basis with all students in the course. Each of the enrolled students who attended the live video conference was asked to post 2 questions for discussion during the conference session. 2.6. Desktop lecture recording system The VCS used in this course allowed lectures to be recorded and archived for viewing at a later time. These recording were
made with a SONY HDD camcorder (HDR-SR12) camera. For the various video conferences held with the students, a 2.0 megapixel web camera and headphone system was used [www.logitech.com; Webcam Pro 9000 and Premium Notebook Headset]. 2.7. Textbook The textbook selected for this course was available in both English and Japanese [Hulley, Cummings, Browner, Grady, Newman. 3rd ed. 2009, Lippencott, Williams and Wilkens; ISBN: 0781782104]. 2.8. Student opinion questionnaire An anonymous student opinion questionnaire was developed and sent out to all 16 students in this course (Table 3). The questionnaire focused on the student attitudes regarding the various online teaching methods used in this course. The questionnaire used a set of statements and asked each student to rate the statement using a likert scale: ( 3) strongly disagree, ( 2) moderately disagree, ( 1) slightly disagree, (0) no opinion, (1) slightly agree, (2) moderately agree, and (3) strongly agree. 3. Results from an online clinical research design course 3.1. Student drop-outs Of the 16 students enrolled, only 11 (69%) completed all aspects of the course on time and received a passing grade (total score 70%; range: 83–96%). There were three major obstacles to the smooth implementation of this course. First, low bandwidth (a bandwidth of >1.0 MBPS uplink was required) which made the audio and video conference sounds/images from the students’ frequently of such poor quality that we had to resort to instant messaging only for several students. To successfully
[()TD$FIG]
[()TD$FIG] G.T. Clark et al. / Journal of Prosthodontic Research 55 (2011) 61–68
Fig. 1. The number of times viewed for the two passive elements of the course: the eight PowerPoint lectures saved in a PDF file format for download; and, the eight streaming videos that were recorded and available for viewing each week.
record and deliver high quality video recordings, it was necessary for the instructors to set-up a high speed dedicated network connection that was outside of the host University’s firewall. Second, University firewall restrictions with the use of nonstandard ports for the LMS [a standard port is usually 8080] made it impossible for some students to log on and view the course materials at their University resulting in their use of their personal home ISP provider. Third, some students had limited spoken English language proficiency, although all students had passed a written English language proficiency test before entering dental school. This resulted in more one-way oral discussions with the students only responding using instant messaging. 3.2. Student online activity data Fig. 1 shows the number of times viewed for the two passive elements of the course: the eight power-point lectures saved in a PDF file format for download; and, the eight streaming videos that were recorded and available for viewing each week. Several observations can be made of this data including the finding that the 16 students in the course viewed each of the PDF lecture handouts several times with the number of viewings decreasing greatly as the course progressed as was true of number of lecture video viewings. We suspect that the reason the PDF files were viewed multiple times since the quiz questions were taken directly from the content in these files. Progressively fewer viewings of the PDF files could be attributed to the possibility that the students were downloading the files or printing them out and thus not logging on each time they wanted to refer to the notes. We also speculate that the video files were not viewed as much as the PDF files because of the difficultly this group of students had with understanding spoken English. The streaming videos were not downloadable and by the fifth lecture some students were not viewing the lectures at all since the total number of views was less than the number of students in the course. It is not possible to know how much this decrease was due to conflicting activities, such as their required teaching duties or home responsibilities since this course was delivered in the evening, which took precedence. The fifth week of the course was when the summer break ended and the new academic year began in
67
Fig. 2. The plot of the individual student’s combined number of viewings of the two passive elements (handouts and videos) against their final score based on the two active elements that had associated scores (quizzes and homework assignment scores combined). A trendline was also calculated and is shown in the plot as the r-squared value (r2 = 0.2461).
Japan and this is usually a very busy time for a junior faculty. Alternatively, the diminished viewing could be related to either a general loss of interest in the course or that the students were more efficient as the course progressed. Regarding the three active elements of the course, the 16 students in this course had the opportunity to take 1 quiz a week for 8 weeks resulting in a total of 128 quizzes possible. The students completed 111 of the quizzes (86.7%). All quizzes were required and each was usually attempted twice by most students, even though the students were advised that only the first attempt would be scored. There were 144 potential homework assignments (16 students times 9 assignments each) and 114 (79.2%) were completed. The possible number of participating events for the video conference recordings were 128 (16 times 8 weeks) but only 94 students (73.4%) logged onto these conferences and participated. The vast majority (84.8%) of the missing active elements (i.e. quizzes, homework assignments and video conference log-ins) were attributed to the five students who did not complete the course. In Fig. 2 we plotted the individual student’s combined number of viewings of the two passive elements (handouts and videos) against their final score based on the two active elements that had associated scores (quizzes and homework assignment scores combined). This plot only includes the data from the 11 students who finished all elements of the course against the total number of views of the PDF files and the video files. Since the 5 excluded students from this plot did not complete most of the active elements of the course (quizzes and homework), they did not have a valid final score. A trendline was also calculated and is shown in the plot as the r-squared value (r2 = 0.2461). This result suggests that the number of informational/passive elements viewed by the students was positively correlated with the students final score in the course. 3.3. Student opinion data Student attitudes collected anonymously from 13 out of 16 students about the various elements of our online course are presented in Table 3. For analysis we grouped the 3 agree and 3
68
G.T. Clark et al. / Journal of Prosthodontic Research 55 (2011) 61–68
disagree responses into two results respectively leaving the neutral response alone. These data show several interesting findings. For example, the students were equally split (agree and disagree = 46.2%) regarding whether face-to-face teaching is as effective as online teaching; a majority of the students (76.9%) stated that they would have liked the class better if it were offered in Japanese; and 69.2% of the students stated their limited spoken English language ability was their biggest problem in the course. In contrast to these responses are the following outcomes: 84.6% of the students agreed that they liked this form of teaching; 100.0% agreed that they would like to try another online course in their field; and, 92.3% of the students agreed that they would recommend an online course to their colleagues. Finally, a strong majority of the students rated the various elements of the course as very helpful (range 61.5–92.3%). 4. Summary and conclusions There is much written about the promise of online teaching by some very distinguished authors. Some Universities and probably many individual faculties have tried and failed to deliver on the promise for various reasons. As it currently exists, web-based teaching may not prove to be cheaper but these extra costs are not the main barrier to using it. Instead, we speculate that the additional demands on the faculty that online teaching presents is likely to be the major barrier. Web-based education will make it easier for the student to acquire knowledge but it will not likely be easier for the faculty to create it; depending on the faculty, it may increase the depth and diversity of content being provided. The concern that students will have access problems in retrieving the content online due to lack of web access is rapidly fading and in most modern Universities access is no longer a problem. Regarding the issue of central support for a comprehensive e-learning software platform, there is one open-source option (e.g. Moodle) for faculty who find their University is not moving forward on this issue with alacrity, although they may need to buy server space from a commercial vendor to mount such a system. Developing and deploying a streaming lecture and hosting live video conferences is also not a major barrier and there are commercial options for faculty to solve this problem. Finally, creating a ‘‘good’’ course requires active elements including online quizzes and weekly homework assignments. Increased student– faculty interaction occurs when question–answer based live video conferences between the participants is an element of the course. Acknowledgement The clinical research design course was supported by Japan Prosthodontic Society (JPS) and Showa University. The authors appreciate significant contributions of Dr. Keiichi Sasaki (President of JPS), Hirofumi Yatani (Director of JPS scientific committee), Takashi Miyazaki (Dean of Showa University
Dental School), and the members of international liaison committee of JPS to the course. References [1] Kerry R. The power of internet learning. Report by the US Congress Web-based Education Commission. www.webcommission.org (posted on 19.12.2000). [2] http://www.mext.go.jp/a_menu/koutou/houjin/03052713.htm (in Japanese), April 2004. [3] http://www.mext.go.jp/a_menu/koutou/itaku/08090305/012.htm (in Japanese), March 2007. [4] Mangan KS. SUNY–Buffalo drops online M.B.A. program. Chronicle of Higher Education 2002. February 20. [5] Garrett R. The real story behind the failure of U.K. eUniversity. Educause Q 2004;27:4–6. [6] Clark GT. Education problems and Web-based teaching: how it impacts dental educators. J Am Coll Dent 2001;68:25–34. [7] Stout PA, Villegas J, Kim H. Enhancing learning through use of interactive tools on health-related websites. Health Educ Res 2001;16:721–33. [8] Schittek M, Mattheos N, Lyon HC, Attstrom R. Computer assisted learning. A review. Eur J Dent Educ 2001;5:93–100. [9] Mattheos N, Schittek M, Attstrom R, Lyon HC. Distance learning in academic health education. Eur J Dent Educ 2001;5:67–76. [10] Chodorow S. Educators must take the electronic revolution seriously. Acad Med 1996;71:221–6. [11] Davis MJ, Wythe J, Rozum JS, Gore RW. Use of World Wide Web server and browser software to support a first-year medical physiology course. Am J Physiol 1997;272:S1–14. [12] Davis LG, Winstanley RB, Duffin R, Griffiths A. Dental education resources on the Web. Eur J Dent Educ 1998;2:19–24. [13] Jaffee D. Institutionalized resistance to asynchronous learning networks. J Asynchron Learn Netw 1998;2:21–32. [14] Allen KW. The use of slides for teaching reference. J Educ Librarian 1965;6:137–9. [15] Croy M, Smelser R, McAlpin V. Report to the provost from the learning management system evaluation committee. May 15, 2009. URL: http:// lmseval.uncc.edu/index.php?option=com_content&task=view&id=62& Itemid=9 (last accessed 15.10.10). [16] Jensen B. History and future of course authoring technologies: including predictions for the 21st century and knowledge portals. Trinity University, 2006. URL: http://www.trinity.edu/rjensen/290wp/290wp.htm# OnsiteOnline. [17] Course Management and Delivery Tools, Web Course Resources, and Web Development Tools. SyllabusWeb Buyers Guide Archive. January 2002. URL: http://www.syllabus.com/syllabusmart/bg_archive.asp. [18] Beginner’s Guide to Hosting. HelpWithHosting.com. March 31, 2002. URL: http://www.helpwithhosting.com/beginners_guide/. [19] English T, Harrison AL, Hart AL. A distance learning model in a physical therapy curriculum. J Allied Health 1998;27:228–32. [20] Woo MA, Kimmick JV. Comparison of Internet versus lecture instructional methods for teaching nursing research. J Prof Nurs 2000;16:132–9. [21] Goldberg HR, McKhann GM. Student test scores are improved in a virtual learning environment. Adv Physiol Educ 2000;23:59–66. [22] Brown BW, Liedholm CE. Can web courses replace the classroom in principles of microeconomics? Lessons from microeconomics. Am Econ Rev 2002;92:444–8. [23] Astin AW. Educational assessment and educational equity. Am J Educ 1990;98:458–78. [24] Astin AW. Assessment for excellence: the philosophy and practice of assessment and evaluation in higher education. American Council on Education/Macmillan Series on Higher Education (#ED327106). Washington, DC: American Council on Education; 1991. [25] Ioannidou MK. Testing and life-long learning: open-book and closedbook examination in a university course. Stud Educ Eval 1997;23:131–9.
Journal of Prosthodontic Research 55 (2011) 61–68 www.elsevier.com/locate/jpor
Review
Developing and providing an online (web-based) clinical research design course in Japan: Lessons learned Glenn T. Clark DDS, MSa, Roseann Mulligan DDS, MSb, Kazuyoshi Baba DDS, PhDc,* b
a Orofacial Pain and Oral Medicine Center, Herman Ostrow School of Dentistry, University of Southern California, Los Angeles, CA, USA Community Dentistry Programs and Hospital Affairs, Herman Ostrow School of Dentistry, University of Southern California, Los Angeles, CA, USA c Department of Prosthodontics, School of Dentistry, Showa University, 2-1-1 Kitasenzoku, Ohta-ku, Tokyo 145-8515, Japan
Received 17 January 2011; accepted 30 January 2011
Abstract This article reports on the lessons learned while teaching an 8-week-long online course about the principles of clinical research design in Japan. Student activity data and how it relates to performance in the course are presented. As prolog, this article focuses on the barriers and solutions to creating and delivering a web-based course and it lists and discusses the most common concerns that educators often have about this process, namely, cost of the system and time requirement of the faculty. Options that must be considered when selecting the support software and hardware needed to conduct live streaming lecture, online video-based conference course are presented. The ancillary role of e-mail based distribution lists as an essential instruction tool within an interactive, instructor-supervised online course is discussed. This article then discusses the inclusion of active learning elements within an online course as well as the pros and cons regarding open-book versus closed book, proctored testing. Lastly, copyright issues the online instructor should know about are discussed. The student tracking data show that as the course progresses, students will reduce the number for page viewings. We speculate that this reduction is due to a combination of conflicting priorities plus increasing efficiency of the students at extracting the critical information. The article also concludes that software and hardware costs to deliver an online course are relatively minor but the faculty’s time requirement is initially substantially higher than teaching in a conventional face-to-face course. # 2011 Japan Prosthodontic Society. Published by Elsevier Ireland. All rights reserved. Keywords: Clinical research design; Online course; Web-based course; Video conference
Contents 1.
2.
Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.1. Why create and deliver a web-based course? . . . . . . . . . . . . . . . . . . . . . . . 1.2. What is active learning and why is it important? . . . . . . . . . . . . . . . . . . . . 1.3. The expense involved . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.4. Faculty time engaged in online education . . . . . . . . . . . . . . . . . . . . . . . . . 1.5. Software platforms for web-based teaching . . . . . . . . . . . . . . . . . . . . . . . . 1.6. Servers—jointly managed or independent . . . . . . . . . . . . . . . . . . . . . . . . . 1.7. Evidence about the relative efficacy of web-based teaching . . . . . . . . . . . . . 1.8. Moving a live-lecture presentation to a streaming online lecture presentation 1.9. The role for E-mail. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.10. How to achieve active learning with online homework assignments? . . . . . . 1.11. Testing types: open or closed book and proctored . . . . . . . . . . . . . . . . . . . 1.12. Avoiding copyright violations in an online course . . . . . . . . . . . . . . . . . . . Case report: design of an online clinical research design course . . . . . . . . . . . . . . 2.1. Structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2. Students . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.3. Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . .
* Corresponding author. Tel.: +81 3 3787 1151; fax: +81 3 9784 6468. E-mail address: [email protected] (K. Baba). 1883-1958/$ – see front matter # 2011 Japan Prosthodontic Society. Published by Elsevier Ireland. All rights reserved. doi:10.1016/j.jpor.2011.01.004
. . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . .
62 62 62 62 63 63 63 63 64 64 64 65 65 65 65 65 66
62
3.
4.
G.T. Clark et al. / Journal of Prosthodontic Research 55 (2011) 61–68
2.4. Learning management system . . . . . . . . . . . . . 2.5. Weekly video conferencing . . . . . . . . . . . . . . . 2.6. Desktop lecture recording system . . . . . . . . . . . 2.7. Textbook . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.8. Student opinion questionnaire . . . . . . . . . . . . . Results from an online clinical research design course . 3.1. Student drop-outs . . . . . . . . . . . . . . . . . . . . . . 3.2. Student online activity data . . . . . . . . . . . . . . . 3.3. Student opinion data . . . . . . . . . . . . . . . . . . . . Summary and conclusions. . . . . . . . . . . . . . . . . . . . . Acknowledgement . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
1. Introduction 1.1. Why create and deliver a web-based course? In December of 2000, a bipartisan, US Congressional committee published a comprehensive report on the issue of Internet usage and education [1] concluding that the promise of web-based education lay in its unique ability to: allow studentcentered rather than classroom-centered learning; focus on the strengths and needs of individual learners; and make life-long learning a reality. They summarized their findings with a call for substantial continuous and relevant training and support for educators in how to develop and deliver web-based content to students. In 2004, the Japanese Ministry of Education, Culture, Sports, Science and Technology established a ‘‘media education development center’’ in order to promote e-Learning in university education [2]. In 2007 this same Ministry published an update on the current status and future prospective of e-Learning in Japan which expressed support for this form of education [3]. In contrast with these actions in 2002 the State University of New York (SUNY) at Buffalo dropped its online M.B.A. program [4] and in 2004 the United Kingdom’s eUniversity (UkeU) shut down as well. Purportedly the SUNY program was ended as online programs weren’t ‘‘worth the expense and hassle’’ whereas the UKeU program was doomed due to multiple factors including substantial up-front investment of public monies, lack of private sector funding, low enrollments that missed targeted projections, brand confusion and an incomplete in-house platform on which to operate the system [5]. In the opinion of one author had impatience for results not taken hold, this elearning venture would probably have covered its costs in another five years and become a major online brand. There are also many success stories in online higher education and herein we will discuss only how to build one good course. One principle that applies is ‘‘a bad course which is put online is still a bad course, only now more expensive.’’ 1.2. What is active learning and why is it important? Probably the most important concept for creating a good course, whether online or face-to-face, is the promotion of an environment of ‘‘active student learning’’ rather than passive listening. Active learning is synonymous with increased student
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
66 66 66 66 66 66 66 67 67 68 68 68
involvement in his/her education. Active learning is not a concept exclusive to web-based delivery of educational content and can occur in any classroom. Presumably interactivity and student engagement is the expectation for on-line educational programs. If such a strategy can be achieved the student will acquire knowledge substantially faster with increased depth and diversity of content. Furthermore the knowledge pathway that the students pursue can focus on the individual interests of the student (thus the term student-centered learning was coined). A search of the literature reveals that several medical and dental educators have described their efforts in this arena and generally all are enthusiastic about the method [6–12]. For the educators interested in becoming involved in this type of content delivery, we suggest their enrolling in 2 or 3 different types of online courses to assess from the student perception what works and what does not before launching their own online course. 1.3. The expense involved There is currently very little evidence to suggest that online teaching will eventually prove to be cheaper than traditional methods as the costs are varied. Although a 1998 report describes both faculty and administrators’ resistance to online instruction in part due to costs [13], in general data is lacking on this topic. Online methodologies are relatively immature and subject to much change in the marketplace; therefore it is premature to attempt an assessment of future real costs. Educators currently developing web-based content in today’s climate would agree that compared to a traditional in-class lecture presentation, creating a computer based, web-accessible learning module is neither faster to develop nor cheaper to provide. The necessity to be cheaper-to-develop and/or deliver has not always been an absolute requirement for educational innovation. Effectiveness is typically used as the metric. For example when slides became readily available for use in teaching, they quickly assumed a prominent role due to studies demonstrating their effectiveness over conventional lecturing [14]. While this added a cost element, it was obviously not an impediment to the widespread, rapid adaptation of this method in education. Perhaps the same will also be said in the future of web-based education. Conversion of a standard PowerPoint lecture to an online lecture can be as simple as buying a
G.T. Clark et al. / Journal of Prosthodontic Research 55 (2011) 61–68
microphone, recording and attaching an audio explanation to each digital slide. Many faculties are in transition, providing hybrid courses where they provide live lectures supplemented with a web site posting of part or all of the lecture. With modern desktop recording systems this process is actually quite easy. Of course, PowerPoint slides, whether delivered live or with an audio track, do not change the fact that both are passive listening methods.
63
Table 1 Programs/platforms for Web-based course development and delivery. Program title Blackboard/WebCT eCollege.com e-Education Moodle (open source) PREP Online SumTotal TopClass
URL www.blackboard.net http://www.ecollege.com www.e-education.com www.moodle.com www.computerprep.com http://www.sumtotalsystems.com/ www.wbtsystems.com
1.4. Faculty time engaged in online education Hidden costs that are uniquely attributable to an ‘‘online course’’ involve the multiple hours it takes to: record an audio note associated with each slide; set up a course web-page inside a learning management system that allows student access to course materials; develop a system for passwords and usernames for each student in the course; and, create (and respond to) an e-mail based distribution list for the students enrolled in the course. Typically these tasks are not needed for a live face-to-face style course. However there are some commercial software programs that attempt to streamline many of the administrative duties just enumerated. 1.5. Software platforms for web-based teaching There are many e-learning software packages that tout their ability to be able to assist faculty and private companies with the development and delivery of online educational modules. The larger ones are essentially designed as an overall learning management system (LMS) that aid the University in student registration, fee collection, grading, and content delivery. The demand for such programs is likely to grow, however presently these platforms are expensive to purchase and set up at the early stages of program development. The larger LMSs often require long learning curves and substantial technical assistance before any educational content is ever delivered online. Moreover, sometimes the individuals that hold the purse strings of a School or University cannot justify the expense since only a few faculties are currently engaged in this type of activity and it is the worry of every administrator that they will spend a large amount of money for a system that sits idle. This expense can be substantial if one considers the annual site license fees for the software programs, the technical support and training staff and the increased capacity of web-server hardware needed. These platforms usually have a large database built-in to the program that allows the program to extensively track a student’s activity with the University (i.e. registration, etc.) as well as their performance in the course (i.e. time online, page hits and student performances on quizzes and tests). These online LMS programs have multiple other features including upgraded security, class bulletin boards, individual student web pages, shared whiteboards and various chat rooms. These features are truly impressive when such a system is fully integrated into the curriculum activity of a University and adequate training and technical support is available. Transformative teaching activities can result. Some examples of the platforms and programs that are available are listed alphabetically in Table 1.
For initial efforts in e-learning, a less expensive program is an open-source system called Moodle. It is not the intent of this article to critique these programs, although others have performed comparative reviews [15–17]. 1.6. Servers—jointly managed or independent One essential fact for a faculty member to establish early on is to determine if his/her school has a server and if so will the faculty member be allowed to host online courses using that resource. If the answer is yes, then move ahead and follow the steps outlined below. If the answer is no, it is still possible to move forward but it will be necessary for the faculty member to either purchase and set up a separate server in the host department or rent server space from a commercial Internet service provider on which to host the online course [18]. Servers are fairly easy to purchase and set up. Once the server availability problem is overcome, then it is logical to move forward with the process of designing and creating educational content for delivery on the Internet. One commonly alleged barrier is that sometimes faculty have the perception that they must wait for their administrators to purchase and implement a School- or University-wide e-learning platform before they can get into online education. This is not so, and online teaching can be achieved once downloading of an program such as an open source software program is completed. Of course, help from a competent Internet/server expert to get started is likely to help solve problems such as low bandwidths and formidable and impenetrable University firewalls. Although these systems are put in place to protect important data from outside hacking, they can be limiting factors when students away from campus want to get access to an online course. The university or school server access restrictions may make it necessary to go through a committee before any changes to security protocols can be made. If these restrictions are able to be negotiated and the server or web-master is astute, then we suggested to the faculty use the University server even if the University has not selected a commercial platform for elearning for there are solutions that can be put in place for a single e-course. 1.7. Evidence about the relative efficacy of web-based teaching Again it is too early to answer this question definitively. There are only a few published articles that directly compare web-based delivery of educational content to traditional lecture
64
G.T. Clark et al. / Journal of Prosthodontic Research 55 (2011) 61–68
based delivery. Some demonstrate no difference in outcome measures [19] or test scores and overall student satisfaction scores for the two methods [20] while other studies showed improvement in test scores for those engaged in a virtual learning environment compared to those receiving the traditional on-site lecture-based education [21]. In contrast with these articles is a recent non-randomized study of students who self-selected for a fully live, hybrid or fully virtual online course in economics [22]. The final scores on their multiplechoice examinations showed a 5–6% point difference between the fully virtual course (lowest) and the other two course types (higher) with no significant differences between the fully live course and the hybrid course outcomes. In our opinion the author conclusion, that choosing a completely on-line course (fully virtual) carries a penalty that would need to be offset by significant advantages in convenience or other factors important to the student is premature until a randomized study is conducted to compare these educational methodologies. Clearly, more assessment is needed along with cost-comparison studies, especially as the methodologies underlying this teaching strategy become more mature. We suspect that the real barriers to online education are the initial difficulty with implementation and reliability of student access to a fast Internet connection, not the relative cost of producing an online lecture versus a real-time lecture or any difference in efficacy between the methods. Moreover, progressively more students have broadband online connections in their homes or dormitories. We also expect that the difficulty of implementation barrier will diminish as the software programs become increasingly user-friendly and the comfort of faculty increases in the domain of digital data manipulation and web-page and website management. 1.8. Moving a live-lecture presentation to a streaming online lecture presentation The methods of providing educational content at a distance are many and most of the time it will be factors such as the target audience and budgetary constraints that will dictate the presentation methodology to be employed. It is our opinion that non-computer based distance education methodologies, which include mailed video and audiotape or CD based programs and mailed self-paced text and image-based student workbook programs solve the problem of the geographically and schedule challenged student. These methods are generally the least expensive way to deliver distance education but cannot be classified as an instructor-lead, interactive course. Without an instructor, such courses are not much better than buying a textbook and reading it. A much more engaging method of online teaching is to use desktop recording software to record a live lecture that is combined with a web-conferencing system so that students can log on while the recording is being made and post-questions, usually via a chat bar. Desktop recording requires a microphone and capture software (e.g. www.camtasia.com) so that the lecturer can provide a voice over as slides (e.g. PowerPoint by Microsoft) are being displayed. If the lecturer has a fast Internet connection, then
Table 2 Programs/platforms for creating streaming video-based lectures. Program title ViVu GoToMeeting WebEx ooVoo Adobe Connect Now Camtasia
URL www.ViVu.tv www.GoToMeeting.com www.webex.com www.oovoo.com www.adobeconnect.com www.techsmith.com/camtasia
adding a webcam or setting up a camcorder allows the lecturer to use web-conferencing software (e.g. www.vivu.tv) (Table 2). This allows the lecturer the option of showing him/herself in a headview or while at the white board if drawing on the board is part of the lecture process. The instructor can share his/her desktop screen or show slides as well as have student interactions via the chatbar while at the same time recording the entire interaction which is then prepared for streaming. In fact, the link to the video feed is often ready for viewing by students who want to review the material again in less than a few minutes. During such reviews there would be no live chatbar feature for the students to ask or answer questions thus resulting in a static model with no interactivity. This model of creating audio/video lessons that can be streamed and delivered over the Internet solves the time zone issues, but unless it is viewed in real time but all students, there will be little interactivity. 1.9. The role for E-mail In the typical course prior to the Internet, student-to-faculty interactivity occurs both during and following the live-lecture via questions asked and answered or during the designated ‘‘office hours’’ of the faculty. This interactivity can be built into an Internet course using Skype or instant messaging. In these sessions students can generate private or semi-public (distributed to the entire class via a distribution list of the students in the course) as part of a standard e-mail program (e.g. Microsoft Outlook). This distribution list is typically assigned a nickname that when entered in the e-mail’s ‘‘Send To’’ field, allows the instructor’s response to the student question (contained in the body of the e-mail communique´) to be automatically sent to everyone listed. Regardless of the method, it must be emphasized that gaining some degree of interactivity between the student and faculty is a necessity in order to help the student internalize the material. 1.10. How to achieve active learning with online homework assignments? Engaging the student in a work project that actually utilizes the concepts being taught promotes ‘‘active learning’’ and helps to achieve the maximum development of a student’s talent [23,24]. An example of an active learning project requiring indepth thinking rather than short term memorization would be a weekly homework activity that involves a task using the concepts taught in the course. The additional advantages for
G.T. Clark et al. / Journal of Prosthodontic Research 55 (2011) 61–68
this type of homework include: one-on-one faculty-to-student interactions; the student must become an active participant; and a record of the degree and nature of the student–faculty interactivity is generated (assuming e-mail is used to receive and respond to the students). 1.11. Testing types: open or closed book and proctored Most educators agree that some student assessment is required regardless of whether a course is live or on the Internet and there are many different methods available to evaluate a student’s mastery of course content. The general rule is that, when there is a need to carefully distinguish and rank student performance, proctored and closed book testing is required. A contrasting method is non-proctored and open-book testing which if done online has significant cost savings and provides immediate feedback to the student. No attempt will be made to compare or contrast these different methods in this article except to cite one study in which student achievement on openbook versus closed-book examinations in a higher education setting found no significant differences in total examination scores between the two types [25]. If the instructor deems that a proctored, closed-book test is not needed or impractical to provide a reasonable alternative online model would be an open-book, non-proctored test, which most LMS’s have the ability to deliver and automatically grade. Timed delivery of the exam is possible as is the automatic recording of the user’s IP address, time spent and time of entry into the exam site. The course instructor can set the quiz for multiple or single attempts and provide individual question feedback as well that can be simple (e.g. correct or incorrect) or more complex (e.g. the rationale for why an answer is correct or incorrect). With this feedback an additional learning opportunity is presented. Nonproctored, open-book online testing is a reasonable strategy for assessment especially if the instructor uses multiple small tests or periodic quizzes as a way to encourage the student to review the virtual-lecture presentations and not as a method to determine the final grade. There are however disadvantages to this exam method (e.g. another unauthorized person could take the test in place of the student or another student could take the test and give out the answers). Such deceitfulness is not preventable with online non-proctored testing however such potential for misuse should be weighed against the advantage of allowing the student to take the test whenever and wherever they want and receive immediate feedback. 1.12. Avoiding copyright violations in an online course While it is not a violation of U.S. copyright law, Title 17 of the United States Code, for an educator to use a small portion of copyrighted material for educational purposes, it is always wise to ask for and receive permission to use this material. This exception to copyright law giving limited permission to educators is called the ‘‘fair use’’ clause. While some may interpret this differently, the generally accepted interpretation is that educators, under most circumstances, may copy a single chapter from a book; an excerpt from a work combining language
65
and illustrations not exceeding two pages or 10% of the work, an article, short story, or essay of 2500 words or less, or excerpts of up to 1000 words or 10% of a longer work, whichever is less; or a single chart, graph, diagram, drawing, cartoon, or picture from a book, periodical, or newspaper. However, once an educator moves from distributing paper copies in a classroom to copying, altering, and/or distributing information collected from the Internet, the ‘‘fair use’’ clause may not apply. There are currently several potential actions being considered by the U.S. Congress which will update this law as it applies to distance education over the Internet for the use of the Internet for teaching has engendered a variety of scenarios that make the interpretation of the ‘‘fair use’’ clause unclear. For example if an educator has a size-limited class who have unique passwords for the Internet course materials and a clear understanding by all students that they cannot recopy or redistribute the material, then this situation is analogous to live-class room activity and the ‘‘fair use’’ may apply. Conversely posting material that is copyrighted without permission on a website where open access can occur by anyone, clearly does not fit within the ‘‘fair use’’ clause. Copyright means that any original work of authorship that is a tangible expression is copyrighted immediately upon creation. An author does not have to register the work, announce that the work is copyright protected, or display the copyright symbol to enjoy copyright protection. When in doubt, assume a work, picture or photo is copyrighted and ask permission to use it. Instructors should not make the mistake of believing that including quotation marks or a line of attribution satisfies copyright requirements. If you properly quote and/or credit a work’s author, experts say, you cannot be accused of plagiarism, but you may still be accused of copyright infringement. 2. Case report: design of an online clinical research design course 2.1. Structure A 10-week course was set up and conducted with the stated purpose of teaching young faculty and residents in the discipline of Prosthodontics about clinical research design. 2.2. Students There were 19 students (5 female and 14 males) who were identified via an email announcement that went to the all members of the Japanese Society of Prosthodontics. Exclusion criteria include: the inability to log in to the website and participate in the video conference because the student did not purchase or have access to a web-camera and a microphoneheadset. Inclusion criteria were being: a member of Japan Prosthodontic Society or Showa Dental Association; under the age of 45 years old; and able to access the Internet using Skype. Potential students submitted an application form and final candidates were selected. From the 19 applicants 3 were excluded because they could not log-on successfully from within their University (firewall issues) or did not have adequate bandwidth at their homes to participate in the video conference.
66
G.T. Clark et al. / Journal of Prosthodontic Research 55 (2011) 61–68
Table 3 Student questionnaire results (n = 13). Statements
Slightly, moderately or strongly disagree
Neutral opinion
Slightly, moderately or strongly agree
I very much enjoyed this form of teaching! I believe online teaching is as effective as live face-to-face teaching! I would have liked this course better if it were offered in Japanese language! I would like to try additional online courses in my field! I would recommend online courses to my colleagues! I found it easy to use the course web-site! I found the textbook very helpful in this course! I found the live video conferences very helpful! I found the videos of the lectures very helpful! I found the PDF handouts of the lectures very helpful! I found the weekly quizzes very helpful! I found the article reviews I had to write very helpful! I found the research protocol I had to write very helpful! The biggest problem I had with this course was my spoken English language ability! The biggest problem with this course was the lack of time necessary for me to complete the work!
0.0% 46.2% 15.4% 0.0% 0.0% 7.7% 0.0% 15.4% 7.7% 7.7% 0.0% 0.0% 0.0% 30.8% 15.4%
15.4% 7.7% 7.7% 0.0% 7.7% 30.8% 7.7% 15.4% 7.7% 7.7% 7.7% 15.4% 7.7% 0.0% 15.4%
84.6% 46.2% 76.9% 100% 92.3% 61.5% 92.3% 69.2% 84.6% 84.6% 92.3% 84.6% 92.3% 69.2% 69.2%
The 16 enrolled students were 4 females and 12 males (mean age 1 SD = 30.2 3.2 years). 2.3. Server All activity associated with the course was hosted on a windows based server running the PHP and Server 2005 operating systems. 2.4. Learning management system The open-source learning management system [www.moodle.com], allowed us to create a course website with a unique log-in password for each student. We placed on this website course information including an introductory statement, a detailed description, a policy on testing and remediation and a recommended textbook as well as weekly activity links. For example, each week contained links for downloading a PDF of the PowerPoint slides used in the lecture, a link to a streaming lecture recording, a multiple choice quiz based on the lecture, a description and sample write up of the homework assignment (total of 9 assignments) and a link for each student to upload their completed homework. 2.5. Weekly video conferencing A video conferencing system using a commercial video conferencing software (VCS) program [http://vivu.tv/vivuweb/ ] was utilized to host a question and answer session on a weekly basis with all students in the course. Each of the enrolled students who attended the live video conference was asked to post 2 questions for discussion during the conference session. 2.6. Desktop lecture recording system The VCS used in this course allowed lectures to be recorded and archived for viewing at a later time. These recording were
made with a SONY HDD camcorder (HDR-SR12) camera. For the various video conferences held with the students, a 2.0 megapixel web camera and headphone system was used [www.logitech.com; Webcam Pro 9000 and Premium Notebook Headset]. 2.7. Textbook The textbook selected for this course was available in both English and Japanese [Hulley, Cummings, Browner, Grady, Newman. 3rd ed. 2009, Lippencott, Williams and Wilkens; ISBN: 0781782104]. 2.8. Student opinion questionnaire An anonymous student opinion questionnaire was developed and sent out to all 16 students in this course (Table 3). The questionnaire focused on the student attitudes regarding the various online teaching methods used in this course. The questionnaire used a set of statements and asked each student to rate the statement using a likert scale: ( 3) strongly disagree, ( 2) moderately disagree, ( 1) slightly disagree, (0) no opinion, (1) slightly agree, (2) moderately agree, and (3) strongly agree. 3. Results from an online clinical research design course 3.1. Student drop-outs Of the 16 students enrolled, only 11 (69%) completed all aspects of the course on time and received a passing grade (total score 70%; range: 83–96%). There were three major obstacles to the smooth implementation of this course. First, low bandwidth (a bandwidth of >1.0 MBPS uplink was required) which made the audio and video conference sounds/images from the students’ frequently of such poor quality that we had to resort to instant messaging only for several students. To successfully
[()TD$FIG]
[()TD$FIG] G.T. Clark et al. / Journal of Prosthodontic Research 55 (2011) 61–68
Fig. 1. The number of times viewed for the two passive elements of the course: the eight PowerPoint lectures saved in a PDF file format for download; and, the eight streaming videos that were recorded and available for viewing each week.
record and deliver high quality video recordings, it was necessary for the instructors to set-up a high speed dedicated network connection that was outside of the host University’s firewall. Second, University firewall restrictions with the use of nonstandard ports for the LMS [a standard port is usually 8080] made it impossible for some students to log on and view the course materials at their University resulting in their use of their personal home ISP provider. Third, some students had limited spoken English language proficiency, although all students had passed a written English language proficiency test before entering dental school. This resulted in more one-way oral discussions with the students only responding using instant messaging. 3.2. Student online activity data Fig. 1 shows the number of times viewed for the two passive elements of the course: the eight power-point lectures saved in a PDF file format for download; and, the eight streaming videos that were recorded and available for viewing each week. Several observations can be made of this data including the finding that the 16 students in the course viewed each of the PDF lecture handouts several times with the number of viewings decreasing greatly as the course progressed as was true of number of lecture video viewings. We suspect that the reason the PDF files were viewed multiple times since the quiz questions were taken directly from the content in these files. Progressively fewer viewings of the PDF files could be attributed to the possibility that the students were downloading the files or printing them out and thus not logging on each time they wanted to refer to the notes. We also speculate that the video files were not viewed as much as the PDF files because of the difficultly this group of students had with understanding spoken English. The streaming videos were not downloadable and by the fifth lecture some students were not viewing the lectures at all since the total number of views was less than the number of students in the course. It is not possible to know how much this decrease was due to conflicting activities, such as their required teaching duties or home responsibilities since this course was delivered in the evening, which took precedence. The fifth week of the course was when the summer break ended and the new academic year began in
67
Fig. 2. The plot of the individual student’s combined number of viewings of the two passive elements (handouts and videos) against their final score based on the two active elements that had associated scores (quizzes and homework assignment scores combined). A trendline was also calculated and is shown in the plot as the r-squared value (r2 = 0.2461).
Japan and this is usually a very busy time for a junior faculty. Alternatively, the diminished viewing could be related to either a general loss of interest in the course or that the students were more efficient as the course progressed. Regarding the three active elements of the course, the 16 students in this course had the opportunity to take 1 quiz a week for 8 weeks resulting in a total of 128 quizzes possible. The students completed 111 of the quizzes (86.7%). All quizzes were required and each was usually attempted twice by most students, even though the students were advised that only the first attempt would be scored. There were 144 potential homework assignments (16 students times 9 assignments each) and 114 (79.2%) were completed. The possible number of participating events for the video conference recordings were 128 (16 times 8 weeks) but only 94 students (73.4%) logged onto these conferences and participated. The vast majority (84.8%) of the missing active elements (i.e. quizzes, homework assignments and video conference log-ins) were attributed to the five students who did not complete the course. In Fig. 2 we plotted the individual student’s combined number of viewings of the two passive elements (handouts and videos) against their final score based on the two active elements that had associated scores (quizzes and homework assignment scores combined). This plot only includes the data from the 11 students who finished all elements of the course against the total number of views of the PDF files and the video files. Since the 5 excluded students from this plot did not complete most of the active elements of the course (quizzes and homework), they did not have a valid final score. A trendline was also calculated and is shown in the plot as the r-squared value (r2 = 0.2461). This result suggests that the number of informational/passive elements viewed by the students was positively correlated with the students final score in the course. 3.3. Student opinion data Student attitudes collected anonymously from 13 out of 16 students about the various elements of our online course are presented in Table 3. For analysis we grouped the 3 agree and 3
68
G.T. Clark et al. / Journal of Prosthodontic Research 55 (2011) 61–68
disagree responses into two results respectively leaving the neutral response alone. These data show several interesting findings. For example, the students were equally split (agree and disagree = 46.2%) regarding whether face-to-face teaching is as effective as online teaching; a majority of the students (76.9%) stated that they would have liked the class better if it were offered in Japanese; and 69.2% of the students stated their limited spoken English language ability was their biggest problem in the course. In contrast to these responses are the following outcomes: 84.6% of the students agreed that they liked this form of teaching; 100.0% agreed that they would like to try another online course in their field; and, 92.3% of the students agreed that they would recommend an online course to their colleagues. Finally, a strong majority of the students rated the various elements of the course as very helpful (range 61.5–92.3%). 4. Summary and conclusions There is much written about the promise of online teaching by some very distinguished authors. Some Universities and probably many individual faculties have tried and failed to deliver on the promise for various reasons. As it currently exists, web-based teaching may not prove to be cheaper but these extra costs are not the main barrier to using it. Instead, we speculate that the additional demands on the faculty that online teaching presents is likely to be the major barrier. Web-based education will make it easier for the student to acquire knowledge but it will not likely be easier for the faculty to create it; depending on the faculty, it may increase the depth and diversity of content being provided. The concern that students will have access problems in retrieving the content online due to lack of web access is rapidly fading and in most modern Universities access is no longer a problem. Regarding the issue of central support for a comprehensive e-learning software platform, there is one open-source option (e.g. Moodle) for faculty who find their University is not moving forward on this issue with alacrity, although they may need to buy server space from a commercial vendor to mount such a system. Developing and deploying a streaming lecture and hosting live video conferences is also not a major barrier and there are commercial options for faculty to solve this problem. Finally, creating a ‘‘good’’ course requires active elements including online quizzes and weekly homework assignments. Increased student– faculty interaction occurs when question–answer based live video conferences between the participants is an element of the course. Acknowledgement The clinical research design course was supported by Japan Prosthodontic Society (JPS) and Showa University. The authors appreciate significant contributions of Dr. Keiichi Sasaki (President of JPS), Hirofumi Yatani (Director of JPS scientific committee), Takashi Miyazaki (Dean of Showa University
Dental School), and the members of international liaison committee of JPS to the course. References [1] Kerry R. The power of internet learning. Report by the US Congress Web-based Education Commission. www.webcommission.org (posted on 19.12.2000). [2] http://www.mext.go.jp/a_menu/koutou/houjin/03052713.htm (in Japanese), April 2004. [3] http://www.mext.go.jp/a_menu/koutou/itaku/08090305/012.htm (in Japanese), March 2007. [4] Mangan KS. SUNY–Buffalo drops online M.B.A. program. Chronicle of Higher Education 2002. February 20. [5] Garrett R. The real story behind the failure of U.K. eUniversity. Educause Q 2004;27:4–6. [6] Clark GT. Education problems and Web-based teaching: how it impacts dental educators. J Am Coll Dent 2001;68:25–34. [7] Stout PA, Villegas J, Kim H. Enhancing learning through use of interactive tools on health-related websites. Health Educ Res 2001;16:721–33. [8] Schittek M, Mattheos N, Lyon HC, Attstrom R. Computer assisted learning. A review. Eur J Dent Educ 2001;5:93–100. [9] Mattheos N, Schittek M, Attstrom R, Lyon HC. Distance learning in academic health education. Eur J Dent Educ 2001;5:67–76. [10] Chodorow S. Educators must take the electronic revolution seriously. Acad Med 1996;71:221–6. [11] Davis MJ, Wythe J, Rozum JS, Gore RW. Use of World Wide Web server and browser software to support a first-year medical physiology course. Am J Physiol 1997;272:S1–14. [12] Davis LG, Winstanley RB, Duffin R, Griffiths A. Dental education resources on the Web. Eur J Dent Educ 1998;2:19–24. [13] Jaffee D. Institutionalized resistance to asynchronous learning networks. J Asynchron Learn Netw 1998;2:21–32. [14] Allen KW. The use of slides for teaching reference. J Educ Librarian 1965;6:137–9. [15] Croy M, Smelser R, McAlpin V. Report to the provost from the learning management system evaluation committee. May 15, 2009. URL: http:// lmseval.uncc.edu/index.php?option=com_content&task=view&id=62& Itemid=9 (last accessed 15.10.10). [16] Jensen B. History and future of course authoring technologies: including predictions for the 21st century and knowledge portals. Trinity University, 2006. URL: http://www.trinity.edu/rjensen/290wp/290wp.htm# OnsiteOnline. [17] Course Management and Delivery Tools, Web Course Resources, and Web Development Tools. SyllabusWeb Buyers Guide Archive. January 2002. URL: http://www.syllabus.com/syllabusmart/bg_archive.asp. [18] Beginner’s Guide to Hosting. HelpWithHosting.com. March 31, 2002. URL: http://www.helpwithhosting.com/beginners_guide/. [19] English T, Harrison AL, Hart AL. A distance learning model in a physical therapy curriculum. J Allied Health 1998;27:228–32. [20] Woo MA, Kimmick JV. Comparison of Internet versus lecture instructional methods for teaching nursing research. J Prof Nurs 2000;16:132–9. [21] Goldberg HR, McKhann GM. Student test scores are improved in a virtual learning environment. Adv Physiol Educ 2000;23:59–66. [22] Brown BW, Liedholm CE. Can web courses replace the classroom in principles of microeconomics? Lessons from microeconomics. Am Econ Rev 2002;92:444–8. [23] Astin AW. Educational assessment and educational equity. Am J Educ 1990;98:458–78. [24] Astin AW. Assessment for excellence: the philosophy and practice of assessment and evaluation in higher education. American Council on Education/Macmillan Series on Higher Education (#ED327106). Washington, DC: American Council on Education; 1991. [25] Ioannidou MK. Testing and life-long learning: open-book and closedbook examination in a university course. Stud Educ Eval 1997;23:131–9.