Critical success factors for the continuation of e-learning initiatives

Internet and Higher Education 22 (2014) 24–36

Contents lists available at ScienceDirect

Internet and Higher Education

Critical success factors for the continuation of e-learning initiatives Tanya J. McGill a,⁎, Jane E. Klobas a,b, Stefano Renzi b a b

School of Engineering and Information Technology, Murdoch University, South St, Murdoch 6150 WA, Australia Carlo F. Dondena Centre for Research on Social Dynamics, Bocconi University, via Roentgen 1, 20136 Milano, Italy

a r t i c l e

i n f o

Article history: Accepted 8 April 2014 Available online 18 April 2014 Keywords: e-Learning Critical success factors Continuance Sustainability Post-secondary education

a b s t r a c t This paper systematically examines conditions associated with continuation of e-learning initiatives in universities. Conditions associated with institutional, developer, instructor, student and technology issues were identified from a review of the literature. Authors of 64 empirical papers describing e-learning initiatives (20 of which had not continued) published in the peer-reviewed literature rated and explained the role of each condition in continuation of their initiative to the time of the study, which was at least three years after all the papers were published. Initiatives reported on at three different times in the development of e-learning between 2000 and 2008 were represented among continued and non-continued initiatives. Conditions associated with learning and student response were well met in both cases. On the other hand, neither continued nor non-continued initiatives were seen to offer much financial advantage to the university. The conditions that distinguished between continued and non-continued initiatives were dominated by characteristics of the technology and institutional support for the initiative, especially financial support. Technology needed to be up to date, but also sufficiently mature or stable, to support continuation. Continued initiatives were also more likely to have involved other people in development and diffusion following the initial implementation. © 2014 Elsevier Inc. All rights reserved.

1. Introduction The term e-learning is widely understood to refer to the use of information and communications technology (ICT) in learning and teaching (Czerniewicz & Brown, 2009; Salmon, 2005). E-learning systems can be observed at both the institutional and the local level in higher education. Institutional systems include learning management systems (LMS), used primarily to manage delivery of course material to enrolled students, and the platforms that support massive online open courses (MOOCs). Local e-learning systems are observed at the level of a single course, class, lesson or learning activity. While investments at both levels can contribute to improvements in learning and teaching (Gunn, 2010), each has its own goals, methods and challenges. Although they might interface with, or use functions within, the LMS of the institution in which the course they are used in is offered, and might later be used beyond the initial course or class for which they were first developed, local e-learning systems are usually developed with a specific teaching or learning purpose in mind and often implemented in the first instance by a single teacher or a small teaching team. When first implemented, they are embedded in learning and teaching in a local level e-learning initiative, the focus of this paper. The broad scope of “e-learning” results in success being studied not only at different levels but also from different points of view. Authors ⁎ Corresponding author. Tel.: +61 8 93602798; fax: +61 8 93602941. E-mail addresses: [email protected] (T.J. McGill), [email protected] (J.E. Klobas), [email protected] (S. Renzi).

http://dx.doi.org/10.1016/j.iheduc.2014.04.001 1096-7516/© 2014 Elsevier Inc. All rights reserved.

who take an institutional point of view often focus on success in terms of the extent of diffusion of e-learning, with success factors related to policy, power, strategy, change management, professional development, the quality and accessibility of institutional technology infrastructure, and pedagogy (Czerniewicz & Brown, 2009; Gunn, 2010; Jenkins, Browne, Walker, & Hewitt, 2011; Nichols, 2008; Salmon, 2005). For other researchers, the technology (whether LMS, MOOC or local level initiative) takes the central role, with the institutional issues constituting the environment in which the e-learning system is used. Success is considered to be determined by system quality and to be an outcome of use and is often described in terms of learning and user satisfaction (e.g. Chiu, Sun, Sun, & Ju, 2007; Chiu & Wang, 2008; Hayashi, Chen, Ryan, & Wu, 2004; Roca, Chiu, & Martínez, 2006). In this paper, we merge both points of view, using a survey to ask an international panel of authors of published papers describing local e-learning initiatives to tell us what made their initiative successful or not. Our indicator of success is continuation of the e-learning initiative, even if in an adapted form, after its initial implementation. In this way, we gather a view of critical success factors that is not biased by a single point of view. 2. Background Academics generally publish descriptions and evaluations of their local level e-learning innovations when they are relatively new and small. The literature therefore abounds with case studies reporting development and implementation of e-learning initiatives, but much less is written about how innovations fare in the long term.

T.J. McGill et al. / Internet and Higher Education 22 (2014) 24–36

The majority of works on the long-term success of e-learning initiatives take an institutional point of view. This literature assumes that successful e-learning initiatives will be adopted more widely following their early implementation. Gunn (2010) made this assumption explicit in her definition of the “sustainability” of e-learning in terms not only of local learning and teaching benefits, but also of “proven potential to be adopted…for use beyond the original development environment” (p. 90). Nonetheless, mechanisms for scaling up from isolated innovations to sustainable e-learning have been difficult to identify (Tham & Werner, 2005). Czerniewicz and Brown (2009) emphasise the need for institutional policy, and Nichols (2008) stresses the importance of top level support, both strategic and financial. Salmon (2005) points out that learning from local e-learning initiatives will inform universities as they make choices about pedagogy and modes of learning, investment in infrastructure, and strategy for institutional change – a theme picked up more recently in light of developments in MOOCs (Stockport, Klobas, & Mackintosh, 2012). Somekh (1998) argues that educational innovations can be subverted and dissipate if there is no longer-term plan for the sustainability and support of the innovation beyond its initial implementation. While this includes financial support for the initiative, it also extends to support in terms of institutional recognition of the time and expertise required to support and maintain initiatives that involve technology. It has been claimed that lack of funding for continuation of elearning initiatives is an issue, even when the educational potential of the initiative has been demonstration in initial implementation (Gunn, 2010). Other authors suggest that, rather than requiring financial support, investment in e-learning should result in financial return and propose that benefits and cost savings are drivers for e-learning (Derouin, Fritzsche, & Salas, 2005; Wang, Wang, & Shee, 2007). Yet others point out that there is little evidence that significant reductions in costs are possible (Romiszowski, 2004; Ruth, 2010) and question whether e-learning technology that is bought on the basis of financial justification has any real chance of delivering financial benefits. E-learning initiatives are also subject to the rapid rate of technology change. Jenkins et al.’s (2011) large survey of e-learning in UK higher education identified that technical problems can be a barrier to continuation. Reliability and robustness of physical infrastructure are thought to be important institutional conditions for successful applications of e-learning technology at all levels (Alexander, 2001; Marshall, 2012). The need to ensure that the technology is consistent with teaching approaches has also been noted (Bates & Poole, 2003; Larsen, Sørebø, & Sørebø, 2009; Salmon, 2005) whilst, at the same time, institutions are advised to formally assess skills and provide participants in e-learning initiatives with targeted training (Marshall, 2012). The existence of suitable institutional technology does therefore not seem likely to be sufficient for continuation of a local e-learning initiative. Technical collaboration and support are also believed to be important as they allow for development effort to be shared and resources to be reused (Gunn, 2011). Lack of awareness of, or failure to use, central support services can contribute to failure to continue (Gunn, 2010). Successive large surveys conducted at the institutional level cite student-driven goals for institutional investment in e-learning: improving the quality of learning and teaching, improving access to off-campus and part-time students and meeting student expectations (Becker & Jokivirta, 2007; Jenkins et al., 2011). Yet, as a guide to assuring continuation of the local initiative (without requiring the initiative to be adopted more widely) these issues may be necessary, but not sufficient, conditions for success. Regardless of discipline or pedagogical stance, there is agreement that local e-learning initiatives (in common with institutional elearning investments) should aim to improve the quality or experience of teaching or learning, or some combination of these outcomes (Alexander, 2001; Bates & Poole, 2003). Students' intentions to continue to use e-learning systems are influenced by their satisfaction with the

25

systems (Limayem & Cheung, 2008), which is in turn affected by the ease of use of the software (Sun, Tsai, Finger, Chen, & Yeh, 2008). It has also been shown that students’ expectations that their teachers use e-learning do influence teachers to adopt e-learning (McGill, Klobas, & Renzi, 2011). Teachers are, of course, critical for both the initial uptake of innovative learning technology (Drent & Meelissen, 2008) and continuation of any e-learning initiative. A major factor believed to be associated with the continuance or otherwise of local e-learning initiatives is the time commitment required of teachers (Alexander, 2001; de Vries et al., 2005; Gunn, 2010; Jenkins et al., 2011; Nichols, 2008). The difficulty of balancing the requirement to maintain research outputs while focussing on teaching innovation can be a problem (Browne, Jenkins, & Walker, 2006; Gunn, 2010). Even if a teacher has no research commitments, blended learning has become the norm in most institutions (Garrison & Vaughan, 2008), and many instructors appear to struggle to balance the demands of their face-to-face teaching with those of online classes or online class components. This issue is exacerbated by the development and maintenance roles that many teachers play in e-learning (Bell & Bell, 2005) and the training that is required to take on these multiple roles (Bell & Bell, 2005; Nichols, 2008). The idea that a teacher should also provide ongoing development and support for technology and systems associated with an e-learning initiative is peculiar to academia. Gunn (2010) emphasises that development requires a different set of skills to teaching, and Guthrie, Griffiths, and Maron (2008) further point out that, in commercial organisations, development is a separate activity to promotion of systems to support diffusion. The literature therefore suggests a mix of conditions or critical success factors for e-learning initiatives, related to institutional support, technology, developers, teachers and student learning and experience, but the relative importance of these factors for continuation of local elearning initiatives in universities is not known. The research described in this paper addresses this gap by directly comparing local e-learning initiatives that have continued with those that have not continued in order to identify the factors that differentiate between them. 3. Method The local level e-learning initiatives of interest in this paper are interventions in which an e-learning innovation that involves new technology, or new (educational) use of existing technology, is introduced. A quasi-experimental research design was used to frame data collection. Rather than attempt to select continued and discontinued initiatives a priori, initiatives were selected from those published in the peer reviewed literature and allocated to the conditions (continuation, non-continuation) on the basis of post hoc author reports of continuation, as described in this section. 3.1. Data collection procedure Two highly ranked international peer-reviewed journals that publish reflective descriptions and evaluations of local e-learning innovations (Journal of Computer Assisted Learning and Computers & Education) and one international e-learning conference series that is included in the Thomson Reuters (previously ISI) Conference Proceedings Citation Index (EdMedia: World Conference on Educational Multimedia, Hypermedia and Telecommunications) provided papers from which elearning initiatives included in this study were identified. The targeted conferences and journals were purposefully selected because, whilst they publish peer-reviewed papers, they also emphasise contributions to practice. Furthermore, they are open to a broad authorship in terms of both country- and discipline-base. Inclusion of the conference proceedings also provided an opportunity to capture some initiatives that might be a little more innovative than those that were published in the formal journal literature, which is typically subject to longer review

26

T.J. McGill et al. / Internet and Higher Education 22 (2014) 24–36

times and offers less space for work in progress. Three publication years – 2000, 2004, 2008 – were chosen to tap into different innovations using the different technologies and learning and teaching methods considered innovative in different time periods at different stages in adoption of e-learning at universities. Much innovation described in 2000 concerned explorations of the potential of the World Wide Web as a somewhat undifferentiated tool for distance learning, online learning, online collaborative learning, blended learning or assessment in these situations (see, for example, the papers listed in the Appendix A and Aggarwal (2000)). By 2004, LMS had become much more prevalent and more innovations sought to take advantage of the opportunities they provided for student/teacher/content/activity interaction in classes of all sizes, modes and subjects; educational researchers were also beginning to experiment with blogs and other online social environments that had been developed without educational purposes in mind (Howard et al., 2005). By 2008, social environments were replacing LMS environments as platforms for educational innovation (Tomei, 2008); notably, however, many of the initiatives listed in the Appendix A mark a return to the concerns of the early years of computer-based learning, taking advantage of adaptive and networked technologies to improve independent learning (and, viewed in retrospect, presaging the development of MOOCs). We set the last publication year for the study at 2008 to ensure that sufficient time had elapsed before data collection in late 2011 for respondents to be able to evaluate if the initiative had continued to become part of normal teaching and learning processes or not. In order to be selected, a paper needed to address teaching innovation in a university setting (rather than school or corporate environments). If the paper described an experiment, the focus had to be on the educational innovation rather than the technology used as a vehicle for testing something else. That is, it needed to be intended to do at least one of the following and to have been shown to have the potential for ongoing use: • • • •

improve learning and teaching outcomes increase student engagement or satisfaction enable more flexible delivery improve quality or ease of assessment.

Over the three years considered, 122 papers were identified that met these criteria. Information about each e-learning initiative that met the criteria for inclusion in the study was obtained from the paper that described it (see the Appendix A for a list of all papers included in the study). The following details were recorded: • • • • • •

Paper citation Description of initiative Publication type Year of publication Country where initiative occurred Discipline studied by students involved.

Contact details for the authors were obtained from the papers or, where the authors were no longer at the institution at which the initiative took place, their current contact email addresses were sought via Internet searches. Invitations to participate were sent to all authors for whom we could identify a current email address. The rationale for this approach was to increase the potential for a response relating to each targeted initiative as well as to obtain different perspectives where they might exist. E-mail messages invited recipients to participate in the study by clicking on a link to a Webbased questionnaire which took approximately 5–10 minutes to complete. 3.2. The questionnaire The core of the questionnaire was a set of items relating to issues that have been proposed in the literature to influence the success, continuation or sustainability of e-learning initiatives. The items were developed specifically for this study and were divided into five sets of factors reflecting the structural division of roles in higher education institutions – institutional factors, developer factors, teacher factors, student factors, and technology factors – for ease of administration. Table 1 identifies source literature that informed the development of the questionnaire. After asking if the e-learning initiative described in their paper had continued or not, the survey software directed participants to the

Table 1 Previous literature on factors associated with continuance of e-learning initiatives. Institutional factors • Management support of the specific innovation (Demirkan, Goul, & Gros, 2010; Gunn, 2010; Marshall, 2012; Nichols, 2008) • Broad management support for e-learning (Gunn, 2010; Marshall, 2012; Nichols, 2008; Somekh, 1998) • Financial support for the ongoing development/operation of the innovation (Gunn, 2010; Nichols, 2008) • Technical support for the ongoing development/operation of the innovation (Bell & Bell, 2005; Gunn, 2011) • Support from colleagues (Derouin et al., 2005; Wang et al., 2007) • Financial benefit (Derouin et al., 2005; Wang et al., 2007) Developer factors • Time available for development (Bell & Bell, 2005; Browne et al., 2006; Gunn, 2010) • Formal recognition of developer role (Alexander, 2001; Birch & Burnett, 2009 • Wider involvement in development/support of the initiative over time (Gunn, 2011) (Derouin et al., 2005; Wang et al., 2007) Teacher factors • Ease of use (Breslin et al., 2007) • Ability to save teachers time (Zhang & Nunamaker, 2003) • Training availability (Bell & Bell, 2005; Marshall, 2012; Nichols, 2008) • Consistency with pedagogical approaches (Larsen et al., 2009; McPherson & Nunes, 2008; Salmon, 2005) • Wider adoption (Demirkan et al., 2010; Gunn, 2010) • Time available for initiative (de Vries et al., 2005; Gunn, 2010; Jenkins et al., 2011; Nichols, 2008) • Formal recognition of teacher’s role (Birch & Burnett, 2009) Student factors • Student satisfaction (Limayem & Cheung, 2008) • Ease of use (Breslin et al., 2007; Sun et al., 2008) • Ability to save students time (Hirscheim, 2005) • Improvement in student learning (Birch & Burnett, 2009) Technology factors • Stability and reliability of technology (Demirkan et al., 2010; Jenkins et al., 2011; Marshall, 2012; McPherson & Nunes, 2008) • Availability of required technology (Alexander, 2001; McPherson & Nunes, 2008) • Affordability (McPherson & Nunes, 2008)

T.J. McGill et al. / Internet and Higher Education 22 (2014) 24–36

items written in present tense and positive direction for continued items and past tense and negative direction for non-continued items. The items are listed in Table 2. The introduction to the teacher set for continued initiatives, and an example from it, is: The list below includes some factors relating to teachers that authors in the field have proposed as potentially influencing the success of elearning innovations and initiatives. Can you please indicate your agreement with the role they played in the continuance of your e-learning initiative? • The innovation is consistent with approaches to teaching taken here

The response scale for all items was a five-point scale anchored only at the end points, to the left: strongly disagree and to the right: strongly agree (no numbers were included with the response scale). Openended questions also gave participants the opportunity to provide additional information about the initiative and their reasons for evaluating it as they did; more than half of the participants took the opportunity to do so. Respondents were also asked to indicate what role or roles they played in the e-learning initiative described in the paper, what training they had received in the use of educational technology, and what training they had received in teaching. After a pilot test of the questionnaire and completion process, undertaken by five academic staff who had participated in e-learning initiatives in two different countries, minor changes were made to clarify some questions.

27

3.3. Sample characteristics Responses to the survey were received from 74 authors associated with 67 initiatives (representing 54.9% of the 122 identified e-learning initiatives). Two surveys with a high percentage of missing responses were omitted as were survey responses from two participants in one initiative whose responses were extreme outliers relative to those referring to other studies. The final data set consisted of responses from 70 authors associated with 64 initiatives. Of the 64 e-learning initiatives included in the analysis, 42 were still ongoing in the original institutions, 20 had not been continued, and two had been discontinued in the original institutions but established elsewhere. The proportion of continued initiatives was consistent across the three years considered (χ2 (1,4) = 0.51, p = .97). The initiatives came from all populated continents, with the greatest concentration from the USA (18.8%), Australia (9.4%), Greece (7.8%), Spain (7.8%) and the UK (7.8%). Four initiatives involved international collaborations. The initiatives also covered a wide range of disciplines (see Table 3), with the most common being ICT (31.3%). This high proportion is consistent with the technical skills often required to develop and implement innovative e-learning. Science (26.6%), education (12.5%) and languages (10.9%) were also well represented, and 10.9% of initiatives related to more than one discipline. Two author responses were received for three of the initiatives included in the analysis. The authors played different roles in the initiatives and had different points of view about the extent to which several of the critical success factors were addressed in the project. Subsequently, because the observations reflected the perspectives of the observers of the initiatives, and there were so few initiatives for which duplicate

Table 2 Items used to evaluate factors associated with continuance of e-learning initiatives. E-Learning initiatives that continued

E-learning initiatives that did not continue

Institutional factors Management supports the continuance of this innovation Management supports e-learning initiatives across the institution Financial support has been made available for the ongoing development/operation of the innovation Technical support has been made available for the ongoing development/operation of the innovation There is support from colleagues The innovation has been financially advantageous for the university (e.g., by saving money or permitting expansion)

Management did not support the continuance of this innovation Management did not support e-learning initiatives Financial support was not made available for the ongoing development/operation of the innovation Technical support was not made available for the ongoing development/operation of the innovation There was no support from colleagues The innovation was not financially advantageous for the university (e.g., by saving money or permitting expansion)

Developer factors The developers have been able to devote sufficient time to the initiative Provision of support for the initiative is a formally recognised part of the developer’s job

The developers did not have sufficient time to support the initiative Provision of support for the initiative was not a formally recognised part of the developer’s job Other people have become involved in the development/support of the initiative over time Other people did not want to get involved in development/support of the initiative Teacher factors The innovation is easy for teachers to use The innovation saves teachers a lot of time Sufficient training for teachers is available The innovation is consistent with the approaches taken to teaching here The initiative has been adopted more widely Teachers have been able to devote sufficient time to the initiative Taking part in the initiative is a formally recognised part of the teacher’s work

The innovation was not easy for teachers to use Incorporating the innovation took teaching staff too much time Insufficient training with the technology was available The was a mismatch between the innovation and the teaching approaches used The initiative was not adopted more widely Teachers were not able to devote sufficient time to the initiative Taking part in the initiative was not a formally recognised part of the teacher’s work

Student factors Students like the innovation The innovation is easy for students to use The innovation saves students a lot of time The innovation has been demonstrated to improve student learning

Students disliked the innovation The innovation was not easy for students to use Using the innovation cost students a lot of time The innovation was not demonstrated to improve student learning

Technology factors The technology used is mature enough to be stable Up to date technology for the innovation is still available The technology used is inexpensive

The technology was not mature Technology changes made the innovation obsolete The technology used was expensive

28

T.J. McGill et al. / Internet and Higher Education 22 (2014) 24–36

responses had been received, it was decided to analyse responses by participant rather than by initiative. The 70 respondents had played a variety of roles in the initiatives, with many having undertaken multiple roles. The most common role was that of teacher (70.0%), followed by online course developer (51.4%) and technical developer (47.1%). Over a fifth of the participants had undertaken all three of these roles (21.4%), and 32.9% had undertaken two of the roles. Other roles that were mentioned included administrative and research roles. Many participants had not received any training in the use of educational technologies (42.9%). Only 11.4% had received training specific to the technology they were using in the initiative and 18.6% had received training through short courses that were not specific to the technology. More than a third of participants (38.6%) had received no teacher training. Just under a third (31.4%) had received teacher training through formal teaching qualification, and the same number had received some teacher training through short courses.

3.4. Data manipulation and analysis The negatively worded items used to collect data for non-continued initiatives were reverse-coded so that responses for both continued and non-continued initiatives were in the same direction. The positive wording was retained for reporting (e.g., Management supported the initiative), and scores can be interpreted as ranging from 1 condition not met to 5 condition well met. We expected that conditions would be more favourable (condition scores would be higher) for continued initiatives than for noncontinued initiatives. The distributions of all condition scores for both groups were approximately normal, so we used t-tests to test for significant differences in the average condition scores for continued and noncontinued initiatives. Responses to open-ended survey questions were broken down into remarks and classified by one researcher using the list of positively worded conditions as a framework. The three authors of this paper agreed 100% on the coding of remarks reported in the results.

4. Results Before identifying the conditions that distinguished between continued and non-continued e-learning initiatives, we report on the extent to which conditions for e-learning continuation were met. This is important in order not to lose sight of the fact that some success factors for e-learning might be met by both continued and non-continued initiatives; these conditions are necessary but not sufficient for success. In the second section, we identify the conditions that distinguish between continued and non-continued initiatives. The third section contains a short summary of the results of statistical tests that showed no significant difference in continuation between ICT-related disciplines and others and no significant effect of participant training on continuation.

Table 3 Summary of the disciplines in which the initiatives were undertaken. Discipline

N

%

Information and communications technologies Science Education Languages Multiple disciplines Other Total

20 17 8 7 7 5 64

31.3 26.6 12.5 10.9 10.9 7.8 100.0

4.1. The extent to which conditions for e-learning continuation were met The relative extent to which each of the 23 conditions for continuation of an e-learning initiative was considered by the respondents to be present across all of the e-learning initiatives is presented in Table 4. In discussing the results, we use responses to the open-ended questions to provide deeper understanding of the scores given to each condition. None of the conditions was extremely poorly met (i.e., had a mean score below 3 on the 5 point scale). On the other hand, many conditions were very well met. We divided the conditions into those that were most prevalent and those that were less prevalent (a plot of the mean scores showed a clear break between Technology is up to date and Management supports this initiative, which we took as the break point for prevalence). The centrality of improving teaching and learning in the initiatives is highlighted by the relatively high levels of: perceptions of improvement of student learning, student liking of the innovations, and consistency with teaching approaches. With the exception of saving students time, conditions associated with learning – including consistency with instructors’ approaches to teaching – and student satisfaction were generally considered by the participants to be well met in the initiatives. Furthermore, technology was rated as predominantly mature or stable, inexpensive, and up to date, and on average, the innovations were considered easy for both teachers and students to use. The following quotes relating to initiatives that did not continue illustrate common perceptions of the benefits of the majority of initiatives, both continued and not continued: The students said they liked [the initiative] a lot, and they wanted more of it. (ID 9, 2004, not continued) Very worthwhile as it changed teacher pedagogy and also allowed students to create knowledge and be innovative. (ID 8, 2004, not continued) The conditions that were generally less well met involved issues of institutional support for the initiative. These included: provision of financial support, technical support and training, formal recognition of participation in the initiative, and the availability of time to devote to

Table 4 Conditions for e-learning success, evaluated by 70 authors of papers describing e-learning initiatives. M

SD

Most prevalent conditions Students like the innovation Innovation is easy for students to use Innovation is consistent with approach to teachinga Technology is sufficiently mature/stable Management supports e-learning Innovation improves student learning Technology is inexpensive Innovation is easy for teachers to use Technology is up to datea

4.34 4.27 4.10 4.00 3.97 3.97 3.97 3.97 3.90

.68 .76 .83 1.12 1.04 0.96 1.05 1.01 1.14

Less prevalent conditions Management supports this initiative Innovation saves students time Instructor training is available Innovation saves instructor time Colleagues provide supporta Developers have sufficient time for the initiative Ongoing technical support has been available Instructor has time to devote to initiative Others involved in development There is formal recognition of instructor involvementa Ongoing financial support has been availablea Support is part of developer's job Initiative is adopted more widely There was financial advantage for the university

3.70 3.64 3.63 3.59 3.52 3.49 3.41 3.34 3.31 3.25 3.22 3.21 3.17 3.10

1.20 1.01 1.02 1.14 .98 1.30 1.36 1.10 1.29 1.35 1.45 1.23 1.25 1.17

Note. an = 69 (otherwise 70).

T.J. McGill et al. / Internet and Higher Education 22 (2014) 24–36

it. The following quotes from initiatives that had continued illustrate the ongoing issues experienced in many initiatives: Technical glitches keep arising or bugs that hadn't been noticed before. These are fixable but require the availability of a technician which is not guaranteed now that the project is finished. (ID 49, 2008, continued) The major problem is time available for the developers to spend on the project. (ID 68, 2004, continued) The role of time in the initiatives was referred to in both positive and negative ways. Whilst many of the innovations contributed to time savings of some sort for students and instructors, participants were very conscious of difficulties in obtaining sufficient time for the ongoing development, support and use of the initiatives. It is not surprising, given the issues relating to financial support, that the lowest ranked condition related to the initiative being financially advantageous. There was also no significant difference between continued and non-continued initiatives in this regard. Despite cost savings having being touted by some authors as a motivation for e-learning initiatives (Derouin et al., 2005; Wang et al., 2007), relatively few of the initiatives appeared to have reached a situation where financial benefits were being realised.

29

availability of ongoing support (2.77), availability of developer time (2.86) and availability of instructor time (2.91). Significant differences between continued and non-continued initiatives were found for 13 of the conditions. Not surprisingly, there was a significant difference between continued and non-continued initiatives for all conditions where the non-continued group had a mean achievement level of less than 3. The size of the difference in access to ongoing financial support for the two groups of initiatives is particularly notable. It appears that elearning initiatives do not achieve financial sustainability easily. Whilst many of the initiatives had obtained some initial funding, initiatives that continued beyond initial funding appeared to require further ongoing financial support. The following quote is typical of the sentiments expressed for initiatives that did not continue and had faced ongoing funding issues: Technology approaches changed in 2005/2006 and we had no financial resources to redesign the platform. (ID 73, 2004, not continued) Even those involved in initiatives that had continued were conscious of limitations imposed by lack of funding: We got a one time financial support which was sufficient for the technology transition, but not sufficient to develop it further. (ID 4, 2000, continued)

4.2. Differences between continued and non-continued initiatives The preceding discussion suggests some similarities in the conditions for sustainable e-learning experienced in both continued and non-continued initiatives. Independent t-tests were used to compare the mean scores of each of the conditions for continued and noncontinued initiatives. Table 5 reports this comparison and is sorted by the size of the difference between the means. Fig. 1 regroups the conditions into the categories we used to present the conditions in the questionnaire to graphically compare the mean scores for continued and non-continued initiatives. Mean scores below 3 can be observed for seven conditions for noncontinued initiatives (only): availability of ongoing financial support (2.24), wider adoption of the initiative (2.50), involvement of others in development (2.64), support as part of the developer’s job (2.68),

In terms of institutional issues, although ongoing financial support, ongoing technical support and management support for the particular initiative were significantly greater for continued initiatives, there was no significant difference in perceptions of general management support for e-learning; the differences were specific to the initiative. The issue of ongoing technical support was also picked up in responses to conditions relating to developer issues. There were significant differences between continued and non-continued initiatives for all three conditions related to development. The environments in which the continued initiatives were developed seem to have offered greater capacity for development. Continued initiatives were more likely to involve more than one person (i.e., other than the study participant) in development, it was more likely that sufficient time was available for developers, and support for the initiatives was more likely

Table 5 Comparison of ratings of conditions for continued and non-continued initiatives. Continued (n = 48)

Ongoing financial support has been available Technology is sufficiently mature/stable Others became involved in development Initiative is adopted more widely Ongoing technical support has been available Developers have sufficient time for the initiative Technology is up to date Support is part of developer's job Management supports this initiative Instructor has time to devote to initiative Innovation is easy for teachers to use Instructor training is available There is formal recognition of instructor involvement Innovation is easy for students to use Innovation saves instructor time Students like the innovation Innovation is consistent with approach to teaching Innovation improves student learning Management supports e-learning Colleagues provide support Technology is inexpensive Innovation saves students time The initiative was financially advantageous

Not continued (n = 22)

t-test Diff

M

SD

M

SD

3.65 4.31 3.63 3.48 3.71 3.77 4.15 3.46 3.94 3.54 4.17 3.79 3.36 4.38 3.65 4.40 4.15 3.98 3.96 3.49 3.94 3.56 3.00

1.25 .78 1.14 1.05 1.29 1.17 .85 1.09 1.06 .94 .93 .92 1.36 .67 1.12 .64 .74 .98 1.03 .91 .98 1.05 1.11

2.24 3.32 2.64 2.50 2.77 2.86 3.33 2.68 3.18 2.91 3.55 3.27 3.00 4.05 3.45 4.23 4.00 3.95 4.00 3.59 4.05 3.82 3.32

1.45 1.43 1.36 1.41 1.31 1.39 1.49 1.36 1.33 1.31 1.06 1.16 1.35 .90 1.18 .75 1.00 .95 1.07 1.14 1.21 .91 1.29

1.41 .99 .99 .98 .94 .91 .81 .78 .76 .63 .62 .52 .36 .33 .19 .17 .15 .02 −.04 −.10 −.11 −.26 −.32

t

df

p (one-tailed)

4.11 3.07 3.16 3.24 2.81 2.83 2.33 2.56 2.55 2.30 2.48 2.01 1.03 1.71 .65 .96 .67 .10 −.16 −.40 −.40 −.99 −1.06

67 27 68 68 68 68 26 68 68 68 68 68 67 68 68 68 67 68 68 67 68 68 68

b.001 .002 .001 .001 .003 .003 .014 .006 .007 .012 .008 .024 .152 .046 .258 .169 .252 .461 .439 .346 .346 .164 .147

30

T.J. McGill et al. / Internet and Higher Education 22 (2014) 24–36

to be a formal part of the developer’s job. The following quote illustrates how a continued initiative transitioned to a more sustainable state in terms of development support: Over time my involvement as course developer and software modifier/ scene builder was reduced. Expertise among users (faculty) to develop courses has grown significantly and has contributed to a collaborative/supportive environment in which faculty work together on course development. The department now has a full time builder to develop scenes for faculty. The project is continuing very well without my ongoing involvement. (ID 41, 2004, continued) The relatively high proportion of initiatives undertaken by teams that involved instructors and/or developers from information technology areas taps into the issues associated with development and technical support. These stakeholders have the interest and skill to successfully develop and support innovations, but the initiatives did not necessarily continue, as illustrated by the following quote from a participant in a non-continued initiative: In our scenario the technical support was not an issue since the research activities were carried out in a computer science faculty. (ID 22, 2004, not continued)

Continued initiatives were also more likely to be built on stable or mature technical platforms that were also perceived to be up to date. Often these were open source products such as PHP and MySQL which are, as one participant noted, “stable and free of charge”. The following quote illustrates the situation that some discontinued initiatives faced: The rapid technology changes made the elearning system become untimely out-dated (ID 59, 2008, not continued) Whilst no significant difference was found with respect to the affordability of the technology used, changes in technology external to the initiatives appeared to have an impact, with continued initiatives being significantly more likely to involve up to date technology. The issue of institutional technology changes flowing on to impact on individual initiatives was also commented on by a number of participants as illustrated by this quote: The development of [the initiative] was not continued because the developer's institution made a technology shift. (ID 58, 2000, not continued) Consistent with responses relating to time for development, instructors in continued initiatives were also more likely to have sufficient time

Fig. 1. Comparison of satisfaction of e-learning sustainability condition scores for continued and non-continued initiatives.

T.J. McGill et al. / Internet and Higher Education 22 (2014) 24–36

to devote to the initiative, and instructor training was more likely to be available. In view of the relatively high numbers of participants who had not received training, and the likelihood that this situation is widespread, the availability of time and training appear to be important factors for continuation, making it possible for wider adoption to occur. Continued initiatives were reported to be more widely adopted than non-continued initiatives. Wider adoption was seen both within the institution and beyond: I retired in July. The project is completely ingrained in the department that housed the initial efforts as the way to offer e-learning and I merely serve as an occasional consultant. I am currently organizing the best users of the initiative software into a consultant group to support other users as they develop their skills in the use of [the learning environment]. (ID 20, 2004, continued) Some participants also mentioned commercialisation of their innovations: Well, getting the commercial partner, and convincing them to invest in a product whose creation we oversaw, that was a very big step for us. If we tried to keep things open-source, we would never have the product we have now, and it would never have seen widespread adoption. (ID 7, 2008, continued) Two other conditions that differed significantly between continued and non-continued initiatives related to ease of use of the innovation. Despite the fact that ease of use was one of the conditions considered to be relatively well met, ease of use for both students and instructors was very high and significantly higher for the continued initiatives. There were no significant differences between continued and noncontinued initiatives for the teaching and learning conditions that received high scores in Table 5: improvement of student learning, student liking of the innovations, and consistency with teaching approaches. The participants in the initiatives appeared to be very committed to using technology to improve learning. There were also no significant differences between continued and non-continued initiatives in time saved for students or instructors. Lack of recognition for the work involved, and contribution made, appeared to be a issue across both continued and non-continued initiatives, with no significant differences found in satisfaction of this condition. The following quotes illustrate the sentiment: I do feel the unit is not really valued at an official level and it has taken a great deal of my time and energy and without official recognition this is not sustainable. (ID 10, 2008, not continued) .. is not recognised as a formal job from the university for the collaborating professors or me. (ID 24, 2008, continued)

4.3. Tests for other effects on continuation There is the potential that other characteristics might influence continuation. These might be unchangeable characteristics of the initiative itself, especially the discipline of study to which it is applied or characteristics of the participant, such as their education and previous training, that affect their ability to successfully implement and sustain the initiative. We tested for potential differences in continuation by participant training and a major disciplinary distinction and found that there were none. Levels of previous training in teaching (χ2(1) = 0.78, p = .38) and educational technology (χ2(1) = 0.03, p = .87) were similar for participants who had been involved in both continued and non-continued initiatives, suggesting that there is no relationship between training and continuation. The importance for continuation of up to date, mature and stable technology, coupled with the relatively high percentage of initiatives in ICT disciplines in the study, raises the question of whether

31

participants in initiatives in IT-related disciplines might be able to compensate for shortcomings in ongoing financial support for development by investing their own time, effort and professional expertise. This, however, was found not to be the case: initiatives in IT-related disciplines were no more likely to continue than those in other disciplines (χ2(2) = 1.76, p = .41). 5. Discussion The sustainability of e-learning is believed to be influenced by a variety of factors. There has, however, been little previous systematic study of the role of these factors (Gunn, 2011; Romiszowski, 2004). The study described in this paper addressed this deficiency by directly comparing e-learning initiatives that have continued with those that have not in order to identify the factors that differentiate between them. The single factor that distinguished most clearly between continued and non-continued initiatives in this study, according to participants in the initiatives, was the availability of ongoing financial support. This finding is not in itself surprising: it makes common sense, and it reflects the concerns of the literature (Gunn, 2010). But, availability of financial support was still low when compared to the extent that other conditions that made the difference between continuing and non-continuing initiatives were met. Continued initiatives are, it seems, able to overcome apparent shortcomings in financial support. The importance of having ongoing financial support is in clear contrast to the perception among participants in both continued and noncontinued initiatives that e-learning initiatives are at best neutral in terms of financial advantage for the university. It also confirms the view of authors who have noted that expected financial advantage is not a valid motivation for investment in e-learning (Romiszowski, 2004; Ruth, 2010). Far from leading to financial benefits, e-learning initiatives often require ongoing financial support to continue. Realistic management of most e-learning initiatives is therefore likely to require careful planning for ongoing financial investment rather than for positive financial returns. Technology issues were also shown to be important in the progression to a sustainable initiative. The maturity/stability of technology used and its currency influenced the continuance of initiatives. This is an interesting finding given the age of the some of the initiatives included in the study (initiatives described in the literature in 2000 would have been more than ten years old at the time of the survey). Those involved in initiatives that were able to continue found ways to ensure that the technology underpinning the initiatives evolved appropriately over time. The higher levels of availability of ongoing technical support for continued initiatives reflect this. The maintenance of technology that is up to date, mature and stable requires investment not only at the level of the local e-learning initiative, but also at the institutional level, as pointed out in open-ended remarks. This observation is consistent with the literature on institutional support for e-learning (Gunn, 2010). Involving others in the development and support of the initiative was also shown to be important for continuance. The vulnerability to staff movements was highlighted in comments from a number of those involved in non-continuing initiatives. This increased involvement provides the stability and continuity that can facilitate wider adoption, with achieving wider adoption of the initiative appearing to be particularly important to sustainability. Participants’ comments about their attempts to gain wider spread adoption suggest that some level of participant commitment to the internal and external promotion of an initiative might differentiate between continued and noncontinued initiatives in some cases. As Gunn (2010) noted, different skills are required to move from initial use in limited circumstances to wider adoption. Organizations can provide support for this shift this once the value of the innovation has been established. Similarly, when teachers and developers felt that they had sufficient time to devote to the initiative, when sufficient training was available to teachers, and when support was a recognised part of the developer’s

32

T.J. McGill et al. / Internet and Higher Education 22 (2014) 24–36

job, the initiative was more likely to continue. High workloads and limited recognition were as freely reported in this study as in others where participants had the opportunity to remark on them. As noted in the institutional literature (Czerniewicz & Brown, 2009; Gunn, 2010; Marshall, 2012), e-learning policies can help to address these issues. 5.1. Limitations and opportunities for further research This survey was the first study of conditions associated with elearning initiatives that reached international participants in a wide variety of initiatives across disciplines. While our approach enabled us to obtain a wide reach, it was also accompanied by some limitations. Only targeting participants who had published on their e-learning initiative was a limitation of the study. Some conditions believed to be associated with e-learning success seemed to be better met than others, regardless of whether the project was able to continue; for example, both continued and non-continued initiatives tended to lead to improvements in student learning and, in general students liked the innovations. These results may, however, reflect the source of the initiatives included in the study. All participants had published a paper about their e-learning initiative and the review process would favour those papers in which the authors were able to clearly describe the learning and teaching outcomes. In addition, restricting the pool to two journals and one conference for reasons of practicality may have compounded this limitation. Future research with a broader range of e-learning initiatives would be valuable. One unexplored explanation of differences between continued and non-continued initiatives is the role of personal characteristics. The observations of Gunn (2010) and Guthrie et al. (2008) that the skills required for diffusion of e-learning differ from those required from teaching and development suggest that some of the personal characteristics associated with marketing and promotion would be valuable. Continued projects may have benefited from the promotional or ‘political’ skills of a participant who was able to ‘sell’ the project to others. This possibility is to some extent supported by ad hoc reference to two survey items which had much higher scores on continued than noncontinued projects: involvement of others in development and wider adoption of the initiative. These items might be indicators of participant behaviours that lead to continuation. Another possibility, for which data are not available, is that participants in continued initiatives felt greater personal ownership and therefore put in more effort toward continuation than others. Neither we, nor anyone else, have specifically set out to study the part played by role, motivation or capability to ‘sell’ an initiative. Research that deliberately studies this, in relation to the prior literature (Gunn, 2010; Guthrie et al., 2008) would be valuable. Another issue for consideration by the scholarly community is definitional. Whilst Gunn (2010) includes wider diffusion of an initiative as a necessary component of the sustainability of e-learning, we would argue that sustainability does not require wider diffusion. Instead, wider diffusion appears to be an indicator of continuation; that is, an initiative that is adopted by others would appear to have a greater possibility of continuation than one that remains in use only in its initial local context. 5.2. Practical implications The relatively high levels of perceptions of improvement of student learning associated with the innovations in this study indicate that learning and teaching concerns are central to e-learning initiatives whether they are sustainable or not. This study therefore highlights the importance for participants in local level e-learning initiatives of developing an initiative that meets the needs of teachers and learners. In addition, plans for continuation need to include plans for financial and technological sustainability. Furthermore, participants should not ignore the internal marketing activities needed to maintain both local

and institutional support for the project (even if they find them time consuming, unattractive or not part of their skill set). Ideally, even a local e-learning initiative should be approached with a social project management mindset in which the importance of demonstrating the value of the initiative to others is acknowledged and acted upon. From an institutional point of view, this empirical study confirms many of the proposals for institutional action to support e-learning initiatives (Czerniewicz & Brown, 2009; Gunn, 2010; Jenkins et al., 2011; Nichols, 2008; Salmon, 2005). Policies that ensure that participation in e-learning initiatives is recognized by the institution as performance of legitimate duties that contribute to promotion are necessary if participants in e-learning initiatives are to be motivated to continue with them. Institutional level technology needs to enable rather than to constrain local level e-learning initiatives. Ongoing institutional support also needs to be felt at the level of the local initiative: financial support for technology and development is needed, along with the institutional level investments in teachers, developers and technology that need to accompany formal acknowledgement of their contribution to ongoing e-learning success. At the institutional level, then, our findings confirm and consolidate those of the authors of much of the institutional literature: successful university e-learning requires mindful investment in structural change. This observation appears to be true whether we consider MOOCs or local level initiatives. 6. Conclusion Having an e-learning initiative that works for students and teachers is not enough to distinguish between sustainable and non-sustainable initiatives. In the initiatives studied here, the institutional factors that differentiated between continued and non-continued initiatives were dominated by institutional support for the initiative (rather than for elearning in general), especially financial support, and formal recognition of development activities as part of the developer’s role. Technology needed to be sufficiently up to date, but also sufficiently mature or stable, to support continuation. Overall, the difference between sustainable and non-sustainable initiatives has more to do with conditions associated with gaining ongoing material support for the initiative and attracting others to become involved in adoption and development than factors associated with teachers’ ability or willingness to participate, or with learning or student response to the initiative. It would appear that it is not just the quality of the initiative that makes the difference – although we assume quality is a necessary condition – nor is it some passive form of ‘management support’ for e-learning. The potential role of ‘political’ or promotional skills that enable an elearning initiative to be ‘sold’ to colleagues, developers and management, has been highlighted as requiring further exploration. Appendix A. Papers included in the study Continued initiatives Alipanahi, F. (2008). Enhanced reading. In J. Luca & E. Weippl (Eds.), Proceedings of World Conference on Educational Multimedia, Hypermedia and Telecommunications 2008 (pp. 964–971). Chesapeake, VA: AACE. (Hypertext to support learning English) Appel, C., & Mullen, T. (2000). Pedagogical considerations for a webbased tandem language learning environment. Computers & Education, 34(3–4), 291–308. (Web-based tandem language learning environment) Armstrong, K. & Retterer, O. (2004). Mi Blog es Su Blog: Implementing community and personal weblogs to encourage writing in Intermediate Spanish. In L. Cantoni & C. McLoughlin (Eds.), Proceedings of World Conference on Educational Multimedia, Hypermedia and Telecommunications 2004 (pp. 1135–1137). Chesapeake, VA: AACE. (Content management system for creating and managing weblogs) Benmimoun, A., Trigano, P., Daouas, T. & Balla, A. (2008). eDalgo: An hypermedia for a self regulated learning of C programming. In J. Luca &

T.J. McGill et al. / Internet and Higher Education 22 (2014) 24–36

E. Weippl (Eds.), Proceedings of World Conference on Educational Multimedia, Hypermedia and Telecommunications 2008 (pp. 1860–1867). Chesapeake, VA: AACE. (Interactive online course for C programming) Boubouka, M., Papanikolaou, K. & Grigoriadou, M. (2008). Integrating peer evaluation in the project based adaptive educational system MyProject. In J. Luca & E. Weippl (Eds.), Proceedings of World Conference on Educational Multimedia, Hypermedia and Telecommunications 2008 (pp. 2238–2246). Chesapeake, VA: AACE. (Peer evaluation in MyProject) Brusilovsky, P. & Spring, M. (2004). Adaptive, engaging, and explanatory visualization in a C programming course. In L. Cantoni & C. McLoughlin (Eds.), Proceedings of World Conference on Educational Multimedia, Hypermedia and Telecommunications 2004 (pp. 1264– 1271). Chesapeake, VA: AACE. (Visualisation to support learning of C programming) Buchanan, T. (2000). The efficacy of a World-Wide Web mediated formative assessment. Journal of Computer Assisted Learning, 16(3), 193–200. (Web based multiple choice package for psychology) Budka, P., Mader, E., Anderl, E. & Stockinger, J. (2008). Strategies for networked learning in social science education. In J. Luca & E. Weippl (Eds.), Proceedings of World Conference on Educational Multimedia, Hypermedia and Telecommunications 2008 (pp. 618–628). Chesapeake, VA: AACE. (Learning environment to support social anthropology learning) Camacho, M. & Gisbert, M. (2004). The N-@ble Initiative: An instructional proposal for teacher training in ICT-based settings. In L. Cantoni & C. McLoughlin (Eds.), Proceedings of World Conference on Educational Multimedia, Hypermedia and Telecommunications 2004 (pp. 3295– 3300). Chesapeake, VA: AACE. (Blended learning for English as a second language) Clothey, R. & Schmitt, C. (2008). Education in motion: Innovating with iPods. In J. Luca & E. Weippl (Eds.), Proceedings of World Conference on Educational Multimedia, Hypermedia and Telecommunications 2008 (pp. 629–639). Chesapeake, VA: AACE. (Increasing mobility of online learning with iPods) Cobos, Y., Sanz, S., Gutiérrez, J., Sanz, S. & Villamañe, M. (2004). Heusklearning: the more innovative learning systems, the better results produce. In L. Cantoni & C. McLoughlin (Eds.), Proceedings of World Conference on Educational Multimedia, Hypermedia and Telecommunications 2004 (pp. 1293–1298). Chesapeake, VA: AACE. (e-learning environment to support learning Basque) Collaud, G., Gurtner, J. L., & Coen, P. F. (2000). Design and use of a hypermedia system at the University level. Journal of Computer Assisted Learning, 16(2), 137–147. (Blended learning in psychology) Contreras-Castillo, J., Favela, J., Perez-Fragoso, C., & Santamaria-delAngel, E. (2004). Informal interactions and their implications for online courses. Computers & Education, 42(2), 149–168. (System to support informal interaction of students and teachers) Demetriadis, S. N., Papadopoulos, P. M., Stamelos, I. G., & Fischer, F. (2008). The effect of scaffolding students' context-generating cognitive activity in technology-enhanced case-based learning. Computers & Education, 51(2), 939–954. (Web-based environment for supporting case-based learning) Depradine, C., & Gay, G. (2004). Active participation of integrated development environments in the teaching of object-oriented programming. Computers & Education, 43(3), 291–298. (Enhanced development environment to support teaching Java) Dreher, H., Dreher, N. & Reiners, T. (2008). Design and integration of an automated assessment laboratory: Experiences and guide. In J. Luca & E. Weippl (Eds.), Proceedings of World Conference on Educational Multimedia, Hypermedia and Telecommunications 2008 (pp. 2858–2863). Chesapeake, VA: AACE. (Automated essay marking) Hauser, G. & Jones, A. (2008). A formative analysis of student perceptions of the standards-based digital portfolio on student learning in educational leadership preparation programs at two private urban universities in the U.S.A. In J. Luca & E. Weippl (Eds.), Proceedings of World Conference on Educational Multimedia, Hypermedia and

33

Telecommunications 2008 (pp. 679–681). Chesapeake, VA: AACE. (Standards-based digital portfolios in education) Hill, C. & Slator, B. (2000). Computer science instruction in a virtual world. In J. Bourdeau & R. Heller (Eds.), Proceedings of World Conference on Educational Multimedia, Hypermedia and Telecommunications 2000 (pp. 441–447). Chesapeake, VA: AACE. (System for teaching computer science online) Höller, H. & Reisinger, P. (2008). Wiki based teaching and learning scenarios at the University of Vienna. In J. Luca & E. Weippl (Eds.), Proceedings of World Conference on Educational Multimedia, Hypermedia and Telecommunications 2008 (pp. 2592–2596). Chesapeake, VA: AACE. (Wiki to support blended learning in physics) Huang, Y.-M., Kuo, Y.-H., Lin, Y.-T., & Cheng, S.-C. (2008). Toward interactive mobile synchronous learning environment with contextawareness service. Computers & Education, 51(3), 1205–1226. (Context-awareness mobile learning) Ishikawa, Y., Nishu, M., Murakami, M., Smith, C. & Shimomura, H. (2008). A case study in curriculum development: Design and evaluation of a new course incorporating CALL and fieldwork research. In J. Luca & E. Weippl (Eds.), Proceedings of World Conference on Educational Multimedia, Hypermedia and Telecommunications 2008 (pp. 707–716). Chesapeake, VA: AACE. (Computer assisted learning of English as a foreign language) Isotani, S., & Brandão, L. d. O. (2008). An algorithm for automatic checking of exercises in a dynamic geometry system: iGeom. Computers & Education, 51(3), 1283–1303. (Tool for the authoring and automatic checking of geometry exercises) Kennedy, D.M., Eizenberg, N. & Kennedy, G.E. (2000). An evaluation of interactive multimedia designed to support problem-based learning in medicine. In J. Bourdeau & R. Heller (Eds.), Proceedings of World Conference on Educational Multimedia, Hypermedia and Telecommunications 2000 (pp. 508–513). Chesapeake, VA: AACE. (Application to support a problem-based learning in medicine) Lara, S. (2008). Two-year use of WebQuest at the university to foster the Bologna process. In J. Luca & E. Weippl (Eds.), Proceedings of World Conference on Educational Multimedia, Hypermedia and Telecommunications 2008 (pp. 3193–3198). Chesapeake, VA: AACE. (WebQuests to help students to study in a more autonomous and active way) Luca, J. & McLoughlin, C. (2004). Using online forums to support a community of learning. In L. Cantoni & C. McLoughlin (Eds.), Proceedings of World Conference on Educational Multimedia, Hypermedia and Telecommunications 2004 (pp. 1468–1474). Chesapeake, VA: AACE. (Online forum to support a community of learning) MacDougall, G., Müldner, T. & Tomek, I. (2004). Acadia Advantage – computerizing a campus. In L. Cantoni & C. McLoughlin (Eds.), Proceedings of World Conference on Educational Multimedia, Hypermedia and Telecommunications 2004 (pp. 41–46). Chesapeake, VA: AACE. (Laptop program) Mackie, B. & Ziemer, N. (2004). CAMS© the collaborative environment. In L. Cantoni & C. McLoughlin (Eds.), Proceedings of World Conference on Educational Multimedia, Hypermedia and Telecommunications 2004 (pp. 1481–1486). Chesapeake, VA: AACE. (Course management system that emphasizes collaboration) Magoun, A.D., Saydam, A., Owens, C., Smith, E. & Richters, S. (2008). Reversing the pattern of low performance in college algebra using the National Center for Academic Transformations’ Roadmap to Redesign approach. In J. Luca & E. Weippl (Eds.), Proceedings of World Conference on Educational Multimedia, Hypermedia and Telecommunications 2008 (pp. 739–744). Chesapeake, VA: AACE. (Initiative to improve learning of algebra) Masuyama, K. & Shea, A. (2004). Another year of successful integration of technology. A case study of a first-year Japanese language course. In L. Cantoni & C. McLoughlin (Eds.), Proceedings of World Conference on Educational Multimedia, Hypermedia and Telecommunications 2004 (pp. 1506–1511). Chesapeake, VA: AACE. (Technology enhanced Japanese language course)

34

T.J. McGill et al. / Internet and Higher Education 22 (2014) 24–36

Muntjewerff, A. & DeTombe, D. (2004). A generic environment for integrating streaming video in legal education. In L. Cantoni & C. McLoughlin (Eds.), Proceedings of World Conference on Educational Multimedia, Hypermedia and Telecommunications 2004 (pp. 527–532). Chesapeake, VA: AACE. (Environment for law students to construct case descriptions) Nguyen, A.V., Gillet, D. & Sire, S. (2004). Sustaining collaboration within a learning community in flexible engineering education. In L. Cantoni & C. McLoughlin (Eds.), Proceedings of World Conference on Educational Multimedia, Hypermedia and Telecommunications 2004 (pp. 1214–1221). Chesapeake, VA: AACE. (eJournal to support learning in engineering) Papadopoulos, G., Voulgari, I. & Houssou, E. (2008). Implementation and assessment of videoconferencing sessions in earth and life sciences: The case of the Agricultural University of Athens. In J. Luca & E. Weippl (Eds.), Proceedings of World Conference on Educational Multimedia, Hypermedia and Telecommunications 2008 (pp. 793–802). Chesapeake, VA: AACE. (Videoconferencing in science courses) Paquette, G., Teja, I.D.l. &; Dufresne, A. (2000). Explora: An open virtual campus. In J. Bourdeau & R. Heller (Eds.), Proceedings of World Conference on Educational Multimedia, Hypermedia and Telecommunications 2000 (pp. 879–884). Chesapeake, VA: AACE. (Learning management system) Pare, D. E., & Joordens, S. (2008). Peering into large lectures: examining peer and expert mark agreement using peerScholar, an online peer assessment tool. Journal of Computer Assisted Learning, 24(6), 526–540. (Online tool used to support peer assessment) Pohjolainen, S., Ruokamo, H., Nykänen, O., Pesonen, K. & Hartikainen, V.M. (2000). A learning experiment using A&O open learning environment. In J. Bourdeau & R. Heller (Eds.), Proceedings of World Conference on Educational Multimedia, Hypermedia and Telecommunications 2000 (pp. 916–921). Chesapeake, VA: AACE. (Learning environment that supports both self-paced and collaborative learning) Riedl, R. (2004). Building a program in a virtual world. In L. Cantoni & C. McLoughlin (Eds.), Proceedings of World Conference on Educational Multimedia, Hypermedia and Telecommunications 2004 (pp. 424–431). Chesapeake, VA: AACE. (3D virtual world for delivery of courses) Sato, K. & Akahori, K. (2004). Enhancing interactivity in face-to-face lecture by using board mediated communication. In L. Cantoni & C. McLoughlin (Eds.), Proceedings of World Conference on Educational Multimedia, Hypermedia and Telecommunications 2004 (pp. 146–153). Chesapeake, VA: AACE. (Tool to promote interactivity in lectures) Scherly, D., Roux, L., & Dillenbourg, P. (2000). Evaluation of hypertext in an activity learning environment. Journal of Computer Assisted Learning, 16(2), 125–136. (Learning tools for biology) Shubert, C., Ceraj, I. & Riley, J. (2008). Bringing research tools into the classroom. In J. Luca & E. Weippl (Eds.), Proceedings of World Conference on Educational Multimedia, Hypermedia and Telecommunications 2008 (pp. 511–519). Chesapeake, VA: AACE. (Use of research tools in the classroom) Shun, L., Csete, J., O'Donoghue, M. & Wong, M. (2008). Drawing from a pool of knowledge: Interactive learning activities linked to a central database. In J. Luca & E. Weippl (Eds.), Proceedings of World Conference on Educational Multimedia, Hypermedia and Telecommunications 2008 (pp. 1833–1838). Chesapeake, VA: AACE. (e-learning tool for anatomy) Stefansson, G. (2004). The tutor-web: An educational system for classroom presentation, evaluation and self-study. Computers & Education, 43(4), 315–343. (System for storage and presentation of materials for classroom use) Stuart, S. A. J., Brown, M. I., & Draper, S. W. (2004). Using an electronic voting system in logic lectures: one practitioner's application. Journal of Computer Assisted Learning, 20(2), 95–102. (Electronic voting system) Vines, R.L. & Bruner, J. (2008). Addressing course accessibility through collaboration and technology. In J. Luca & E. Weippl (Eds.), Proceedings of World Conference on Educational Multimedia, Hypermedia and Telecommunications 2008 (pp. 568–573). Chesapeake, VA: AACE.

(Use of technology to provide access to hematology lectures on several campuses) Zeiller, M., Jelinek, M. & Eschenbach, S. (2008). Campus IB – a peopleoriented e-learning environment. In J. Luca & E. Weippl (Eds.), Proceedings of World Conference on Educational Multimedia, Hypermedia and Telecommunications 2008 (pp. 1979–1984). Chesapeake, VA: AACE. (Learning environment to support situated learning)

Discontinued initiatives Andrews, K. & Hughes, J. (2008). Mediating student – teacher interaction in areas of conflict and political strife through the use of DVD delivered video. In J. Luca & E. Weippl (Eds.), Proceedings of World Conference on Educational Multimedia, Hypermedia and Telecommunications 2008 (pp. 2810–2814). Chesapeake, VA: AACE. (Use of video to provide a virtual presence for faculty unable to travel to conflict zones) Armstrong, D. (2008). THE SUBURB AS SITE: A virtual photographic collaboration project. In J. Luca & E. Weippl (Eds.), Proceedings of World Conference on Educational Multimedia, Hypermedia and Telecommunications 2008 (p. 300). Chesapeake, VA: AACE. (Photographic collaboration presented via a virtual gallery) Chen, X., Cotton, D., Roeber, R. & Guru, A. (2004). Building interactive graphic simulations to help students understand nitrogen transformations in the soil. In L. Cantoni & C. McLoughlin (Eds.), Proceedings of World Conference on Educational Multimedia, Hypermedia and Telecommunications 2004 (pp. 1280–1284). Chesapeake, VA: AACE. (System to simulate soil nitrogen transformations) El-Deghaidy, H., & Nouby, A. (2008). Effectiveness of a blended elearning cooperative approach in an Egyptian teacher education programme. Computers & Education, 51(3), 988–1006. (Blended e-learning approach for a science teaching methods course) Fritze, P., Kemm, R.E., Kavnoudias, H., Weaver, D., Stone, N. & Williams, N. (2000). Collaborative learning: an effective and enjoyable experience! A successful computer-facilitated environment for tertiary students. In J. Bourdeau & R. Heller (Eds.), Proceedings of World Conference on Educational Multimedia, Hypermedia and Telecommunications 2000 (pp. 44–55). Chesapeake, VA: AACE. (Collaborative learning environment for physiology) García-Luque, E., Ortega, T., Forja, J. M., & Gómez-Parra, A. Using a laboratory simulator in the teaching and study of chemical processes in estuarine systems. Computers & Education, 43(1–2), 81–90. (Laboratory simulator for teaching chemical processes in estuarine systems) Kaburlasos, V. G., Marinagi, C. C., & Tsoukalas, V. T. (2008). Personalized multi-student improvement based on Bayesian cybernetics. Computers & Education, 51(4), 1430–1449. (Student self-assessment from an item bank) Kalles, D. (2008). Students working for students on programming courses. Computers & Education, 50(1), 91–97. (Programming laboratory course) LeBaron, J., Pulkkinen, J. & Scollin, P. (2000). Problems of student communication in a cross-cultural, international internet course setting. In J. Bourdeau & R. Heller (Eds.), Proceedings of World Conference on Educational Multimedia, Hypermedia and Telecommunications 2000 (pp. 558–563). Chesapeake, VA: AACE. (International web-based course created as a joint initiative of 3 universities) Lockyer, L. & Kerr, Y. (2000). Learner as designer-producer: Physical and health education students experience Web-based learning resource development. In J. Bourdeau & R. Heller (Eds.), Proceedings of World Conference on Educational Multimedia, Hypermedia and Telecommunications 2000 (pp. 626–630). Chesapeake, VA: AACE. ('Learner as designerproducer' approach in a physical and health education course) Lopez, J., Sanchez, J., Gonzalez, I., Cabello, R., Gomez, F. & Martinez, J. (2004). A distance training laboratory for digital circuit design and prototyping of control systems. In L. Cantoni & C. McLoughlin (Eds.), Proceedings of World Conference on Educational Multimedia, Hypermedia

T.J. McGill et al. / Internet and Higher Education 22 (2014) 24–36

and Telecommunications 2004 (pp. 678–685). Chesapeake, VA: AACE. (Laboratory tool set for distance learning) Lorenz, S., Horstmann, W., Oesker, M., Gorczytza, H., Dieckmann, A. & Egelhaaf, M. (2004). The monist project - A system for learning and teaching with educational simulations. In L. Cantoni & C. McLoughlin (Eds.), Proceedings of World Conference on Educational Multimedia, Hypermedia and Telecommunications 2004 (pp. 1444–1451). Chesapeake, VA: AACE. (Simulations for neural and cognitive science education) Medélez Ortega, E., Cuggia, M., Le Cuziat, P., Honnorat, C. & Le Beux, P. (2004). Experiment in Virtu@l Consult@tion, a collaborative and multimedia learning environment based on simulation in medicine. In L. Cantoni & C. McLoughlin (Eds.), Proceedings of World Conference on Educational Multimedia, Hypermedia and Telecommunications 2004 (pp. 1544–1549). Chesapeake, VA: AACE. (Learning environment for medical education) Read, J., Sim, G., Gregory, P. & Taylor, K. (2004). Using a web based tool to innovate assessment practices. In L. Cantoni & C. McLoughlin (Eds.), Proceedings of World Conference on Educational Multimedia, Hypermedia and Telecommunications 2004 (pp. 707–712). Chesapeake, VA: AACE. (Web based assessment tool for computing) Renner, D. (2008). The language assessment and placement test: Project development process. In J. Luca & E. Weippl (Eds.), Proceedings of World Conference on Educational Multimedia, Hypermedia and Telecommunications 2008 (pp. 3024–3028). Chesapeake, VA: AACE. (Online language placement test) Shittu, A. & Hassan, W.H. (2008). Developing an e-learning application for a distributed environment. In J. Luca & E. Weippl (Eds.), Proceedings of World Conference on Educational Multimedia, Hypermedia and Telecommunications 2008 (pp. 2024–2029). Chesapeake, VA: AACE. (e-learning system that encourages student participation) Sitthiworachart, J., & Joy, M. (2008). Computer support of effective peer assessment in an undergraduate programming class. Journal of Computer Assisted Learning, 24(3), 217–231. (Web-based peer assessment tool) Taylor, M. J., Pountney, D. C., & Baskett, M. (2008). Using animation to support the teaching of computer game development techniques. Computers & Education, 50(4), 1258–1268. (Animation to support the teaching of computer game development techniques) Tedesco, R., Cesarini, M., Guinea, S. & Sbattella, L. (2004). Innovative learning and teaching scenarios in Virtual Campus. In L. Cantoni & C. McLoughlin (Eds.), Proceedings of World Conference on Educational Multimedia, Hypermedia and Telecommunications 2004 (pp. 180–187). Chesapeake, VA: AACE. (Innovative e-learning platform) Wessner, M., Seeberg, C. & Steinmetz, R. (2000). Taking our own medicine: Learning "Learning with Multimedia" with multimedia. In J. Bourdeau & R. Heller (Eds.), Proceedings of World Conference on Educational Multimedia, Hypermedia and Telecommunications 2000 (pp. 1161–1167). Chesapeake, VA: AACE. (Course on multimediabased learning with emphasis computer-based collaboration) Winther Bech, C., Bouvin, N.O., Graves Petersen, M. & Grønbæk, K. (2004). An open portal for blended learning. In L. Cantoni & C. McLoughlin (Eds.), Proceedings of World Conference on Educational Multimedia, Hypermedia and Telecommunications 2004 (pp. 734–739). Chesapeake, VA: AACE. (An open configurable web portal that supports distributed learning) Other – discontinued at one institution and subsequently transferred or restarted at another Liebig, H.C. & Effelsberg, W. (2004). Computer-supported formation of virtual learning groups based on proficiency levels. In L. Cantoni & C. McLoughlin (Eds.), Proceedings of World Conference on Educational Multimedia, Hypermedia and Telecommunications 2004 (pp. 1171– 1178). Chesapeake, VA: AACE. (Web-based system that supports virtual learning groups)

35

Strobel, J., Jonassen, D. H., & Ionas, I. G. (2008). The evolution of a collaborative authoring system for non-linear hypertext: A design-based research study. Computers & Education, 51(1), 67–85. (Collaborative knowledge construction environment)

References Aggarwal, A. K. E. (2000). Web-based learning and teaching technologies: Opportunities and challenges. Hershey, PA Idea Group Publishing. Alexander, S. (2001). E-learning developments and experiences. Education and Training, 43(4/5), 240–248. Bates, A. W. T., & Poole, G. (2003). Effective teaching with technology in higher education: Foundations for success. San Francisco Jossey Bass. Becker, R., & Jokivirta, L. (2007). Online learning in universities: Selected data from the 2006 Observatory survey – November 2007. The Observatory on Borderless Higher Education (OBHE) (http://www.obhe.ac.uk). Bell, M., & Bell, W. (2005). It's installed … now get on with it! Looking beyond the software to the cultural change. British Journal of Educational Technology, 36(4), 643–656. http://dx.doi.org/10.1111/j.1467-8535.2005.00541.x. Birch, D., & Burnett, B. (2009). Bringing academics on board: Encouraging institution wide diffusion of e-learning environments. Australasian Journal of Educational Technology, 25(1), 117–134. Breslin, C., Nicol, D., Grierson, H., Wodehouse, A., Juster, N., & Ion, W. (2007). Embedding an integrated learning environment and digital repository in design engineering education: Lessons learned for sustainability. British Journal of Educational Technology, 38(5), 805–816. http://dx.doi.org/10.1111/j.1467-8535.2007.00754.x. Browne, T., Jenkins, M., & Walker, R. (2006). A longitudinal perspective regarding the use of VLEs by higher education institutions in the United Kingdom. Interactive Learning Environments, 14(2), 177–192. Chiu, C. -M., Sun, S. -Y., Sun, P. -C., & Ju, T. L. (2007). An empirical analysis of the antecedents of Web-based learning continuance. Computers & Education, 49(4), 1224–1245. Chiu, C. -M., & Wang, E. T. G. (2008). Understanding Web-based learning continuance intention: The role of subjective task value. Information & Management, 45(3), 194–201. Czerniewicz, L., & Brown, C. (2009). A study of the relationship between institutional policy, organisational culture and e-learning use in four South African universities. Computers & Education, 53(1), 121–131. de Vries, F. J., Kester, L., Sloep, P., van Rosmalen, P., Pannekeet, K., & Koper, R. (2005). Identification of critical time-consuming student support activities in e-learning. ALT-J: Research in Learning Technology, 13(3), 219–229. Demirkan, H., Goul, M., & Gros, M. (2010). A reference model for sustainable e-learning service systems: Experiences with the Joint University/Teradata Consortium. Decision Sciences Journal of Innovative Education, 8(1), 151–189. http://dx.doi.org/10. 1111/j.1540-4609.2009.00250.x. Derouin, R. E., Fritzsche, B. A., & Salas, E. (2005). e-Learning in organizations. Journal of Management, 31(6), 920–940. Drent, M., & Meelissen, M. (2008). Which factors obstruct or stimulate teacher educators to use ICT innovatively? Computers & Education, 51(1), 187–199. Garrison, D. R., & Vaughan, N. D. (2008). Blended learning in higher education: Framework, principles, and guidelines. San Francisco, CA Jossey-Bass. Gunn, C. (2010). Sustainability factors for e-learning initiatives. ALT-J: Research in Learning Technology, 18(2), 89–103. Gunn, C. (2011). Sustaining elearning innovations. In G. Williams, S. Statham, N. Brown, & B. Cleland (Eds.), Changing Demands, Changing Directions. Proceedings ascilite Hobart 2011 (pp. 509–519). (http://www.ascilite.org.au/conferences/hobart11/procs/Gunnfull.pdf). Guthrie, K., Griffiths, R., & Maron, N. (2008). Sustainability and revenue models for online academic resources: An Ithaka report. Bristol, UK JISC/Strategic Content Alliance. Hayashi, A., Chen, C., Ryan, T., & Wu, J. (2004). The role of social presence and moderating role of computer self efficacy in predicting the continuance usage of e-learning systems. Journal of Information Systems Education, 15(2), 139–154. Hirscheim, R. (2005). The Internet-based education bandwagon: Look before you leap. Communications of the ACM, 48(7), 97–101. Howard, C., Boettcher, J. V., Justice, L., Schenk, K., Rogers, P. L., & Berg, G. A. (Eds.). (2005). Encyclopedia of distance learning. Hershey, PA Information Science Publishing. Jenkins, M., Browne, T., Walker, R., & Hewitt, R. (2011). The development of technology enhanced learning: findings from a 2008 survey of UK higher education institutions. Interactive Learning Environments, 19(5), 447–465. http://dx.doi.org/10.1080/ 10494820903484429. Larsen, T. J., Sørebø, A. M., & Sørebø, Ø. (2009). The role of task-technology fit as users' motivation to continue information system use. Computers in Human Behavior, 25(3), 778–784. Limayem, M., & Cheung, C. M. K. (2008). Understanding information systems continuance: The case of Internet-based learning technologies. Information & Management, 45(4), 227–232. Marshall, S. (2012). Improving the quality of e-learning: lessons from the eMM. Journal of Computer Assisted Learning, 28(1), 65–78. http://dx.doi.org/10.1111/j.1365-2729. 2011.00443.x. McGill, T., Klobas, J., & Renzi, S. (2011). LMS use and instructor performance: The role of task-technology fit. International Journal on E-Learning, 10(1), 43–62. McPherson, M. A., & Nunes, J. M. (2008). Critical issues for e-learning delivery: what may seem obvious is not always put into practice. Journal of Computer Assisted Learning, 24(5), 433–445. http://dx.doi.org/10.1111/j.1365-2729.2008.00281.x.

36

T.J. McGill et al. / Internet and Higher Education 22 (2014) 24–36

Nichols, M. (2008). Institutional perspectives: The challenges of e-learning diffusion. British Journal of Educational Technology, 39(4), 598–609. http://dx.doi.org/10.1111/ j.1467-8535.2007.00761.x. Roca, J. C., Chiu, C. -M., & Martínez, F. J. (2006). Understanding e-learning continuance intention: An extension of the Technology Acceptance Model. International Journal of Human-Computer Studies, 64(8), 683–696. Romiszowski, A. J. (2004). How's the e-learning baby? Factors leading to success or failure of an educational technology innovation. Educational Technology, 44(1), 5–27. Ruth, S. (2010). Is e-learning really working? The trillion-dollar question. IEEE Internet Computing, 14(2), 78–82. Salmon, G. (2005). Flying not flapping: a strategic framework for e-learning and pedagogical innovation in higher education institutions. ALT-J: Research in Learning Technology, 13(3), 201–218. Somekh, B. (1998). Supporting information and communication technology innovations in higher education. Journal of Information Technology for Teacher Education, 7(1), 11–32. Stockport, G., Klobas, J. E., & Mackintosh, J. B. (2012). Beyond the rhetoric: Planning for the future university. Proceedings of the IADIS International Conference on Higher Education. Perth, Western Australia IADIS.

Sun, P., Tsai, R., Finger, G., Chen, Y., & Yeh, D. (2008). What drives a successful e-learning? An empirical investigation of the critical factors influencing learner satisfaction. Computers & Education, 50(4), 1284–1303. Tham, C. M., & Werner, J. M. (2005). Designing and evaluating e-learning in higher education: a review and recommendations. Journal of Leadership and Organizational Studies, 11(2), 15–26. Tomei, L. (Ed.). (2008). Online and distance learning: Concepts, methodologies, tools, and applications. Hershey, PA Information Science Reference. Wang, Y. -S., Wang, H. -Y., & Shee, D. Y. (2007). Measuring e-learning systems success in an organizational context: Scale development and validation. Computers in Human Behavior, 23(4), 1792–1808. Zhang, D., & Nunamaker, J. F. (2003). Powering e-learning in the new millennium: An overview of e-learning and enabling technology. Information Systems Frontiers, 5(2), 207–218. http://dx.doi.org/10.1023/A:1022609809036.