Examining student characteristics, goals, and engagement in Massive Open Online Courses

Computers & Education 126 (2018) 433–442

Contents lists available at ScienceDirect

Computers & Education journal homepage: www.elsevier.com/locate/compedu

Examining student characteristics, goals, and engagement in Massive Open Online Courses

T

Kyle M. Williamsa,∗, Rose E. Stafforda, Stephanie B. Corlissb, Erin D. Reillyc a b c

Educational Psychology, The University of Texas at Austin, 1 University Station, Austin, TX, USA Dell Medical School, The University of Texas at Austin, 1 University Station, Austin, TX, USA Edith Nourse Rogers Memorial Veterans Hospital, USA

A R T IC LE I N F O

ABS TRA CT

Keywords: Motivation MOOCs Engagement Goals

Massive Open Online Courses (MOOCs) have emerged with much popularity in the last five years, yet many questions remain about whom MOOCs best serve and what constitutes learner success. Completion rates, a common metric of student success, remain low, averaging less than 8%, and may be a misleading measure of success unless learner intentions are considered. This research addresses the relationships among learner characteristics and goals for enrolling in MOOCs, and the impacts on student persistence and completion in varying disciplines. We examined learner self-reported goals for taking a MOOC, characteristics, and rate of completion of 15,655 participants in eight MOOC courses. Results revealed that while age was positively associated with MOOC participation, motivation differed across course disciplines. The relationship between learner goals and engagement differed between those enrolled in Humanities/Liberal Arts (HLA) and STEM courses. Most notably, while taking the course due to personal interest or usefulness to a participant's career held a positive relationship with engagement in HLA courses, the endorsement of these same goals was predictive of less engagement in STEM courses. Our findings indicate that learner goals impact engagement and success, and that there are differences in engagement and goals between course disciplines. Suggestions for future MOOC research and potential course improvement to better align with learner goals are also provided.

1. Introduction As a result of our knowledge about online learning and the open access education movement, online courses offered for free and to anyone with an internet connection have become a popular arena for large-scale education (see Yuan & Powell, 2013). Massive open online courses, or MOOCs, began when George Siemens and Stephen Downes created what they termed a “connectivist MOOC” or “cMOOC”, where learning occurred within a network. Students used technology to make connections with the content and other learners to create and construct knowledge. MOOCs gained more attention in 2011 when Stanford professors opened enrollment to their Artificial Intelligence course and got approximately 160,000 enrolled students. This was the first of the “xMOOCs”, which function similarly to higher education flipped classroom experiences where an instructor delivers content to students through recorded lectures or interactive activities and student learning is assessed through quizzing (see Terras & Ramsay, 2015 for further reading on this). Since that time, millions of people have registered for hundreds of MOOCs delivered primarily through the edX,

∗

Corresponding author. E-mail addresses: [email protected] (K.M. Williams), rose.staff[email protected] (R.E. Stafford), [email protected] (S.B. Corliss), [email protected] (E.D. Reilly). https://doi.org/10.1016/j.compedu.2018.08.014 Received 12 February 2017; Received in revised form 8 August 2018; Accepted 10 August 2018 Available online 11 August 2018 0360-1315/ © 2018 Published by Elsevier Ltd.

Computers & Education 126 (2018) 433–442

K.M. Williams et al.

Coursera, and Udacity platforms. In essence, MOOCs were created with the idea that people across the globe could obtain access to high quality topics in professional fields and higher education, regardless of their country of residence, educational background, and socioeconomic status as long as they have access to the internet. This introduction marked the beginning of a debate among education researchers and practitioners around the appropriateness of the Web to be a vehicle for quality education in a class of thousands (Mahraj, 2012). Since that time, much of the fervor has died down, but the opportunity for useful and innovative educational assessment remains. The affordances (as well as constraints) of this platform combined with the variety of learners from different backgrounds, cultures, and lived experiences creates an exciting opportunity for researchers to learn more about the impact of goals on course engagement, which we argue is an indicator of student persistence or perseverance with a task despite challenges that arise. 1.1. Measuring success in MOOCS The subject of learning via MOOCs has been one of the most hotly debated topics recently in higher education, with proponents suggesting that MOOCs could render traditional brick-and-mortar universities obsolete and opponents maintaining that high attrition rates and limited quality measures make MOOCs an ineffective learning platform (Watters, 2013). Much of this may be related to how success is measured in online environments, and particularly specialized online environments like MOOCs. Success in MOOCs has generally been measured in one of two ways: as either the completion of a required amount of material or the receipt of a certificate of completion (e.g. Trumbore, 2014). Some researchers suggest that the low completion rates associated with MOOCs are not worrisome for a number of reasons, one being that students enter MOOCs with different intentions other than course completion (Koller, Ng, Do, & Chen, 2013). Others suggest that intention to continue in MOOC courses is impacted by individual expectationconfirmation factors such as perceived reputation and perceived openness (Alraimi, Zo, & Ciganek, 2015). In other words, the intention to continue with a MOOC may be influenced by the perceived reputation of the school hosting the course, which is an expectation-confirmation factor. MOOC learners are highly diverse individuals with multiple reasons for pursuing their learning in open online courses. Many MOOC students may be browsing to find something they like (Kolowich, 2013), and some students view MOOCs as a “leisure activity”, much like watching television or reading a book (Watters, 2013). Not only do MOOC students differ in their reasons for registering for MOOCs, but without any restrictions to registration they also vary greatly in prior knowledge of the subject area, age, education level, and country of residence (Breslow et al., 2013). MOOC students differ from those in traditional courses so much that some researchers have suggested that typical online learning metrics and educational terminology cannot be applied to both environments (DeBoer, Ho, Stump, & Breslow, 2014; Jordan, 2014; Seaton, Bergner, Chuang, Mitros, & Pritchard, 2014). Recent MOOC research has begun to focus less on completion rates and more on learner participation and engagement patterns to better define MOOC student success (Breslow et al., 2013; Coffrin, Corrin, de Barba, & Kennedy, 2014; DeBoer et al., 2014). The metrics used by different MOOC completion studies vary widely, in terms of what constitutes completion or MOOC learner engagement. DeBoer et al. (2014) explored various definitions for success in the first edX MOOC, “Circuits and Electronics”, such as the grades students earned on all assignments and assessments in the course, obtaining a course certificate, and persistence throughout the duration of the course. Using these metrics, 4.5% of student enrolled in the course “passed”, causing researchers to re-think what success in a MOOC means. Using data gathered from that same course, DeBoer et al. (2014) categorized participants who enrolled in the course but failed to complete as “shoppers” if they attended 5 days or fewer, “dabblers” if they attended between 6 and 15 days, and “auditors” if they attended 16 or more days. The researchers inferred from their actions that students can have goals for achievement that are different than those set out by a course instructor. Coffrin et al. (2014) used advanced learning analytics techniques to analyze course engagement patterns and success on assessments within the first two to three weeks of MOOCs to identify students who had both the prior knowledge and intention to actively participate and persist in the course. In each of these studies, the researchers did not gather data about participants' goals for enrolling in the MOOCs, but point out that these intentions impact student behaviors in the course. 1.2. Measuring motivation in MOOCs In addition to measuring completion rates, other researchers have gathered data about student motives for enrolling in MOOCs. In a review of recent MOOC studies, Hew and Cheung (2014) report that students sign up for MOOCs to learn more about a new topic or to increase their knowledge in a certain area, to earn a certificate, or because they are curious about MOOCs. Kizilcec and Schneider (2015) found, in addition to the reasons listed above, that students enrolled in MOOCs to meet others interested in the same topic, to improve English skills, and to learn knowledge and/or skills to advance in schooling or in a career. They found differences in learner motivation across courses, and that each motivation predicted key behavioral outcomes for learners. Few studies have looked at retention rates with an eye toward better understanding the learner goals and demographic characteristic that predict behavioral completion of MOOCs beyond single-course case studies (e.g., Hone & El Said, 2016). In the current study, we built upon previous research and examine student motivation and behavior in MOOCs from various fields of study and across multiple courses. Although “motivation” and “goals” are not synonymous, we measure goals as a way of assessing motivation, or learner motives for enrolling in and continued participation in MOOCs. In this way, we examine the reasons students take part in a MOOC, also referred to as learner goals. One area of study that has been neglected is how to define engagement in a course that is, for the most part, lacking accountability or even the requirement of completing course activities. Additionally, although these courses have credentialing possibilities, they are not comparable to the expense and dedication required to complete a college degree, and as 434

Computers & Education 126 (2018) 433–442

K.M. Williams et al.

such, are even less likely to extrinsically motivate learners. Because of this, we wanted to explore ways to measure engagement, without using the definitions that have already been examined in the literature above. Specifically, we explored relationships among learner characteristics, goals, and engagement in the course to answer the following research questions: 1. What are the primary goals of xMOOC participants and do they differ by discipline (Humanities/Liberal Arts and STEM)? 2. How do learner characteristics relate to participants' primary goal for taking the course? 3. How do learner goals and characteristics relate to course engagement? 2. Methods 2.1. Participants Participants were 15,655 edX users across eight UTAustinX courses that were categorized as either Humanities/Liberal Arts (HLA) or STEM courses. HLA courses included Ideas of the 20th Century, Age of Globalization, and Jazz Appreciation. STEM courses were Take Your Medicine (a pharmaceuticals course), Energy 101, Linear Algebra: Foundations to Frontiers, Embedded Systems, and Effective Thinking Through Mathematics. To simplify reporting and adjust for large discrepancies in enrollment, we characterized each course by the discipline or field with which it corresponds according to the university offering the course. For example, the university offers each of the HLA courses within the liberal arts school, while the other courses (STEM) are found in the school of natural sciences, which helped establish the general discipline. For enrollment numbers by course and demographic information, see Table 1. The demographics of the participants ranged widely, but the majority identified as U.S. citizens and native English speakers (56%). Humanities/Liberal Arts courses were evenly divided between men and women participants (49–53% men), whereas STEM courses ranged from 54% to 92% men (Table 1). Additionally, participants across courses had an average age of 32–41 years, with the exception of an average of 49 years of age in Jazz Appreciation (see Table 2). 2.2. Materials A survey developed for MOOCs at the Center for Teaching and Learning at the University of Texas at Austin was given to all participants at the beginning of their MOOC experience. The survey was designed to collect general demographic information as well as information about MOOC related intentions and their reasons for taking the MOOC. We based this survey on earlier open-ended information gathered in the first semester of course offerings, instructor requests for particular data, and our own questions about learners' reasons for enrolling and persisting in the course. The learner characteristics gathered from this survey and used in the analyses are age (measured continuously in years), sex, and highest level of educational attainment. Education level was measured as an ordered categorical variable with six levels: (a) less than high school, (b) high school degree or equivalent, (c) some college but less than a bachelor's college degree, (d) bachelor's degree, (e) a professional or master's degree, and (f) a doctoral degree (see Table 1). It is important to note that because these courses are offered to a global audience, we took careful steps to make our survey culturally sensitive. In other words, because not all countries are familiar with different categories of gender, we simply asked participants to report their sex in one of three categories: Female, Male, or Other. Further, we asked participants to identify their country of residence, and asked for ethnicity information from respondents who reported their country of residence as the United States, because not all countries use the same ethnicity labels that are used in the US. Learner goals were assessed using two different items. The first item asked participants to select their primary goal for taking the course, and indicate an additional goal that was not included in the options if none of the provided options accurately described their primary goal. The second goal question contained the same answer options and asked participants to select all of their goals (i.e., select all that apply item) for enrolling in the MOOC. The open-ended information generated during the first semester of UTAustinX MOOCs led to the creation of new options to better reflect the participants' goals. For example, in early administrations of the survey, dozens of participants wrote in that their general goal for taking the course was lifelong learning; as such, we added this as a potential reason for enrolling in the course to subsequent surveys. Some goal options slightly varied across semesters, so similar categories were combined for consistency across all courses. The goal items contained six possible answer choices: (a) personal interest and lifelong learning, (b) learn more about MOOCs and online learning, (c) prepare for a credit or placement exam, (d) use this knowledge in my area of study or career, (e) take a course from the specific professor(s) or from UT-Austin, and (f) connect with others interested in this topic. Because of the developing nature of this survey, we focused on the six response choices above that were consistently reported on across courses and semesters. Student achievement on course assignments and activities was gathered at the termination of the course from the gradebook of each edX course. Rather than focus on final course grade or whether a participant earned a certificate for the course, we sought to investigate the relationships of learner attributes with their engagement in the course. We defined course engagement as the percentage of assignments and activities attempted by each participant throughout the course. Each activity or assignment for a course was given a new indicator of a 0 or 1, with 0 indicating that the participant did not answer any questions of an assignment or participate in an activity and 1 indicating that they engaged with the assignment or activity at least partially. The sum of these engagement scores were divided by the number of assignments and activities to calculate an engagement percentage for each participant. Therefore, higher course engagement scores indicate greater participation in course activities and assignments. Percentages were used so that all courses would be on a common scale since each course had varying numbers of assignments and activities. 435

436

5283 2693 2410 180 4257 666 1797 33 93 1668

5065 2393

2489

183 991 55 320

23 78

515

21

0 8

2 46 1 16

101

185 82

276

10 10

17 609 100 213

825

1646 804

199

5 31

45 724 130 359

405

827 377

Some College (n = 1551)

769

13 46

93 1780 249 703

1629

3510 1788

Bachelor's Degree (n = 5290)

800

20 69

165 1787 217 681

1652

3534 1717

Prof/Master's Degree (n = 5321)

119

8 7

43 305 22 149

291

650 316

Doctoral Degree (n = 955)

Note. Sample sizes for sex and education level do not equal total sample size (N = 15,655) due to missing responses on these items. HS = High School, Prof = Professional Degree.

HLA (n = 10,388, 66.4%) Age of Globalization (n = 5,099, 32.6%) Ideas of the 20th Century (n = 4,922, 31.4%) Jazz Appreciation (n = 367, 2.3%) STEM (n = 5,267, 33.6%) Embedded Systems (n = 721, 4.6%) Linear Algebra: Foundations to Frontiers (n = 2,127, 13.6%) Take Your Medicine (n = 56, 0.4%) Effective Thinking Through Mathematics (n = 172, 1.1%) Energy 101 (n = 2,191, 14.0%)

HS or Equivalent (n = 2255)

Less Than HS (n = 231)

Women (n = 6056)

Men (n = 9540)

Education Level (n = 15,603)

Sex (n = 15,596)

Table 1 Frequencies of sex and education level by discipline and course.

K.M. Williams et al.

Computers & Education 126 (2018) 433–442

Computers & Education 126 (2018) 433–442

K.M. Williams et al.

Table 2 Descriptive statistics: age by course and primary goal.

Humanities/Liberal Arts Age of Globalization Ideas of the 20th Century Jazz Appreciation STEM Embedded Systems Linear Algebra: Foundations to Frontiers Take Your Medicine Effective Thinking Through Mathematics Energy 101 Primary Goal Personal interest & lifelong learning Learn more about MOOCs/online learning Prepare for a credit or placement exam Use in area of study or career Take a course from UT/specific professor Connect with others interested in topic Other Total

N

Mean Age

SD

Min

Max

10,043 4899 4777 367 5116 721 2127 54 148 2066

34.32 31.79 35.83 48.55 36.22 33.54 37.87 39.04 41.40 35.02

14.15 11.95 15.20 15.86 14.05 12.57 14.51 16.81 15.51 13.57

11 11 11 19 16 19 19 19 19 12

89 89 85 79 88 86 88 77 80 85

8395 571 188 5239 253 78 224 15,159

37.46 37.69 25.61 31.08 32.27 37.05 39.81 35.96

15.20 15.46 9.64 10.92 13.06 16.42 16.89 14.14

11 11 12 11 12 16 13 11

89 88 62 84 81 75 83 89

Note. Total age descriptive statistics based on the age responses of all participants who answered this question.

2.3. Data analysis Data analysis and collection was limited due to the format of edX log data and the iterative nature of our survey. As such, our data analysis approach was carefully selected to make the most of our data and create the most useful and parsimonious interpretation of our results. It is important to note that the sample sizes in each of these analyses varied. These sample size fluctuations were due to unanswered survey items and participants not providing the necessary information to link their survey responses to their course performance. Though we could have deleted cases so that a single sample with full information was used across all analyses, we felt that it was more important to use all the information possible so that we could gain the most accurate picture of the relationships between our variables of interest. To examine the first research question, a χ2 test for independence was conducted to determine whether a participant's primary goal for taking the course differed by course discipline. Three separate analyses were conducted to answer our second research question, which was whether a participant's primary goal was related to sex, education level and age. A separate analysis was conducted for each to explore how each of these learner characteristics independently contributed to a participant's primary goal. We conducted χ2 tests for independence to investigate the relationships between learner goals and categorical learner characteristics (sex and education level). For any analysis that revealed significant and practically important relationships, we examined standardized residuals (SR) of the frequencies for each combination of goal selected and level of the other categorical variable of interest to determine where important differences existed. Standardized residuals exceeding an absolute value of two were considered to be sufficiently unlikely due to chance (Agresti, 2013). Multinomial logistic regression was used to determine whether age was predictive of a participant's primary goal. A total of seven logistic regressions were conducted, which were identical with the exception of the primary goal that served as the reference group so that we could determine age was related to the odds of choosing each goal in comparison to all other primary goal options. While a more cohesive analysis of the relationship of participant characteristics and course discipline to primary goal could include all variables in a single model, this would be a cumbersome model that lacked clarity due to the large number of categorical variables in this analysis. The difficulty of interpretation of such a model would be exacerbated by the requirement that dozens of equivalent models that vary the reference category of education level and primary goal to make all group comparisons. Therefore, the analyses used to answer the first two research questions are exploratory in nature, while the third research question delves deeper into the relationships and combined effect of these variables. To answer the last research question regarding whether learner characteristics and goals predicted course engagement, we conducted a multiple regression analysis, in which engagement was predicted by course discipline, sex, education level, age, and the seven learner goals. The learner goals included in this analysis were from the “select all that apply” item, meaning that participants could choose multiple goals for the course. This allowed us to determine the relationship of each goal to a participant's engagement with course activities without restricting participants to select a single goal for the course. Due to the small number of students reporting that they had less than a high school education, this category was combined with having a high school degree or an equivalent. Education level was dummy coded into four categorical variables (high school or less, some college, masters or professional degree, and doctorate degree) with the middle education level, having a bachelor's degree, serving as the comparison group. This model also considered the interactions between each of these variables to determine whether learner characteristics had an interactive effect on course engagement or whether the effect of learner characteristics differed by course discipline. A model that included all interactions was the first model considered. From here we used a backward model selection approach to finding the final model. Redundant predictor variables were identified using collinearity statistics (i.e., Tolerance and VIF) and partial correlation examination. Non-significant interaction terms were also dropped from the model. Many models were examined in this iterative 437

Computers & Education 126 (2018) 433–442

K.M. Williams et al.

Table 3 Frequencies of reported primary goal for each course discipline.

Personal interest & lifelong learning Learn more about MOOCs/online learning Prepare for a credit/placement exam Use in area of study or career Take a course from UT/specific professor Connect with others interested in this topic Other Total

HLA

STEM

Total

5934 (57.3%) 479 (4.6%) 115 (1.1%) 3452 (33.3%) 185 (1.8%) 61 (0.6%) 130 (1.3%) 10356 (67.1%)

2705(53.2%) 112 (2.2%) 81 (1.6%) 1978 (38.9%) 80 (1.6%) 22 (0.4%) 103 (2.0%) 5081 (32.9%)

8639 (55.2%) 591 (3.8%) 196 (1.3%) 5430 (34.7%) 265 (1.7%) 83 (0.5%) 233 (1.5%) 15437

Note. Percentages for HLA and STEM columns are the percentage of students who selected this primary goal within their course discipline. Percentages given in the total column are percentage of students who selected each goal across all courses.

model building process, with predictors entered in up to 10 blocks to determine whether there was a combined effect of a set of dummy-coded predictors representing a single categorical variable or its interaction with other predictors. Due to the overwhelming nature of presenting the results of this series of regressions, only those of the final model are provided in this manuscript. Due to the very large sample sizes across analyses (N > 9000), we placed emphasis on effect sizes over statistical significance in our interpretation of all analyses. Following Cohen's (1998) recommendations on the interpretation of effect sizes, we considered a statistically significant measure with a small effect size or greater to indicate a meaningful relationship. The definition of a small effect size was ϕ = 0.10 for the χ2 analyses, η2 = 0.01 for the ANOVA, and R2 = 0.10 for the multiple regression analysis. All analyses were conducted in SPSS version 23. 3. Results 3.1. Learner goals and academic discipline Our first research question asked: What are the primary goals of xMOOC participants and do they differ by discipline (Humanities/Liberal Arts and STEM)? Frequencies of reported primary goal for each course can be found in Table 3. We ran a χ2 test for independence to see if course discipline and a participant's primary goal were related (N = 15,437). Results revealed that the primary goal for taking the course significantly varies depending on the discipline of the course being taken, χ2 = 114.76, p < .001. The size of this effect is small but approaches practical importance, ϕ (i.e., phi coefficient) = 0.09 (p < .001). An examination of the standardized residuals revealed that HLA students were more likely than STEM participants to have primary goals of personal interest and lifelong learning (SRHLA = 4.80, SRSTEM = −4.80) and learning more about MOOCs/online learning than STEM participants (SRHLA = 7.40, SRSTEM = −7.40). STEM participants were more likely than HLA participants to have primary goals of preparing for a credit or placement exam (SRHLA = −2.50, SRSTEM = 2.50), to use what they learn in their area of study or career (SRHLA = −6.80, SRSTEM = 6.80). There were no meaningful differences between HLA and STEM students in having a primary goal of connecting with others (SRHLA = 1.20, SRSTEM = −1.20) or for taking a course from UT or the specific professor (SRHLA = 1.00, SRSTEM = −1.00). 3.2. Learner characteristics and goals Our second research question asked: How do learner characteristics relate to participants' primary goal for taking the course? We ran three separate analyses to determine the relationship between a participant's primary goal for taking the course and their sex (N = 15,381), level of education (N = 15,392), and age (N = 14,948). A χ2 test for independence determined that there was a significant relationship between a participant's sex and primary goal but that the size of this effect was trivial χ2(6) = 39.27, p < .001, ϕ = 0.05. We conducted a separate χ2 test to investigate the relationship between education level and primary goal for taking the course, across both HLA and STEM courses. This analysis indicated a significant and practically important relationship with a small effect size, χ2(30) = 178.430, p < .001, ϕ = 0.11. Examination of the standardized residuals for each combination of education level and primary goal revealed that participants whose highest educational attainment was a high school degree were less likely to choose personal interest and lifelong learning as a goal than would be expected by chance (SR = −4.20). Learners with professional or master's degrees were less likely to have learning more about MOOCS and online learning as a primary goal (SR = −2.20), while students with a doctoral degree were more likely to choose this goal (SR = 3.60). Participants without college degrees were more likely to be taking the course to prepare for a credit or placement exam (less than high school: SR = 5.40, high school or equivalent: SR = 7.00, and college without a bachelor's degree: SR = 3.60) while those with upper level degrees were less likely to have this as their main goal (professional and master's degrees: SR = −6.90, doctoral degrees: SR = −2.40). Participants were less likely to be taking the course with the goal of using it in their current or future study or career if they had not completed high school (SR = −2.10) or had taken some college courses without obtaining a bachelor's degree (SR = −2.40). Lastly, participants who had not completed high school were more likely to select ‘Other’ as a choice (SR = 3.50). Table 2 contains summary statistics of age by primary goal. We conducted multinomial logistic regression to examine the 438

Computers & Education 126 (2018) 433–442

K.M. Williams et al.

Table 4 Coefficients for final regression model predicting course engagement. Independent Variables

B

SE

β

t

STEM Male Age Education Level ≤ High School or Equivalent College without bachelor's degree Professional or master's degree Doctoral degree Learner Goals Learn more about MOOCs/online learning Use in my area of study or career Prepare for a credit or placement exam Personal interest & lifelong learning Take a course from UT/specific professor Connect with others interested in topic Interaction Effects Male × Age STEM × Male STEM × ≤ High School or Equivalent STEM × College without bachelor's degree STEM × Professional or master's degree STEM × Doctoral degree STEM × Learn more about MOOCs/online learning STEM × Use in my area of study or career STEM × Prepare for a credit or placement exam STEM × Personal interest & lifelong learning STEM × Take a course from UT/specific professor STEM × Connect with others interested in topic

27.85 10.59 0.63

3.49 2.53 0.06

0.33 0.12 0.22

7.99∗∗∗ 4.19∗∗∗ 11.38∗∗∗

2.58 −3.73 0.77 −1.98

1.71 2.26 1.36 2.56

0.02 −0.03 0.01 −0.01

1.51 −1.65 0.56 −0.77

−1.39 2.41 −7.07 4.50 −1.09 8.72

0.99 1.23 3.75 1.12 1.66 1.89

−0.01 0.03 −0.04 0.05 −0.01 0.07

−1.41 1.96∗ −1.88∗ 4.04∗∗∗ −0.66 4.62∗∗∗

−0.19 −4.59 −1.77 −3.65 4.90 8.33 −0.20 −5.57 6.75 −9.50 10.58 −7.01

0.06 2.02 2.79 3.07 2.08 3.90 0.07 1.81 4.40 1.82 2.34 2.61

−0.09 −0.05 −0.01 −0.02 0.04 0.03 −0.10 −0.06 0.03 −0.11 0.07 −0.04

−2.88∗∗∗ −2.28∗∗∗ −0.63 −1.19 2.35∗ 2.14∗ −2.88∗∗∗ −3.07∗∗∗ 1.54 −5.22∗∗∗ 4.52∗∗∗ −2.68∗∗

Note. STEM represents the effect of course discipline, reflecting the effect of being enrolled in a STEM course relative to a HLA course. Education level comparison group is bachelor's degree. p < .05 = ∗, p < .01 = ∗∗, p < .001 = ∗∗∗, N = 9074.

relationship between age and primary goal for taking the course (N = 14,948), which indicated that there was a significant but small relationship between these variables, χ2(6) = 865.77, p < .001, McFadden R2 = 0.06. Table 3 presents the odds that a participant chose a goal as their primary goal rather than each other goal option for every one-year increase in age. As age increases by one year, participants were significantly less likely to choose preparing for a credit of placement exam as their primary goal in comparison to every other primary goal option (all p < .001)). Similarly, as age increased, participants were significantly less likely to have the primary goal of using what they learn in their area of study or their career rather than personal interest and lifelong learning, learning more about MOOCs and online learning, connecting with others interested in this topic, or writing in a different goal (all p < .001). The odds of having a having a primary goal of personal interest and lifelong learning, to learn more about MOOCs, or connecting with others all increased as age increased relative to exam preparation, using what they learn in their future career, or taking a course from a specific professor (all p < .001). 3.3. Learner goals and course engagement Our final research question asked: How do learner goals and characteristics relate to course engagement? We used multiple regression to investigate the relationship between learner goals and course engagement. As previously described, all predictor variables (i.e., course discipline, sex, education level, age, and the seven course goals) were included and two-way interactions between all variables were considered in the initial model. An iterative backward model selection process was used to eliminate nonsignificant interaction variables and redundant predictors that produced multicollinearity. Listwise deletion was used for these analyses meaning that only participants with complete demographic information (i.e., sex, education level, and age) that had indicated at least one learning goal were included. This produced a sample of 9074 participants. Table 4 presents the coefficients of the final model, which was significant, F(25, 9048) = 25.64, p < .001, R2 = 0.07). Course engagement was significantly predicted by the main effects of course discipline (B = 27.85, t(9048) = 7.99, p < .001), sex (B = 10.59, t(9048) = 4.19, p < .001), and age (B = 0.63, t(9048) = 11.38, p < .001). These results indicate that, controlling for other predictors, course engagement was significantly higher for males and those that were enrolled in STEM courses, and that course engagement increased with participant age. Education level did not have a significant main effect on engagement. Three of the learner goals had a significant main effect, which all positively predicted course engagement: to use this knowledge in my current or future career (B = 2.41, t(9048) = 1.96, p < .05), to learn more about a topic in which I am personally interested (B = 4.50, t (9048) = 4.04, p < .001), and to connect with others interested in this topic (B = 8.72, t(9048) = 4.62, p < .001). The interaction between sex and age was a significant predictor of engagement (B = −0.19, t(9048) = −2.28, p < .001), indicating that the positive effect of age on course engagement was stronger for females in comparison to male students. There was also 439

Computers & Education 126 (2018) 433–442

K.M. Williams et al.

a significant interaction between course discipline and sex (B = −4.59, t(9048) = −2.28, p < .05), which indicated that the positive main effects of being male and in a STEM course were slightly lower than the increase in engagement seen by adding the main effects of these variables. Having a graduate degree significantly increased engagement in comparison to students with a bachelor's degree, but only for students enrolled in STEM courses (master's or professional degree: B = 4.90, t(9048) = 2.35, p < .05; doctorate degree B = 8.33, t(9048) = 2.14, p < .05). Having a goal to learn more about MOOCs and online learning was not a significant predictor as a main effect, but did have a significant relationship with engagement when looking at its interaction with course discipline (B = −0.20, t(9048) = −2.88, p < .001), which revealed that goal was only a positive predictor of engagement for students in HLA courses. Having a goal of using this knowledge in their current or future career had a significant main effect (B = 2.41, t(9048) = 1.96, p < .05), indicating that this goal was a positive predictor of engagement. However, its interactive effect with course discipline indicated that this positive effect was only present for HLA students and was the reverse for STEM students, who were actually less likely to stay engaged with the course if they had this course goal. Wanting to learn more about a topic of personal interest was also a positive predictor of engagement (B = 4.50, t(9048) = 4.04, p < .001), though a significant interaction with course discipline again revealed that this only true for HLA students (B = −9.5, t (9048) = −5.22, p < .001). Therefore, STEM students actually had lower engagement if they held this goal for the course. Taking the course because of the specific professor or university teaching the course did not have a significant main effect, but did have a significant interaction with course discipline (B = 10.58, t(9048) = 4.52, p < .001), which indicates that this goal is positive predictor of engagement for STEM students but not HLA students. Having a goal to connect with others interested in the course topic was linked to increased course engagement (B = 8.72, t (9048) = 4.62, p < .001). However, as with the majority of other learner goals, a significant interaction between this goal and course discipline revealed that this goal was only linked to increased engagement for participants in HLA courses (B = −7.01, t (9048) = −2.68, p < .01). Preparing for a credit or placement exam was the only learner goal to not have a significant effect, either as a main effect or as through an interaction with course discipline. 4. General discussion The results of this study suggest that investigation of MOOC success and completion should include a better understanding of learner characteristics and their goals for enrolling in MOOC courses from different disciplines. In a typical course taken for college credit or to earn a certification, students are extrinsically motivated to do the coursework and to perform well on assignments in order to pass the course and earn their credit. In these traditional courses, success is often measured by performance on summative assessments. However, given that the characteristics of MOOC participants and their motivation vary greatly from students in traditional courses, the same metric of success does not apply. The findings from this study emphasize the importance of learner motivation and characteristics in predicting course engagement and completion, as these better align with students' course goals for MOOCs. In this study, we found that MOOC participant motivation differed across course discipline. Specifically, those in HLA courses were more likely to select personal interest, lifelong learning, and learning more about MOOCs or online learning as a goal for enrolling in the course. Those in STEM courses were more likely to have a goal of preparing for a credit or placement exam and gaining knowledge for use in future studies or career. These results align with the importance placed on STEM skills in today's society and the demand within the current job market. The knowledge gained through studying philosophy, globalization, and jazz may be extremely interesting to many as indicated by participant goals for taking these courses, but perhaps is perceived as less marketable or useful to current jobs or future careers. Further investigation of the relationship among goals and learner characteristics revealed that, in general, older participants were more likely to select personal interest, lifelong learning, learning about MOOCs/online learning, and connecting with others interested in the topic as goals for enrolling in a MOOC, and reported higher engagement in both HLA and STEM courses. On the other hand, younger participants were more likely to select goals such as preparing for a credit or placement exam and using the knowledge in their future studies or career. These findings suggest that younger students may be using MOOCs to prepare for credit or placement exams for future college courses, or to prepare in some other way for their eventual career or coursework. This coincides with the results indicating that this goal was more common with students who had not obtained a college degree. It is likely that MOOCs may not be useful as exam preparation for older adults who are more likely to be established in their career than younger adults simply as a function of age. Thus, it also makes sense that older adults are engaging in MOOCs for personal interest and lifelong learning. Age may also be a factor in the difference between the likelihood of setting a goal to learn more about MOOCs and online learning, as younger adults are more likely to be digital natives (Prensky, 2001) and more familiar with online learning environments than older adults. This goal was also more frequently selected for participants who had already earned a doctoral degree, meaning that they are finished with institutionalized education and may be curious about recent online educational innovations. When exploring how these learner characteristics and goals relate to course engagement and completion, our results reveal that it depends on course discipline. One common finding was the positive influence of age on course engagement, even after accounting for educational level, sex, and participant goals. Participants were much more likely to continue with coursework as their age increased in both HLA and STEM courses. Though males attempted about 11% more activities than females in HLA courses, there was interestingly a smaller difference between males and females in STEM courses (6%) though this area is frequently viewed as being an unequal playing field. However, the tendency for females to avoid STEM discipline could be seen in the course enrollment numbers (see Table 1). HLA and STEM courses also differed in the educational levels that contributed to course engagement. In STEM courses, 440

Computers & Education 126 (2018) 433–442

K.M. Williams et al.

participants with graduate degrees took part in more activities than those with a bachelor's degree or lower education level. However, a relation between higher educational attainment and engagement was not found for HLA participants, where PhD students were less likely to stay engaged with the course. This difference between HLA and STEM courses in the importance of educational attainment is unknown, but may be a product of the differences observed in the learner goals between the two disciplines. For example, it is possible that more graduate students with interest in humanities and liberal arts in particular participated in course activities, compared to graduate students from STEM areas of study. Alternately, high school and undergraduate students may have been more interested in some of the content of STEM courses as a method of obtaining course credit compared to graduate students in those same courses. Possible reasons for this finding are a promising area of emerging research in terms of disciplinary and demographic differences in MOOC engagement. Our results further reveal that the goals predictive of engagement differ across disciplines; with the exception of exam preparation, no learner goals had the same relationship with engagement across HLA and STEM courses. In STEM courses, taking the course to learn more about MOOCs and online learning had a negative relationship with partaking in course activities while there was no relationship for HLA students. Across courses, participants who selected this as a learner goal generally had lower engagement, indicating that they may have dropped out of the course once their curiosity was satisfied, perhaps never having long-term goals for the course. Having a goal of exam preparation also had a negative relationship with engagement across courses, which may signify that participants became disengaged with the material if it did not seem pertinent to the credit or placement exam for which they were preparing. On the other hand, HLA participants who were already interested in the topic, due to either personal or career reasons, had a higher level of engagement in the course. In the absence of the extrinsic motivator to earn a passing grade for credit or to obtain certification, perhaps it is the intrinsic motivation of being personally interested in the topic that keeps a participant engaged in a MOOC. This aligns with much of the literature on motivation, including self-determination theory (Ryan & Deci, 2000) and goal orientation theory (Elliot & Church, 1997), both of which depict internalized and externalized mechanisms of motivation to engage with an activity. While the learner goals associated with engagement in HLA courses align educational theory and general reasoning, the relationships discovered for STEM courses were more puzzling. Opposing the patterns seen in HLA courses, the two goals reflecting an interest in the course due to it being the participant's career area or a topic of personal interest had a negative relationship with engagement in STEM courses. The only learning goal that was a positive predictor of course engagement in STEM courses was the desire to take a course at the university of with a specific professor. Both personal interest/lifelong learning and use of information in future career or studies were two goals that were significant predictors of engagement across disciplines but they had opposing relationships. Participants who had lifelong learning or personal interest goals and those who planned on using what they learned in their career were more engaged students in HLA courses. However, participants with these goals in STEM courses performed more poorly than those without these goals. It is difficult to theorize the mechanism behind these results. One possibly contributing factor is that low survey response rate in STEM courses, which was four times lower than that of HLA courses. It is possible that the low response rate may have led to results that are not representative of the true relationships between these learner goals and course engagement. It is also possible that this survey did not measure the learner motivations that are the clearest source of success in STEM courses or most important to those who choose to enroll in these courses. Though this study leaves many questions, it appears that the motivational causes of engagement in HLA and STEM MOOCs differ, which may be due to either the course material itself, differences in translation of these topics through an online format, or other variations in course organization and assessment. Further investigation should be done into the motivational factors linked to success in STEM MOOCs. It is clear, however, that students' goals impact their success at taking part in course activities and completing them, which indicate the importance of this form of motivation to learner success in MOOCs. Further, findings such as these indicate a need for more useful measurements of MOOC participant success that consider student performance, completion, or engagement in light of their individual characteristics and goals. In other words, rather than focusing on retention rates or certificates of completion, a better model for measuring MOOC participant success might include understanding their reasons for taking the course in the first place, which then might help inform the degree to which we should consider different meanings of “success”. This study contributes to our understanding of these courses by attempting to measure engagement not only in a novel way, but across multiple courses. Up to this point, even when studies measured engagement using some other metric that certification, they did so using a single course to collect data and without considering the discipline or field of the course as part of their analysis (e.g. Hone & El Said, 2016). Further, researchers who have measured engagement and motivation across multiple courses have typically used certification or activity completion as a measurement proxy (e.g. Kizilcec & Schneider, 2015). Terras and Ramsay (2015) also suggest including psychological perspectives in examining student engagement in MOOCs, and we agree that this new environment requires a broader method of assessing learning and performance. A better understanding of MOOC learners might also aid in advertising and improving such courses by including content that is interesting, topical, and aligns with the goals of learners beyond the “traditional” college demographic. Future research should also examine MOOC environments through the lenses of different theories of motivation and non-cognitive skill development to gain a more complete profile of MOOC participants. Although the study of MOOCs is still emerging, the ideological notions about how these courses will democratize and replace current forms of higher education have for the most part subsided. In its place, new ideas about how to engage students and use MOOC materials as supplemental resources in many different contexts are emerging (e.g. Kizilcec, Piech, & Schneider, 2013; Trumbore, 2014). A better understanding of student motivation for enrolling in MOOCS or accessing particular MOOC resources can help practitioners and researchers alike design open online materials that are better aligned with learner goals, motivations, and 441

Computers & Education 126 (2018) 433–442

K.M. Williams et al.

interests – and ultimately lead to more successful learning. References Agresti, A. (2013). Categorical data analysis(3rd ed.). Hoboken, NJ: John Wiley & Sons. Alraimi, K. M., Zo, H., & Ciganek, A. P. (2015). Understanding the MOOCs continuance: The role of openness and reputation. Computers & Education, 80, 28–38. Breslow, L., Pritchard, D. E., DeBoer, J., Stump, G. S., Ho, A. D., & Seaton, D. T. (2013). Studying learning in the worldwide classroom research into edX's first MOOC. Research and Practice in Assessment, 8, 13–25. Coffrin, C., Corrin, L., de Barba, P., & Kennedy, G. (2014, March). Visualizing patterns of student engagement and performance in MOOCs. Proceedings of the fourth international conference on learning analytics and knowledge (pp. 83–92). ACM. Cohen, J. (1988). Statistical power analysis for the behavioral sciences(2nd ed.). Hillsdale, NJ: Lawrence Earlbaum Associates. DeBoer, J., Ho, A. D., Stump, G. S., & Breslow, L. (2014). Changing “course”: Reconceptualizing educational variables for massive open online course. Educational Researcher, 42(2), 74–84. Elliot, A. J., & Church, M. A. (1997). A hierarchical model of approach and avoidance achievement motivation. Journal of Personality and Social Psychology, 72(1), 218–232. Hew, K. F., & Cheung, W. S. (2014). Students' and instructors' use of massive open online courses (MOOCs): Motivations and challenges. Educational Research Review, 12, 45–58. Hone, K. S., & El Said, G. R. (2016). Exploring the factors affecting MOOC retention: A survey study. Computers & Education, 98, 157–168. Jordan, K. (2014). Initial Trends in enrolment and completion of massive open online courses. International Review of Research in Open and Distance Learning, 15(1), 133–160. Kizilcec, R. F., Piech, C., & Schneider, E. (2013, April). Deconstructing disengagement:analyzing learner subpopulations in massive open online courses. Proceedings of the third international conference on learning analytics and knowledge (pp. 170–179). ACM. Kizilcec, R. F., & Schneider, E. (2015). Motivation as a lens to understand online learners: Toward data-driven design with the OLEI scale. AMC Transactions on Computer-human Interaction, 22(2), 1–24. Koller, D., Ng, A., Do, C., & Chen, Z. (2013, June 3). Retention and intention in massive open online courses: In depth. EduCause Review Online. Retrieved from http://www. educause.edu/ero/article/retention-and-intention-massive-open-online-courses-depth-0. Kolowich, S. (2013). Coursera takes a nuanced view of MOOC dropout rates. The Chronicle of Higher Education. Retrieved from http://chronicle.com/blogs/ wiredcampus/coursera-takes-a-nuanced-view-of-mooc-dropout-rates/43341. Mahraj, K. (2012). Using information expertise to enhance massive open online courses. Public Services Quarterly, 8(4), 359–368. Prensky, M. (2001). Digital natives, digital immigrants, 9, On the Horizon MCB University Press1–6 5. Ryan, R. M., & Deci, E. L. (2000). Intrinsic and extrinsic motivations: Classic definitions and new directions. Contemporary Educational Psychology, 25, 54–67. Seaton, D. T., Bergner, Y., Chuang, I., Mitros, P., & Pritchard, D. E. (2014). Who does what in a massive open online course? Communications of the ACM, 57(4), 58–65. Terras, M. M., & Ramsay, J. (2015). Massive open online courses (MOOCs): Insights and challenges from a psychological perspective. British Journal of Educational Technology, 46(3), 472–487. Trumbore, A. (2014). Rules of engagement: Strategies to increase online engagement at scale. Change: The Magazine of Higher Learning, 46(4), 38–45. Watters, A. (2013). MOOC mania: Debunking the hype around massive open online courses. The Digital Shift. Retrieved from http://www.thedigitalshift.com/2013/04/ featured/got-mooc-massive-open-online-courses-are-poised-to-change-the-face-of-education. Yuan, L., & Powell, S. (2013). MOOCs and open education: Implications for higher education [White Paper]. Retrieved from http://www.smarthighered.com/wp-content/ uploads/2013/03/MOOCs-and-Open-Education.pdf.

442