Assessing the effectiveness of a voluntary online discussion forum on improving students’ course performance

Computers & Education 56 (2011) 253–261

Contents lists available at ScienceDirect

Computers & Education journal homepage: www.elsevier.com/locate/compedu

Assessing the effectiveness of a voluntary online discussion forum on improving students’ course performanceq Cho Kin Cheng a, *, Dwayne E. Paré a, Lisa-Marie Collimore b, Steve Joordens a a b

Department of Psychology, University of Toronto, Toronto, Ontario, Canada M5L3G3 Department of Human Development and Applied Psychology, Ontario Institute for Studies in Education, University of Toronto, Ontario, Canada M5S1V6

a r t i c l e i n f o

a b s t r a c t

Article history: Received 21 April 2010 Received in revised form 28 July 2010 Accepted 28 July 2010

The purpose of this study was to evaluate the effectiveness of voluntary discussion forums in a higher education setting. Specifically, we examined intrinsic forum participation and investigated its relation to course performance across two experiments. In Experiment 1 (N ¼ 1284) an online discussion forum was implemented at the beginning of an undergraduate introductory psychology course, and measures of course performance (i.e., writing assignment grades, exam grades, and extra-credits obtained) were compared with measures of forum participation. In Experiment 2 (N ¼ 1334) an online discussion forum was implemented halfway through a second undergraduate introductory psychology course, after an initial measure of course performance was obtained, to control for the potential confound of student engagement (e.g., students who perform better in the course use the forum more). Overall, the results showed that students who participated in the forum tended to have better performance in the course, and furthermore that participating in the discussion forum, particularly reading posts on the forum, slightly improved exam performance. This study provides empirical support for the theoretical proposition that there is a facilitation effect of discussion forum participation on course performance. The results also suggest that implementation of an online discussion forum is beneficial even if a teacher only invests minimal time on the forum. Ó 2010 Elsevier Ltd. All rights reserved.

Keywords: Discussion forum Performance Page hits Lurking

1. Introduction As information and communication technologies become ubiquitous, their use has steadily increased in education (Bures, Abrami, & Amundsen, 2000; Rau, Gao, & Wu, 2008). These technologies provide educational institutions with new instructional possibilities (Hewitt, 2005) and open unprecedented opportunities for educational interactivity (Harasim, 1990). They have changed the traditional boundaries and limitations of teaching and learning, and have become essential components of modern education. Web-based asynchronous discussion forums are one example of these new technologies that have been employed to complement traditional methods and techniques (Garrison, Anderson, & Archer, 2003; Peters, 2003), and it is the technology of primary interest in this work. A discussion forum is a web application that provides a virtual environment supporting discussion and debate among student peers, as well as between teacher and student (Brower, 2003) without temporal or geographical barriers (Taradi & Taradi, 2004). The asynchronous nature of discussion forums, meaning that all participants are not required to be present or logged on at the same moment in time, allows students to work at times and places of their own choice, providing more flexibility in scheduling for the students (Harasim, 1989; Kaye, 1989). Discussion forums also extend student–teacher interactions allowing academic dialogues and social interactions to occur outside of a traditional classroom. These advantages allow discussion forums to supplement or even totally replace face-to-face interaction, particularly in large-sized courses and web-based courses. Unlike face-to-face or other synchronous interaction, there is less demand on turn taking and making immediate responses (Hammond, 1999; Kaye, 1989). An asynchronous forum allows students to have more time to structure and organize their thoughts before asking q This research was supported by a grant from the Natural Sciences and Engineering Research Council of Canada to Steve Joordens. Part of this project was presented in the form of a poster at the 38th annual meeting of the Society for Computer in Psychology, Chicago, Illinois. The full project was also presented in form of a poster at the 30th annual meeting of the Society for Teaching and Learning in Higher Education, Toronto, Ontario. * Corresponding author. E-mail address: [email protected] (C.K. Cheng). 0360-1315/$ – see front matter Ó 2010 Elsevier Ltd. All rights reserved. doi:10.1016/j.compedu.2010.07.024

254

C.K. Cheng et al. / Computers & Education 56 (2011) 253–261

a question or making a statement. In addition, students can communicate simultaneously or even participate in multiple discussions at the same time (Kaye, 1989). Some suggest that discussion forums create a more comfortable environment for some students to engage in discussion. This perceived comfort in discussion forums may stem from the fact that, like other types of computer-mediated communications, the focus is often placed on the words in the message with less emphasis on other aspects of communication (Sproull & Keisler, 1991). Because of the reduced social cues within the discussion forum, students, especially those who are introverted, feel less threatened to express their views or to ask for help from teachers and peers (Bures et al., 2000; Rau et al., 2008). It has also been suggested that the scant social cues reduce the inequalities in participation and allows for a more egalitarian mode of communication (Dubrovsky, Kiesler, & Sethna, 1991). Yet, interactions in online discussion forums are not always pleasant and mellow. Flaming, a hostile expression of strong emotion such as swearing, insults, and name-calling, is also common in forums (Lee, 2005). The support for enhanced communication provided by discussion forums may be especially important because of the sort of learning that may occur as a function of the discussions. That is, students do not only learn from course materials and lectures, they also learn a great deal from interacting with one another. What they learn is not static knowledge but rather is a creative cognitive process, mainly the process of coming up with ideas, exposing these ideas to others’ comments and criticisms, and being able to reshape ideas in light of peer discussions (Rowntree, 1995). Through this cognitive process, collaborative knowledge construction is taking place; students collaboratively develop and explore hypotheses, negotiate differing perspectives, and work towards common understandings. This is consistent with the idea that learning emerges out of efforts to construct shared meanings (Roschelle, 1992). By supporting extensive interactions outside the classroom environment, discussion forums can effectively enhance the process of acquiring, sharing and exchanging knowledge among students, thereby improving learning outcomes and performance (Leidner & Jarvenpaa, 1995). Some research studies on discussion forums have focused on students’ engagement within a discussion forum and have examined whether students would engage in pedagogically meaningful conversations or discussions in an online environment. A number of studies have employed content analyses to assess the level of cognitive and social engagement in discussion forums (e.g., Garrison, Anderson, & Archer, 1999, 2001; Henri, 1992; Newman, Webb, & Cochrane, 1995). Although there have been methodological concerns with these content analyses in term of their validities and reliabilities (De Wever, Schellens, Valcke, & Van Keer, 2006; Schrire, 2006; Strijbos, Martins, Prins, & Jochems, 2006; Valcke & Martens, 2006; Weinberger & Fischer, 2006), the results of these studies converged generally showing that students would engage in meaningful discussions in online discussion forums, at least to some extent. Some studies further suggest that the level of engagement in discussion forums can be moderated by factors such as teacher’s involvement and possibility of receiving course credits (Garrison et al., 1999, 2001, 2003; Guzdial & Turns, 2000; Taradi & Taradi, 2004). Yet, it should be noted that these analyses focused heavily on the actual text itself; they might have overlooked the effect of other forms of participation. While the sorts of content analyses described above are informative, the content of posted messages is just one part of the way forums can affect learning. It is indeed the case that posted messages set the tone of the forum, but posting the messages and replying to posts reflect only a portion of the activities in the forum. Students have to read the messages of other students before they can give relevant comments. Students who post messages would probably want to come back to read what others have said about their posts. Furthermore, many students are so-called lurkers (Beaudoin, 2002); they participate by reading, but perhaps not by posting, similar to students who listen to lectures but never ask questions. One would expect that any pedagogical benefits associated with lurking would depend on the richness of post content, but by focusing only on posted messages some benefits of forum use may be overlooked. Thus, the larger question is whether the level of communication supported by online discussion forums, as indicated by both posts and page hits, is sufficient to promote collaborative learning and facilitate knowledge acquisition as reflected by improved academic outcomes. Several researchers have attempted to answer this question. Patel and Aghayere (2006) investigated the relation between forum participation and course performance in two undergraduate civil engineering courses. In their study, they found that forum participation, defined by the number of posts and post views in the discussion forum, were weakly but positively correlated with the students’ course grade. However, one important confound of the study is that forum participation accounted for 5% of the course grade, and the authors failed to factor out this 5% contribution from the course grade in the analysis. Thus, those who participated more in the forum might have done better in the course because of their higher participation mark alone. This confound could sufficiently explain the correlations found in the study, leaving the effectiveness of discussion forums still largely unknown. A similar study by Taradi and Taradi (2004) examined the effect of forum membership on course performance for a physiology class among medical students. The authors compared the final exam grade of students who were members in a discussion forum to those who were non-members. The findings revealed that members of the forum obtained significantly higher grades on their final exam than the nonmembers. The authors did not, however, further elaborate on or quantify the level of participation in the forum. Therefore, suggestions about the effectiveness of discussion forums could only be made in a very general sense. Taken together, the findings from these previous studies suggest that online discussion forums can be beneficial, but they leave several questions unanswered. Aiming to fill the gap between the theoretical propositions, which suggests participation in discussion forums has a positive impact on course performance and learning in general, and the available empirical evidence, the current study was conducted to evaluate the effectiveness of discussion forums and to assess the effect of forum participation on course performance. This was achieved by implementing online discussion forums in introductory psychology courses and comparing writing assignment grades, exam grades and extra-credits obtained with the various quantitative measures of participation in the forum. Most of the studies on discussion forums have examined the type of discussion forums that provide a platform for discussion or debate over a given topic initiated by a teacher. Participation in this type of forum is usually a part of the course requirements; students are either given credits for participating or graded according to their level of contributions. Relatively few studies have focused on another common type of discussion forum which is implemented as a medium for students to ask question or discuss anything related to the course - but is not directly associated with any grades. This type of forum provides opportunities for students to discuss questions about the course materials and issues that extend from the materials. It is often adopted to compensate for the diminishing opportunity for interaction in a large-sized course or a web-based course. Participation in this type of forum is usually voluntary and intrinsic. Students are given no external incentive or course credit for participating, and the extra materials discussed in the forum are not required for the exams. Students participate because of their own willingness to learn. The current study was intended to examine this type of forum participation, therefore no credit was awarded for participation, and teacher/teaching assistants only played a passive role in the forums. In accordance to the idea of

C.K. Cheng et al. / Computers & Education 56 (2011) 253–261

255

collaborative learning, we hypothesized that participation in online discussion forums would be related to a higher level of performance in a course (Experiment 1), and also it would lead to a facilitation effect and improve students’ course performance (Experiment 2). 2. Experiment 1 2.1. Method 2.1.1. Participants A group of undergraduate students (N ¼ 1284) enrolled in an introductory psychology course participated in this experiment. Among these students, about a quarter of them attended the regular in-class session, while the other three-quarters watched the recorded lectures over the internet; recorded lectures were made available on the same day as the in-class sessions and were posted on the internet for about one week. Students were allowed to switch between these two options at any point in the semester as they wished. This version of blended learning is referred to as the WebOption (see Bassili, 2008; Bassili & Joordens, 2008 for a detailed discussion of the webOption approach to blended learning). 2.1.2. Materials 2.1.2.1. Online discussion forum. An online discussion forum was setup using the discussion board function provided within the Blackboard Learning System. It was structured to be accessible only to students enrolled in the course, the teaching assistants, and the professor. The anonymous post function was disabled in order to keep students accountable. The forum was separated into three main sections: 1) individual chapters and general psychology related topics (i.e., content), 2) administrative and technical inquiries about exams, writing assignments, extra credits, and online lectures (i.e., administrative), and 3) subsections that did not fit into the first two sections, which included setting up study groups and topics outside psychology (i.e., other). The layout of the Blackboard forum is shown in Fig. 1. 2.2. Procedure The online discussion forum was made available to students at the beginning of the course through the course website, and remained accessible until the end of the course. It was available 24 h a day, 7 days a week throughout this period. Participation in the forum was voluntary; no course credit was associated with posting materials in the forum. Also, knowledge of the materials discussed in the forum was not required for the exams or writing assignments. The teaching assistants (graduate students in the Department of Psychology) and the professor took a passive role in the forum. They monitored discussions in the forum throughout the course and only intervened when questions were not answered fully, were answered incorrectly, or when questions were directed explicitly to them. To fulfill the course requirements, students had to complete two written peerScholar assignments (see Paré & Joordens, 2008), each accounting for 5% of their final grade, and two multiple-choice exams; the midterm and final exams were worth 40% and 50% respectively. Students were also given the opportunity throughout the course to participate in research experiments in the psychology department to earn additional credits, which could give students a maximum of 3% extra-credits. 2.3. Results Forum participation was assessed by the number of posts, and a post was denoted as a message or a comment a student wrote, or replied to, in the forum (Harasim, 1993). In total there were 450 posts. One hundred and forty three students (11.1%) posted at least once. The number of posts an individual posted in the forum ranged from 0 to 26, with a mean of 0.4 and standard deviation of 1.64 (for those who posted at least one, M ¼ 3.1, SD ¼ 3.94). Due to the large number of students who never participated in the forum, the distribution was positively skewed, g1 ¼ 8.94, SEM ¼ 0.068. Splitting the course into halves around the midterm exam, 216 (48%) posts were logged in the first half of the course and 234 (52%) posts were logged in the second half. The difference between number of posts in the first and the second half of the course was not significant, t(1283) ¼ 0.40, d ¼ 0.011, ns. The correlation of posts in the first half and the second half was weak but significant, r ¼ 0.272, r2 ¼ 0.074, p < 0.001. Thus, students who used the forum in the first half of the course tended to also be the ones who used it in the second half. Bassili (2008) showed that instructional mode choice was not related to course performance, therefore data from in-class and online session were analyzed together. Course performance was measured by students’ performance on various assessment components of the course, including the scores on the two peerScholar assignments, the scores of the midterm and final exams, and the amount of extra-credits obtained. The mean of the first and second peerScholar assignments was 3.4 (SD ¼ 0.83) and 3.1 (SD ¼ 1.24) out of 5, respectively. The mean of the midterm exam was 25.9 (SD ¼ 6.78) out of 40 and the mean of the final exam was 29.2 (SD ¼ 10.48) out of 50. Students on average obtained 1.7 (SD ¼ 1.27) extra-credits out of a possible 3. The distribution of the scores on all these assessment measures were negatively skewed as fewer students received scores lower than the means: for the first peerScholar assignment, g1 ¼ 3.13, SEM ¼ 0.068; for the second peerScholar assignment, g1 ¼ 1.92, SEM ¼ 0.068; for the midterm exam, g1 ¼ 0.84, SEM ¼ 0.068; for the final exam, g1 ¼ 0.73, SEM ¼ 0.068; and for the extra-credit, g1 ¼ 0.28, SEM ¼ 0.068. A series of linear regression analyses were conducted. The numbers of posts were paired with the standardized scores of each assessment measure. The summary of the results are presented in Table 1. The results revealed a significant correlation between forum participation and performance on the midterm exam, r ¼ 0.106, r2 ¼ 0.011, p < 0.001; b ¼ 0.064, t ¼ 3.803, p < 0.001, and between forum participation and performance on the final exam, r ¼ 0.087, r2 ¼ 0.008, p < 0.005; b ¼ 0.053, t ¼ 3.115, p < 0.005. Forum participation was also significantly related to the scores students received for the two peerScholar assignments: first assignment, r ¼ 0.065, r2 ¼ 0.004, p < 0.05; b ¼ 0.039, t ¼ 2.319, p < 0.05; second assignment, r ¼ 0.058, r2 ¼ 0.003, p < 0.05; b ¼ 0.035, t ¼ 2.085, p < 0.05. Lastly, the students who posted more in the forum were likely to obtain more extra-credits, r ¼ 0.119, r2 ¼ 0.014, p < 0.001; b ¼ 0.073, t ¼ 4.298, p < 0.001. None of the correlations are very large, but if one imagines all the factors that likely have large effects on a student’s performance in the course (e.g., intellect, study habits, study time, vocabulary, etc.), one should not expect large correlations between performance and intrinsic participation on an online forum.

256

C.K. Cheng et al. / Computers & Education 56 (2011) 253–261

Fig. 1. The online discussion forum (provided by the Blackboard Learning System) in Experiment 1.

Overall, our results suggest that the students who participated in the forum tended to perform slightly better on all five aspects of the course. Although there was variability, one post translated to 0.064 and 0.053 standard deviations higher grade on the midterm and final exam respectively. Posting in the forum was also related to higher scores on the peerScholar assignments, one post translated to 0.039 and 0.035 standard deviation higher on the first and second peerScholar assignment respectively. Finally, the results also showed that students who posted in the forum were likely to get more extra-credits for the course; one post translated to 0.073 standard deviations more extra-credits. 2.4. Discussion This experiment examined the relation between discussion forum participation and course performance by implementing an online discussion forum in an introductory psychology course and correlating students’ performance on various aspects of the course with the Table 1 Summary of regression analyses between forum participation and course performance in Experiment 1. Course Performance

r bpost **Denoted p < 0.01. *Denoted p < 0.05.

Midterm

Final

1st Assignment

2nd Assignment

Extra-credit

0.106** 0.064**

0.087** 0.053**

0.065* 0.039*

0.058* 0.035*

0.119** 0.073**

C.K. Cheng et al. / Computers & Education 56 (2011) 253–261

257

participation in the forum. The results showed that the students who participated in the forum tended to perform better. This might suggest that participating in discussion forum improves course performance. However, it could also have been the students who participated in the discussion forum were the better performing students regardless (Taradi & Taradi, 2004). Students who are more engaged in the course are probably more likely to use the discussion forum to study and work harder in the course. Consequently, both forum participation and better performance may entirely reflect this variable of engagement. Experiment 1 does not rule out such a possibility. Thus, to better investigate the effectiveness of a discussion forum on course performance a second experiment was conducted to better address the potential of an engagement confound. In the second experiment, the discussion forum was introduced in the second half of the course after the first peerScholar assignment and the midterm exam. This allowed students’ performance on the first peerScholar assignment and the midterm exam, which were administrated before the discussion forum was implemented, to serve as a baseline measurement of performance, and the differences in performance between the first and second measures could be used as a means to assess the effectiveness of a discussion forum. Also missing from the first experiment was the factor of post viewing. Post viewing occurs when an individual browses through an online forum only reading other people’s posts and not actually posting any messages or replying to any questions. Those who engage in post viewing behaviour still obtain extra information which might not have otherwise been available to them. It is even quite probable that students’ view posts more than they write posts in the forum. This can be hinted from the end-of-term evaluation survey; among students indicating that they used the forum (160 out of 808 respondents), 94.4% of them indicated that they usually read others’ posts, while only 34.4% and 11.9% of them indicated they usually asked questions or answered questions on the forum. Without a post viewing measure, a large portion of forum participation was not accounted for. Therefore, in the second experiment, in addition to the number of posts, post viewing, defined as the number of page views, was also used as a measure of forum participation. 3. Experiment 2 3.1. Method 3.1.1. Participants Another group of undergraduate students (N ¼ 1334) enrolled in the introductory psychology course in a different semester participated in this experiment. As in the previous experiment, the course was in WebOption format resulting in about a quarter of the students attending in-class sessions, while the rest watched the recorded lectures over the internet. Otherwise, all students were treated in an identical manner. 3.1.2. Materials 3.1.2.1. Online discussion forum. An open source discussion forum (phpBB) was adopted in this experiment because the discussion board function within the Blackboard Learning System did not compile data for number of page views. As an additional feature, this open source forum allowed students to change their screen name and add an avatar to hide their identity from their peers, while still allowing teaching assistants and the professor to maintain accountability by having access to student identities through the database. Other than that, the accessibility settings and the structures of the forum remained the same as Experiment 1. The layout of the open source forum is shown in Fig. 2. 3.2. Procedure The online discussion forum was made accessible via the course website beginning in the second half of the course, after the first peerScholar assignment and the multiple-choice midterm exam had been administered. Students were made aware of the forum through an announcement in class and on the course website. Otherwise, the forum was administrated the same way as in Experiment 1. As in the previous experiment, students had to complete two peerScholar assignments, each worth 5% of their final grade, and two multiple-choice exams; a midterm worth 40% and a final worth 50%. Students were also given the opportunity throughout the course to obtain extra-credit through participating in research experiments in the psychology department; however, in contrast to the previous experiment, students could only get a maximum of 2% extra-credits. 3.3. Results Participation in the discussion forum was assessed by the number of posts and the number of page views. A post was defined the same way as the previous experiment. A page view was denoted as each forum page opened by a student. In total, 927 posts were logged in the forum for the second half of the course (i.e., the duration of the semester that the forum was available). One hundred and fifty six (11.7%) students posted at least once. The number of posts an individual student posted in the forum ranged from 0 to 70, with a mean of 0.7 and standard deviation of 4.15 (for those who posted at least once, M ¼ 5.9, SD ¼ 10.79). The distribution was positively skewed, g1 ¼ 11.22, SEM ¼ 0.067. Compared to those who posted in Experiment 1 (M ¼ 0.4, SD ¼ 1.64), students in Experiment 2 posted significantly more often in the forum, t(2616) ¼ 2.77, d ¼ 0.054, p < 0.01. Interestingly, students in Experiment 2 posted on the forum more even though the duration being examined was half the length of the duration being examined in Experiment 1. We attribute this increased activity to the better interface provided by the open-source discussion forum software; a more complete explanation is presented in the discussion. In total, there were 31,806 page views among all students. The number of page views per student ranged from 0 to 990, with a mean of 23.8 and standard deviation of 76.0 (for those who viewed at least one page, M ¼ 48.6, SD ¼ 102.88). Six hundred and fifty four (49.0%) students viewed at least one page. On average, students had 34.3 times more page views than posts, t(1333) ¼ 11.65, d ¼ 0.319, p < 0.001. The distribution was positively skewed, g1 ¼ 7.21, SEM ¼ 0.067. This result supports the suggestion that most of the participation in the discussion forum was in the form of page views, which was not accounted for in the previous experiment. However, there was a strong

258

C.K. Cheng et al. / Computers & Education 56 (2011) 253–261

Fig. 2. The open source online discussion forum (provided free of charge by phpBB) in Experiment 2.

correlation between these two forum participation measures, r ¼ 0.829, r2 ¼ 0.687, p < 0.001, suggesting students who had more page views tended to make more posts as well. This also supports the idea that posts and page views were interactive. Course performance was assessed by the scores of the two peerScholar assignments, the midterm and final exams, and the amount of extra-credit obtained. On average, students received 3.3 (SD ¼ 1.07) out of 5 on their first peerScholar assignment, 3.3 (SD. ¼ 0.96) out of 5 on their second peerScholar assignment, 25.5 (SD ¼ 7.14) out of 40 on their midterm exam, 27.4 (SD ¼ 9.59) out of 50 on their final exam, and students obtained 1.4 (SD. ¼ 0.86) out of 2 extra credits. Similar to the previous experiment, the distribution of the scores in all of these assessment measures were negatively skewed: for the first peerScholar assignment, g1 ¼ 2.35, SEM ¼ 0.067; for the second peerScholar assignment, g1 ¼ 2.64, SEM ¼ 0.067; for the midterm exam, g1 ¼ 1.11, SEM ¼ 0.067; for the final exam, g1 ¼ 0.82, SEM ¼ 0.067; and for extra-credit, g1 ¼ 0.90, SEM ¼ 0.067. A series of multiple regression analyses were performed to relate the standardized score of each of the course performance measures with the two discussion forum participation measures. The summary of the results are presented in Table 2. Discussion forum participation was correlated to all of the course performance measures: first peerScholar assignment, R ¼ 0.105, R2 ¼ 0.011, p < 0.005; second peerScholar assignment, R ¼ 0.113, R2 ¼ 0.013, p < 0.001; midterm exam, R ¼ 0.104, R2 ¼ 0.011, p < 0.005; final exam, R ¼ 0.161, R2 ¼ 0.026, p < 0.001; and extra credit, R ¼ 0.127, R2 ¼ 0.016, p < 0.001. These relations between forum and participation and course performance can be explained mostly by the number of page views rather than posts, which were supported by the significant regression coefficients of the number of page views in these regression analyses: first peerScholar assignment, b ¼ 0.002, t ¼ 3.02, p < 0.005; second peerScholar assignment; b ¼ 0.002, t ¼ 3.58, p < 0.001; final exam, b ¼ 0.003, t ¼ 4.29, p < 0.001; and extra credit, b ¼ 0.002, t ¼ 3.53, p < 0.001 (the regression coefficient of pages views for the midterm exam was only marginally significant, b ¼ 0.001, t ¼ 1.77, p ¼ 0.077). The unique contributions of posts to the various course performance measures were insignificant: first peerScholar assignment, b ¼ 0.014, t ¼ 1.17, ns.; second peerScholar assignment, b ¼ 0.021, t ¼ 1.79, ns.; midterm exam, b ¼ 0.005, t ¼ 0.41, ns.; final exam, b ¼ 0.014, t ¼ 1.24, ns.; and extracredits, b ¼ 0.014, t ¼ 1.23, ns.

C.K. Cheng et al. / Computers & Education 56 (2011) 253–261

259

Table 2 Summary of regression analyses between forum participation and course performance in Experiment 2. Course Performance

R bpage bpost

view

Midterm

Final

1st Assignment

2nd Assignment

Extra-credit

0.104** 0.001þ 0.005

0.161** 0.003** 0.014

0.105** 0.002** 0.014

0.113** 0.002** 0.021

0.127** 0.002** 0.014

**Denoted p < 0.01. þ Denoted p < 0.1.

As discussed previously, these relations may reflect a third variable of course engagement, rather than reflecting a benefit of forum participation per se. With this in mind we performed additional analyses that focused on performance improvements across the peerScholar assignments and the high-stakes exams. The notion is that while a more deeply engaged student may tend to perform well on all assignments and exams, there is no reason to suppose that a more deeply engaged student would improve more across assignments or exams. Thus, if such an improvement is related to forum use, it is less likely that this relation is due to a confound with student engagement. In order to investigate whether participating in a discussion forum might improve course performance, the standardized difference scores between the midterm and final exams were computed. A multiple regression was conducted using these difference scores as the dependent variable and the two measures of forum participation as the independent variables. The analysis showed a significant correlation between discussion forum participation and the difference scores in exams, R ¼ 0.074, R2 ¼ 0.005, p < 0.05. The regression coefficient for the number of page views was significant, b ¼ 0.002, t ¼ 2.58, p < 0.05, while the coefficient for the number of posts was insignificant, b ¼ 0.019, t ¼ 1.70, ns. These results suggest that an improvement in course performance was related to page views in an online discussion forum. Consistently, students who had above mean number of page views (M ¼ 23.8) improved more between exams than the students who had less page views, t(1322) ¼ 2.531, p < 0.05. Another regression analysis was conducted on the difference scores of the peerScholar assignments and the two measures of forum participation. Neither the regression analysis nor the t comparison revealed a significant result, R ¼ 0.016, R2 ¼ 0.000, ns.; t(1322) ¼ 0.347, ns. The summary of the regression analyses between forum participation and changes in course performance are presented in Table 3. 3.4. Discussion Experiment 2 investigated the relation between online discussion forum participation, overall course performance, and improvement in course performance. This was achieved by implementing an online discussion forum midway through the semester. Then the performance differences in exams and peerScholar assignments were compared with participation in the forum. This manipulation allowed for the assessment of the baseline course performance when no discussion forum was available, and the change of course performance when the forum was made available. One unexpected result was the increased use of the discussion forum overall in Experiment 2 relative to Experiment 1. The number of students posting on the forum stayed roughly the same at 11%, but the number of posts doubled in half the time. Given that forum use was voluntary in both cases, and that we used roughly the same wording when introducing the forums to the students, this enhanced use should not be due to any procedural change in implementation. Instead, we believe it is due primarily to the ease of use of the discussion forum interface utilized in Experiment 2. The interface provided by the open-source program we used was more straightforward in its setup, making it, we believe, easier to learn and use. Also, the feature of allowing students to change their screen name and avatar to personalize their account and hide their true identity might have created a more comfortable environment for students to interact. While this was not the focus of our work, it is a highly relevant point because, if discussion forums truly are beneficial to learning, then it is important that any barriers to use of the forum should be minimized. With respect to the effects of forum participation, the results presented in Experiment 2 largely replicate those presented in Experiment 1. Consistent with the first experiment, it was found that students who participated in the discussion forum did better in the course overall. These students had higher grades on exams and peerScholar assignments, and they also earned more extra-credits. These relations between forum participation and course performance can be accounted mostly by the number of pages students viewed while the number of posts students posted did not matter as much. This finding is different from the previous experiment where only the number of posts was recorded and analyzed. That said, clearly someone needs to be posting for others to be viewing, and it is likely the case that the interaction between the posts and their follow-ups are what make the viewing beneficial to the degree it is. Thus, to some extent, our findings show how entwined these forum activities are. This is supported by the high correlation between posts and page views.

Table 3 Summary of regression analyses between forum participation and changes in course performance in Experiment 2. Change in Course Performance

R bpage bpost

views

*Denoted p < 0.05.

Final  Midterm

2nd Assignment  1st Assignment

0.074* 0.002* 0.019

0.016 0.000 0.007

260

C.K. Cheng et al. / Computers & Education 56 (2011) 253–261

The unique aspect of Experiment 2 was the potential to get closer to identifying a causal role of forum participation by examining improvement in performance rather than raw performance alone. The idea here is that while we would expect more engaged students to perform better on all writing assignments and exams, and also to be more likely to engage in the online forum, we would not necessarily expect them to improve any more from the first to the second writing assignment and from the midterm to the final. Thus, improvement in performance may provide a more relevant measure of learning that may be less susceptible to a confound with engagement. Our results suggested that participating in the discussion forum was related to improvement in exam performance. Again, this improvement was mostly accounted for by the number of page views. Roughly 63 pages views translated to 0.1 standard deviation improvement in exam performance. Once again, the correlation was small, and there was a lot of variability between the exam performance and the number of page views, but the correlation was reliable. As mentioned before, there are a lot of factors that likely have large effects on a student’s performance in the course, thus it is not unexpected to have small correlations between performance and intrinsic participation on an online forum. Moreover, there was no evidence for a relation between discussion forum participation and improvement in assignment performance. This finding was not surprising because of the nature of the assignments. In particular, students were required to write a critical response to an article that posed a strong position on a controversial topic. Assignments were thus not as tightly related to the course materials compared to the two exams; instead, they assessed the students’ ability to think critically. 4. General discussion It has been assumed that discussion forums facilitate students’ learning and improve their performance in a course through expanding the opportunity for discussion and communication among students and teacher. This assumption is often based on the idea that discussion promotes collaborative learning and the construction of shared meanings which, in turn, facilitates better understanding of the course material and improves course performance (Leidner & Jarvenpaa, 1995). While this all makes a great deal of sense theoretically, empirical demonstrations supporting the beneficial effects of intrinsic forum participation are few, and are limited in terms of the ability to generalize their findings or to rule out third variable explanations. The current study evaluated the effectiveness of intrinsic discussion forums in a higher education setting by assessing the relation between forum participation and course performance in two introductory psychology courses. In the first experiment, the discussion forum was implemented at the beginning of the course. The results showed that the number of posts students made was positively related to the scores of all the assessment components in the course. In the second experiment, the discussion forum was implemented halfway through the course and the forum participation measures were expanded to include the number of page views. The results again showed a positive relation between forum participation and course performance, carried primarily by the number of pages students viewed rather than by their posting behaviour. Experiment 2 also showed that forum participation was not only related to raw performance scores, but also to improvements in performance across two high-stakes exams. These results add to the existing research and provide even stronger support for the idea that forum participation, particularly post viewing, promotes and leads to better outcomes in exam performance. These results suggest that extra opportunities for discussion and interaction provided by an online discussion forum can enhance learning and facilitate understanding of course materials, which, in turn, lead to better course performance. This is consistent with the notion of collaborative learning which suggests that learning is a process of constructing shared knowledge through interaction between individuals (Leidner & Jarvenpaa, 1995; Roschelle, 1992). One important implication of the collaborative learning model is that receiving feedback from peers and the teacher is a crucial part of learning. Students expose their ideas to others’ comments and criticisms, and with these new perspectives, they reshape their ideas and reconstruct their knowledge to achieve a higher level of understanding (Rowntree, 1995). This implies that learning might have a close relation with the amount of feedback received, which is supported by the relation found between page views and improvement in exam grades along with the evidence that page views and posts were interactive. These results support the idea that students who read others’ posts and provide comments and feedbacks, and those students who posted would likely return later to read what others would say about their posts. These interactions provide the basis for facilitating learning and underpin the idea of collaborative learning. These results also suggest that the previous works that focused on analyzing the content in discussion forums might have overlooked the contribution of reading posts on learning and failed to take into account the component of engagement. An important merit of this study was the large number of students in the two experiments. Having over a thousand students in each experiment, this study had very high statistical power. Such high power enabled the discovery of even small effects of forum participation on course performance. Given that most studies typically have substantially fewer numbers of students, lack of statistical power could be the reason why little research has been reported on the facilitation effect of discussion forums. This study is not without limitations and so caution should be taken when evaluating the results and making conclusions about the facilitation effects of discussion forums. Firstly, although we found that forum participation promotes better course performance, the weak correlations suggest that there are substantial variabilities in the data. For instance, there were students who had never used the discussion forum and yet they performed very well in the course. On the other hand, there were also students who used the forum yet did poorly in the course. Although, it was not very surprising to find only weak correlations given that discussion forum is only a relatively minor component of the course while the majority of the assessment materials were based on the lectures and textbook. Secondly, the distributions of the data might elicit some concern. The distributions of forum participation measures and course performance measures were deviated from a normal distribution. Such deviations might have taken a toll on the accuracy of the statistical analyses. However, it should be noted that the skewed distributions of the data reflected the realistic distributions of these measures in a real-life setting. When participation in a discussion forum is not mandatory and does not bear any course credit, a large proportion of the students in class would not use the discussion forum at all, while a few students would use it a lot more than the other students. This pattern gives rise to the highly positive skewed data. While slightly negative skewed distribution of the course performance measures were also consistent with typical grade distribution of typical undergraduate courses. Finally, the results this study also tapped into two issues about discussion forum use in higher educational settings. First, it is often a challenge to get students involved in class discussions in any educational environment. Although this study found that participating in

C.K. Cheng et al. / Computers & Education 56 (2011) 253–261

261

a discussion forum can help students learn and improve their grades, only half of the class voluntarily went onto the forum at some point during the semester and only about 11% of the students actually made a post. Similar results were found in a previous study in which participation in a discussion forum was voluntary (Taradi & Taradi, 2004). Furthermore, it is even more difficult to reach lower performing and unmotivated students, which was suggested by the positive correlation between forum participation and various assessment measures. Yet, they are probably the ones that would benefit the most from participating in the forum. Second, there is an enormous difference between the numbers of posts between the two experiments. One possible reason might be the difference in functionality and usability of the two forums. The open source discussion forum used in the second study seems to have a cleaner layout and simpler interface which could allow for easier browsing and reading of posts. It also allows users to personalize their own accounts including screen name and avatar. Future studies should investigate the effects of these functions and factors on the forum participation rate. However, despite these limitations, the current results provide the strongest support to date that forums can be beneficial for many students and that, on average, they enhance learning. The correlation between forum use and improvement in grades takes previous findings a step further by making it far less likely that the correlations obtained are due to a more general student engagement variable. Rather, those who participated more in a discussion forum that was not available until after the midterm showed more of an improvement in performance. Perhaps the presence of a forum transforms less engaged students into more engaged students, which is clearly the hope, a hope seemingly supported by our data. References Bassili, J. N. (2008). Motivation and cognitive strategies in the choice to attend lectures or watch them online. Journal of Distance Education, 22(3), 129–148. Bassili, J. N., & Joordens, S. (2008). Media player tool usage, satisfaction with online lectures and examination performance. Journal of Distance Education, 22(2), 93–108. Beaudoin, M. F. (2002). Learning or lurking? Tracking the “invisible” online student. The Internet and Higher Education, 5(2), 147–155. Brower, H. H. (2003). On emulating classroom discussion in a distance-delivered BHR course: creating an on-line learning community. Academy of Management Learning and Education, 2(1), 22–36. Bures, E. M., Abrami, P. C., & Amundsen, C. (2000). Student motivation to learn via computer conferencing. Research in Higher Education, 41(5), 593–621. De Wever, B., Schellens, T., Valcke, M., & Van Keer, H. (2006). Content analysis schemes to analyze transcripts of online asynchronous discussion groups: a review. Computers & Education, 46(1), 6–28. Dubrovsky, V. J., Kiesler, S., & Sethna, B. N. (1991). The equalization phenomenon: status effects in computer-mediated and face-to-face decision-making groups. HumanComputer Interaction, 6, 119–146. Garrison, D. R., Anderson, T., & Archer, W. (1999). Critical inquiry in a text-based environment: computer conferencing in higher education. The Internet and Higher Education, 2(2–3), 87–105. Garrison, D. R., Anderson, T., & Archer, W. (2001). Critical thinking, cognitive presence, and computer conferencing in distance education. The American Journal of Distance Education, 15(1), 7–23. Garrison, D. R., Anderson, T., & Archer, W. (2003). A theory of critical inquiry in online distance education. In M. G. Moore, & W. G. Anderson (Eds.), Handbook of distance education. London, England: Laurence Erlbaum. Guzdial, M., & Turns, J. (2000). Effective discussion through a computer-mediated anchored forum. Journal of the Learning Sciences, 9(4), 437–469. Hammond, M. (1999). Issues associated with participation in on line forums – the case of the communicative learner. Education and Information Technologies, 4(4), 353–367. Harasim, L. M. (1989). On-line education: a new domain. In R. Mason, & A. Kaye (Eds.), Mindweave: Communication, computers, and distance education. Elmsford, New York: Pergamon. Harasim, L. M. (1990). Online education: an environment for collaboration and intellectual amplification. In L. M. Harasim (Ed.), Online education: Perspectives on a new environment. New York, New York: Praeger Publishers. Harasim, L. M. (1993). Collaborating in cyberspace: using computer conferences as a group learning environment. Interactive Learning Environments, 3, 119–130. Henri, F. (1992). Computer conferencing and content analysis. In A. R. Kaye (Ed.), Collaborative learning through computer conferencing. The Najadan papers (pp. 117–136). London, England: Springer-Verlag. Hewitt, J. (2005). Toward an understanding of how threads die in asynchronous computer conferences. Journal of the Learning Sciences, 14(4), 567–589. Kaye, A. (1989). Computer-mediated communication and distance education. In R. Mason, & A. Kaye (Eds.), Mindweave: Communication, computers, and distance education. Elmsford, New York: Pergamon. Lee, H. (2005). Behavioral strategies for dealing with flaming in an online forum. Sociological Quarterly, 46(2), 385–403. Leidner, D. E., & Jarvenpaa, S. L. (1995). The use of information technology to enhance management school education – a theoretical view. MIS Quarterly, 19(3), 265–291. Newman, D. R., Webb, B., & Cochrane, C. (1995). A content analysis method to measure critical thinking in face-to-face and computer supported group learning. Journal of Interpersonal Computing and Technology, 3, 56–77. Paré, D. E., & Joordens, S. (2008). Peering into large lectures: examining peer and expert mark agreement using peerScholar, an online peer-assessment tool. Journal of Computer Assisted Learning, 24(6), 526–540. Patel, J., & Aghayere, A. (2006). Students’ perspective on the impact of a web-based discussion forum on student learning. In: Paper presented at the 36th Annual Frontiers in Education, San Diego, California. Peters, O. (2003). Learning with new media in distance education. In M. G. Moore, & W. G. Anderson (Eds.), Handbook of distance education. London, England: Laurence Erlbaum. Rau, P. L. P., Gao, Q., & Wu, L. M. (2008). Using mobile communication technology in high school education: motivation, pressure, and learning performance. Computers & Education, 50(1), 1–22. Roschelle, J. A. (1992). Learning by collaborating: convergent conceptual change. Journal of the Learning Sciences, 2(3), 235–276. Rowntree, D. (1995). Teaching and learning online: a correspondence education for the 21st century? British Journal of Educational Technology, 26(3), 205–215. Schrire, S. (2006). Knowledge building in asynchronous discussion groups: going beyond quantitative analysis. Computers & Education, 46(1), 49–70. Sproull, L., & Keisler, S. (1991). Connections: New ways of working in the networked organisation. Cambridge, Massachusetts: MIT Press. Strijbos, J.-W., Martins, R. L., Prins, F. J., & Jochems, W. M. G. (2006). Content analysis: what are they talking about? Computers & Education, 46(1), 29–48. Taradi, S. K., & Taradi, M. (2004). Expanding the traditional physiology class with asynchronous online discussions and collaborative projects. Advances in Physiology Education, 28(2), 73–78. Valcke, M., & Martens, R. (2006). The problem area of researching computer supported collaborative learning: introduction to the special section. Computers & Education, 46(1), 1–5. Weinberger, A., & Fischer, F. (2006). A framework to analyze argumentative knowledge construction in computer-supported collaborative learning. Computers & Education, 46(1), 74–95.