Comparing synchronous and asynchronous online discussions for students with disabilities: The impact of social presence

Journal Pre-proof Comparing synchronous and asynchronous online discussions for students with disabilities: The impact of social presence Ibrahim Dahlstrom-Hakki, Zachary Alstad, Manju Banerjee PII:

S0360-1315(20)30042-7

DOI:

https://doi.org/10.1016/j.compedu.2020.103842

Reference:

CAE 103842

To appear in:

Computers & Education

Received Date: 10 June 2019 Revised Date:

1 February 2020

Accepted Date: 7 February 2020

Please cite this article as: Dahlstrom-Hakki I., Alstad Z. & Banerjee M., Comparing synchronous and asynchronous online discussions for students with disabilities: The impact of social presence, Computers & Education (2020), doi: https://doi.org/10.1016/j.compedu.2020.103842. This is a PDF file of an article that has undergone enhancements after acceptance, such as the addition of a cover page and metadata, and formatting for readability, but it is not yet the definitive version of record. This version will undergo additional copyediting, typesetting and review before it is published in its final form, but we are providing this version to give early visibility of the article. Please note that, during the production process, errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain. © 2020 Published by Elsevier Ltd.

Ibrahim Dahlstrom-Hakki: Conceptualization, Methodology, Formal analysis, Investigation, Resources, Writing - Original Draft, Writing - Review & Editing, Visualization, Supervision, Project administration, Funding acquisition. Zachary Alstad: Validation, Formal analysis, Investigation, Resources, Data Curation, Writing - Original Draft, Writing - Review & Editing, Visualization, Project administration. Manju Banerjee: Conceptualization, Investigation, Supervision, Funding acquisition.

Running head: COMPARING SYNCHRONOUS AND ASYNCHRONOUS DISCUSSIONS

1

Comparing Synchronous and Asynchronous Online Discussions for Students with Disabilities: The Impact of Social Presence

Ibrahim Dahlstrom-Hakki Zachary Alstad Manju Banerjee Landmark College 19 River Road South Putney, VT 05346

Author Note This material is based upon work supported by the National Science Foundation under Grant No. HRD-1420198. Ibrahim Dahlstrom-Hakki is now at TERC. Correspondence concerning this article should be addressed to Ibrahim Dahlstrom-Hakki, EdGE at TERC, 2067 Massachusetts Avenue, Cambridge, MA 02140. Contact: [email protected]

COMPARING SYNCHRONOUS AND ASYNCHRONOUS DISCUSSIONS

2

Abstract The affordances of online learning have the potential to improve learning outcomes for students with disabilities by allowing customization and flexibility to meet individual needs. However, there are significant executive function and language processing demands that may be particularly challenging for this population. With that in mind, interventions guided by social presence theory may increase students’ ability to engage in online settings. This study implemented a computer mediated, blended classroom setting in order to assess students’ understanding of statistics concepts following synchronous and asynchronous online, videobased discussion sessions. Social presence theory predicts that the immediacy of synchronous interactions should improve social presence, thereby increasing student engagement and performance. Data was collected from 105 students with high-incidence disabilities using a mixed methods experimental design with a within subjects quantitative component. Our findings indicate that while students with disabilities expressed preference for synchronous discussions, including self-reported greater engagement and self-reported improved comprehension, their performance on assessments of conceptual understanding was slightly better following asynchronous discussions. Implications for these disparities between preference and performance are discussed.

COMPARING SYNCHRONOUS AND ASYNCHRONOUS DISCUSSIONS

3

1. Introduction While the pervasiveness of online learning has grown considerably in recent years, its overall effectiveness as an educational medium has still not been fully explored in at-risk populations (Means, Toyama, Murphy, Bakia, & Jones, 2009; Phan, 2012; Nguyen, 2015). Online learning offers the potential for customization and flexibility to meet the unique needs of students with disabilities, however, those same students are potentially uniquely vulnerable to the shortcomings of online educational environments (Basham, Stahl, Ortiz, Rice, & Smith, 2015). Indeed, the Center on Online Learning and Students with Disabilities has identified nine key areas of concern regarding the participation of students with disabilities in online environments, including inconsistent policies, lack of monitoring and accountability, and failure to provide access, both physical and cognitive to students with disabilities (Deshler, East, Rose, & Greer, 2012). With this disparate research in mind, it is evident that the specifics of how students with disabilities are best engaged in online environments is still largely unexplored. There are many complexities surrounding the labels used to describe this population. Labels need to encompass both the difficulties that these students face, as well as their complex heterogeneity. For the purposes and context of this study, the term “students with high-incidence disabilities” is an appropriate compromise. This term includes those students with the most common diagnoses receiving accommodations in educational settings under the Americans with Disabilities Act (ADA, 1990). Students with high-incidence disabilities includes primarily those facing significant challenges with language processing, attention, and/or social interactions often with a diagnosis of a Learning Disability (LD), Attention Deficit Hyperactivity Disorder (ADHD), and Autism Spectrum Disorder (ASD) (Leko, Brownell, & Lauterbach, 2010). These students comprise the largest segment of individuals with disabilities, constituting more than

COMPARING SYNCHRONOUS AND ASYNCHRONOUS DISCUSSIONS

4

10% of the entire population of students in the United States (U.S. Department of Education, National Center for Education Statistics, 2018). This population's challenges primarily manifest in educational settings and overlap in terms of academic, behavioral, and social needs (Friend & Bursuck, 2018). There is little evidence on the effectiveness of online learning for students with highincidence disabilities, and there is little research on its impact on student performance (Vasquez & Straub, 2016). While some work indicates that productive use of digital tools can potentially level the playing field for students with disabilities in some online learning contexts (Lancaster, Schumaker, Lancaster, & Deshler, 2009), other evidence suggests that students with disabilities tend to perform more poorly than neurotypical students in distance education settings (Richardson, 2015). Online learning tends to be particularly challenging for students with disabilities because of its high demands on executive function skills, including goal directed behavior, response inhibition, and problem solving (Miyake & Friedman, 2012). Ineffective executive function skills and behaviors are a hallmark of the aforementioned high-incidence disabilities (Barkley, 2012; Denckla, 2007; Pellicano, 2012). These skills are critical for comprehension, task persistence, and successful academic performance. Indeed, research suggests deficits in executive function skills are significant predictors for lack of persistence in online courses (Lee, Choi, & Kim, 2013). In particular, students often need a great deal of self-regulation and intrinsic motivation to be successful in online courses (Bol & Garner, 2011; Cho & Shen, 2013). Therefore, students with high-incidence disabilities need more intensive support in this area in order to be successful in online learning.

COMPARING SYNCHRONOUS AND ASYNCHRONOUS DISCUSSIONS

5

With this in mind, several mechanisms can be used to provide executive function supports to students with disabilities online. For example, there is active research in the field on the use of cognitive skills training programs (Jaeggi, Buschkuehl, Jonides, & Shah, 2011; Kane & Engle, 2002) as well as the provision of executive function coaching support (DuPaul et al., 2017; Parker & Boutelle, 2009), but these methods have not been explored in online settings and are implemented outside of the course itself. A different way of providing executive function support that can be integrated directly into a course is through the use of certain types of online social interactions. Ybarra and Winkielman (2012, p. 6) suggest that, “EF [Executive Function] benefits are selectively conferred by certain on-line dynamic social interactions, which require participants to mentally engage with another person and actively construct a model of their mind.” In particular, they note that these benefits are observed in real time dynamic social interactions with others rather than static threaded interactions. This study therefore aims to explore whether incorporating real time dynamic online social interactions, referred to in this paper as synchronous interactions, leads to increased engagement and improved conceptual understanding for students with high-incidence disabilities in a statistics course over similar online interactions that are not in real time, referred to here as asynchronous interactions. Some evidence suggests that students with high-incidence disabilities express a desire for greater synchronous interactions in online courses. For example, a needs assessment conducted on Universal Design and online learning found the lack of face-to-face interactions to be a key impediment to online learning for students with LD and ADHD (Madaus, Banerjee, McKeown, & Gelbar, 2011). These supports can be provided through instructor-mediated synchronous discussions that allow for virtual face-to-face interactions with the immediacy and instructor

COMPARING SYNCHRONOUS AND ASYNCHRONOUS DISCUSSIONS

6

presence described by Baker (2010). They also provide opportunities for face-to-face time with peers, which the Means et al. (2013) meta-analysis found to significantly improve student outcomes for adult learners. Research on the effectiveness of synchronous online interactions for students with disabilities is scant, but many practitioners perceive it as critical for the online success of their students. Vasquez and Slocum (2012) found that synchronous online tutoring in reading was effective in improving student reading skills. Furthermore, Archambault et al. (2010) looking at how K-12 online programs support at-risk students, reported that synchronous learning activities and web-conferencing are commonly used as learning scaffolds. The benefits of synchronous interactions can be understood through the framework of social presence theory as discussed in the next section. The goal of the research described in this paper was to experimentally assess the impact of synchronous interactions on students’ conceptual understanding and to assess whether the benefits predicted by social presence theory are realized for students with high-incidence disabilities in online learning. 1.1 Social Presence Theory One of the main theories guiding the development of effective online learning interaction over the past decade is Social Presence Theory. First introduced by Short, Williams, and Christie (1976) well before the advent of the Internet, social presence was described by them as the, “degree of salience of the other person in the interaction and the consequent salience of the interpersonal relationships”. Put more simply, social presence describes the extent to which we view other individuals in a technology mediated interaction as real people, and as a result the extent to which we view our interactions with them as authentic social interactions. The degree

COMPARING SYNCHRONOUS AND ASYNCHRONOUS DISCUSSIONS

7

to which the online facsimile of the classroom approximates the real experience can be seen as a function of the type of online system used. That initial conceptualization of social presence has evolved into a theoretical framework over the past few decades. Gunawardena and Zittle (1997) defined two key elements of social presence that they termed immediacy and intimacy. Under their conceptualization, immediacy referred to the latency between responses in a technology mediated interaction. They theorized that the longer it took an individual to receive a response, the less social presence they perceived in that interaction and the less engaged they were likely to be. Intimacy was viewed as the quality of the interpersonal relationship between the individuals in the technology mediated interaction. They saw things like humor or the sharing of personal experiences as ways to promote a personal connection and as a result increase one’s social presence and the subsequent quality of the social interaction. Evidence indicates that increasing the immediacy and intimacy of responses from an instructor in online courses increases students’ self-reported satisfaction with interactions in those courses (Gunawardena, 1995; Gunawardena and Zittle, 1997). In 1999, Rourke, Anderson, Garrison, and Archer, articulated a framework for measuring the level of social presence in online interactions (Rourke, Anderson, Garrison, and Archer, 1999; Rourke, Anderson, Garrison, & Archer, 2001). They operationally defined 12 indicators of social presence that fall within three categories of response types: affective responses, interactive responses, and cohesive responses. Affective responses tended to involve reference to emotional state, humor, or personal information. Interactive responses typically involved some sort of response or comment to another individual’s statement. And finally, cohesive responses tended to include purely social statements or reference to the group as a cohesive unit. This work was further extended by Whiteside (2007) who defined five elements of social presence: affective

COMPARING SYNCHRONOUS AND ASYNCHRONOUS DISCUSSIONS

8

investment, cohesiveness, interaction level, knowledge and experience, and instructor involvement. While researchers have sought to refine how social presence should be defined and measured, the core elements of interaction immediacy and intimacy introduced by Gunawardena and Zittle (1997) remain at the center of all subsequent conceptualizations. Research has sought to empirically assess the impact of social presence and has generally found a positive relationship between increased social presence and student satisfaction with online learning (Tu & McIssac, 2002; Richardson & Swan, 2003; Swan & Shih, 2005; Wei, Chen, & Kinshuk, 2012; Richardson, Maeda, Lv, & Caskurlu, 2017). However, most research in this area has relied on observational research and self-reported measures with a neurotypical student population in assessing the impact of social presence, rather than measures of learning outcomes. While definitions of social presence have varied, the immediacy of participant interaction is consistently seen as a key component of social presence. This study therefore experimentally manipulated social presence by manipulating the immediacy of interactions and measured its impact on the learning outcomes of students with disabilities. Furthermore, for the purposes of this study, high social presence was operationally defined as interactions one encounters in an online, whole group, real time classroom, such as those found in our synchronous discussion condition. Social presence was manipulated in this study by altering the immediacy of students’ social interactions in online discussions. This was done by conducting video-based synchronous and asynchronous discussions. Synchronous interactions, all other things being equal, are predicted to have greater social presence than asynchronous interactions under any of the existing social presence frameworks. Synchronous interactions are also generally used in

COMPARING SYNCHRONOUS AND ASYNCHRONOUS DISCUSSIONS

9

education as a means to make online learning more accessible to students with disabilities despite the lack of evidence supporting this stance. To assess the impact of social presence on the development of conceptual understanding of statistics topics among students with disabilities, we collected both self-report and performance measures from students who participated in synchronous and asynchronous online discussions of those concepts. Below we report on this NSF grant funded work [BLINDED]. 2. Methods 2.1 Participants Data in this study were collected from college students taking an introductory level course in statistics at a college that exclusively serves students with LD, ADHD, and Autism. In all, 105 students participated in this study from eight sections of the same course across four semesters. Participants volunteered to take part in this study by signing informed consent forms and were offered $100 gift cards for participating in the data collection sessions outlined in the procedure section below. Of those participants, 7-10 students participated in each of the four qualitative focus group conducted at the end of each semester, and 5 participated in 1-on-1 interviews. Demographically, 62 participants identified as male and 43 identified as female. Racially, 78 identified as White, 4 identified as Hispanic/Latino, 4 as African American, 1 as Asian American, 3 as Mixed Racial, and 15 as Other/No Response. In terms of disability based on formal diagnostic documentation, 19 students had a diagnosed learning disability, 23 had an ADHD diagnosis, 12 an Autism diagnosis, 26 had an LD and an ADHD diagnosis, 6 had an LD and Autism diagnosis, 9 had an ADHD and Autism diagnosis, and 2 had all three. The remaining eight students had disabilities that were outside of these three diagnostic categories.

COMPARING SYNCHRONOUS AND ASYNCHRONOUS DISCUSSIONS

10

2.2 Materials To assess students’ understanding of the discussed statistics concepts, the research team selected items from two existing instruments designed specifically to assess conceptual understanding in statistics: The Statistics Concept Inventory (SCI) (Allen, 2006), and the Levels Of Conceptual Understanding in Statistics (LOCUS) Project (Jacobbe et al., 2014). We avoided assessments with significant computational demands, as we did not want to confound computational fluency with conceptual understanding. To create topic specific assessments, we selected two items from the SCI and four items from LOCUS for each of the following topic areas: sampling, central tendency, sampling distribution, confidence interval, significance testing, and correlation. These items were selected by two content area experts and only items independently identified by both experts were considered for the assessments. The resultant 36 items were then reviewed by a panel of three disability education experts, and three statistics education experts to ensure that the items were appropriate for the topic and the participant pool. To allow for consistent pre versus post assessment of each topic, parallel items were generated for each of the 36 items. These parallel items assessed the same concepts and used the same structure as the items selected from the SCI and LOCUS but differed in surface features such as context and values. This resulted in another 36 items for a total of 72 items split into two sets of 6 parallel tests. All items were reviewed by the advisory panel for content validity and accessibility. Each test was composed of 3 items selected from the SCI and LOCUS and 3 parallel items, and the tests were counterbalanced between participants across sections and semesters (see [BLINDED CITATION] for additional information on the assessment items). The item development and experimental design were intended to reduce potential differences in

COMPARING SYNCHRONOUS AND ASYNCHRONOUS DISCUSSIONS

11

difficulty between the two tests, furthermore, the counterbalancing of the items ensures that all items appear equally as pre and post-test items. The tool selected for online synchronous discussions was BigBlueButton for its ease of use, recording capability and customizability. BigBlueButton is a video based, online classroom and discussion tool that integrates with the Canvas LMS. All the interactions in BigBlueButton happen in real time. The tool selected for asynchronous discussions was VoiceThread, again for its ease of use, variety of modalities for participation and relative similarity to BigBlueButton. Here students participate in their own time in a threaded video-based discussion, so long as the teacher has a discussion thread open. Interactions in VoiceThread do not happen in real time. 2.3 Design This study used a within subjects, repeated measures, mixed methods design to assess the impact of social presence on the conceptual understanding of students with high-incidence disabilities. The dependent variable for this study was student performance on the conceptual assessments described in the Materials section. Students completed a series of pre/post tests for each of six topics areas: Sampling, Central Tendency, Sampling Distributions, Confidence Intervals, Significance Testing, and Correlations. Quantitative analysis was based on gains in assessment scores from pre- to post-tests following synchronous versus asynchronous discussions. To support the interpretation of the quantitative data, several sources of qualitative data were collected including interviews and student focus groups. 2.4 Procedure Data collection occurred as part of a residential, introductory, college level statistics course across four semesters over two years. All courses were taught by the same instructor and followed the same structure and content. Data in each semester were collected from two sections,

COMPARING SYNCHRONOUS AND ASYNCHRONOUS DISCUSSIONS

12

taught one after the other. Students were asked to complete an informed consent form and a demographic survey at the beginning of each semester. Each participant was then asked to participate in 12 data collection periods across each semester as part of their coursework, a pre and a post assessment for each of the six aforementioned topics. All six discussions were conducted online with each section engaging in three synchronous discussions, alternating with three asynchronous discussions. For each topic, one section engaged in a synchronous discussion and one in an asynchronous discussion with the format and the assessment versions counterbalanced across sections and semesters (see Figure 1). Students were invited to participate in focus groups and interviews during the end of the semester to share feedback regarding the synchronous and asynchronous discussions. In order to participate in the synchronous discussion, students would log in from their own computers to an online classroom in BigBlueButton. Students were required to have both their webcams and microphones active in order to maximize the means of interaction with the instructor and their peers. Students were asked to wear headphones and participate in a quiet environment to minimize potential environmental distractions. BigBlueButoon also allowed text communication through a chat dialogue that was present concomitantly with the spoken dialogue. In the asynchronous discussion format, students would login to a threaded video-based discussion in VoiceThread. They were given a 24-hour period within which to participate in the asynchronous discussion. They were asked to respond to a video prompt presented by the teacher and to respond to their classmates and further participate in discussion. Students were able to participate via video, audio, and/or text in VoiceThread, the same modalities available to them in BigBlueButton. After the 24-hour period, the discussion for that section was closed.

COMPARING SYNCHRONOUS AND ASYNCHRONOUS DISCUSSIONS

13

2.5 Analysis The analysis of the data from this study included both quantitative and qualitative elements. For the quantitative analysis, a model comparison approach was used due to the complex structure of the data. The data was modeled using Generalized Linear Mixed Effects Models (GLMEM) (McCulloch & Neuhaus, 2001). GLMEMs are appropriate because of the nested nature of the data and because they are robust to several of the potential issues in the data including high between participant variability, item variability, and differential attrition (Matuschek, Kliegl, Vasishth, Baayen, & Bates, 2017). The modeling and analysis was performed in the R statistical software environment (R Core Team, 2013) using the LME4 package (Bates, Maechler, Bolker, & Walker, 2015). A theory driven, model building process was used to build progressively more complex models using the available predictors while maximizing parsimony of their relationships. The models were compared for best fit while accounting for model complexity using the Akaike Information Criterion (AIC) and Log Likelihood scores. A model was considered more parsimonious if its log likelihood was significantly higher (based on a chi-square test) and if its AIC score (Akaike, 1974) was at least 4 points lower (Burnham & Anderson, 2002). The GLMEM models use a binary item response accuracy (correct vs. incorrect) from the assessments described under the materials section as the dependent variable. Models for comparison were built using 6 independent variables. Three binary variables that coded for the presence of a documented diagnosis of each of the following: LD, ADHD, or Autism while allowing for co-occurrence of overlapping diagnostic categories. In addition, a binary variable coded for discussion type (synchronous vs. asynchronous), a binary variable coded for test (pre vs. post), and a categorical variable coded for topic (sampling, central tendency, sampling

COMPARING SYNCHRONOUS AND ASYNCHRONOUS DISCUSSIONS

14

distribution, confidence intervals, significance testing, and correlation). A baseline model (Model 1) was created using only participants and test items as random variables. Model fit was assessed by comparing AIC scores and Log Likelihood scores from models incorporating the aforementioned independent variables to this baseline model. An informal thematic analysis was used to analyze the various qualitative modalities with an eye towards providing elaborative information to help interpret the quantitative findings. Qualitative analysis was conducted using focus groups and interviews at the end of each semester. The study sought to gather additional voluntary information regarding participants’ self-reported level of engagement, stated format preference, and perceived conceptual understanding of the topics following synchronous and asynchronous discussions. 3. Results 3.1 Quantitative results Models were created based on the aforementioned six variables which included three for Disability status (ADHD, LD, Autism), the Discussion type (Synchronous or Asynchronous), pre or post test, and Statistics topic. Two-way interactions of all these variables were also considered. In assessing the impact of the independent variables, the analysis focused on models in which the variable was a statistically significant predictor and on models that significantly improved fit over the baseline model. The use of both criteria is necessary because the impact of independent variables crossed with random variables is only evident by looking for significant improvements in model fit. The three diagnostic binary variables of LD, ADHD, and Autism were not found to significantly contribute to any of the models either independently or in interaction with any other

COMPARING SYNCHRONOUS AND ASYNCHRONOUS DISCUSSIONS

15

variables, nor did they significantly improve model fit. Therefore, these three variables are not included in the models presented below. 3.1.1 Model 1 The baseline model (Model 1) includes only the random variables of participant and item. This model serves as a floor in terms of model fit and a benchmark against which to assess the fit of models incorporating the independent variables. Equations for the five models included in this section can be seen in Table 1. The intercept for the baseline model indicates that the log odds of answering correctly across the entire sample was -0.44 which implies that participants had a higher likelihood of answering the items incorrectly. 3.1.2 Model 2 This model adds the pre/post test variable to the baseline model allowing comparisons to assess for gains in conceptual understanding. Model 2 also includes a pre/post by items component allowing an individual pre/post line of best fit for each assessment item. This allows the model to better fit the data as can be seen in the significant gain in log likelihood, χ2(1) = 12.2, p < 0.001, and AIC score (see Table 2). The fixed component indicates that the average gain from pre to post is modest, this is primarily because performance on the assessments is relatively low and variability is high (see Table 3), also note that each data collection point in Table 3 is gathered from half the sample (i.e. 1 group) as groups alternate between conditions across the semester. However, the significant increase in model fit with the pre/post by items component indicates that student performance gains from pre to post are consistent for individual items and that there is high variability in how students performed on the various assessment items and this variability is being accounted for in our model.

COMPARING SYNCHRONOUS AND ASYNCHRONOUS DISCUSSIONS

16

3.1.3 Model 3 This model adds the variable of topic to the baseline model, this variable is categorical and codes for each of the six topics in this study. As can be seen in Table 3, participant performance differed significantly across statistics topics as one would expect. Model 3 significantly improves on the baseline model with a significant gain in log likelihood, χ2(1) = 24.4, p < 0.0001, and AIC (see Table 2). Central tendency is the benchmark topic with statistically significant coefficients for other topics indicating significant performance differences with respect to that benchmark. Looking at the fixed effect coefficient estimates in Table 2, the data indicate that participants performed similarly in the Correlation and Sampling topics as they did in Central Tendency. Participants tended to do significantly worse in the Confidence Interval, Sampling Distribution, and Significance Testing topics. This is consistent with typical performance on assessments of these topics. 3.1.4 Model 4 This model adds the discussion format variable to Model 2 to allow for an analysis of gains in student conceptual understanding across synchronous and asynchronous discussions. This results in an improvement in model fit that is significant based on gains in log likelihood, χ2(1) = 5.0, p < 0.05, but fails to meet our threshold of a 4 point gain in AIC score. Note here the comparison is with respect to Model 2, a model that already significantly improves on the baseline Model 1. The model comparison results indicate that while model fit is significantly improved by adding the synchronous component, the improvement is modest and may not justify the additional model complexity. Looking at the fixed effect coefficients (see Figure 2), the data indicate that participants typically saw a modest gain from pre to post in the asynchronous condition but saw no gain in the synchronous condition. The interaction coefficient falls just shy

COMPARING SYNCHRONOUS AND ASYNCHRONOUS DISCUSSIONS

17

of the 0.05 level of significance, this combined with the modest gain in AIC indicates that the impact of format is fairly modest. 3.1.5 Model 5 Finally Model 5 is a model that combines all the factors that were found to significantly improve model fit. Specifically, this model incorporates Model 3 and Model 4 and significantly improves over both with a significant increase in AIC and in log likelihood, χ2(1) = 18.2, p < 0.0001 versus Model 3, and χ2(1) = 25.4, p < 0.0001 versus Model 4. The fixed effect coefficients remain relatively unchanged compared to their values in Model 3 and Model 4 indicating that the contributions of the variables are fairly independent. 3.2 Qualitative results 3.2.1 Focus Groups and Interviews Data was collected from four focus groups, one conducted at the end of each semester of data collection. If a student was unable to attend a focus group and wanted to share their feedback with the research team, they were invited to participate in a 1-on-1 interview. Participation in focus groups and interviews was completely voluntary. Participants were queried on three broad areas: (1) preference for synchronous vs. asynchronous discussions; (2) social dynamics and connectedness; and (3) technical aspects of participation in the two formats. These sessions were kept informal to allow students to participate in whatever way was most comfortable for them. Students participated at their own pace and not every student was required to respond to every question. The qualitative component of this study was generally conceived to provide perspective on the quantitative results and were not intended to be prescriptive. As such, the informal thematic analysis was based on the impressions of the

COMPARING SYNCHRONOUS AND ASYNCHRONOUS DISCUSSIONS

18

researcher conducting each session and not of strict coding criteria. Broad themes from these analyses are summarized below. 3.2.2 Synchronous vs. Asynchronous Preference Most participants expressed a preference for the synchronous format over the asynchronous format for online discussions. Participants’ stated reasons for this preference include the ability to ask the instructor clarifying questions and the ability to interact directly with the instructor and their peers. While participants were certainly able to ask clarifying questions in both formats, the response in the synchronous condition was immediate. Some participants spoke of a sense of increased accountability in the synchronous discussions because they occurred at a specific time each week and non-participation would be immediately evident to the instructor and to their peers. Participants who felt that sense of increased accountability stated that it helped them remain more organized, motivated, and on task. Individual disparities in student motivation level may have impacted student preference for synchronization type. A minority of participants, however, expressed a preference for the asynchronous sessions. One of the main reasons some participants did not prefer synchronous discussions was distractibility and by contrast they found the asynchronous environment to be less distracting. For students who have social anxiety or are prone to distractibility in social environments, the asynchronous condition may provide a means of avoiding these confounds. Within the asynchronous discussions, participants had control over which video feeds they watched and had an opportunity to prepare their responses at their own pace. Several students expressed difficulty in maintaining focus when they had access to live visual feeds of a number of their peers, particularly if those peers were participating in the discussion from locations with a lot of background noise or visual stimulation. In addition,

COMPARING SYNCHRONOUS AND ASYNCHRONOUS DISCUSSIONS

19

technical issues tended to be more common in the synchronous sessions and their impact was more prominent. Immediate resolution of technical issues was necessary in the synchronous condition to allow a student’s participation in the discussion, whereas in the asynchronous sessions, students could still participate fully even if resolution of the technical issues was not immediate. Finally, some students expressed that synchronous discussions were too fast for them to be able to follow the conversation, whereas in the asynchronous session they had more control over the pacing of the discussion. 3.2.3 Student Perceptions of Connectedness Participants were asked to describe factors that they perceived impacted their active engagement with classroom discussions in both the synchronous and asynchronous conditions. Discussions were often structured around problems posed by the instructor to stimulate discussion. Some participants expressed difficulty in understanding the problems and chose not to engage with the content as they expressed skepticism in their ability to deepen their comprehension of the topic. Some participants explicitly expressed a lack of desire to understand the concepts and wanted the instructor to merely provide them with the procedure needed to compute a response. “Give me the procedure and how to apply it” was the thought articulated by one of the students. Many participants made comparisons between the two online discussion formats and traditional classroom discussions even though those comparisons were outside the scope of this study. Participants almost universally expressed preference for traditional face-to-face discussions although some speculated that it may be because that is the format they are most familiar with. Participants found synchronous discussions to be closer to traditional classroom discussions and therefore close to the format they were more familiar and comfortable with.

COMPARING SYNCHRONOUS AND ASYNCHRONOUS DISCUSSIONS

20

Most felt more included and perceived a greater sense of belonging to a peer group during synchronous discussions. By and large, even the minority of participants who expressed a preference for the asynchronous format felt a stronger connection and engagement with the instructor and their peers in the synchronous discussions. 3.2.4 Technical Considerations Participants were asked to comment on the technical aspects and challenges of the platforms used to conduct the synchronous and asynchronous discussions. The two software platforms were selected to offer students ease of use and to allow interaction using text, video, and audio. Overall, students had very little tolerance for technical issues and would disengage from the content in the face of any technical challenge. Students felt that at times dealing with the technology hindered their ability to fully participate in the discussions. Technical difficulties may have significantly impacted the ability of some students to engage with the discussions and may therefore have impacted students’ perceptions of social presence and learning outcomes. While technical difficulties had similar prevalence across the two discussion formats, they may have had greater impact on the synchronous discussions given the time sensitive nature of that format. There were also differences in how students used the synchronous and asynchronous platforms. While many students would join synchronous discussions by video, far fewer chose to do so in the asynchronous discussions. In fact, most students used text in the asynchronous format even when they had learning disabilities. Students reported that they found it easier to type a response rather than record a video. Some also felt that the persistence of the video in the asynchronous format made them far more self-conscious than in the synchronous format. Students who did use the video format in the asynchronous platform felt pressure to think

COMPARING SYNCHRONOUS AND ASYNCHRONOUS DISCUSSIONS

21

through their response and re-record their video if they were unhappy with the initial recording. Their responses in the synchronous discussion were more casual and spontaneous. 4. Discussion The aim of this study was to assess the impact of social presence on the conceptual understanding of students with disabilities during online discussions. Social presence was manipulated by comparing participants’ performance following synchronous discussions versus parallel asynchronous discussions. Based on social presence theory, synchronous discussions were predicted to increase social presence as they allowed for more immediate and intimate communication. This prediction was supported by the qualitative data, which indicated that a majority of participants preferred the synchronous format, felt more engaged and motivated during synchronous discussions, and perceived improved conceptual understanding following synchronous discussions. These findings are in line with the existing literature, primarily based on self-report, involving neurotypical student populations (Richardson, Maeda, Lv, & Caskurlu, 2017). Paradoxically, the quantitative data indicated no improvement in conceptual understanding for students participating in the synchronous discussion. The model comparisons provide several key insights into the data. The results for Model 2 indicate that gains from pre to post were consistent for individual items but there was no consistent significant gain from pre to post across all items, participants, and conditions. Furthermore, the statistically significant negative intercept indicates that in general, students were more likely to respond incorrectly than correctly on the assessments. This is not uncommon for students with high-incidence disabilities who tend to perform poorly on assessments and to exhibit high variability in their performance (Meyen, Poggio, Seok, & Smith, 2006). Model 3 indicates that the tendency to respond

COMPARING SYNCHRONOUS AND ASYNCHRONOUS DISCUSSIONS

22

incorrectly was mainly driven by poor performance in the more difficult topics of Confidence Intervals, Sampling Distributions, and Significance Testing. Most relevant to the hypothesis however is Model 4. As can be seen in Figure 2, students’ odds of responding correctly to items, once individual and item specific variability have been accounted for, tended to increase following asynchronous but not synchronous discussions. There are several reasons worth considering for understanding this outcome. First, it is worth noting that, while performance following the asynchronous discussions generally improved, this effect met only some and not all our criteria for statistical significance. Model 4’s log likelihood was statistically superior to that of Model 2 but the AIC gain did not meet the threshold of 4 point established a priori. Furthermore, the interaction of format with pre/post test was just shy of the level of statistical significance. Therefore, one can be confident that the data do not support the superiority of the synchronous format, but additional research is needed to determine whether the superiority of the asynchronous format in this context and with this population is robust. It is also worth cautioning against drawing conclusions from these results for other student populations. The sample of this study consisted exclusively of students with highincidence disabilities. Therefore, one cannot assume that similar findings would be observed with a neurotypical population. These findings however, highlight the need to look carefully at learning outcomes and not merely self-reported measures of engagement and performance when evaluating the impact of social presence on the conceptual understanding of student populations. Several limitations of this study may also explain the unexpected findings. One possibility is that there was not a higher degree of social presence in the synchronous condition in the context of our study. While the qualitative data provide some evidence that students felt

COMPARING SYNCHRONOUS AND ASYNCHRONOUS DISCUSSIONS

23

more engaged and connected with their peers during the synchronous discussions, thereby suggesting that the manipulation was successful, incorporating a sample wide survey of social presence could provide stronger evidence of successful manipulation. Future work should incorporate other means of assessing students’ perceptions of social presence to ensure that the current finding is not an artifact of the qualitative approach used in this study. These results have several implications for researchers studying social presence theory. First and foremost is highlighting the need for objective measures of student performance. A majority of research on social presence theory has relied primarily on student self-report (Richardson, Maeda, Lv, & Caskurlu, 2017), and the findings of this study indicate that student learning outcomes may not always align with student perceptions. Researchers should be particularly careful in this regard when looking at student subpopulations. Researchers should also be clear on the types of measures used to assess student learning outcomes. This study looked at assessments of conceptual understanding and it is possible that the performance discrepancy observed in this study is unique to these forms of assessment. The data indicate that diagnostic category did not impact student performance in either synchronous, or asynchronous conditions. This finding is not unique and is in line with other work cited in the introduction. This indicates a significant amount of overlap in functional limitations across categories of students with high-incidence disabilities (Friend & Bursuck, 2018). While there are likely to be multiple reasons leading to poor performance in the synchronous condition, they lead to an overall similar pattern of performance across all categories of high-incidence disabilities observed in this study. 4.1 Conclusions

COMPARING SYNCHRONOUS AND ASYNCHRONOUS DISCUSSIONS

24

The outcomes of this study point to the complex nature of online interactions and the interplay between student engagement and information processing. Students’ self-perceptions based on their comments during focus groups and interviews were in line with the study’s prediction of higher social presence during the synchronous discussions. A majority of students self-reported greater engagement in and preference for the synchronous discussions. However, the quantitative analysis revealed that performance following asynchronous discussions tended to be superior for this student population in this context. This could be due to a number of factors. The experimental manipulation may not have provided significantly greater social presence in the synchronous condition for this population. Furthermore, students may not be appropriately evaluating which condition they are actually performing better in and simply preferring the synchronous condition for superficial reasons. It is also possible that the asynchronous discussions had greater social presence due to other relevant factors such as intimacy and efficiency, or due to the fact that the discussions were confined to a 24-hour period during which all students were expected to be active. Finally, there may be other factors unrelated to social presence impeding this population's learning during synchronous discussions. The authors intend to perform a more thorough content analysis of the discussions in a future publication to better understand the impact of the discussion formats on student interactions. Synchronous discussions require the processing of fast back and forth conversations. This processing is particularly challenging for students with learning disabilities, especially when discussing topics they are not very familiar with and ones that involve a lot of new, specialized terminology (Baxter, Woodward, & Olson, 2001). Synchronous discussions also involve a greater demand on social dynamics, an area that is especially challenging for some individuals

COMPARING SYNCHRONOUS AND ASYNCHRONOUS DISCUSSIONS

25

who have social pragmatic difficulties such as autistic learners (Harrower & Dunlap, 2001). Finally, synchronous discussions tend to place greater demands on students’ attention as they often need to filter out multiple video and audio streams while attending to the conversation. For students with disabilities and others who may find these demands particularly challenging, the benefit gained as a result of the increased engagement due to higher social presence may be outweighed by the additional processing demands of the discussion format. More research is needed to determine whether these additional potential cognitive loads are the reason for the discrepancy between the findings of this study and the prediction of social presence theory.

COMPARING SYNCHRONOUS AND ASYNCHRONOUS DISCUSSIONS

26

References Akaike, H. (1974). A new look at the statistical model identification. IEEE Transactions on Automatic Control, 19(6), 716–723. Allen, K. (2006). The Statistics Concept Inventory: Development and Analysis of a Cognitive Assessment Instrument in Statistics (SSRN Scholarly Paper No. ID 2130143). Retrieved from Social Science Research Network website: http://papers.ssrn.com/abstract=2130143 Americans With Disabilities Act of 1990, Pub. L. No. 101-336, 104 Stat. 328 (1990). Archambault, L., Diamond, D., Brown, R., Cavanaugh, C., Coffey, M., Foures-Aalbu, D., … Zygouris-Coe, V. (2010). Research committee issues brief: An exploration of at-risk learners and online education. International Association for K-12 Online Learning. Retrieved from https://eric.ed.gov/?id=ED509620 Baker, C. (2010). The impact of instructor immediacy and presence for online student affective learning, cognition, and motivation. Journal of Educators Online, 7(1), n1. Barkley, R. A. (2012). Executive functions: What they are, how they work, and why they evolved. Retrieved from https://books.google.com/books?hl=en&lr=&id=7c39F6qD38IC&oi=fnd&pg=PP2&dq =barkley+2012&ots=ayXdhxInZu&sig=hT2YoBr2pL_K5tMtyo6IakfE8bs Basham, J. D., Stahl, W., Ortiz, K. R., Rice, M. F., & Smith, S. J. (2015). Equity matters: Digital and online learning for students with disabilities. University of Kansas Center on Online Learning and Students with Disabilities.

COMPARING SYNCHRONOUS AND ASYNCHRONOUS DISCUSSIONS

27

Bates, D., Mächler, M., Bolker, B., & Walker, S. (2015). Fitting linear mixed-effects models using lme4. Journal of Statistical Software, 67(1), 1–48. https://doi.org/10.18637/jss.v067.i01 Baxter, J. A., Woodward, J., & Olson, D. (2001). Effects of reform-based mathematics instruction on low achievers in five third-grade classrooms. The Elementary School Journal, 101(5), 529–547. Bol, L., & Garner, J. K. (2011). Challenges in supporting self-regulation in distance education environments. Journal of Computing in Higher Education, 23(2), 104–123. https://doi.org/10.1007/s12528-011-9046-7 Burnham, K. P., & Anderson, D. R. (2002). Information and likelihood theory: A basis for model selection and inference. Model Selection and Multimodel Inference: A Practical Information-Theoretic Approach, 49–97. Cho, M.-H., & Shen, D. (2013). Self-regulation in online learning. Distance Education, 34(3), 290–301. https://doi.org/10.1080/01587919.2013.835770 [BLINDED CITATION] Denckla, M. B. (2007). Executive function: Binding together the definitions of attentiondeficit/hyperactivity disorder and learning disabilities. In Executive Function in Education: From Theory to Practice. (pp. 5–18). New York: Guilford Press. Deshler, D., Rose, D., East, W., & Greer, D. (2012). Concerning Participation. Retrieved November 11, 2016, from http://centerononlinelearning.org/openletter/ DuPaul, G. J., Dahlstrom-Hakki, I., Gormley, M. J., Fu, Q., Pinho, T. D., & Banerjee, M. (2017). College students with ADHD and LD: Effects of support services on academic performance. Learning Disabilities Research & Practice, 32(4), 246–256.

COMPARING SYNCHRONOUS AND ASYNCHRONOUS DISCUSSIONS

28

Friend, M., & Bursuck, W. D. (2018). Including students with special needs. Allyn & Bacon Boston, MA, USA. Gunawardena, C. N. (1995). Social presence theory and implications for interaction and collaborative learning in computer conferences. International Journal of Educational Telecommunications, 1(2), 147–166. Gunawardena, C. N., & Zittle, F. J. (1997). Social presence as a predictor of satisfaction within a computer-mediated conferencing environment. American Journal of Distance Education, 11(3), 8–26. Harrower, J. K., & Dunlap, G. (2001). Including children with autism in general education classrooms: A review of effective strategies. Behavior Modification, 25(5), 762–784. Jacobbe, T., Case, C., Whitaker, D., & Foti, S. (2014). Establishing the content validity of the LOCUS assessment through evidence centered design. Proceedings of the 9th International Conference on Teaching Statistics. Jaeggi, S. M., Buschkuehl, M., Jonides, J., & Shah, P. (2011). Short-and long-term benefits of cognitive training. Proceedings of the National Academy of Sciences, 108(25), 10081–10086. Kane, M. J., & Engle, R. W. (2002). The role of prefrontal cortex in working-memory capacity, executive attention, and general fluid intelligence: An individual-differences perspective. Psychonomic Bulletin & Review, 9(4), 637–671. Lancaster, P. E., Schumaker, J. B., Lancaster, S. J. C., & Deshler, D. D. (2009). Effects of a computerized program on use of the Test-Taking Strategy by secondary students with disabilities. Learning Disability Quarterly, 32(3), 165–179. https://doi.org/10.2307/27740366

COMPARING SYNCHRONOUS AND ASYNCHRONOUS DISCUSSIONS

29

Lee, Y., Choi, J., & Kim, T. (2013). Discriminating factors between completers of and dropouts from online learning courses. British Journal of Educational Technology, 44(2), 328–337. https://doi.org/10.1111/j.1467-8535.2012.01306.x Leko, M. M., Brownell, M. T., & Lauterbach, A. A. (2010). High-incidence disabilities. Best Practices for the Inclusive Classroom, 25–50. Madaus, J. W., Banerjee, M., McKeown, K., & Gelbar, N. (2011). Online and blended learning: the advantages and the challenges for students with learning disabilities and attention deficit/hyperactivity disorder. Learning Disabilities: A Multidisciplinary Journal, 17(2), 69–76. Matuschek, H., Kliegl, R., Vasishth, S., Baayen, H., & Bates, D. (2017). Balancing Type I error and power in linear mixed models. Journal of Memory and Language, 94(Supplement C), 305–315. https://doi.org/10.1016/j.jml.2017.01.001 McCulloch, C. E., & Neuhaus, J. M. (2001). Generalized linear mixed models. Retrieved from http://onlinelibrary.wiley.com/doi/10.1002/0470011815.b2a10021/full Means, B., Toyama, Y., Murphy, R., & Baki, M. (2013). The effectiveness of online and blended learning: A meta-analysis of the empirical literature. Teachers College Record, 115(3), 1–47. Means, B., Toyama, Y., Murphy, R., Bakia, M., & Jones, K. (2009). Evaluation of Evidence-Based Practices in Online Learning: A Meta-Analysis and Review of Online Learning Studies. Retrieved from https://eric.ed.gov/?id=ED505824 Meyen, E., Poggio, J., Seok, S., & Smith, S. (2006). Equity for students with high-incidence disabilities in statewide assessments: A technology-based solution. Focus on Exceptional Children, 38(7), 1–8.

COMPARING SYNCHRONOUS AND ASYNCHRONOUS DISCUSSIONS

30

Miyake, A., & Friedman, N. P. (2012). The nature and organization of individual differences in executive functions: Four general conclusions. Current Directions in Psychological Science, 21(1), 8–14. https://doi.org/10.1177/0963721411429458 Nguyen, T. (2015). The effectiveness of online learning: Beyond no significant difference and future horizons. MERLOT Journal of Online Learning and Teaching, 11(2), 309– 319. Parker, D. R., & Boutelle, K. (2009). Executive function coaching for college students with learning disabilities and ADHD: A new approach for fostering self-determination. Learning Disabilities Research & Practice, 24(4), 204–215. Pellicano, E. (2012). The development of executive function in autism. Autism Research and Treatment, 2012. Retrieved from https://www.hindawi.com/journals/aurt/2012/146132/abs/ Phan. (2012). Online Learning Literature Review: Effectiveness and Outcomes. Retrieved from www.reed.edu/registrar/pdfs/online_litrev.pdf R Core Team. (2013). R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing. Richardson, J. C., Maeda, Y., Lv, J., & Caskurlu, S. (2017). Social presence in relation to students’ satisfaction and learning in the online environment: A meta-analysis. Computers in Human Behavior, 71, 402–417. Richardson, J., & Swan, K. (2003). Examining Social Presence in Online Courses in Relation to Students’ Perceived Learning and Satisfaction. Retrieved from https://www.ideals.illinois.edu/handle/2142/18713

COMPARING SYNCHRONOUS AND ASYNCHRONOUS DISCUSSIONS

31

Richardson, J. T. E. (2015). Academic attainment in students with dyslexia in distance education. Dyslexia: An International Journal of Research and Practice, 21(4), 323– 337. https://doi.org/10.1002/dys.1502 Rourke, L., Anderson, T., Garrison, D. R., & Archer, W. (1999). Assessing social presence in asynchronous text-based computer conferencing. Journal of Distance Education, 14(2), 50–71. Rourke, L., Anderson, T., Garrison, D. R., & Archer, W. (2001). Methodological issues in the content analysis of computer conference transcripts. International Journal of Artificial Intelligence in Education (IJAIED), 12, 8–22. Short, J., Williams, E., & Christie, B. (1976). The Social Psychology of Telecommunications. Retrieved from http://www.citeulike.org/group/1662/article/327279 Swan, K., & Shih, L. F. (2005). The development of perceptions of social presence in online course discussions. EdMedia: World Conference on Educational Media and Technology, 836–843. Retrieved from https://www.learntechlib.org/f/20184/ Tu, C.-H., & McIsaac, M. (2002). The relationship of social presence and interaction in online classes. American Journal of Distance Education, 16(3), 131–150. https://doi.org/10.1207/S15389286AJDE1603_2 U.S. Department of Education, National Center for Education Statistics. (2018). Digest of Education Statistics, 2016 (NCES 2017-094), Retrieved from https://nces.ed.gov/fastfacts/display.asp?id=64

COMPARING SYNCHRONOUS AND ASYNCHRONOUS DISCUSSIONS

32

Vasquez, E. I., & Straub, C. (2016). Online writing instruction for children with disabilities: A review of the empirical literature. Reading & Writing Quarterly: Overcoming Learning Difficulties, 32(1), 81–100. https://doi.org/10.1080/10573569.2014.951502 Vasquez, E., & Slocum, T. (2012). The evaluation of synchronous online tutoring for students at-risk of reading failure. Exceptional Children, 78, 221–235. Wei, C.-W., Chen, N.-S., & Kinshuk. (2012). A model for social presence in online classrooms. Educational Technology Research and Development, 60(3), 529–545. https://doi.org/10.1007/s11423-012-9234-9 Whiteside, A. L. (2007). Exploring social presence in communities of practice within a hybrid learning environment: A longitudinal examination of two case studies within the School Technology Leadership graduate -level certificate program (UNIVERSITY OF MINNESOTA). Retrieved from http://gradworks.umi.com/32/75/3275052.html Ybarra, O., & Winkielman, P. (2012). On-line social interactions and executive functions. Towards a Neuroscience of Social Interaction, 188.

Table 1 Description of compared models. Model

Description

Formulae

Model 1

Baseline model

𝑙𝑜𝑔𝑖𝑡(Y() ) = β0 + µ0( + 𝜈0) + ϵ()

Model incorporating a Model 2 pre/post by items component Model 3

Model incorporating topic

𝑙𝑜𝑔𝑖𝑡(Y() ) = β0 + µ0( + ν0) + (β1 + ν1) )(𝑇𝑒𝑠𝑡) + ϵ()

𝑙𝑜𝑔𝑖𝑡(Y() ) = β0 + µ0( + ν0) + β1...5 (𝑇𝑜𝑝𝑖𝑐) + ϵ()

Model incorporating a pre/post by discussion format 𝑙𝑜𝑔𝑖𝑡(Y() ) = β0 + µ0( + ν0) + (β1 + ν1) )(𝑇𝑒𝑠𝑡) Model 4 interaction with a pre/post by

+ β9 (𝐹𝑜𝑟𝑚𝑎𝑡) + β> (𝑇𝑒𝑠𝑡)(𝐹𝑜𝑟𝑚𝑎𝑡) + ϵ()

items component 𝑙𝑜𝑔𝑖𝑡(Y() ) = β0 + µ0( + ν0) + (β1 + ν1) )(𝑇𝑒𝑠𝑡) + β2...6 (𝑇𝑜𝑝𝑖𝑐) Model 5

Combined model

+ β7 (𝐹𝑜𝑟𝑚𝑎𝑡) + β? (𝑇𝑒𝑠𝑡)(𝐹𝑜𝑟𝑚𝑎𝑡) + ϵ()

Y() = Probability of participant i responding correctly to item j β = Fixed component µ = Participant random component ν = Item random component ϵ = Residual Error

Table 2 GLMEM results for compared models. Fixed Effects (Estimate)

Intercept

Model 1

Model 2

Model 3

Model 4

Model 5

-0.44***

-0.40**

-0.05

-0.30*

0.09

Topic: Confidence Intervals

-0.55*

-0.50†

Topic: Correlations

0.002

-0.04

Topic: Sampling

0.02

0.04

Topic: Sampling Distribution

-0.79**

-0.81**

Topic: Significance Testing

-0.99***

-1.00***

Test: Pretest

-0.08

-0.20*

-0.21*

Format: Synchronous

-0.20*

-0.20*

Test: Pretest x Format: Synchronous

0.25†

0.25†

Random Effects (Variance) Model 1

Model 2

Model 3

Model 4

Model 5

Participant (Intercept)

0.55

0.56

0.55

0.56

0.55

Item (Intercept)

0.52

0.68

0.35

0.69

0.50

0.14

0.14

Test: Pretest (Item)

0.14

Model Fit Model 1

Model 2

Model 3

Model 4

Model 5

Log Likelihood

-3243.9

-3237.8

-3231.7

-3235.3

-3222.6

AIC

6493.8

6487.6

6479.4

6486.6

6471.2

Note. β coefficient significance level: † p<0.1, * p<0.05, ** p<0.01, *** p<0.001

Table 3 Descriptive statistics of test scores by discussion format and topic. Pre-Test

Synchronous Topic

Post-test

n

Mean

SD

n

Mean

SD

Sampling

34

3.12

1.27

38

2.92

1.73

Central Tendency

38

2.11

1.39

38

2.21

1.34

Sampling Distribution

40

2.75

1.51

31

2.94

1.65

Confidence Intervals

44

2.84

1.49

47

2.74

1.44

Significance Testing

40

1.93

1.49

36

1.97

1.36

Correlation

33

1.76

1.50

29

1.83

1.63

2.43

1.52

2.46

1.57

Average

Pre-Test

Asynchronous Topic

Post-test

n

Mean

SD

n

Mean

SD

Sampling

35

2.57

1.80

42

3.05

1.75

Central Tendency

34

2.53

1.56

31

2.23

1.76

Sampling Distribution

30

2.80

1.19

27

3.33

1.59

Confidence Intervals

43

3.05

1.63

43

3.37

1.59

Significance Testing

41

1.80

1.25

44

2.18

1.24

Correlation

33

1.64

0.96

30

1.80

0.85

2.40

1.52

2.68

1.61

Average

Topic Sampling Central Tendency Sampling Distribution Confidence Intervals Significance Testing Correlation

Synchronous Discussion Group A Group B Group A Group B Group A Group B

Asynchronous Discussion Group B Group A Group B Group A Group B Group A

Figure 1. Counterbalancing of discussion format across groups and topics for the within-subjects experimental design.

0 Pretest

Posttest

Beta Coefficient

-0.05

-0.1

-0.15

-0.2

-0.25 Synchronous

Asynchronous

Figure 2. Predicted beta coefficient gains (Model 4) from pre- to post-test for the synchronous and asynchronous conditions.

Submission highlights • • • • •

Social presence was manipulated by comparing synchronous and asynchronous discussions Students with disabilities expressed preference for synchronous discussions Greater comprehension was self-reported in the synchronous condition Performance on test of conceptual understanding was better in asynchronous condition Cognitive demands were higher in the synchronous condition