Formative computer-based feedback in the university classroom: Specific concept maps scaffold students' writing

Accepted Manuscript Formative Computer-Based Feedback in the University Classroom: Specific Concept Maps Scaffold Students’ Writing

Andreas Lachner, Christian Burkhart, Matthias Nückles PII:

S0747-5632(17)30157-7

DOI:

10.1016/j.chb.2017.03.008

Reference:

CHB 4833

To appear in:

Computers in Human Behavior

Received Date:

08 December 2016

Revised Date:

01 March 2017

Accepted Date:

02 March 2017

Please cite this article as: Andreas Lachner, Christian Burkhart, Matthias Nückles, Formative Computer-Based Feedback in the University Classroom: Specific Concept Maps Scaffold Students’ Writing, Computers in Human Behavior (2017), doi: 10.1016/j.chb.2017.03.008

This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

ACCEPTED MANUSCRIPT Highlights   

Concept map feedback induced lowest levels of difficulty during revisions Concept map feedback and outline feedback helped students improve text cohesion Acceptance of feedback is crucial for students’ revision implementation

ACCEPTEDFeedback MANUSCRIPT Running head: Formative Computer-Based in the University Classroom Formative Computer-Based Feedback in the University Classroom: Specific Concept Maps Scaffold Students’ Writing Author note Andreas Lachner, Department of Educational Science, University of Freiburg, Germany; Leibniz-Institut für Wissensmedien, Tübingen, Germany; and Department of Psychology, University of Tübingen, Germany; Christian Burkhart, Department of Educational Science, University of Freiburg, Germany; Matthias Nückles, Department of Educational Science, University of Freiburg, Germany. We would like to thank Lena Fischer, Coralie Köncke-Medlin, Sarah Lozano, and Oliver Steinhilber for helping us with coding the data. Correspondence concerning this article should be addressed to Andreas Lachner, Leibniz-Institut für Wissensmedien, Schleichstraße. 6, D-72076 Tübingen. E-mail: [email protected], Phone: +49(0)7071-979-356; Fax: +49(0)7071-979-200 Ethical statement The current research was not funded. The authors declare that they have no conflict of interest. All procedures performed in this study were in accordance with the 1964 Helsinki declaration, and the German Psychological Society’s (DGPS) ethical guidelines. According to the DGPS guidelines, experimental studies only need approval from an institutional review board if participants are exposed to risks that are related to high emotional or physical stress or when participants are not informed about the goals and procedures included in the study. As none of these conditions applied to the current study, we did not seek approval from an institutional review board. Informed consent was obtained from all individual participants included in the study.

ACCEPTEDFeedback MANUSCRIPT Running head: Formative Computer-Based in the University Classroom Formative Computer-Based Feedback in the University Classroom: Specific Concept Maps Scaffold Students’ Writing

Manuscript Submission: 01 March 2017

ACCEPTED MANUSCRIPT Formative Computer-Based Feedback in the University Classroom

2

Abstract Formative feedback can be regarded as a crucial scaffold for students’ writing cohesive texts. However, especially in large lectures students rarely receive feedback on their writing product. Thus, computer-based feedback could be an alternative to provide formative feedback to students. However, it is less clear, how computer-based feedback should be designed to help students writing cohesive texts. For that purpose, we implemented three different computer-based feedback methods within an authentic large lecture class. We investigated effects of the format (outline versus concept map) and the specificity (specific versus general) of the feedback on students’ perceived difficulty and the generation of cohesive texts. We found that specific concept map feedback was perceived as less difficult as compared to the general feedback or the specific outline feedback. Additionally, students who received specific feedback wrote explanatory texts that were more cohesive as compared to students with the general feedback. However, the format of the feedback (concept map versus outline) did not account for improvements of cohesion. Evidently, specific concept map feedback can be regarded as an efficient scaffold to provide cohesive explanations. Keywords: computer-based feedback, writing, cohesion, concept maps

ACCEPTED MANUSCRIPT Formative Computer-Based Feedback in the University Classroom

3

1. Introduction Through the advent of recent information technologies, written communication has become a prevalent method of delivering information both in formal, as well as in informal learning settings. For instance, at schools, teachers use learning management systems to provide instructional explanations to their students (Almarashdeh, 2016); in organizations, employees explain distinct organizational routines within knowledge management systems to potential freshmen (Wang & Wang, 2016), or in informal settings, such as free online encyclopedias (e.g., Wikipedia), millions of voluntary authors write explanations on any conceivable topic in the world (Oeberst, Halatchliyski, Kimmerle, & Cress, 2014). These examples illustrate that providing written explanations has become paramount for delivering subject matter information to others (National Commission on Writing, 2004; Rowan, 1988). University faculty, however, often lament that students often struggle with the demand of writing explanatory texts that convey the intended information in a structured and comprehensible manner that effectively supports a reader’s understanding (Cho & MacArthur, 2011; Lachner & Nückles, 2015). One practical limitation is that students rarely receive feedback for their writing products, as feedback is considered as very laborious and time-consuming. This especially holds true for large-lecture classes - still the dominant type of learning environments at public universities - with hundreds of students and a relatively low degree of interaction between instructor and students. Hence, these lectures may only provide marginal opportunities for students to receive feedback on their writing products. However, formative feedback would be relevant to further develop their writing skills (Cho & MacArthur, 2011). One alternative to feedback by instructors could be to implement computer-based feedback in such large university lectures (Moore & MacArthur, 2016). Computer-based feedback has been shown to be effective to foster students’ writing in diverse instructional

ACCEPTED MANUSCRIPT Formative Computer-Based Feedback in the University Classroom

4

settings (e.g., Graham, Hebert, & Harris, 2015; Lachner, Burkhart, & Nückles, 2017; Roscoe & McNamara, 2013; Wade-Stein & Kintsch, 2004). Despite the potential of computer-based feedback, however, the effects of computer-based feedback on students’ writing were mainly tested in small-scale field studies with rather highly selective groups of students (e.g., Caccamise, Franzke, Eckhoff, Kintsch, & Kintsch, 2007; Wade-Stein & Kintsch, 2004), or in very controlled laboratory single-session experiments (e.g., Lachner et al., 2017), which potentially restrict the transfer of these findings to large-scale lectures which are perceived as more anonymous and less engaging (Anderson & Shattuck, 2012; Trees & Jackson, 2007). Furthermore, it is less clear, how computer-based feedback should be designed to optimally foster students’ writing. Therefore, the aim of the experimental field-study reported in this paper was to investigate how different formats of computer-based feedback affect students’ cohesive writing in an authentic large-scale university lecture. 2. Cohesion as Central Textual Feature of Comprehensible Explanatory Texts Research on text comprehension showed that the cohesion of a text plays a crucial role to enhance text comprehensibility and foster a (novice) reader’s understanding (Graesser, Millis, & Zwaan, 1997; McNamara & Kintsch, 1996; O’Reilly & McNamara, 2007; Wittwer & Ihme, 2014). Cohesion can be achieved by textual devices that help readers connect ideas and sentences within a text. Local cohesion refers to textual connections between neighboring sentences of a text, so that a reader can relate ideas of a subsequent sentence to the ideas of the prior sentence (McNamara, Louwerse, McCarthy, & Graesser, 2010). Local cohesion can be established either by using rather simple syntactic cohesive ties, such as using connectives (e.g., therefore, and, because), or by using more sophisticated cohesive ties that require the writer to relate neighboring sentences semantically, for instance by using common noun phrases (e.g., by reiterating arguments), using near-synonyms, or by inserting bridging information (e.g., “as previously mentioned”, “regarding the latter

ACCEPTED MANUSCRIPT Formative Computer-Based Feedback in the University Classroom

5

argument”) which explicitly explains the semantic relation between two neighboring sentences (Halliday & Hasan, 1976; McNamara et al., 2010). Global cohesion refers to the overall-structure (i.e., the macrostructure) of a text, so that the main relations between the central ideas become explicit. Global cohesion can be enhanced by making the key relations among the central concepts of an explanatory text explicit, so that readers can establish an integrated macrostructure of the text (Graesser et al., 1997). This can particularly be established by arranging the relevant concepts of the text in a way that is in accordance with the genre-typical rhetorical macrostructure of the text (Graesser et al., 1997; Lachner et al., 2017). For instance, with regard to explanatory texts, Lachner et al. (2017) suggested structuring the concepts of the explanations around the central principles and concepts of the subject matter to be explained (see also, Wittwer & Renkl, 2008). For that purpose, explanatory texts should contain general-level concepts that allow readers to subsume subsequent concepts and identify relations among the concepts within the explanation. Additionally, the concepts and relations should be exemplified by more detailed concepts in order to provide examples for the general-level concepts (Kalyuga, Renkl, & Paas, 2010; Leinhardt, 2001). However, students are often challenged by the demand of writing cohesive explanatory texts (see Concha & Paratore, 2011; Lachner & Nückles, 2015). During writing, students are involved in simultaneous and alternating processes of planning, drafting, and revising (Hayes & Flower, 1986). These alternating and simultaneous writing processes, however, put high demands on the limited capacity of students’ working memory, which can be detrimental for the provision of high-cohesive texts (De Smet, Brand-Gruwel, Broekkamp, & Kirschner, 2012; Kellogg, Whiteford, Turner, Cahill, & Merlens, 2013). The problem of cognitive overload could be reduced or even suppressed, when students are provided with formative computer-based feedback on their writing. Formative computer-based feedback could guide

ACCEPTED MANUSCRIPT Formative Computer-Based Feedback in the University Classroom

6

students’ attention during the process of their revisions, and ease the realization of more germane revision activities, such as evaluating potential cohesion deficits or planning and implementing appropriate cohesion strategies (e.g., enhancing argument overlap, integrating missing concepts). 3. Computer-Based Feedback Approaches to Enhance Students Writing Cohesive Texts As formative feedback is very laborious and time-consuming, computer-based approaches of formative feedback could assist students during their writing of cohesive texts. However, computer-based feedback approaches for the cohesion of students’ texts are scarce. One exception is Writing-Pal (Roscoe & McNamara, 2013). Writing-Pal is an intelligent tutoring system that assists students to revise and improve their texts. Besides a direct strategy training, Writing-Pal provides automated feedback on students’ drafts with regard to central measures of text quality (e.g., essay length, global structure, elaboration, or relevance of the topic). The feedback consists of a rather general rating for the text quality from poor to great (6-point scale) and a number of prompt-like recommendations based on the deficits identified by the system (such as: “One way to expand your essay is to add additional relevant examples and evidence”, see Roscoe & McNamara, 2013). In a longitudinal classroom study, Roscoe and McNamara found that students could significantly enhance their writing over time, as indicated by the feedback measures of Writing-Pal. However, the students also concerned about the rather general character of the feedback provided by Writing-Pal. Nearly 25 % of the students complained that the automated Writing-Pal feedback was too general and thus difficult to apply. Against this background, Lachner et al. (2017) developed CohViz, which provides formative feedback on students’ drafts in the format of a concept map. To that end, the students’ texts are segmented into single propositions and visualized as a node-link structure. Nodes represent the concepts of the explanation, and links with arrows indicate the structural

ACCEPTED MANUSCRIPT Formative Computer-Based Feedback in the University Classroom

7

relationships between these concepts based on their grammatical function (e.g., subject, object) within a sentence (see Figure 1 for an example). The concept maps provide a writer with specific information about both the local and the global cohesion of their draft. With regard to the local cohesion, the potential cohesion gaps are represented as unrelated fragments in the concept map (see Figure 1). Regarding global cohesion, the concept map represents the key concepts of the text (i.e., the nodes of the concept map), as well as the central relationships (i.e., the arrows) between them which form the global macrostructure of the text. As such writers can use their concept map to monitor their draft in order to detect thematically relevant concepts and relations between these concepts which they potentially missed in their initial draft (Berlanga, van Rosmalen, Boshuizen, & Sloep, 2012). Lachner et al. (2017) conducted three studies to investigate the effectiveness of the concept map feedback on students’ generation of cohesive texts. Using think-aloud-protocols, in a pilot study, the authors examined the students’ cognitive processes while using the concept map feedback. Lachner et al. found that the concept map primarily triggered students to search for deficits of local and global cohesion in their text. These search processes resulted in the additional planning of revision strategies to remedy the deficits of local and global cohesion. In a further experimental laboratory study, the authors could show that students who were provided with the concept map feedback also wrote more locally and globally cohesive explanations as compared with students who had revised their drafts without concept map feedback. Additionally, Lachner et al. could replicate the main findings of their laboratory study in a classroom-experiment with a rather small sample size of 27 advanced teacher students. More importantly, the authors could show that students who had received concept map feedback before could transfer their acquired knowledge during the feedback to another explanation about a different topic without such feedback. Despite these promising results by Lachner et al., however, there are some open questions:

ACCEPTED MANUSCRIPT Formative Computer-Based Feedback in the University Classroom

8

First, as Lachner et al. (2017) compared the concept map condition to a no-feedback condition, the question remains, whether it was the specific information provided within the concept map, or the spatial format of the concept maps that helped students improve the cohesion of their texts. On the one hand, due to the spatial format, concept maps could better assist students to exploit perceptual processes, and facilitate students’ search processes (Ainsworth, 2006; Larkin & Simon, 1987), such as the search of cohesion gaps within a text as compared to textual formats, such as outlines which present the identical information in a serial format. On the other hand, the beneficial effects of concept map feedback could alternatively be explained by the fact that students received specific information by the concept map feedback that directed students to particular problems in their explanations with regard to cohesion (for a related study see, Ritzhaupt & Kealy, 2015). Thus, regardless of the feedback format, specific feedback approaches could better assist students’ writing as compared to more general feedback approaches that only provide general advice of how to improve their explanatory texts (e.g., Roscoe & McNamara, 2013). Second, as most of the studies on computer based feedback were implemented in the laboratory or in rather interactive and engaging learning environments, such as seminars or school classrooms (Lachner et al., 2017; Roscoe & McNamara, 2013; Wade-Stein & Kintsch, 2004), the question remains, whether the beneficial effects of concept map feedback would replicate in large-scale lectures, still a dominant type of instruction at universities. Commonly, lectures can be characterized as university courses which are attended by a large number of students who are often anonymous strangers to each other (Anderson & Shattuck, 2012; Trees & Jackson, 2007). From a motivational perspective, students’ higher levels of anonymity may result, however, in lower levels of social involvement (Deci & Ryan, 2002), and potentially decrease students’ perceived responsibility and willingness to participate in learning activities, such as practicing writing, in and outside the classroom

ACCEPTED MANUSCRIPT Formative Computer-Based Feedback in the University Classroom

9

(Cuseo, 2007; Geske, 1992; Gleason, 1986; Trees & Jackson, 2007). These negative effects of anonymity may even be increased by students’ implicit beliefs about the nature of lectures, and more importantly, about their anticipated instructional role within such lectures. Students may view their instructional role as rather passive, as in lectures students are often only required to attend the lecture sessions, rehearse the particular lecture contents, and finally prepare for a test at the end of the lecture (Anderson & Shattuck, 2012). As such, even when active opportunities (e.g., practicing writing skills with individualized feedback) that go beyond the common lecture activities are available during the semester, a lot of students may be less engaged or even refrain to deliberately participate in such learning activities (Trees & Jackson, 2007). Empirical evidence about the effects of large lectures on students’ motivation can be found in the study by Wulff, Nyquist, and Abbott (1987). The authors asked 800 university students about their perceptions of large-lecture classes. They found that students reported relatively low levels of motivation to participate in learning activities in and outside the lecture due to the rather anonymous nature of large lectures (see also Carbone & Greenberg, 1998, for similar findings). Similarly, in a recent study by Chapman and Ludlow (2010) the authors found that the perceived utility of the lecture was negatively related with the lecture size. Thus, smaller university courses were perceived to contribute more to one’s individual learning than larger lectures. Redirecting these findings to the use of concept map feedback in large-lecture classes, one may assume that students should be less inclined to implement the formative feedback provided by the feedback tool, as the anonymous character of large-lecture classes may decrease students’ motivation to actively participate in practicing their writing skills with formative computer-based feedback. As such, the effects of the feedback tool can be assumed

ACCEPTED MANUSCRIPT Formative Computer-Based Feedback in the University Classroom

10

to be smaller in large-scale lectures as compared to smaller and less anonymous settings, such as seminars (Trees & Jackson, 2007). 4. Overview of the Current Study To answer these open questions, we conducted an experimental field study implemented within a regular large-scale lecture on Educational Science of a German university over two instructional sessions. Our aims of the study were twofold: First, we wanted to investigate whether the beneficial effects of concept map feedback could be replicated in an authentic large-scale university lecture. Second, as we implemented different kinds of feedback methods in the lecture, we examined whether primarily the format (spatial format: concept map versus textual format: outline) or the specificity of computer-based feedback better helped students improve their drafts for local and global cohesion (general feedback versus specific feedback). By implementing those different feedback methods, we were able to disentangle whether students would rather depend on the specific information provided within the computer-based feedback while revising their explanatory texts, or whether the type of representational format (concept map versus outline) would additionally contribute to students’ improvements of cohesion. The study was implemented during the first two weeks of the lecture. Students were asked to write an explanation about the current topic of the lecture, as additional homework assignment. Additionally, to assist students during their writing, students randomly received one of the three feedback methods to inform them about the level of local and global cohesion of their texts. 4.1 Research Questions and Hypotheses Based on the findings by Lachner et al. (2017) and Roscoe and McNamara (2013), we expected that revising with specific feedback would be less difficult than revising with general feedback. These lower levels of difficulty should contribute to larger levels of local

ACCEPTED MANUSCRIPT Formative Computer-Based Feedback in the University Classroom

11

and global cohesion of the revised explanations by students with specific feedback as compared with students with general feedback (feedback specificity hypothesis). With regard to differences between the outline feedback and the concept map feedback, we refrained from making a clear prediction, as it is less clear what to expect. On the one hand, following Larkin and Simon (1987), concept map feedback could better facilitate students’ search processes of cohesion deficits within a text which potentially could result in more locally and globally cohesive explanations as compared with the outline feedback. On the other hand, as both feedback types contained identical information about the cohesion of their explanations, and students had ample time to process their feedback, students in both conditions could benefit from the feedback to a similar extent during their revisions, which would result in comparable cohesive explanations between the concept map feedback and the outline condition (representation format hypotheses). 5. Method 5.1 Study Site and Participants The lecture, in which we implemented our study, was a basic lecture on Educational Science of the general pedagogical curriculum at a university in south-west Germany. In the lecture, students were educated in teaching and learning processes at schools, general didactics, and developmental psychology. Three hundred-ninety-nine teacher students were enrolled in the lecture. The lecture consisted of weekly face-to-face sessions (comprising 90 minutes) and online voluntary homework assignments in between the face-to-face-sessions. In total, the lecture lasted a complete semester (approximately three months). The study took place during the first two weeks of the lecture. During the study, students were asked to provide explanations about the current topic of the lecture as homework assignments. Furthermore, students were asked to revise their explanations. For that purpose, they randomly received one of the three computer-based feedback methods. The students’ mean

ACCEPTED MANUSCRIPT Formative Computer-Based Feedback in the University Classroom

12

age was 22.44 (SD = 2.77); 172 of them were male. They were in their sixth semester on average (SD = 3.57). 5.1.1. Screening of participants. 348 of the 399 students provided informed consent to participate in the study. In a next step, we inspected our data, whether the students adhered to the study instructions to enhance the treatment fidelity of our study. Based on our study instructions, the minimum requirements for including students in the final data sample was, that they 1) uploaded their homework assignments within the given deadlines in order to keep time on task comparable across conditions, and 2) that the students were engaged in revising their explanations, measured by the number of revisions. Based on these criteria, we excluded 68 students based on the deadline criterion, and further 29 students due to missing revisions. Thus, our final sample was based on 251 students, which we included for further analyses. The remaining students’ mean age was 22.19 (SD = 2.76); 172 of them were male. They were in their sixth semester on average (SD = 4.02). 5.1.2. Analyses for systematic differences between included and excluded students. As our selection procedure could have resulted in rather biased data (e.g., we could have selected a subpopulation of only high-skilled or low-skilled students), we additionally conducted several ANOVAs to check whether there were any systematic differences between students who were included or excluded from the current study. None of the analyses reached statistical significance. There were no differences between included and excluded students with regard to their subjective prior-knowledge in Educational Science, F(1, 397) = 2.47, p = .12, partial η2 = .01, and their subjective writing skills, F(1, 397) = 0.67, p = .41, partial η2 = .00. Similarly, a MANOVA revealed that the excluded and included students did not differ concerning their average level of local and global cohesion of their submitted homework assignments: F(2, 396) = 1.68, p = .19, partial η2 = .01. Thus, as there were no systematic

ACCEPTED MANUSCRIPT Formative Computer-Based Feedback in the University Classroom

13

differences between included and excluded students, we can conclude that our data was not biased by our exclusion criteria of students. 5.2. Design We used a one-factorial between-subjects design with the two revised explanations as within-subjects-variable. Type of feedback (specific concept map feedback versus specific outline feedback versus general feedback) was the independent variable. Dependent variables encompassed the level of local cohesion indicated by the proportion of cohesion gaps per explanation. As a measure for global cohesion, we used a holistic rating of principleorientation of the students’ explanations (Lachner et al., 2017). 5.3. The Feedback System We provided the students with a completely automated feedback system that was based on CohViz by Lachner et al. (2017). Within the feedback system students wrote their explanations, and subsequently received feedback via the feedback system. The feedback system consisted of a pre-processing engine and a visualization engine. 5.3.1. Pre-processing engine. For all three feedback types, first the original explanatory text was segmented into sentences. Second, all concepts (i.e., nouns) and their grammatical function (i.e., subject, possessor, direct object, and indirect object) were determined and extracted per sentence. For this purpose, we used a natural language processing technology called RFTagger (Schmid & Laws, 2008) which automatically determines the grammatical functions of words based on hidden Markov models. Additionally, all concepts that could be treated as near-synonyms were replaced with GermaNet, a large lexical database of German synonyms (Hamp & Feldweg, 1997). Third, relations between the concepts of each sentence were drawn according to their grammatical function (van Valin, 2001): a) subject proceeds possessor, b) subject proceeds direct objects, c) subject proceeds indirect object, and d) direct object proceeds indirect object (see Lachner

ACCEPTED MANUSCRIPT Formative Computer-Based Feedback in the University Classroom

14

et al., 2017). These relations were then stored in a simple list containing the sequences of concepts, such as “concept x → concept y”. Fourth, in the merging phase, duplicate concepts between sentences were detected and merged (see Lachner et al., 2017, for a step-by-stepexample). 5.3.2. Visualization engine. On the basis of the list of sequences, dependent on the experimental conditions, students received one of the tree different types of feedback. For the concept map feedback, the sequences of concepts and their relations were visualized as a concept map using the layout tool d3 (http://d3js.org/). A cohesion gap within the concept map was presented as an isolated fragment that was not related to the rest of the conceptual representation (see Figure 1A). To keep the visualizations as parsimonious as possible, we refrained from labeling the links between concepts (i.e., explaining their logical or semantic relations) to minimize the risk of creating a redundancy effect (see Sweller, 2010). For the outline feedback, we similarly used the sequence list of the concepts. The concepts of each fragment were printed below each other. Cohesion gaps were marked by a blank line between the subsequent sentences (see Figure 1B). Thus, the information with regard to the levels of cohesion provided in the outline representation was identical to the concept map condition, but was presented in a linear format. In the general feedback condition (see Figure 1C), we provided the students only with a standardized feedback statement which included the number of explained concepts, and the number of cohesion gaps of their draft (e.g., “Your explanation contains eight concepts and has three cohesion gaps”). The number of concepts and cohesion gaps were also based on the sequence list derived from the pre-processing engine. In contrast to the more specific concept maps or outlines, students provided with the general feedback, however, needed to infer where the cohesion gaps were allocated and which concepts were present in their drafts. By

ACCEPTED MANUSCRIPT Formative Computer-Based Feedback in the University Classroom

15

implementing the general feedback condition in our experiment, this allowed us to disentangle effects of the specificity and the format of feedback on students’ writing performance. 5.3.3. Perceived difficulty. After students had finished their revision, the students assessed their perceived difficulty during revising the draft. For this purpose, we used an adapted short questionnaire, originally developed by Berthold and Renkl (2009). Students’ perceived difficulty was assessed by four items on a 5-point rating-scale (1 = easy, 5 = difficult: 1) “How easy or difficult was it for you to revise your explanation?”; 2) “How easy or difficult was it for you to work with the feedback?”; 3) “How easy or difficult was it for you to distinguish important and unimportant information of the feedback?”; “4) How easy or difficult was it for you to identify cohesion deficits in the explanation?”). The reliability of the questionnaire was very good, α = .87 (Cronbach’s α). Four students had one missing value on the perceived difficulty rating, which were replaced by the series mean. Unfortunately, due to technical errors, the difficulty rating form was not displayed after the second explanation task. Thus, our analyses with regard to perceived difficulty, reported in this study, are solely based on the first explanation task. 5.4. Procedure For an overview of the procedure see Table 1. Before the study started, we informed the students about the goal of the study and obtained informed consent to use their data for scientific purposes. The first lecture was about how to effectively explain subject matter to students. The main objective was to provide the students with conceptual knowledge about learning from instructional explanations (Leinhardt, 2001; Wittwer & Renkl, 2008). Furthermore, students were informed about appropriate explanation strategies to establish local and global cohesion (Lachner et al., 2017; McNamara et al., 2010). At the end of the lecture, the instructor assigned the first explanation task as homework to be completed

ACCEPTED MANUSCRIPT Formative Computer-Based Feedback in the University Classroom

16

outside the class within seven days. The students were asked to write an instructional explanation about how to provide effective instructional explanations that would be intelligible to novice students, and to upload their draft to the feedback system within seven days. The students were informed that they would receive feedback after uploading their draft. At home, students logged in the feedback system, and submitted their draft of the explanations. After submitting their drafts, students automatically received feedback dependent on the experimental condition (see Figure 1, for an example). The general feedback condition (n = 96) received only general information about the number of concepts and the number of cohesion gaps in their explanations, and a prompt to revise their explanation for local and global cohesion according to their feedback. The concept map feedback group (n = 77), in contrast, received a message containing the prompt and a specific concept map of their explanation. The outline feedback group (n = 78) received a message containing the prompt and an outline of their explanation which contained the identical information as the concept map with regard to cohesion gaps and explained concepts, but in the linear format of an outline. During the feedback, all the students were provided with additional information on how to read and interpret their feedback (see Appendix A, for the entire instruction). Subsequently, they revised their explanations. After submitting their revisions, students answered the perceived difficulty questionnaire. The second face-to-face-session was a lecture on assessment strategies in the classroom. The students were provided with conceptual knowledge about assessment strategies (e.g., Herppich, Wittwer, Nückles, & Renkl, 2014; Ruiz‐Primo & Furtak, 2007). At the end of this session, the instructor assigned the second explanation task as homework. The students were asked to write an instructional explanation about how to implement assessment strategies in the classroom that would be suitable for novice students. Again, after the students submitted their draft of the diagnosis explanations, students automatically received feedback dependent

ACCEPTED MANUSCRIPT Formative Computer-Based Feedback in the University Classroom

17

on the particular experimental condition. After the students had received the feedback, they revised their explanations and uploaded their revised explanation no later than the start of the next face-to-face session. * Insert Table 1 about here * 5.5. Treatment fidelity To ensure that our feedback was delivered as intended, we implemented the following safeguards. First, as we implemented our study with the help of an automated computerbased feedback system which allowed a standardized procedure across participants, we could ensure equal conditions for all participants. Second, to maximize the probability that the students attended to the homework instruction, we additionally sent the homework instruction and additional reminders via E-Mail. Third, with regard to the quality of the feedback, a trained student assistant who was blind to the experimental conditions counted the number of cohesion gaps of 30 % of the first explanation. The correlation between the hand-coded cohesion gaps and the cohesion gaps derived by the feedback system was very high, r = .85, p < .001. 5.6. Analysis and coding 5.6.1. Local cohesion. To obtain a measure for local cohesion, we used the same algorithm which was used to provide the feedback via our feedback system, as it has been shown to be a reliable indicator for local cohesion (see also paragraph on treatment fidelity). Thus, the feedback system automatically tracked and stored the number of cohesion gaps of the explanations in a separate database. Cohesion gaps were operationalized as concepts within the sequence list that had no relation to the rest of the concept list (see Figure 1, for an example). Additionally, as the number of cohesion gaps could be sensitive to the particular text length, we followed the procedure by McNamara et al. (2010), and computed proportions

ACCEPTED MANUSCRIPT Formative Computer-Based Feedback in the University Classroom

18

of the number of cohesion gaps by dividing the absolute frequencies of cohesion gaps by the total number of sentences of the particular explanation. 5.6.2. Global cohesion. As “principle-orientation” can be regarded as the central genrespecific rhetorical principle of instructional explanations which forms the global cohesion within a text, we used a holistic rating scheme developed by Lachner et al. (2017), and assessed global cohesion on the following four dimensions. On the first dimension, we assessed, whether the students provided general-level concepts that allowed readers to subsume more specific concepts (e.g., instructional explanations as a central instructional method). On the second dimension, we rated, whether the explanation contained the relevant domain principles and concepts with regard to the topic to be explained (e.g., definition of instructional explanations, features of effective explanations such as cohesion, student characteristics, such as prior knowledge). Third, we rated whether the explanations described important relations between the concepts by relating them to the central domain principles (e.g., the relation between student characteristics and features of instructional explanations). Fourth, we rated whether the explanations contained sub-concepts that illustrated superordinate concepts (e.g., examples for cohesive devices). For each dimension, students could receive either one point (dimension fully present), half a point (dimension partly present) or zero points (dimension not present at all), yielding a maximum score of four points. Three trained raters rated 25 % of the explanations. Inter-rater agreement was very good, ICC = .84 (Wirtz & Caspar, 2002). Thus, one rater coded the rest of the explanations. 6. Results We used an alpha level of .05 for all statistical analyses. As effect size measure, we used partial η2 qualifying values < .06 as small effect, values in the range between .06 and .14 as medium effect, and values > .14 as large effect (see Cohen, 1988). 6.1. Preliminary Analyses

ACCEPTED MANUSCRIPT Formative Computer-Based Feedback in the University Classroom

19

A series of ANOVAs and χ² tests revealed no significant differences between the experimental conditions concerning gender, χ²(2) = 0.26, p = .88; age, F(2, 248) = 0.45, p = .64, partial η2 = .00; number of enrolled semesters, F(2, 248) = 0.13, p = .87, partial η2 = .00; and subjective writing experience, F(2, 248) = 0.27, p = .76, partial η2 = .00. Additionally, a MANOVA with the average number of sentences for the four explanations as dependent variables showed no significant differences among experimental conditions, F(8, 492) = 0.88, p = .54, partial η2 = .01. Furthermore, the students’ explanation drafts for the first homework assignment did not differ significantly between the experimental conditions with regard to the level of local cohesion, F(2, 248) = 0.09, p = .91, partial η2 = .00, and the degree of global cohesion as measured by the rating of principle-orientation, F(2, 248) = 0.30, p = .74, partial η2 = .00. Therefore, we can conclude that students possessed comparable writing skills with regard to establishing local and global cohesion across conditions, before we started with the feedback intervention. Table 2 gives an overview of the descriptive statistics for our dependent variables. *Insert Table 2 about here* 6.2. Analyses with Regard to Perceived Difficulty To examine whether the students perceived revising their texts as less difficult when they had concept map feedback at hand, we computed planned contrasts with students’ perceived difficulty as dependent variable, and type of feedback (general versus concept map versus outline) as between-subjects-factor. There was no effect between the specific feedback (outline, concept map) and the general feedback condition, F(1, 248) = 1.01, p = .32, partial η2 = .00, but a significant effect between the concept map condition and the outline condition, F(1, 248) = 15.30, p < .001, partial η2 = .06. These findings indicate that the outline feedback was perceived as most difficult, followed by the general feedback, whereas revising with the concept map feedback was seen as least difficult (see Table 2).

ACCEPTED MANUSCRIPT Formative Computer-Based Feedback in the University Classroom

20

6.3. Analyses with Regard to Local Cohesion To investigate whether students with specific feedback (outline versus concept map feedback) generated more locally cohesive explanations as compared with students with general feedback, we performed a repeated measures ANOVA with the local cohesion scores of the revised homework assignments (revised explanation 1 and revised explanation 2) as within-subjects factor, and experimental condition (general versus concept map versus outline) as between-subjects-factor. There was a main effect of time, F(1, 248) = 6.23, p = .01, partial η2 = .03, but no significant interaction effect, F(1, 248) = 0.31, p = .73, partial η2 = .00, indicating that students improved the local cohesion of their drafts from the first to the second explanation. Similarly, we found a main effect of experimental condition, F(1, 248) = 4.65, p = .01, partial η2 = .04, indicating that there were significant differences among the feedback conditions. To investigate differences between the experimental conditions, we additionally computed planned contrasts, and first contrasted the specific feedback conditions (outline and concept map) to the general feedback condition. As expected, students who were provided with specific information about the level of local cohesion (outline and concept map) generated more locally cohesive explanations (i.e., fewer cohesion gaps) as compared to students with general feedback, F(1, 248) = 8.81, p = .01, partial η2 = .03. However, there were no significant differences between outline feedback and concept map feedback, F(1, 248) = 0.51, p = .47, partial η2 = .00. Apparently, students profited from the specific information within the feedback, but not from the spatial format of concept map feedback with regard to their local cohesion (see Table 2). Thus, as students had ample time to revise their texts for cohesion, apparently students could compensate the higher levels of difficulty induced by the outline feedback, and generate as locally cohesive explanations as the students with concept map feedback due to the comparable information provided within the feedback.

ACCEPTED MANUSCRIPT Formative Computer-Based Feedback in the University Classroom

21

6.4. Analyses with Regard to Global Cohesion Similarly to latter analyses, with regard to the level of global cohesion, we additionally performed a repeated measures ANOVA with the global cohesion scores of the two homework assignments (revised explanation 1 and revised explanation 2) as within-subjects factor, and experimental condition (general versus concept map versus outline) as betweensubjects-factor. Again, there was a main effect of time, F(1, 248) = 32.72, p < .001, partial η2 = .12, but no significant interaction effect, F(1, 248) = 0.63, p = .53, partial η2 = .00, indicating that students improved the global cohesion of their drafts from the first to the second explanation. However, we did not find a main effect of experimental condition, F(1, 248) = 1.21, p = .30, partial η2 = .01. These findings indicate that all students comparably benefited from receiving formative feedback to improve the global cohesion of their text, regardless of the specificity or the representation format of the feedback (see Table 2). Apparently, providing feedback itself was sufficient to trigger students’ evaluation processes about potential deficits of global cohesion of their texts which similarly helped them revise their texts for global cohesion. 7. Discussion We conducted a field-experimental study to investigate effects of computer-based feedback on students’ writing of cohesive texts in a large-lecture class. For that purpose, we implemented three different computer-based feedback methods that primarily differed with regard to the specificity of the provided feedback information (specific versus general), and the representation format (graphical: concept map feedback, textual: outline feedback) in an authentic university lecture on Educational Science. By and large, the results showed clear efficiency benefits of concept map feedback in terms of perceived difficulty compared to the other conditions. We could show that the concept map feedback induced the lowest levels of difficulty while students revising their texts as compared to the outline feedback and the

ACCEPTED MANUSCRIPT Formative Computer-Based Feedback in the University Classroom

22

general feedback. Furthermore, we found that students who received specific formative feedback (concept map feedback and outline feedback) generated more locally cohesive explanations as compared with students who only had general feedback about the number of cohesion gaps and the number of concepts at hand. Therefore, we can conclude that specific concept map feedback may be a feasible and efficient feedback methodology to help students revise their text for local cohesion. These beneficial effects, however, do not hold true for the global cohesion, as all students comparably improved the global cohesion of their texts, regardless of the type of feedback. Thus, one may conclude that, in order to assist students revising for global cohesion, feedback may suffice when it sensitizes students towards global deficits of their drafts, without the necessity of providing specific information about the allocation of global cohesion gaps (Shute, 2008). However, this interpretation is rather speculative, as we did not include a control group of students who did not receive any feedback. Thus, alternatively, the improvements of global cohesion could also be explained due to the repeated practice of writing skills and not due to the provision of feedback. Overall, we also need to acknowledge that as compared to the findings by Lachner et al. (2017), in this study the overall effects of our feedback methods were by far less pronounced. Whereas Lachner et al. reported medium to large effects of the concept map feedback, we “only” found small effects of our feedback methods. We attribute the smaller effects to the particular learning environment in which we implemented our feedback system. Due to students’ potential higher levels of anonymity in the lecture (Trees & Jackson, 2007), apparently students were less inclined to actively participate during the semester, for instance by regularly accomplishing instructional explanations as homework outside the classroom. As such, the rather small effects of the current study can be explained by the fact that students potentially were less engaged to revise their explanatory texts for cohesion.

ACCEPTED MANUSCRIPT Formative Computer-Based Feedback in the University Classroom

23

Additionally, an alternative explanation of the smaller effects could be that students were provided with computer-mediated feedback (Golke, Dörfler, & Artelt, 2015). In contrast to the recent studies by Lachner et al. (2017), in which students received person-mediated concept map feedback either by an instructor or by an experimenter, in the current study, the feedback was entirely provided within a computer-based system. Such computer-mediated feedback could have additionally resulted in weaker commitments by the students, as students potentially perceived the computer-mediated feedback as less valuable compared to person-mediated feedback. As such, students may have been less willing to process and implement the computer-based feedback than the person-mediated feedback. However, these assumptions are highly speculative, as we cannot provide data about students’ affective and motivational states while accomplishing the homework assignments. For instance, empirical evidence for this claim can be found in a recent study by Golke et al. (2015). The authors investigated effects of person-mediated feedback versus computermediated feedback on students’ comprehension of texts. For that purpose, while learning from texts, the authors provided students with inference prompts as specific feedback to foster students’ comprehension. Additionally, they varied the type of mediation, as the feedback was either provided by a computer system or by a real person. Thus, the feedback types were comparable with regard to the level of provided information, but significantly differed in terms of type of mediation of information (computer-mediated versus personmediated). Golke et al. found that students who received computer-based feedback spent significantly less time on processing the feedback as compared with students who received feedback by an instructor. More importantly, the shallow processing of the feedback resulted in lower levels of comprehension by students with computer-mediated feedback as compared with students who received person-mediated feedback.

ACCEPTED MANUSCRIPT Formative Computer-Based Feedback in the University Classroom

24

Therefore, further research is needed to investigate how the provision of computerbased feedback can be enhanced from a motivational and social perspective, to fully exploit the potential of computer-based feedback. For instance, during processing the feedback, students could be provided with relevance instructions (McCrudden & Schraw, 2007; Roelle, Lehmkuhl, Beyer, & Berthold, 2015) to make the beneficial effects of computer-based feedback on students’ writing explicit. Relevance instructions could increase the perceived value of computer-mediated feedback which should also contribute to a higher acceptance of the computer-based feedback by the students. Furthermore, it could be worth exploring how pedagogical agents, as often used in intelligent tutoring systems (Goldberg & CannonBowers, 2015; Moreno, Mayer, Spires, & Lester, 2001), could be implemented as an alternative to person-mediated feedback to enable social grounding of the computer-mediated feedback to enhance the acceptance of computer-based feedback approaches. A further promising approach could be to supplement computer-based feedback with peer-feedback (Cho & MacArthur, 2011). As such, in individual phases, such as homework assignments, students could revise their texts with formative computer-based feedback to deliberately practice their writing skills. Additionally, students’ revised texts could be discussed with their peers to further enhance the quality of their texts (Cho & MacArthur, 2011). By following this approach, person-mediated as well as computer-mediated feedback could be combined to fully exploit the potential of computer-based feedback on students’ writing. However, it is up for further research to more directly investigate how peer-feedback and computer-based feedback should be orchestrated in instructional classrooms to effectively support students developing their writing skills (Kellogg et al., 2013). 7.1. Further Study Limitations and Future Research In our study, we sought to examine effects of computer-based feedback on students’ writing in an authentic field-setting, namely large-lecture-classes. Conducting such studies in

ACCEPTED MANUSCRIPT Formative Computer-Based Feedback in the University Classroom

25

authentic settings can be regarded as highly ecologically valid. However, they potentially lack control for other confounding variables, such as students’ time on task, or their commitment during participation. As such, we have to acknowledge that the variability within the groups was relatively high (see Table 2), which could have additionally impaired the effects of our feedback intervention. That said, as we conducted our study in an authentic learning environment, it was not possible to include an additional control group of students who did not receive any feedback, as those students would be disadvantaged as compared to the students of the experimental conditions. Such a disadvantage, however, would have been against the APA standards for ethical treatment of human participants (American Psychological Association, 2002). Hence, laboratory studies with an additional control group of students, as well as with higher levels of control are needed to clearly disentangle effects of the specificity and representation format of formative feedback on students’ writing. For instance, scenario approaches (Bolzer, Strijbos, & Fischer, 2014; Strijbos, Narciss, & Dünnebier, 2010), in which students revise experimenter-generated text materials, could be used to better control for potential inter-individual differences among students. Such scenario approaches would allow to experimentally disentangling effects of the representation format and the specificity of computer-based feedback on students’ revision activities, while keeping the text to be revised for cohesion constant across conditions. In addition, we have to admit that our sampling procedure may have contributed to a biased sample, as the participation in our study was voluntary. However, it has to be noted that we were able to recruit more than 80 % of the lecture students to provide informed consent. That said, the participatory rate in our study can be regarded as rather high as compared to other studies which are often based on ad-hoc occupational samples with lower participatory rates of the total sample. Nevertheless, more research with randomized

ACCEPTED MANUSCRIPT Formative Computer-Based Feedback in the University Classroom

26

controlled trials is needed to further investigate effects of the specificity and the format of feedback on students’ writing. Additionally, from a technological perspective, we suggest that the feedback tool should be further developed to fully exploit the potential of formative computer-based feedback in the classroom. So far, the automatic construction of the sequence list of the concept map is based on lexical re-iteration or the use of near-synonyms. However, the concept map feedback technology is not able for example to separate homonyms (i.e., words with the identical pronunciation, but different meanings). That said, recent machine learning technologies from advanced computer linguistics (e.g., Ziai, de Kuthy, & Meurers, 2016; Ott, Ziai, & Meurers, 2012) may be an alternative to provide students with more accurate concept map representations of their texts. A final caveat refers to the generalizability of our findings to other text genres. As our students only wrote explanatory texts, the question is left open whether the findings would replicate to other text genres, such as argumentative texts (De Smet et al., 2012). We would assume that automated feedback may also help students revise their argumentative essays for cohesion, as such representation should be amply appropriate to visualize the logic and interrelations of students’ arguments (see Nussbaum & Schraw, 2007). Hence, the generalizability of our findings should be examined by future research. 7.2. Conclusion The study reported in this paper shows how computer technologies can be used to support students with formative feedback in large-scale lecture classes. Our concept map feedback approach has been shown to be an easy-to-implement and feasible approach to support students developing relevant writing skills in large lecture classes. However, despite the valuable findings, our findings are suggestive of ways that some technical and

ACCEPTED MANUSCRIPT Formative Computer-Based Feedback in the University Classroom

27

instructional scaffolds need to be addressed beforehand to safeguard the effectiveness of computer-based feedback in large-scale lectures. First, to heighten the acceptance of the particularly technology, we suggest that the technology should be accessible to all students without any barriers, such as lacking internet access or limited availability of computer technology. By providing a feasible and easy-touse technological environment, students more likely engage in acting with the technology, which potentially results in more genuine learning experiences. Second, while implementing computer-based feedback in the classroom, instructors should also consider potential second-order barriers, and integrate potential remedial strategies (Ertmer, Ottenbreit-Leftwich, Sadik, Sendurur, & Sendurur, 2012). Second-order barriers comprise both students’ attitudes towards the additional benefits of using the technology (Ertmer et al., 2012; Ifenthaler & Schweinbenz, 2016), as well as their general beliefs about the learning environment in which the feedback should be implemented (Staub & Stern, 2002; Wegner, & Nückles, 2016). As our findings may be interpreted that way that students’ beliefs about the learning environment, such as lectures, largely impacted students’ engagement to deliberately use the technology (for related findings, see; Roscoe, Wilson, Johnson, & Mayra, 2017), we suggest to implement potential remedial instructional strategies before-hand, such as relevance instruction (Roelle et al., 2015) to heighten students’ engagement while using the technology. Finally, we propose that computer-based feedback should be supplemented with other instructional approaches (e.g., tutorials, or peer-feedback, instructor feedback) in order to strengthen the benefits of computer-based feedback. For instance, our findings are suggestive of ways that computer-based feedback should be combined with a-priori tutorials in which students are taught about adequate writing strategies which are practiced in subsequent practice phases with computer-based feedback. By combining computer-based feedback with

ACCEPTED MANUSCRIPT Formative Computer-Based Feedback in the University Classroom

28

a-priori tutorials, students can be enabled to develop sustainable skills in writing locally and globally cohesive texts (Kellogg et al., 2013). All in all, our study is a promising starting point for further research on computerbased feedback on students’ writing skills. Our findings show that specific concept map feedback can be an efficient instructional scaffold to support students’ writing cohesive text. They further show that feedback has to be implemented properly to fully exploit the potential of formative computer-based feedback.

ACCEPTED MANUSCRIPT Formative Computer-Based Feedback in the University Classroom

29

References Ainsworth, S. (2006). DeFT: A conceptual framework for considering learning with multiple representations. Learning and Instruction, 16(3), 183-198. doi:10.1016/j.learninstruc.2006.03.001 Almarashdeh, I. (2016). Sharing instructors experience of learning management system: A technology perspective of user satisfaction in distance learning course. Computers in Human Behavior, 63, 249-255. doi: 10.1016/j.chb.2016.05.013 American Psychological Association. (2002). Ethical principles of psychologists and code of conduct. American Psychologist, 57, 1060–1073. doi: 10.1037/0003-066X.57.12.1060 Anderson, T., & Shattuck, J. (2012). Design-based research a decade of progress in education research?. Educational researcher, 41(1), 16-25. doi: 10.3102/0013189X11428813 Berlanga, A. J., van Rosmalen, P., Boshuizen, H. P., & Sloep, P. B. (2012). Exploring formative feedback on textual assignments with the help of automatically created visual representations. Journal of Computer Assisted Learning, 28(2), 146-160. doi: 10.1111/j.1365-2729.2011.00425.x Berthold, K., & Renkl, A. (2009). Instructional aids to support a conceptual understanding of multiple representations. Journal of Educational Psychology, 101(1), 70–87. doi:10.1037/a0013247 Bolzer, M., Strijbos, J. W., & Fischer, F. (2014). Inferring mindful cognitive-processing of peer-feedback via eye-tracking: role of feedback-characteristics, fixation-durations and transitions. Journal of Computer Assisted Learning. doi: 10.1111/jcal.12091 Caccamise, D., Franzke, M., Eckhoff, A., Kintsch, E., & Kintsch, W. (2007). Guided practice in technology-based summary writing. In: D. McNamara (Ed.), Reading comprehension strategies: Theories, interventions, and technologies. (pp. 375-396). Mahwah, NJ: Erlbaum. Carbone, E., & Greenberg, J. (1998). Teaching large classes: Unpacking the problem and responding creatively. To Improve the Academy. Paper 399. http://digitalcommons.unl.edu/podimproveacad/399 Chapman, L., & Ludlow, L. (2010). Can downsizing college class sizes augment student outcomes?: An investigation of the effects of class size on student learning. The Journal of General Education, 59(2), 105-123. doi: 10.1353/jge.2010.0012 Cho, K., & MacArthur, C. (2011). Learning by reviewing. Journal of Educational Psychology, 103(1), 73-84. doi: 10.1037/a0021950

ACCEPTED MANUSCRIPT Formative Computer-Based Feedback in the University Classroom

30

Cohen, J. (1988). Statistical power analysis for the behavioural sciences. Hillside. New York, NJ: Lawrence Earlbaum Associates. Concha, S., & Paratore, J. R. (2011). Local coherence in persuasive writing: An exploration of Chilean students’ metalinguistic knowledge, writing process, and writing products. Written Communication, 28(1), 34-69. doi: 10.1177/0741088310383383 Cuseo, J. (2007). The empirical case against large class size: adverse effects on the teaching, learning, and retention of first-year students. The Journal of Faculty Development, 21(1), 5-21. De Smet, M. J. R., Brand-Gruwel, S., Broekkamp, H., & Kirschner, P. A. (2012). Write between the lines: Electronic outlining and the organization of text ideas. Computers in Human Behavior, 28(6), 2107-2116. doi:10.1016/j.chb.2012.06.015 Deci, E. L., & Ryan, R. M. (2002). Handbook of self-determination research. Rochester, NY: University Rochester Press. Ertmer, P. A., Ottenbreit-Leftwich, A. T., Sadik, O., Sendurur, E., & Sendurur, P. (2012). Teacher beliefs and technology integration practices: A critical relationship. Computers & Education, 59(2), 423-435. doi: 10.1016/j.compedu.2012.02.001 Geske, J. (1992). Overcoming the drawbacks of the large lecture class. College teaching, 40(4), 151-154. doi: 10.1080/87567555.1992.10532239 Gleason, M. (1986). Better communication in large courses. College Teaching, 34(1), 20-24. doi: 10.1080/87567555.1986.10532325 Goldberg, B., & Cannon-Bowers, J. (2015). Feedback source modality effects on training outcomes in a serious game: Pedagogical agents make a difference. Computers in Human Behavior, 52, 1-11. doi:10.1016/j.chb.2015.05.008 Golke, S., Dörfler, T., & Artelt, C. (2015). The impact of elaborated feedback on text comprehension within a computer-based assessment. Learning and Instruction, 39, 123-136. doi:10.1016/j.learninstruc.2015.05.009 Graesser, A. C., Millis, K. K., & Zwaan, R. A. (1997). Discourse comprehension. Annual Review of Psychology, 48(1), 163-189. doi:10.1146/annurev.psych.48.1.163 Graham, S., Hebert, M., & Harris, K. R. (2015). Formative assessment and writing: A metaanalysis. The Elementary School Journal,115(4), 523-547. doi: 10.1086/681947 Halliday, M. A. K., & Hasan, R. (1976). Cohesion in english. London: Longman Hamp, B., & Feldweg, H. (1997). GermaNet - a Lexical-Semantic Net for German. Proceedings of the ACL workshop Automatic Information Extraction and Building of Lexical Semantic Resources for NLP Applications. Madrid.

ACCEPTED MANUSCRIPT Formative Computer-Based Feedback in the University Classroom

31

Hayes, J. R., & Flower, L. S. (1986). Writing research and the writer. American Psychologist, 41(10), 1106-1113. doi: 10.1037/0003-066X.41.10.1106 Herppich, S., Wittwer, J., Nückles, M., & Renkl, A. (2014). Addressing knowledge deficits in tutoring and the role of teaching experience: Benefits for learning and summative assessment. Journal of Educational Psychology. 106(4), 934-945. doi: 10.1037/a0036076 Ifenthaler, D., & Schweinbenz, V. (2016). Students' acceptance of tablet pcs in the classroom. Journal of Research on Technology in Education, 48(4), 306-321. doi: 10.1080/15391523.2016.1215172 Kalyuga, S., Renkl, A., & Paas, F. (2010). Facilitating flexible problem solving: A cognitive load perspective. Educational Psychology Review,22(2), 175–186. doi: 10.1007/s10648010-9132-9 Kellogg, R. T., Whiteford, A. P., Turner, C. E., Cahill, M., & Merlens, A. (2013). Working Memory in Written Composition: An Evaluation of the 1996 Model. Journal of Writing Research, 5(2), 159-190. doi: 10.17239/jowr-2013.05.02.1 Lachner, A., Burkhart, C., & Nückles, M. (2017). Mind the gap! Automated concept map feedback supports students in writing cohesive explanations. Journal of Experimental Psychology: Applied. Advance Online Publication, doi: 10.1037/xap0000111 Lachner, A., & Nückles, M. (2015). Bothered by abstractness or engaged by cohesion? Experts' explanations enhance novices' deep-learning. Journal of Experimental Psychology: Applied, 21(1), 101-115 doi: 10.1037/ /xap0000038 Larkin, J. H., & Simon, H. A. (1987). Why a diagram is (sometimes) worth ten thousand words. Cognitive Science, 11(1), 65-100. doi: 10.1111/j.1551-6708.1987.tb00863.x Leinhardt, G. (2001). Instructional explanations: A commonplace for teaching and location for contrast. In V. Richardson (Ed.), Handbook for research on teaching (pp. 333–357). Washington, DC: American Educational Research McCrudden, M. T., & Schraw, G. (2007). Relevance and goal-focusing in text processing. Educational psychology review, 19(2), 113-139. doi: 10.1007/s10648-006-9010-7 McNamara, D. S., & Kintsch, W. (1996). Learning from text: Effects of prior knowledge and text coherence. Discourse Processes, 22(3), 247–287. doi:10.1080/01638539609544975 McNamara, D. S., Louwerse, M. M., McCarthy, P. M., & Graesser, A. C. (2010). Cohmetrix: Capturing linguistic features of cohesion. Discourse Processes, 47(4), 292-330. doi: 10.1080/0

ACCEPTED MANUSCRIPT Formative Computer-Based Feedback in the University Classroom

32

Moore, N. S., & MacArthur, C. A. (2016). Student use of automated essay evaluation technology during revision. Journal of Writing Research, 8(1). doi: 10.17239/jowr2016.08.01.05 Moreno, R., Mayer, R. E., Spires, H. A., & Lester, J. C. (2001). The case for social agency in computer-based teaching: Do students learn more deeply when they interact with animated pedagogical agents?. Cognition and instruction, 19(2), 177-213. doi: 10.1207/S1532690XCI1902_02 National Commission on Writing. (2004, September). Writing: A ticket to work... or a ticket out. Available on www.collegeboard.com Nussbaum, E. M., & Schraw, G. (2007). Promoting argument-counterargument integration in students' writing. The Journal of Experimental Education, 76(1), 59-92. doi: 10.3200/JEXE.76.1.59-92 Oeberst, A., Halatchliyski, I., Kimmerle, J., & Cress, U. (2014). Knowledge construction in Wikipedia: A systemic-constructivist analysis. Journal of the Learning Sciences, 23, 149-176. doi: 10.1080/10508406.2014.888352 O’Reilly, T., & McNamara, D. S. (2007). Reversing the reverse cohesion effect: Good texts can be better for strategic, high-knowledge readers. Discourse Processes, 43(2), 121152. doi: 10.1207/s15326950dp4302_2 Ott, N., Ziai, R., & Meurers, D. (2012). Creation and analysis of a reading comprehension exercise corpus: Towards evaluating meaning in context. In T. Schmidt & K. Wörner (eds), Multilingual Corpora and Multilingual Corpus Analysis, Hamburg Studies in Multilingualism (HSM), pp. 47–69. Amsterdam: Benjamins Ritzhaupt, A. D., & Kealy, W. A. (2015). On the utility of pictorial feedback in computerbased learning environments. Computers in Human Behavior, 48, 525-534. doi: 10.1016/j.chb.2015.01.037 Roelle, J., Lehmkuhl, N., Beyer, M.-U., & Berthold, K. (2015). The role of specificity, targeted learning activities, and prior knowledge for the effects of relevance instructions. Journal of Educational Psychology, 107, 705–723. doi:10.1037/edu0000010 Roscoe, R., & McNamara, D. (2013). Writing Pal: Feasibility of an intelligent writing strategy tutor in the high school classroom. Journal of Educational Psychology, 105, 1010-1025. doi: 10.1037/a0032340

ACCEPTED MANUSCRIPT Formative Computer-Based Feedback in the University Classroom

33

Roscoe, R. D., Wilson, J., Johnson, A. C., & Mayra, C. R. (2017). Presentation, expectations, and experience: Sources of student perceptions of automated writing evaluation. Computers in Human Behavior. doi: 10.1016/j.chb.2016.12.076 Rowan, K. E. (1988). A contemporary theory of explanatory writing. Written Communication, 5(1), 23-56. doi: 10.1177/0741088388005001002 Ruiz‐Primo, M. A., & Furtak, E. M. (2007). Exploring teachers' informal formative assessment practices and students' understanding in the context of scientific inquiry. Journal of Research in Science Teaching, 44(1), 57-84. doi:10.1002/tea.20163 Schmid, H., & Laws, F. (2008). Estimation of conditional probabilities with decision trees and an application to fine-grained POS tagging. Proceedings of the 22nd International Conference on Computational Linguistics. Shute, V. J. (2008). Focus on formative feedback. Review of educational research, 78(1), 153-189. doi: 10.3102/0034654307313795 Staub, F. C., & Stern, E. (2002). The nature of teachers' pedagogical content beliefs matters for students' achievement gains: Quasi-experimental evidence from elementary mathematics. Journal of educational psychology, 94(2), 344. doi: 10.1037/00220663.94.2.344 Strijbos, J. W., Narciss, S., & Dünnebier, K. (2010). Peer feedback content and sender's competence level in academic writing revision tasks: Are they critical for feedback perceptions and efficiency? Learning and Instruction, 20(4), 291-303. doi:10.1016/j.learninstruc.2009.08.008 Sweller, J. (2010). Cognitive load theory: Recent theoretical advances. In J. L. Plass, R. Moreno, & R. Brünken (Eds.), Cognitive load theory (pp. 29–47). New York: Cambridge University Press. Trees, A. R., & Jackson, M. H. (2007). The learning environment in clicker classrooms: student processes of learning and involvement in large university‐level courses using student response systems. Learning, Media and Technology, 32(1), 21-40. doi: 10.1080/17439880601141179 Van Valin, R. D. (2001). An introduction to syntax. New York: Cambridge University Press. Wade-Stein, D., & Kintsch, E. (2004). Summary Street: Interactive computer support for writing. Cognition and Instruction, 22(3), 333-362. doi: 10.1207/s1532690xci2203_3 Wang, Y. M., & Wang, Y. C. (2016). Determinants of firms' knowledge management system implementation: An empirical study. Computers in Human Behavior, 64, 829-842. doi: 10.1016/j.chb.2016.07.055

ACCEPTED MANUSCRIPT Formative Computer-Based Feedback in the University Classroom

34

Wegner, E., & Nückles, M. (2016). Training the brain or tending a garden? Students' metaphors of learning predict self-reported learning patterns. Frontline Learning Research, 3(4), 95-109. doi: 10.14786/flr.v3i4.212 Wirtz, M. A., & Caspar, F. (2002). Beurteilerübereinstimmung und Beurteilerreliabilität: Methoden zur Bestimmung und Verbesserung der Zuverlässigkeit von Einschätzungen mittels Kategoriensystemen und Ratingskalen [Interrater Agreement und interrater reliability: Methods for calculating and improving the reliability of ratings by category systems and rating scales]. Göttingen: Hogrefe. Wittwer, J., & Ihme, N. (2014). Reading skill moderates the impact of semantic similarity and causal specificity on the coherence of explanations. Discourse Processes, 51(1-2), 143-166. doi: 10.1080/0163853X.2013.855577 Wittwer, J., & Renkl, A. (2008). Why instructional explanations often do not work: A framework for understanding the effectiveness of instructional explanations. Educational Psychologist, 43(1), 49–64. doi:10.1080/00461520701756420 Wulff, D. H., Nyquist, J. D., & Abbott, R. D. (1987). Students' perceptions of large classes. New directions for teaching and learning, 1987(32), 17-30. doi: 10.1002/tl.37219873204 Ziai, R., de Kuthy, K., & Meurers, D. (2016). Approximating givenness in content assessment through distributional semantics. Proceedings of the Fifth Joint Conference on Lexical and Computational Semantics (*SEM) (pp. 209-218), Berlin, Germany: Association for Computational Linguistics.

ACCEPTEDFeedback MANUSCRIPT Running head: Formative Computer-Based in the University Classroom Appendix A Translated Instruction for the Revision Activities Feedback of your draft Now, we kindly ask you to revise your explanation. For that purpose, we additionally provide you with formative feedback about the quality of your explanation (see below). When you are finished, press SEND. C-Map Condition: The feedback contains the central concepts of your explanations, and their relations to other concepts. Separate concepts without a relation to the rest of the concept map, mark cohesion gaps of your explanation. 1) Identify the depicted cohesion gaps in your text, and close them in your revision. 2) Pay attention to potential missing concepts, which do not show up in your current feedback, and include them in your revision. Outline Condition: The feedback contains the central concepts of your explanations, and their relations to other concepts. Blank lines between the concept maps, mark cohesion gaps of your explanation. 1) Identify the depicted cohesion gaps in your text, and close them in your revision. 2) Pay attention to potential missing concepts, which do not show up in your current feedback, and include them in your revision. General Condition: The feedback contains information about the number of concepts, and the number of cohesion gaps your explanations. 1) Identify the mentioned cohesion gaps in your text, and close them in your revision. 2) Pay attention to potential missing concepts, which do not show up in your current feedback, and include them in your revision.

ACCEPTEDFeedback MANUSCRIPT Running head: Formative Computer-Based in the University Classroom

Figure 1. Original text passage of a student’s draft about instructional explanations and examples of the different types of feedback. Figure 1A presents the concept map feedback, Figure 1B presents the outline feedback, and Figure 1C presents the general feedback condition.

ACCEPTEDFeedback MANUSCRIPT Running head: Formative Computer-Based in the University Classroom Table 1 Conditions and Materials used in the study

Week 1

2

General Feedback

Concept Map Feedback

Outline Feedback

Condition

Condition

Condition

Lecture 1

Lecture 1

Lecture 1

Pre-Questionnaire

Pre-Questionnaire

Pre-Questionnaire

Draft on explanation 1

Draft on explanation 1

Draft on explanation 1

General Feedback

Concept Map Feedback

Outline Feedback

Revision of explanation 1

Revision of explanation 1

Revision of explanation 1

Perceived difficulty

Perceived difficulty

Perceived difficulty

Lecture 2

Lecture 2

Lecture 2

Draft on explanation 2

Draft on explanation 2

Draft on explanation 2

General Feedback

Concept Map Feedback

Outline Feedback

Revision of explanation 2

Revision of explanation 2

Revision of explanation 2

Note. Bold items varied across experimental conditions.

ACCEPTED MANUSCRIPT Formative Computer-Based Feedback in the University Classroom

2

Table 2 Means and Standard Deviations for our Dependent Measures Dependent Variables

General Feedback

Concept Map Feedback

Outline Feedback

Training Explanation 1 Perceived difficultya

3.62 (0.85)

3.24 (0.83)

3.81 (0.73)

Draft Number of

12.00 (3.76)

11.14 (3.69)

11.69 (4.32)

Draft Local Cohesion

.23 (.15)

.25 (.15)

.23 (.12)

Draft Global Cohesion

1.38 (0.76)

1.37 (0.75)

1.43 (0.78)

12.95 (4.14)

12.70 (4.49)

13.06 (5.19)

.20 (.12)

.18 (.14)

.19 (.13)

1.48 (0.78)

1.47 (0.79)

1.58 (0.82)

12.90 (4.50)

12.72 (4.26)

12.54 (4.21)

Draft Local Cohesion

.27 (.15)

.25 (.14)

.25 (.15)

Draft Global Cohesion

1.81 (0.72)

1.86 (0.77)

1.93 (0.75)

13.46 (4.49)

13.13 (4.87)

13.10 (4..21)

.24 (.15)

.21 (.13)

.22 (.13)

1.86 (0.73)

1.93 (0.75)

2.05 (0.85)

Sentences

Revision Number of Sentences Revision Local Cohesion Revision Global Cohesion Training Explanation 2 Draft Number of Sentences

Revision Number of Sentences Revision Local Cohesion Revision Global Cohesion

Note. aPerceived difficulty was measured on a scale from 1 = low to 5 = high.