Electronic outlining as a writing strategy: Effects on students' writing products, mental effort and writing process

Computers & Education 78 (2014) 352e366

Contents lists available at ScienceDirect

Computers & Education journal homepage: www.elsevier.com/locate/compedu

Electronic outlining as a writing strategy: Effects on students' writing products, mental effort and writing process €lle Leijten b, c, Paul A. Kirschner a Milou J.R. de Smet a, b, *, Saskia Brand-Gruwel a, Marie a

Open University in the Netherlands, Welten Institute, Research Centre for Learning, Teaching and Technology, P.O. Box 2960, 6401, DL Heerlen, The Netherlands University of Antwerp, Faculty of Applied Economics, Department of Management, Prinsstraat 13, 2000, Antwerp, Belgium c Research Foundation e Flanders (FWO), Belgium b

a r t i c l e i n f o

a b s t r a c t

Article history: Received 28 September 2013 Received in revised form 16 June 2014 Accepted 20 June 2014 Available online 5 July 2014

This study addresses to what extent and how electronic outlining enhances students' writing performance. To this end, the focus of this study is not only on students' final writing products but also on the organisation of the writing process (i.e., planning, translating, and reviewing) and perceived mental effort during writing. In addition, effects of repeated electronic outlining were examined. A combined within and between subjects design was implemented in which 93 10th-grade students wrote two argumentative texts with or without using electronic outlining. Analyses showed that using electronic outlining for planning and writing significantly improved the presentation of the argumentative structure. However, effects were less clear for correctly and completely establishing a text structure and no effects were found on the elaboration of students' argumentation. Process data showed that electronic outlining increased total process time, but no effect was found on students' overall planning and revision activities. Finally, self-reports showed no effect of electronic outlining on students' perceived mental effort. Nevertheless, repeated use of the same writing strategy enhanced writing fluency. © 2014 Elsevier Ltd. All rights reserved.

Keywords: Secondary education Writing strategies Writing process Electronic outlining Mental effort

1. Introduction In the current knowledge-driven society, writing elaborated and coherent texts is an important skill for both one's educational and professional career (U.S. Department of Education, 2003). The development of students' writing skills is therefore an explicitly formulated educational goal. At the same time, writing is a complex and demanding skill to master (Flower & Hayes, 1980; Hayes, 1996; Kellogg, 1994). The complexity of writing can be explained by the fact that writers must simultaneously perform a set of distinctive cognitive activities. During composition, writers must simultaneously plan, translate, and review their text; they should consider a content problem of what to write, and a rhetorical problem of how to express their ideas in a way that suits both the topic and the audience (Flower & Hayes, 1980; Lindgren & Sullivan, 2005; Torrance, Thomas, & Robinson, 2000). The alternation of these cognitive processes puts a heavy load on a writer's working memory. Empirical studies by Kellogg (1996) and Baddeley (2010), have shown that this working memory has only limited capacity to process and store information. Writers, therefore, often experience cognitive overload while composing a text and may be unable to adequately attend to any of these processes at all (Flower & Hayes, 1981; Kellogg, 1988). Such writing may be inefficient and might lead to poorly structured and/or incoherent texts. Kellogg (2008) showed that the efficiency of writing is affected by expertise as certain processes become automated with expertise (i.e., they no longer require cognitive processing). Moreover, expert writers make use of well-developed writing strategies to enhance their writing. Different studies have shown that these writing strategies, in particular outlining, may improve students' writing products and decrease mental effort during writing (e.g., Erkens, Kanselaar, Prangsma, & Jaspers, 2002; Kellogg, 1988, 1990; Kozma, 1991). However, until now, not much is known about how these effects are achieved. To understand how outlining exerts influence on writing, the three main components of the writing process e planning, translating, and reviewing e serve as a starting point in this study. The purpose of this study is to examine the effects of electronic outlining on students' organisation of the writing process and its influence on students' writing products and perceived mental effort. * Corresponding author. University of Antwerp, Faculty of Applied Economics, Department of Management, Prinsstraat 13 (C.459), B2000, Antwerp, Belgium. Tel.: þ32 3 265 4116. E-mail addresses: [email protected], [email protected] (M.J.R. de Smet), [email protected] (S. Brand-Gruwel), [email protected] (M. Leijten), [email protected] (P.A. Kirschner). http://dx.doi.org/10.1016/j.compedu.2014.06.010 0360-1315/© 2014 Elsevier Ltd. All rights reserved.

M.J.R. de Smet et al. / Computers & Education 78 (2014) 352e366

353

Traditionally, many studies have focussed on whether writing strategies work, but not on how they are used and how they work. Understanding how electronic outlining is used and how it affects the organisation of the writing process is an important first step which may help to fill this gap and begin to provide a deeper understanding of how and why electronic outlining works. Understanding what works in electronic outlining may subsequently help indicate where teachers can and should provide extra support in writing education. This is an important step towards developing effective pedagogical instruction for using electronic outlining in education. 1.1. The writing process Since the early 1980's, emphasis in writing research shifted from the writing product to the writing process, focussing mainly on the cognitive processes involved in writing so as to unravel what happens in the writer's mind during composition (Flower & Hayes, 1981; Hayes & Flower, 1980). From then on, several researchers attempted to grasp the complexity of the writing process in a model. The first and most prominent model is Hayes and Flower's (1980) cognitive model of the writing process which contains three main components: the task environment, the writer's long term memory and the writing process. Through the years, various revisions have been made of the initial Hayes and Flower model (Chenoweth & Hayes, 2001; Flower & Hayes, 1981; Hayes, 1996, 2012; Leijten, Van Waes, Schriver, & Hayes, 2014). However, as Hayes (2012) states, “despite its age, the [initial] model contains features that are still current in modern representation of writing” (p.370). For the purpose of this study, the writing process component from the original model is used as a theoretical base because it clearly distinguishes three subcomponent processes, namely planning, translating, and reviewing. This study focuses on the interaction between, and the management of these three subprocesses in the writing process. Hayes and Flower's model makes clear that the writing process is not so much a linear process but much more a recursive cyclical one in which planning, translating, and reviewing may occur at any time. They constantly alternate and interact with one another throughout composition (Flower & Hayes, 1981). Regarding the alternation of the subprocesses, several researchers (Braaksma, 2002; Graham & Harris, 2000; McCutchen, 2000; McCutchen, Covill, Hoyne, & Mildes, 1994) used the term ‘orchestration’ to emphasise the temporal management of the writing processes and the fact that the subprocesses in writing can be activated and coordinated by the monitor. Several studies (Beauvais, Olive, & Passerault, 2011; Berninger, Fuller, & Whitaker, 1996; Braaksma, Rijlaarsdam, Van den Bergh, & Van Hout-Wolters, 2004; Breetvelt, Van den Bergh, & Rijlaarsdam, 1994; Levy & Ransdell, 1995; McCutchen, 1988) have shown that the ability to manage and distribute the interacting subprocesses of writing is a decisive factor for both text quality and cognitive load. It might therefore be expected that writing performance can be enhanced by using writing strategies that help writers efficiently manage the different subprocesses during writing. 1.2. Writing strategies Student writers, who are considered to be novice writers, may especially benefit from using writing strategies that help manage orchestrating the writing process (Kozma, 1991; Torrance, Thomas, & Robinson, 1994). The premise underlying this is that novices profit most from strategies that divide the writing process into separate stages, allowing them to focus effort on one single subtask a time, reducing the number of simultaneous constraints (Kellogg, 2008). In this study, a novice is defined as a writer who may have sufficient domain or genre knowledge but lacks process and procedural skills to effectively and efficiently write a coherent text. For these novices, little of the writing process is automated and therefore they must devote close attention to a variety of tasks and processes simultaneously (Flower & Hayes, 1981; Kellogg, 2008). This study focuses on the effects of using a planning strategy which was expected to positively influence writing performance through not only organising but also generating content to set up an elaborated and structured text. In line with this, Olive and Passerault (2012) suggested that structuring ideas was important for generating new ideas. Also, Pouit and Golder (2002) found that in argumentative texts, students included more properties for the defended position when they drew up a list of ideas in advance. Finally, Walvoord et al. (1995) found that outlining was helpful for generating text. 1.3. Outlining as a writing strategy Outlining is possibly the most recommended planning strategy for novices to enhance writing performance (Galbraith, Ford, Walker, & Ford, 2005; Hayes, 2006; Murray, 2011). An outline is a specific type of text plan drawn up by the writer before fully elaborating a text. It is a vertical list of items that is organised in the sequence which the writer intends to use for the final text, using one or more levels of hierarchy (University of Chicago, 2003; Walvoord et al., 1995). Outlining allows generating, clustering, and ordering ideas at an early stage in the writing process, and forces writers to consider both hierarchical and structural relations. Piolat and Roussey (1996) found that when students set up an organised list of ideas, the chronology helped them to linearise their ideas during the translation phase leading to essays with significant higher grades. Outlining can improve text quality because it shifts the writer's focus from a lower level of text-bound considerations to a higher, more structural level, leading to better structured texts (Kozma, 1991). Outlining not only influences text quality, but according to Favart and Coirier (2006) the cognitive load induced during the writing process might be reduced when a text structure has been established in a prewriting phase. A written outline serves as an external representation of the plan and allows the writer to focus on non-planning activities during later phases of the writing process (Kellogg, 1988). Making a written outline prior to composing a full text could thus reduce cognitive load during writing (Collins & Gentner, 1980; Galbraith & Rijlaarsdam, 1999; Glynn, Britton, Muth, & Dogan, 1982). 1.4. Electronic outlining Although outlines are traditionally set up with pen and paper, writers can nowadays profit from using electronic outline tools embedded in standard word-processing programs, such as MS® Word (Deacon, Jaftha, & Horwitz, 2004; Kozma, 1991; Price, 1997). These electronic tools enable writers to easily create outlines in which they arrange the sequence and subordination of their ideas. In this study, the use of

354

M.J.R. de Smet et al. / Computers & Education 78 (2014) 352e366

such an electronic outline tool and the corresponding outline strategy is referred to as ‘electronic outlining’. The advantage of electronic outlining above using pen and paper lies in its increased flexibility in that it enables writers to (1) create better visible hierarchical structures through automatic formatting, (2) directly attach elaborated text to the different original headings from the outline, which automatically become part of the text, (3) present the outline and the full text simultaneously on the computer screen (cf. Erkens, Jaspers, Prangsma, & Kanselaar, 2005; Price, 1997), (4) fold and unfold parts of the text to selectively display what they are working on and hide intervening text, (5) navigate easily through the text on the basis of the outline, and (6) easily and continuously revise their outlines (i.e., insert, delete, promote, demote and move topics and subtopics). Price suggests that a traditional outline is a fixed document “that acts as a rigid blueprint the student must follow when drafting” (p. 410) while an electronic outline can be part of the ongoing writing- and thinking process. He states that the purpose of outlining (i.e., developing a meaningful structure for a text) becomes lost amid the use of pen and paper which made changing difficult. This study focuses on the effects of electronic outlining on students' argumentative writing. Results from previous studies (De Smet, Brand-Gruwel, Broekkamp, & Kirschner, 2012; De Smet, Broekkamp, Brand-Gruwel, & Kirschner, 2011) suggest positive effects of electronic outlining on the quality of students' argumentative texts. The tool helped students better present their texts' structure and decreased perceived mental effort. These studies showed that it was important to practise using the tool in order to benefit from its effects. However, these studies only focused on the effects on students' final writing products and not on how these texts were produced or how the outlines affected the organisation of the writing process. Moreover, in these studies, a control group was missing which hindered a clear differentiation between the effects of electronic outlining and a common learning effect of repeatedly performing a similar writing task. To understand the functioning of electronic outlining, this study examines the extent to which the effects of electronic outlining on text quality may be caused by differences in the organisation of the writing process and alleviation of attentional (over)load. Pre-task planning such as electronic outlining may result in greater fluency (cf. Ellis & Yuan, 2004) and reduce students' online planning and reviewing behaviour. Students could use their initial outlines as a guide while elaborating their text, which may influence the course of the writing process. Students who do not use electronic outlining in advance may lack such an explicit text structure, which might cause the writing process to be more fragmented and leading to a higher degree of recursion. That is, these students may need to consider content, structure and formulations in a more extensive way while writing. Less initial planning might affect pausing and revision behaviour in the further writing process. 1.5. Analysing the writing process The recursive nature of the writing process often leaves traces as pauses and revisions in the process output. Differences in pausing and revision behaviour have been found to indicate differences in the underlying cognitive processes in sentence production (Quinlan, Loncke, Leijten, & Van Waes, 2012; Van Waes, Leijten, & Quinlan, 2010) and full text production (Butterworth, 1980; Leijten, Janssen, & Van Waes, 2010; Schilperoord, 1996; Wengelin, 2005). Many researchers analysed pause patterns to study the underlying cognitive processes in writing and used different definitions of a pause. Spelman Miller (2005) defines a pause as a “visible trace of nonwriting activity” (p. 24) which, as such, offers measureable clues to cognitive processes such as planning activities. Wengelin (2005) takes the practical interference of the computer into account when defining a pause. According to her, a pause is a transition time between two keystrokes that is longer than the time that may be expected for a writer to find the next key. In her view, a pause threshold of 2000 ms is twice as long as the normal transition time between two keystrokes, even for the slowest typist and it is, therefore, sufficient to eliminate the practical problem of pauses between keystrokes. In this study, a pause is defined as a period of 2000 ms (2 s) or more (Wengelin, 2005) between two consecutive keystrokes. Although recent research has shown that there may be important cognitive activities between 200 ms en 2000 ms (Van Waes & Leijten, 2013) for this study, we were more interested in longer pauses as they are considered indicators of higher-level global planning processes (Schilperoord, 1996). Using a pause threshold of 2000 ms allows the elimination of shorter pauses that are caused by activities associated with lexical retrieval and other lower-level local planning. As discussed, electronic outlining may shift a writer's focus to a higher more structural level of planning (Kozma, 1991) which would suggest that electronic outlining leads to longer, globally-oriented pauses and reduces lower level planning and shorter pauses. Since the 1980's, researchers also focused on revision processes in writing research (Lindgren & Sullivan, 2006; Lindgren, Sullivan, & Spelman Miller, 2008). Matsuhashi (1982) defines a revision as an episode in which the writer stops writing and makes a change in the previously written text. The purpose of a revision is emphasised in the definition of Leijten and Van Waes (2006) who define revisions as changes in the text aiming at the text's content, wording and lay-out. The analysis and description of pausing and revision behaviour may be a first step towards understanding the effects of electronic outlining on students' cognitive strategies that subtend writing achievement. 1.6. Research questions and hypotheses The present study investigates the effect of electronic outlining on students' (1) writing products, (2) perceived mental effort, and (3) the organisation of the writing process. In addition, it examined the effect of repeated use of electronic outlining as a writing strategy. Based on the theoretical notions and prior studies on (electronic) outlining described above (De Smet et al., 2011; 2012; Kozma, 1991; Walvoord et al., 1995) it is expected (H1) that electronic outlining leads to better structured texts. Second, it is hypothesised (H2) that electronic outlining decreases perceived mental effort while writing (cf. Favart & Coirier, 2006; Kellogg, 2008). Furthermore, it is hypothesised (H3) that electronic outlining causes a less fragmented organisation of the writing process in which initial planning through electronic outlining reduces the need to plan or revise later in the writing process. Finally, based on previous studies (De Smet et al., 2012) it is expected (H4) that repeated electronic outlining enhances effective use of this strategy and thus leads to further improvement of text quality. Tool-practice might reduce the demands of tool appropriation to free up working memory space for the writing process.

M.J.R. de Smet et al. / Computers & Education 78 (2014) 352e366

355

2. Method 2.1. Participants Participants were 93 Dutch 10th-grade students from Higher General Secondary Education (HGSE). The group consisted of 38 male and 55 female participants between the age of 14 and 17 (M ¼ 15.50; SD ¼ 0.69). Four intact classes were randomly assigned to one of the three conditions in this experiment. In order to equalise the number of participants in each condition, the two largest classes were each assigned to one condition and the two smallest classes were both assigned to one and the same condition. Participation in the experiment was compulsory and part of students' regular curriculum. 2.2. Design The central research question was examined by means of a combined within-subjects and between-subjects design in which three conditions were compared. In each condition, students carried out two argumentative writing tasks with an interval of one week. According to the design as shown in Table 1, students were or were not required to use the electronic outline tool to make an outline before fully elaborating their text. This combined design allowed determination of the effects of electronic outlining while controlling for learning effects and possible pretest differences between conditions. The control condition (in which students used no electronic outlining for both tasks) helped to distinguish between the effects of electronic outlining and a common learning effect of repeatedly performing a similar writing task. Finally, the design allowed determination of the effects of repeatedly using electronic outlining (repeated outline condition). Tool- and strategy practice might enhance effective use of electronic outlining and enhance students' writing performance. 2.3. Materials 2.3.1. Questionnaire concerning control variables Before carrying out the writing tasks, all participants filled out a general self-report questionnaire which focused on students' (1) knowledge of available writing tools in their word processor (2) self-efficacy for argumentative writing (3) report grade in the Dutch language, and (4) their writing style. Questions on knowledge of available writing tools were partly based on the questionnaire used by Leijten (2007). Students indicated whether they had previously used any of 12 writing functions incorporated in their word processor (e.g., generating a table of contents, using headings, using the outline tool, etcetera). Questions on self-efficacy for argumentative writing were based on an existing self-efficacy test for argumentative writing (Braaksma, Rijlaarsdam, & Van den Bergh, 2009). Self-efficacy beliefs may predict actual writing performance (cf. Duijnhouwer, 2010) and are therefore important as a control variable. Cronbach's alpha for the self-efficacy questionnaire (19 items) in this study was a ¼ .92. A 10-point scale was used instead of the original 100-point scale used by Braaksma et al. because the Dutch educational rating system runs from 1 to 10. Students were therefore expected to be more familiar with this scale. Questions on students' writing style were based on a writing process questionnaire developed and tested by Kieft, Rijlaarsdam, and Van den Bergh (2006). Using a 5-point Likert scale, students indicated the degree to which they usually engage in planning (12 items, a ¼ .71) and revising activities (10 items, a ¼ .63) when composing an argumentative text. 2.3.2. Writing tasks Students carried out two argumentative writing tasks about two timely topics (i.e., surveillance cameras in inner city areas and mobile phone use (cf. Van Weijen, 2009)). Texts were all written in Dutch, which was students' L1. Apart from the specific topics and the explicit instruction to use the outline tool or not, all other task features (e.g., genre, goal, target audience, instructions, available time and information, topic difficulty, and students' involvement in the topic) were kept constant. To control for possible topic-effects, the topics of the writing tasks were counterbalanced. Students were randomly assigned to a specific order of topics. For each task, 75 min were given to write a text of 600e800 words; students were allowed to stop earlier when they completed the task before the end of the lesson. They all knew in essence how to write an argumentative task as this was part of their regular curriculum. The genre of argumentative writing is particularly interesting in this context, due to its complexity and the natural requirement to hierarchically connect and order ideas. In an argumentative text, writers should support their standpoint with arguments which in turn should be supported with subordinated arguments. However, writers often have problems establishing a well ordered coherent argumentative structure (Favart & Coirier, 2006). In argumentative writing, writers cannot structure their text through simple approaches such as causal, chronological, or spatial organisation. Instead, to satisfy complex argumentative constraints, they must transpose their often multidimensional mental structures into a linguistic sequence of sentences and an organised text (Coirier, Andriessen, & Chanquoy, 2000; Coirier, Favart, & Chanquoy, 2002; Erkens et al., 2005; McCutchen, 1987). Setting up an outline may help writers to organise their argumentative structure. Table 1 Design of the study.

Control condition (N ¼ 28) Single outline condition (N ¼ 29) Repeated outline condition (N ¼ 36)

Task 1

Task 2

Without outline tool Without outline tool With outline tool

Without outline tool With outline tool With outline tool

356

M.J.R. de Smet et al. / Computers & Education 78 (2014) 352e366

Students who were not required to use electronic outlining started writing their text immediately after receiving the instructions, while students using the tool spent the first part of the writing task on their outlines. No specific time instruction was given for setting up the outline. Students could divide their own time between electronic outlining and writing the full text. To guarantee high ecological validity, students performed the writing tasks in their own school's computer room in the presence of their own teachers. 2.3.3. Tool instruction Prior to the first writing task that required students to use the outline tool, students received a 10-min training in which the tool and its working were explained. For the repeated outline condition, the training was given prior to the first writing task, while for the single outline condition training was only given prior to the second task. The training was restricted to a technical instruction showing the available buttons and their functions, and not on how to use the outline tool to write an argumentative text so as to prevent differences between conditions caused by instruction instead of the effects of electronic outlining. The instruction was followed by a 5-min practice session where students individually practised using the outline tool. The tool used in this study is the outline function incorporated in the ‘view’ menu in MS® Word 2007 (see Fig. 1) which was available in the school's computer room and with which the students were familiar. The advantage of this specific outline tool is that it is directly available in the standard settings of a widely used word processor. Moreover, students were familiar with composing writing tasks in MS® Word, so they only needed specific instruction on the outline tool and not on the writing environment in general. 2.3.4. Keystroke logging via Inputlog Detailed data on the writing process was collected via the keystroke logging program Inputlog (Leijten & Van Waes, 2013). Inputlog is a research tool for logging and analysing writing process data in Windows™ environments. It gives researchers the opportunity to capture detailed information concerning the organisation of the writing process. Inputlog records students' writing activity on the computer and registers and stores, from beginning to end, all keystrokes and mouse movements and their distribution across time. The writing process can, thus, be studied in real time as it unfolds, enhancing the understanding of how a text develops. Using keystroke logging as an observational tool has several advantages above, for example, think aloud procedures or retrospective interviews in that (1) it yields a detailed picture of the writing process, (2) results from different respondents are quantitatively comparable, and (3) it is an ecological, nonintrusive method that does not interfere with the writing process or influence students' course of writing. 2.3.5. Retrospective self-reports on perceived mental effort To measure perceived mental effort, students indicated after each writing task how much mental effort they perceived during the composition of the task on a 5-point Likert scale (1 ¼ very little effort, 5 ¼ very much effort). The scale is based on the 9-point cognitive load scale developed and validated by Paas (1992). 2.4. Data analysis 2.4.1. Writing products The quality of the writing products was evaluated with an analytic assessment protocol for argumentative text quality. This protocol was adapted from Erkens et al. (2002) and distinguishes the degree to which students (1) correctly and completely establish text structure, (2) clearly present this structure, and (3) produce complex and elaborated argumentation. A high score on these three characteristics would indicate a well-structured and elaborated argumentative text. Credit for Total Text Structure was given when students differentiated between title, introduction, text body, and conclusion. Moreover, they needed to elaborate the different characteristics of each text part. A scoring rubric was used in which the presence (1) or absence (0) of these different characteristics were scored. The maximum total score for Total Text Structure was 16 points (see Appendix A). Credit for Structure Presentation was given when students established an explicit presentation of the argumentative structure by correctly and sufficiently distinguishing paragraphs, using headings and subheadings, by correctly and sufficiently using linguistic organisers such as connectives and anaphora (cf. Chanquoy, 1996) and by presenting the text towards the reader. The maximum score for Structure Presentation was 14 points (see Appendix A).

Fig. 1. Screen dump of outline function in MS® Word 2007.

M.J.R. de Smet et al. / Computers & Education 78 (2014) 352e366

357

The validity and reliability of this assessment instrument has been established by De Smet et al. (2011) who used the protocol for the same population and the same text genre. Moreover, to calculate interrater reliability for the present study, a random sample of 20 texts (i.e. 10.8% of the total sample) was independently scored by the first author and a research assistant who had a Master degree in modern Dutch language and literature and had experience with teaching Dutch language in secondary education. A significant correlation at the .01 level was found for both Total Text Structure (r ¼ .84) and for Structure Presentation (r ¼ .78) between both raters. The third and last variable for the writing products was Hierarchical Elaboration of Arguments which focused on students' ability to produce complex and elaborated argumentation. The procedure for scoring this variable started with segmentating all writing products. Each text was manually divided into segments with regard to distinct idea units based on their argumentative function. In some cases this would mean that sentences were split, based on argumentative and organisational markers, so that the different constituents of a sentence could be properly coded. Subsequently, all individual segments were evaluated in terms of their relevance to the main issue and were coded according to an extensive coding system (see Appendix B) which incorporated 13 different labels focussing on the segments' argumentative function. A unique code was assigned to each segment (cf. Reznitskaya, Kuo, Glina, & Anderson, 2009). In total 10% of all texts was independently coded by the first and second author. The number of segments per text ranged from 25 to 46, with an average of 35 segments per text, thus giving a large amount of data that were used to measure interrater reliability. Overall, interrater agreement was 78%. Eventually, analyses concentrated on the segments focussing on the arguments and their elaboration. Based on coded segments, students' hierarchical elaboration of arguments was measured. All arguments were counted and a mean score was calculated as follows. For each (elaborated) main argument, students could obtain 0 or 1 point (0 ¼ a main argument is presented without any elaboration, 1 ¼ a main argument is elaborated with at least one subargument). In other words, students could obtain a score for their argument only when an argument was hierarchically elaborated and underpinned. Finally, an average hierarchical elaboration score (between 0 and 1) was calculated for each student. 2.4.2. Writing process To analyse the effects of electronic outlining on students' organisation of the writing process, the focus is mainly on the pausing and revision behaviour during writing. Data collection via Inputlog allows observing, detecting, and analysing students' pausing-, and revision behaviour in extensive ways. To analyse students' pausing behaviour, a threshold of 2000 ms (2 s) was used to define a pause. First, to describe the writing process data, general process information (i.e., Total Process Time and the number of Words Produced per Minute (WPM)) was analysed and compared between conditions. The average number of WPM is a standard measure used to analyse verbal fluency (Chenoweth & Hayes, 2001; Gould, 1980; Kellogg, 1996; Le Bigot, Passerault, & Olive, 2012; Van Waes & Leijten, 2013). The analyses of students' pausing behaviour are based on Mean Pause Time and the Pause Ratio. Regarding students' revision activities, a global measure of revision (i.e., the Revision Ratio) was calculated. Students generally produce more words during the process of writing than what appears in the final text. The Revision Ratio takes this difference into account and consists of the total number of words in the final text divided by the total number of words produced during the writing process, subtracted from 1. Higher scores on the revision ratio indicate greater amounts of revision (cf. Baaijen, Galbraith, & De Glopper, 2010). In addition to these global process measures, it is important to study students' pausing behaviour per interval. Several researchers (e.g., Beauvais et al., 2011; Braaksma, 2002; Braaksma et al., 2004; Breetvelt et al., 1994) have shown that intervention effects on the writing process are best visible in the temporal organisation e the orchestration e of the writing process. Global analyses may not show these differences in the writing process. Electronic outlining as a preplanning strategy might above all influence the first part of the writing process. Therefore, the temporal organisation of writing activities was taken into account to analyse the writing process in a more sensitive way (Breetvelt et al., 1994; Olive, Kellogg, & Piolat, 2008). Because of technical issues, process information of two students (both on the second writing task) was excluded from analyses. However, for these two students, the product and self-report data were retained. We carefully controlled the writing process data by means of the replay function in Inputlog to insure that students actually made an outline before elaborating their text. These process data showed that in four cases students who were required to use electronic outlining did not start making an outline on their computer. Though explicitly instructed to do so, and in spite of supervision in classrooms, they started writing their texts immediately. For these students, both product and process information were excluded from analyses since these data do not reflect the effects of electronic outlining. For the first writing task, this was the case for one student and for the second, data from three students were excluded from analyses for this reason. The final data set used for analyses thus only includes texts that were performed as required by the design. 2.5. Statistics To study the effect of electronic outlining, an ANOVA was performed on the first writing task to compare the repeated outline condition with the control and single outline condition. On the second writing task, an ANOVA was performed to compare the effect of nonoutlining, single electronic outlining and repeated electronic outlining. Furthermore, ANOVAs were used to analyse the difference scores for the control and the single outline condition to compare the effects of electronic outlining with a mere learning effect of repeatedly performing a similar task. Repeated measures could not be used in this study, because there was no equal starting point for the three conditions. 3. Results 3.1. Control variables Before presenting the results related to the main research question, the general findings from the control variables are reported. Results of the ANOVA comparing the three conditions (see Table 2) revealed that there were no significant a priori differences between conditions. Moreover, no difference was observed in gender distribution between conditions (c2(2) ¼ 3.171; p ¼ .205).

358

M.J.R. de Smet et al. / Computers & Education 78 (2014) 352e366

Table 2 Means, Standard Deviations and ANOVA results of the Control Variables. Control M (SD) Knowledge of writing tools (max ¼ 12) Self-efficacy for argumentative writing (max ¼ 190) Planning strategy (max ¼ 60) Revision strategy (max ¼ 50) Report grade Dutch language (max ¼ 10) Student age (years)

7.23 147.15 33.72 31.04 7.00 15.65

(2.01) (11.61) (6.21) (4.68) (.61) (.85)

Single outline M (SD)

Repeated outline M (SD)

E^2

F

p

8.11 135.54 35.15 32.26 6.73 15.63

7.52 138.08 35.71 32.28 7.03 15.52

.034 .067 .021 .017 .034 .008

1.452 2.960 .823 .703 1.398 .341

.240 .057 .443 .498 .253 .712

(2.05) (25.44) (6.64) (4.45) (.85) (.63)

(1.73) (15.53) (4.31) (3.99) (.74) (.62)

3.2. Effects of electronic outlining on students' writing products Students' performance on the first writing task was analysed and compared to answer the first part of the research question concerning the effect of electronic outlining on students' writing products (see Table 3). 3.2.1. First writing task Students in the control condition and in the single outline condition carried out the first task without electronic outlining while students in the repeated outline condition used the tool during the first task to organise and write their text. Results of the ANOVA showed a significant effect of condition on Total Text Structure, F(2, 90) ¼ 11.312, p < .001, s2 ¼ .201. Post hoc analyses (Bonferroni) indicated that students in the repeated outline condition elaborated the different characteristics of an argumentative text better than students in both the control condition (p < .001) and the single outline condition (p < .001). No significant differences were found between both nonoutline conditions regarding Total Text Structure (p ¼ .989). Second, a significant effect was found for electronic outlining on Structure Presentation, F(2, 90) ¼ 8.972, p < .001, s2 ¼ .166. Post hoc analyses revealed that students in the repeated outline condition scored significantly higher on the variable Structure Presentation than students in both the control condition (p ¼ .001) and the single outline condition (p ¼ .003). This suggests that electronic outlining enhances the presentation of the argumentative structure. There were no significant differences between the two non-outline conditions regarding Structure Presentation (p ¼ .926). Finally, a marginal effect was found for Elaboration of Argumentation F(2, 90) ¼ 2.855, p ¼ .063, s2 ¼ .060. A post hoc analysis was conducted, but showed no significant differences between conditions. 3.2.2. Second writing task Students' second writing tasks were analysed using a between-subject analysis (ANOVA) comparing all three conditions. Focus is on the difference between the control condition and the single outline condition to determine the effect of electronic outlining as compared to a learning effect. Furthermore, the difference between the single outline condition and the repeated outline condition is examined to compare the effects of first time electronic outlining to the effects of repeatedly using this strategy for writing. Analyses of the second writing task showed a significant effect on Total Text Structure, (F(2, 88) ¼ 10.77, p < .001, s2 ¼ .197). Post hoc analyses revealed that students in the repeated outline condition scored significantly higher on Total Text Structure than both other conditions (p < .001 for the control condition; p ¼ .013 for the single outline condition). However, no difference was found between the control and the single outline condition (p ¼ .278). Regarding Structure Presentation, a significant effect of condition (F(2, 88) ¼ 28.39, p < .001, s2 ¼ .392) was found. Post hoc analyses revealed that the control condition scored significantly lower than both conditions using electronic outlining (p < .001 for both outline conditions). No significant differences were found between both outline conditions (p ¼ .388). This suggests that electronic outlining improves the presentation of the argumentative structure, but no increased beneficial effect was found for repeated use. Finally, analyses on the second writing task revealed no effects of electronic outlining on students' Elaboration of Argumentation (p ¼ .817). 3.2.3. Comparing difference scores: electronic outlining vs. learning effects Difference scores in this study indicate the amount of change between the first and the second writing task. Comparing difference scores between the control condition and the single outline condition helps distinguishing between common learning effects of repeatedly performing a similar writing task, and effects of electronic outlining.

Table 3 Mean scores and standard deviations of text quality on the first and second writing task (T1 ¼ Task 1; T2 ¼ Task 2). Control (N ¼ 28)

T1: T2: T1: T2: T1: T2:

Total Text Structure Total Text Structure Structure Presentation Structure Presentation Elaboration Arguments Elaboration Arguments

Single outline (N ¼ 29)

Repeated outline (N ¼ 36)

M

SD

M

SD

M

SD

8.46 8.04 7.86 7.68 0.68 0.73

2.69 2.67 1.88 1.96 0.15 0.19

8.55 8.96 8.07 10.46 0.71 0.73

2.35 1.94 2.40 1.72 0.21 0.21

10.89 10.63 9.89 11.09 0.79 0.76

2.05 2.13 2.11 1.85 0.20 0.17

M.J.R. de Smet et al. / Computers & Education 78 (2014) 352e366

359

Table 4 Means and standard deviations of perceived mental effort. Control

Task 1 (max ¼ 5) Task 2 (max ¼ 5)

Single outline

Repeated outline

M

SD

M

SD

M

SD

2.93 2.37

0.78 0.84

3.00 2.67

1.07 1.06

2.86 2.57

0.83 0.96

Regarding Total Text Structure, results from the ANOVA showed a marginally significant difference between the control and the single outline condition (F(2, 54) ¼ 1.51, p ¼ .058, s2 ¼ .045) suggesting a marginal beneficial effect for electronic outlining as compared to a simple learning effect. For Structure Presentation, results from the ANOVA showed a significant difference between the control condition and the single outline condition (F(1, 54) ¼ 22.64, p < .001, s2 ¼ .295) suggesting positive effects of electronic outlining for Structure Presentation over a simple learning effect. Finally, analyses revealed no significant differences for the variable Elaboration of Argumentation between the control and the single outline condition (p ¼ .320). 3.3. Effects of electronic outlining on students' perceived mental effort The second part of the research question focused on the effect of electronic outlining on students' perceived mental effort while carrying out the writing tasks. Results of students' self-reports on mental effort are presented in Table 4. Results from the ANOVA showed no significant differences between the three conditions both on the first (p ¼ .888) and the second writing task (p ¼ .531). Electronic outlining did not significantly affect perceived mental effort during writing. Similarly, comparing the difference scores between the control and the single outline condition, showed no significant effect of electronic outlining (p ¼ .555). However, within-subject analyses, comparing students' perceived mental effort over both writing tasks, pointed out that students in the control condition (t(25) ¼ 2.487, p ¼ .01) and students in the repeated outline condition (t(36) ¼ 2.157, p ¼ .019) reported a significant decrease in perceived mental effort over time. Students in the single outline condition reported only a marginal decrease in perceived mental effort over both writing tasks (t(28) ¼ 1.440, p ¼ .085). This suggests beneficial effects on perceived mental effort as a result of repeatedly using the same writing strategy (i.e., repeated electronic outlining or repeated non-outlining). 3.4. Effects of electronic outlining on students' writing process 3.4.1. General process information The final part of the research question focused on the effect of electronic outlining on students' organisation of the writing process. First, the Total Process Time and the number of Words Produced per Minute (WPM) were analysed and compared between conditions and over tasks (see Table 5). Participants were given a manimum of 75 min to write their texts, but they were allowed to stop earlier when they finished their text. 3.4.1.1. First writing task. Analyses between the three conditions on the first writing task revealed a significant main effect of condition on total process time (F(2, 90) ¼ 12.97, p < .001, s2 ¼ .224). Post hoc analyses showed that electronic outlining significantly increased the Total Process Time. Students in the repeated outline condition showed a significantly longer process time than students in both the control (p < .001) and the single outline condition (p ¼ .004). No differences were found between both non-outline conditions (p ¼ .270). It is, however, also relevant to relate the Total Process Time to the number of words in the final product to examine overall writing fluency. A marginal effect on WPM was found (F(2, 90) ¼ 2.53, p ¼ .086, s2 ¼ .054). A post hoc analysis, however, revealed no significant differences between conditions. Although students in the outline condition showed a significant longer process time, electronic outlining did not affected overall writing fluency. 3.4.1.2. Second writing task. Total Process Time on the second writing task was analysed and compared between conditions. Analyses on the second writing task revealed a significant effect for condition (F(2, 87) ¼ 11.76, p < .001, s2 ¼ .217). Post hoc analyses confirm the finding on the first writing task that electronic outlining significantly increases Total Process Time. Here, both conditions in which students used electronic outlining, showed a longer total process time as compared to the non-outline condition (single outline condition, p < .001; repeated outline condition, p ¼ .001). No significant difference was found between both outline conditions (p ¼ .527). Table 5 Means and standard deviations of total process time and words produced per minute (WPM). Control

T1: T2: T1: T2:

Total process time Total process time WPM WPM

Single outline

Repeated outline

M

SD

M

SD

M

SD

420 3300 320 0900 14.86 20.72

90 0900 110 1100 4.86 8.36

470 0900 460 4600 13.93 14.19

100 1700 120 4500 4.61 5.44

560 2400 430 2600 12.31 16.26

130 0200 110 3800 4.31 5.94

360

M.J.R. de Smet et al. / Computers & Education 78 (2014) 352e366

Table 6 Means and standard deviations of pausing- and revision behaviour. Control

T1: T2: T1: T2: T1: T2:

Pause ratio Pause ratio Mean pause time (s) Mean pause time (s) Revision ratio Revision ratio

Single outline

Repeated outline

M

SD

M

SD

M

SD

.44 .37 10.37 9.37 .24 .16

.11 .11 3.74 3.30 .12 .06

.44 .36 11.47 8.96 .24 .22

.10 .11 4.07 3.77 .10 .10

.46 .36 10.71 8.77 .22 .21

.10 .10 3.69 2.77 .09 .08

Analyses on the WPM showed a significant difference between conditions (F(2, 88) ¼ 7.10, p ¼ .001, s2 ¼ .140). Post hoc analyses revealed a significant effect of electronic outlining on the second writing task. Students in the control condition produced significantly more words per minute compared to the single outline condition (p ¼ .001) and the repeated outline condition (p ¼ .027). No significant difference was found between both outline conditions (p ¼ .444). 3.4.1.3. Comparing difference scores: electronic outlining vs. learning effects. Difference scores from students in the control and single outline condition were analysed and compared to examine the difference between electronic outlining and learning effects. Regarding Total Process Time, analyses showed a significant main effect for condition (F(2, 54) ¼ 16.17, p < .001, s2 ¼ .230) suggesting that the decrease in total process time was significantly larger for students in the control condition. Regarding difference scores on WPM, no significant differences were found between conditions (p ¼ .950). Although students in the control condition showed a larger decrease in total process time, their improvement in fluency did not differ from the single outline condition. 3.4.2. Pausing- and revision behaviour Data collection via Inputlog allowed analyses of students' pausing and revision behaviour (see Table 6). Because there was a significant difference between conditions on Total Process Time, it was important to use relative measures (e.g., Pause Ratio) instead of absolute measures (e.g., Total Pause Time, Number of Pauses) of the writing process. A longer process time may cause more pauses during the writing process. Using a relative measure, initial differences between conditions are controlled and the data become more comparable (cf. Leijten & Van Waes, 2005). The Pause Ratio indicates the relative time students spent pausing as opposed to active writing time. The higher the Pause Ratio, the more time students spent relatively on pausing and the less they spent on active writing. 3.4.2.1. First writing task. Analyses of the Pause Ratio revealed that there were no significant between-subject differences (p ¼ .588). Also, analyses regarding the Mean Pause Time in seconds, showed no difference between conditions (p ¼ .539). Finally, students' Revision Ratio was analysed and compared between conditions. Analyses showed no significant difference regarding students' revision ratio (p ¼ .768). Electronic outlining did not influence students' overall revision activities on this first writing task. 3.4.2.2. Second writing task. With respect to the Pause Ratio, analyses on the second task showed no significant differences between conditions (p ¼ .808). Moreover, no effect was found on the Mean Pause Time (p ¼ .773). However, analyses on the Revision Ratio showed a significant difference between conditions (F(2, 85) ¼ 5.03, p ¼ .009, s2 ¼ .106). Post hoc analyses revealed that students in the control condition had a significant lower Revision Ratio as compared to both the single outline (p ¼ .012) and repeated outline condition (p ¼ .035). Students who used electronic outlining as a writing strategy made here more revisions during writing, compared to students who did not use electronic outlining. 3.4.2.3. Comparing difference scores: electronic outlining vs. learning effects. No difference was found on the difference scores for Pausing Ratio (p ¼ .621) and Mean Pause Time (p ¼ .215). However, analyses on the difference scores for Revision Ratio showed a significant difference between the control and the single outline condition (F(1, 54) ¼ 4.19, p ¼ .046, s2 ¼ .073). Students in the control condition demonstrated a significantly larger decrease of revisions during writing as compared to the single outline condition. 3.4.3. Temporal organization of pausing activities In addition to global process measures, students' pausing behaviour (Number of Pauses and Pause Time) across intervals was studied. Electronic outlining as a preplanning strategy might above all influence the first part of the writing process. Therefore, the temporal Table 7 Means and standard deviation of pausing behaviour per interval. Control

Single outline

Repeated outline

M (SD)

M (SD)

M (SD)

Interval

1

2

3

4

5

1

2

3

4

5

1

2

3

4

5

T1: Nr of Pauses T2: Nr of Pauses T1: Pause Time (s) T2: Pause time (s)

24.54 (7.12) 18.93 (8.89) 4.22 (0.87) 4.58 (2.16)

22.77 (8.53) 15.57 (8.34) 4.36 (1.44) 3.83 (1.02)

22.00 (23.19) 14.71 (8.54) 4.80 (1.66) 4.13 (1.53)

23.19 (5.51) 15.57 (7.30) 4.73 (1.51) 4.28 (1.25)

17.85 (8.18) 12.43 (7.25) 4.50 (1.24) 4.18 (1.53)

26.41 (8.40) 27.66 (8.78) 4.39 (0.88) 5.35 (2.17)

24.90 (8.22) 24.21 (7.99) 4.16 (1.38) 3.76 (1.07)

23.62 (10.35 21.83 (10.10) 4.18 (1.00) 3.45 (0.62)

22.66 (8.03) 22.10 (9.50) 4.53 (1.38) 4.05 (1.01)

13.97 (6.38) 17.10 (9.51) 4.42 (1.18) 4.23 (1.16)

34.44 (10.99) 24.03 (9.62) 4.35 (0.91) 4.84 (1.76)

27.73 (8.12) 22.97 (10.86) 4.07 (1.03) 3.56 (0.69)

26.07 (9.37) 18.43 (8.56) 3.94 (0.84) 3.76 (1.03)

28.27 (10.06) 19.37 (8.05) 3.76 (0.80) 3.62 (0.95)

16.70 (9.08) 14.13 (7.69) 3.95 (1.02) 3.95 (1.13)

M.J.R. de Smet et al. / Computers & Education 78 (2014) 352e366

361

organisation of writing activities was taken into account to analyse the writing process in a more valid and sensitive way (Breetvelt et al. 1994; Leijten & Van Waes, 2006; Olive et al., 2008). In this study, each writing process was split into five equally long time intervals. In this study, five intervals were used because a detailed study of all the general analyses files in Inputlog showed that students spent on average 10 min outlining. Dividing a writing process of approximately 50 min (i.e., the average total process time in this sample of students using the outline tool) in five intervals results in five episodes of 10 min each, meaning, thus, that the first interval could roughly be considered as the outline interval. Regarding students' Pause Time, median pause times were used instead of mean pause times. Because all types of pausing times (e.g., within words, between sentences) were combined, the duration of the pauses varied extremely because times considerably differ between higher and lower level locations (Schilperoord, 1996; Spelman Miller, 2005). Due to this high deviation on pause times, mean pause times were less suitable for analyses and therefore a more stable measure, the median, was used. A mixed design ANOVA with repeated measures for the five intervals across time (factor interval) and with experimental condition as between subjects factor (factor condition) was conducted for the first and second writing task. Results are shown in Table 7. Analyses of the Number of Pauses per interval on the pretest (as shown in Fig. 2) showed a significant main effect of interval (Wilks Lambda ¼ .274, F(4, 79) ¼ 52,255, p < .001, s2 ¼ .726) indicating that the number of pauses significantly differed across intervals. Furthermore, a significant main effect of condition was found here (F(2, 82) ¼ 3.84, p ¼ .025, s2 ¼ .086). Post hoc analyses comparing the three conditions showed that the repeated outline significantly differed from the control condition (p ¼ .018) and the single outline condition (p ¼ .021). Electronic outlining increased the number of pauses overall. Finally, a significant interaction effect between interval and condition was found (Wilks Lambda ¼ .712 F(8, 158) ¼ 3,651, p ¼ .001, s2 ¼ .156). To understand where in the writing process conditions differed, difference scores between intervals (within conditions) were calculated and computed. Post hoc analyses of these difference scores revealed a significant difference between conditions from the first to the second interval (p ¼ .013) in which the repeated outline condition significantly differed from the control (p ¼ .003) and the single outline condition (p ¼ .012). Students using electronic outlining showed a significant decrease in the number of pauses between the first and the second interval while the number of pauses did not decrease for students who did not use electronic outlining. This suggests that electronic outlining as a preplanning strategy decreased online planning. On the second writing task, there is again a significant main effect of interval (Wilks Lambda ¼ .424, F(4, 81) ¼ 27,479, p < .001, s2 ¼ .726). Also, a significant main effect of condition was found (F(2, 84) ¼ 6.48, p ¼ .002, s2 ¼ .134). Post hoc analyses indicated that the control condition significantly differed from the single outline condition (p ¼ .001) and the repeated outline condition (p ¼ .031). Students in both outline conditions showed more pauses overall. However, on this second writing task, no significant interaction effect was found between condition and interval. The finding on the first task in which preplanning in the form of electronic outlining reduced online planning was not confirmed in this second task. Regarding the Median Pause Time per interval (see Fig. 3), results on the first writing task showed that there was no significant main effect of interval. A significant main effect of condition was found (F(2, 67) ¼ 3.70, p ¼ .030, s2 ¼ .100). Post hoc analyses revealed that the repeated outline condition significantly differs from the control condition (p ¼ .010) and marginally differs from the single outline condition (p ¼ .086). Electronic outlining decreased the Median Pause Time across overall. No significant interaction effect between interval and condition was found. On the second writing task, a significant main effect of interval was found (Wilks Lambda ¼ .715, F(4, 64) ¼ 6,39, p < .001, s2 ¼ .285). No significant main effect was found, however, of condition. On the second writing task, there was no longer a difference between students who did or did not use electronic outlining. Again, no significant interaction effect was found. 3.4.4. Repeatedly using the same writing strategy The results from the writing process analyses, as described above, suggest an appropriation effect for both the control and the repeated outline condition. Students in these two conditions seem to become more fluent across tasks. However, this does not seem to be the case for the single outline condition. Judging from this first impression, it would be interesting to examine the effect of repeatedly using the same writing strategy. Therefore, additional paired-samples t-tests for all three conditions were performed to examine the change within subjects across both writing tasks. Analyses on students' Total Process Time showed a significant decrease over both writing tasks for the control (t(27) ¼ 5.956, p < .001) and the repeated outline condition (t(30) ¼ 6.119, p < .001). No significant decrease in time was found for the single outline condition (p ¼ .333). A similar result was found for WPM. A significant increase in WPM over both writing tasks was found for students in the control (t(27) ¼ 4.743, p < .001) and the repeated outline condition (t(31) ¼ 4.161, p < .001). No significant difference was found for the single outline condition across tasks (p ¼ .314).

Fig. 2. Number of pauses per interval for the first and second writing task.

362

M.J.R. de Smet et al. / Computers & Education 78 (2014) 352e366

Fig. 3. Median pause time per interval for the first and second writing task.

Regarding Total Number of Pauses, analyses demonstrated a similar finding. Students in the control condition showed a decrease in total number of pauses (t(27) ¼ 5.549, p < .001) as did the repeated outline condition (t(30) ¼ 6.640, p < .001). Again, this decrease was not found for the single outline condition (p ¼ .186). With respect to Total Pause Time, a significant decrease was found for the control (t(27) ¼ 3.524, p ¼ .001), the single outline (t(27) ¼ 3.775, p < .001) and the repeated outline condition (t(30) ¼ 7.321, p < .001). Analyses of the Mean Pause Time, showed no significant decrease over writing tasks for the control condition (p ¼ .156), but a decrease was found for both the single outline (t(27) ¼ 2.950, p ¼ .003) and repeated outline condition (t(30) ¼ 3.689, p < .001). Students' Pause Ratio decreased for the control (t(27) ¼ 2.461, p ¼ .011), the single outline (t(30) ¼ 4.652, p < .001) and the repeated outline condition (t(30) ¼ 6.162, p < .001). During the second writing task, all conditions spent relatively more time actively writing compared to the first writing task. Finally, analyses from the paired-samples t-test revealed a significantly decreased Revision Ratio for the control condition (t(26) ¼ 4.040, p < .001). However, no significant difference was found across tasks for either the single outline (p ¼ .621) or the repeated outline condition (p ¼ .678). Both conditions using electronic outlining did not change across tasks regarding the revision ratio, though students in the control condition showed a decreased Revision Ratio indicating less revisions during the second writing task.

4. Discussion The goal of this study was to determine how electronic outlining affects students' writing. In order to do so, it examined the effects of repeated and single use of electronic outlining on the quality of students' argumentative texts, the organisation of the writing process and perceived mental effort. Consistent with the hypothesis (H1), beneficial effects were found for electronic outlining on the variable Structure Presentation which focused on reader orientation and correct and sufficient use of headings, paragraphs and connectives. On the first writing task, the repeated outline condition significantly outperformed both conditions in which students did not use electronic outlining, and on the second task both conditions in which students used electronic outlining scored significantly higher than students in the control condition. Furthermore, difference scores regarding structure presentation showed that across both tasks, there were positive effects of electronic outlining compared to mere learning effects. Electronic outlining helped to improve Structure Presentation, and although this technique explicitly encourages students to use headings in their text as the original headings from the outline become part of the elaborated text, additional analyses showed that the effects of electronic outlining on Structure Presentation remain intact when headings are discounted. This allows making more general conclusions about the role of electronic outlining on linguistic features beyond those relating to headings. Results showed, thus, that electronic outlining enhances the presentation of the argumentative structure. Second, on the first writing task, students in the repeated outline condition scored significantly higher than both non-outline conditions on the variable Total Text Structure, which focused on the presence of all argumentative characteristics in the text. Electronic outlining improved students' elaboration of the different characteristics of an argumentative text. On the second writing task, students in the repeated outline condition again scored significantly higher than the other two conditions. Regarding difference scores on Total Text Structure, students in the single outline condition showed a marginally greater improvement over time compared to the control condition suggesting a marginal positive effect of electronic outlining over a learning effect. However, students in the single outline condition who used the tool for the first time during the second task did not score significantly higher than the control condition. This is somewhat at odds with the hypothesis since one would expect this condition also to perform better when using electronic outlining, as was the case for the repeated outline condition. This may partly be accounted for by a lack of practice in electronic outlining. Previous research (De Smet et al., 2012) showed that beneficial effects of electronic outlining on text quality were most prominent for repeated tool-use. During the second writing task, students in the single outline condition used the tool for the first time which might explain why electronic outlining did here not improve their scores on Total Text Structure. However, it is not clear why electronic outlining did enhance Total Text Structure for students in the repeated outline condition when using the tool for the first time but not for students in the single outline condition. A priori differences between conditions should not be the cause of this result, since control variables for relevant pretest differences were analysed and no a-priori differences were found between conditions. An alternative explanation is that changing strategies over time might here have interfered with the learning process. Dembo and Praks Seli (2004) indicated that changing learning strategies requires considerable commitment, effort and time. Further research must then be done to investigate the effects of changing writing strategies over time.

M.J.R. de Smet et al. / Computers & Education 78 (2014) 352e366

363

Regarding students' Elaboration of Arguments, no effects of electronic outlining were found. This can probably be accounted for by the minimal instruction students received in this experiment. Instruction focused only on the technical aspects of electronic outlining and not on how to produce elaborated argumentation with an outline tool. Although argumentative writing and elaborated argumentation was part of students' regular curriculum, it was not explicitly repeated here. As a consequence, students may not have used the outline tool effectively for setting up an elaborated argumentative structure. For educational purposes, it would therefore be important to focus instruction on hierarchical and structural relations and integrate this instruction on elaborated argumentation with the technical outline instruction. Research has shown that for higher-order skills such as general problem solving, metacognitive processing and writing, integrated instruction is much more effective than separate instruction (Brown, 1997). Future studies should therefore offer students a more integrated instruction focussing not only on the technical aspects of tool use, but also on the theoretical principles of argumentative writing, setting up an argumentative structure and how an electronic outline tool can help planning and structuring a complex and hierarchical argumentation. In conclusion, the first hypothesis regarding the quality of students' writing products was partially confirmed. Although there are clear positive effects of electronic outlining on Structure Presentation, the effects are still indistinct regarding Total Text Structure and absent for Elaboration of Argumentation. Analyses of students' Perceived Mental Effort (H2) showed no significant differences between conditions on either the first or the second writing task. However, a significant beneficial effect was found for repeatedly using the same writing strategy. Practice in applying a specific writing strategy (electronic outlining, or not) decreased perceived mental effort over time. This is in line with other studies (Shaffer, Doube, €nboer, & Paas, 1998) which found that once a particular skill is acquired, automatic processing can & Tuovinen, 2003; Sweller, Van Merrie circumvent the limits of working memory. It is therefore important to practise using a specific strategy to decrease perceived mental effort while writing an argumentative text. Previous research (De Smet et al., 2012) concluded that beneficial effects on perceived mental effort resulted from repeated use of electronic outlining. However, the present study suggests that it is not only the effect of repeated electronic outlining, but the effect of repeatedly using the same writing strategy in general which reduces mental effort across writing tasks. Using electronic outlining for the first time, or changing strategies, did not reduce perceived mental effort. The process of strategy exploration and appropriation might here have interfered with the writing process. Since working memory is limited, when a learner is confronted with a complex task and a new tool designed to aid in carrying out the task, the learner may at first experience extraneous load caused by the tool. Van Bruggen, Kirschner, and Jochems (2002) found this, for example, when students were required to produce external visual representations that were actually intended to reduce load. Analyses regarding the hypothesis on students' writing process (H3) showed that electronic outlining significantly increased Total Process Time. On the first writing task, the repeated outline condition showed a significantly longer writing process than the control and the single outline condition. Similarly, on the second writing task, both outline conditions showed a significantly longer process time than the control condition. Students using electronic outlining spent on average more than 10 min longer on their writing task than students who did not use electronic outlining. This difference in Total Process Time might be the result of the time students need to plan and set up their outlines. It might, however, also be the case that students using electronic outlining think more deeply about what they are doing due to the requirement to plan and that they therefore showed a longer writing process, even when excluding the outlining phase. Regarding the latter, Kellogg (1988) indeed found that students' mean time spent writing was about seven minutes greater for students who outlined (excluding their prewriting time). Regarding the number of Words Produced per Minute, no significant effects were found on the first writing task; though students using electronic outlining showed significantly longer process times, their writing fluency did not differ from students in both non-outline conditions. Nevertheless, on the second writing task, students in the control condition produced significantly more words per minute compared to both outline conditions. Contrary to our hypothesis, students performing both tasks without electronic outlining became more fluent on the second writing task as compared to both outline conditions. Similar to findings in Kellogg's (1988) study, electronic outlining improved the quality of the writing product, but there was no evidence that electronic outlining improved the efficiency of the writing process. It is, however, important to note that the production of the outline is included in these overall analyses. Setting up an outline, which requires the determination of discrete elements, may initially involve high cognitive effort and may thus be a less fluent activity than directly starting the translation of a text; a strategy which may have become automated across time. This could consequently decrease students' average fluency on the whole writing task. Overall analyses do not take into account these possible differences in fluency across ‘sub phases’ of outlining and text elaboration (cf. Van Waes & Leijten, 2012). Future research may focus on these different phases of writing in order to confirm this finding. Regarding students' pausing behaviour, results on the Pause Ratio showed that there are no significant differences on both the first and the second writing task regarding students' overall pausing ratio. Also, regarding the Mean Pause Time, no effects of electronic outlining were found on the first and second writing task. Although it was hypothesised (H3) that electronic outlining might lead to less but longer pauses, instead of many short pauses, this was not confirmed by means of students' overall pausing behaviour. This is not surprising because intervention effects on the writing process are mostly not found on global measures but much more in the temporal organisation. This study, therefore, also analysed students' pausing behaviour on interval level. However, results from this temporal analysis did not provide clear answers to the question how electronic outlining influenced students' pausing behaviour. Results on the first writing task suggested that electronic outlining in advance decreased the number of pauses between the first and second interval once the outline was finished. In other words, preplanning through electronic outlining reduced online planning, but only immediately after the outlining phase. This was only a short effect. Moreover, this effect was not affirmed by the results on the second writing task. These analyses therefore not clearly provide an answer to the effects of electronic outlining on students' pausing behaviour. Analyses on students' revision behaviour showed no effects of electronic outlining on the first writing task, however, on the second writing task, both outline conditions showed a significantly higher Revision Ratio indicating more revisions during writing. Contrary to the hypothesis, students using electronic outlining revised more during writing than students who did not use this strategy. Although more revision is often seen as an indication for better writing, it was expected in this study that students using electronic outlining would need less revision since they planned their text more extensively in advance. Apparently, this was not the case. Students in the control condition, who wrote both texts without electronic outlining, may have focused less on revision during the second writing task because their writing became more automated over time without focussing extensively on planning or revising activities.

364

M.J.R. de Smet et al. / Computers & Education 78 (2014) 352e366

Analyses on the writing process revealed that practice in using a specific writing strategy (i.e., repeated electronic outlining, repeated direct writing) enhanced writing fluency in terms of total process time, words produced per minute, total number of pauses, total pause time and perceived mental effort. Results thus underline the importance of strategy practice. With respect to the last hypothesis on the effects of repeated electronic outlining (H4), it was found that repeated use showed beneficial effects on the writing product regarding Total Text Structure. However, as mentioned above, it may not be repeated electronic outlining that caused this, but rather repeated use of a writing strategy that enhanced writing fluency. In conclusion, results from this study suggest beneficial effects of electronic outlining on students' writing products. However, no clear answer was found on the question how these effects are achieved. To this end, the following adaptations could be made concerning the methodology to insure univocality of results found. First, future studies should elaborate instruction and not only focus on the technical aspects of electronic outlining but incorporate instruction on setting up a complex and elaborated argumentation. Furthermore in addition to the questionnaire concerning control variables, future studies should include a pretest writing task to insure that the performance profile of all three conditions is equal at the beginning. This may gain better insight in current results on Total Text Structure. Results from this study help understand how electronic outlining affects argumentative writing. However, not much is known about the quality and characteristics of these outlines. Future research should also analyse the outlines apart from the final text. Students' outlines may naturally differ in terms of completeness, elaborateness, and hierarchical organisation. A study of the General Analyses file from Inputlog revealed that students' outlines range from very generic to content specific (cf. Walvoord et al., 1995). Similar to Piolat and Roussey (1996), who distinguish different types of drafts (i.e., note drafts, organised drafts, and composed drafts), a distinction can be made for different types of outlines that may evoke different effects for the final text. Future studies should consider these different types of outlines, analysing them in terms of content and structure and relate this to the quality of the final texts. Finally, although keystoke logging has advantages for research purposes, it also causes some challenges. The data can measure the occurrence of pauses and revisions and show exactly when and how often they occur, but it remains difficult to relate these events to the underlying cognitive processes and the reasons for pausing and revision activities (Baaijen et al., 2012; Lindgren & Sullivan, 2005; Wengelin, 2005). The issue with cognitive processes is that they can never be directly observed. Future studies may therefore complement keystroke logging data with think aloud protocols (i.e., concurrent and retrospective) or with eye tracking data. This may allow researchers to examine why students perform a writing task in a certain way and where students focus on during pauses and revisions. This study suggests that electronic outlining has the potential to improve students' text quality and, thus, creates perspectives for further research on optimising the effects of electronic outlining and setting up an effective instruction for electronic outlining and argumentative writing in the classroom. Based on the findings in this study, it can be recommended for educational practice that future instruction regarding electronic outlining should provide sufficient practice and explicit strategy instruction. Although results were not univocal, it seemed that more pausing at the beginning of the writing process, reduced online planning which may result in better quality texts. Instruction should thus focus on the importance of initial planning and the support of electronic outlining in this preplanning process. Role of the funding source The funding source had no role in the study design; in the collection, analysis and interpretation of data; in the writing of the report; and in the decision to submit the article for publication. Acknowledgement This research project was generously funded by the Kennisnet Foundation, the knowledge centre for information and communication technologies and education in the Netherlands. Appendix A. Supplementary data Supplementary data related to this article can be found at http://dx.doi.org/10.1016/j.compedu.2014.06.010. References Baaijen, V. M., Galbraith, D., & De Glopper, K. (2010). Writing: the process of discovery. In S. Ohlsson, & R. Catrambone (Eds.), Proceedings of the 32nd Annual Conference of the Cognitive Science Society (pp. 1774e1779). Austin, TX: Cognitive Science Society. Baaijen, V. M., Galbraith, D., & De Glopper, K. (2012). Keystroke analysis: reflections on procedures and measures. Written Communication, 29, 246e277. http://dx.doi.org/ 10.1177/0741088312451108. Baddeley, A. (2010). Working memory. Current Biology, 20(4), 136e140. Beauvais, C., Olive, T., & Passerault, J.-M. (2011). Why are some texts good and others not? Relationship between text quality and management of the writing process. Journal of Educational Psychology, 103, 415e428. Berninger, V. W., Fuller, F., & Whitaker, D. (1996). A process model of writing development across life span. Educational Psychology Review, 8, 193e218. http://dx.doi.org/ 10.1007/BF01464073. Braaksma, M. A. H. (2002). Observational learning in argumentative writing. Unpublished doctoral dissertation. The Netherlands: University of Amsterdam. Braaksma, M. A. H., Rijlaarsdam, G., & Van den Bergh, H. (2009, August). Hypertext writing: Effects on writing processes and writing products. Paper presented at the EARLI Conference, Budapest, Hungary. Braaksma, M. A. H., Rijlaarsdam, G., Van den Bergh, H., & Van Hout-Wolters, B. H. A. M. (2004). Observational learning and its effects on the orchestration of writing processes. Cognition and Instruction, 22(1), 1e36. Breetvelt, I., Van den Bergh, H., & Rijlaarsdam, G. (1994). Relations between writing processes and text quality: when and how? Cognition and Instruction, 12(2), 103e123. Brown, A. (1997). Transforming schools into communities of thinking and learning about serious matters. American Psychologist, 52, 399e413. Butterworth, B. L. (1980). Evidence from pauses in speech. In B. L. Butterworth (Ed.), Language production 1: Speech and talk (pp. 155e176). Londen, England: Academic Press. Chanquoy, L. (1996, October). Connectives and argumentative text: A developmental study. Paper presented at the first international workshop on argumentative text processing, Barcelona, Spain. Chenoweth, N. A., & Hayes, J. R. (2001). Fluency in writing: generating text in L1 and L2. Written Communication, 18(1), 80e98. Coirier, P., Andriessen, J. E. B., & Chanquoy, L. (2000). From planning to translating: the specificity of argumentative writing. In P. Coirier, & J. E. B. Andriessen (Eds.), Foundations of argumentative text processing (pp. 1e29). Amsterdam, the Netherlands: Amsterdam University Press.

M.J.R. de Smet et al. / Computers & Education 78 (2014) 352e366

365

Coirier, P., Favart, M., & Chanquoy, L. (2002). Ordering and structuring ideas in text: from conceptual organization to linguistic formulation. European Journal of Psychology of Education, 17(2), 157e175. Collins, A., & Gentner, D. (1980). A framework for a cognitive model of writing. In L. W. Gregg, & E. R. Steinberg (Eds.), Cognitive processes in writing (pp. 51e72). Hillsdale, NJ: Lawrence Erlbaum Associates. Deacon, A., Jaftha, J., & Horwitz, D. (2004). Customising Microsoft Office to develop a tutorial learning environment. British Journal of Educational Technology, 35, 223e234. Dembo, M. H., & Praks Seli, H. (2004). Students' resistance to change in learning strategies courses. Journal of Developmental Education, 27(3), 2e11. De Smet, M. J. R., Brand-Gruwel, S., Broekkamp, H., & Kirschner, P. A. (2012). Write between the lines: electronic outlining and the organization of text ideas. Computers in Human Behavior, 28(6), 2107e2116. De Smet, M. J. R., Broekkamp, H., Brand-Gruwel, S., & Kirschner, P. A. (2011). Effects of electronic outlining on students' argumentative writing performance. Journal of Computer Assisted Learning, 27(6), 557e574. Duijnhouwer, H. (2010). Feedback effects on students' writing motivation, process, and performance. Unpublished doctoral dissertation. the Netherlands: University of Utrecht. Ellis, R., & Yuan, F. (2004). The effects of planning on fluency, complexity and accuracy in second language narrative writing. Studies in Second Language Acquisition, 26, 59e84. http://dx.doi.org/10.10170/S0272263104261034. Erkens, G., Jaspers, J. G. M., Prangsma, M. E., & Kanselaar, G. (2005). Coordination processes in computer supported collaborative writing. Computers in Human Behavior, 21, 463e486. Erkens, G., Kanselaar, G., Prangsma, M. E., & Jaspers, J. (2002). Using tools and resources in computer supported collaborative writing. In G. Dr. Stahl (Ed.), Computer support for collaborative learning: Foundations for a CSCL community (pp. 389e399). Hillsdale, NJ: Lawrence Erlbaum Associates. Favart, M., & Coirier, P. (2006). Acquisition of the linearization process in text composition in third to ninth graders: effects of textual superstructure and macrostructural organization. Springer, 35, 305e328. Flower, L. S., & Hayes, J. R. (1980). The dynamics of composing: making plans and juggling constraints. In L. W. Gregg, & E. R. Steinberg (Eds.), Cognitive processes in writing: An interdisciplinary approach. Hillsdale, NJ: Lawrence Erlbaum Associates. Flower, L. S., & Hayes, J. R. (1981). A cognitive process: theory of writing. College Composition and Communication, 32(4), 365e387. Galbraith, D., Ford, S., Walker, G., & Ford, J. (2005). The contribution of different components of working memory to knowledge transformation during writing. Educational Studies in Language and Literature, 5, 113e145. Galbraith, D., & Rijlaarsdam, G. (1999). Effective strategies for the teaching and learning of writing. Learning and Instruction, 9, 93e108. Glynn, S. M., Brittton, B. K., Muth, K. D., & Dogan, N. (1982). Writing and revising persuasive documents: cognitive demands. Journal of Educational Psychology, 74(4), 557e567. Graham, S., & Harris, K. R. (2000). The role of self-regulation and transcription skills in writing and writing development. Educational Psychologist, 35(1), 3e12. Gould, J. D. (1980). Experiments on composing letters: some facts, some myths, and some observations. In L. W. Gregg, & E. R. Steinberg (Eds.), Cognitive processes in writing (pp. 97e127). Hillsdale, NJ: Lawrende Erlbaum Associated. Hayes, J. R. (1996). A new framework for understanding cognition and affect in writing. In C. M. Levy, & S. Ransdell (Eds.), The science of writing: Theories, methods, individual differences, and applications (pp. 1e28). Mahwah, NJ: Erlbaum. Hayes, J. R. (2006). New directions in writing theory. In C. A. MacArthur, S. Graham, & J. Fitzgerald (Eds.), Handbook of writing research (pp. 28e40). New York, NY: The Guilford Press. Hayes, J. R. (2012). Modeling and remodeling writing. Written Communication, 29, 369e388. http://dx.doi.org/10.1177/0741088312451260. Hayes, J. R., & Flower, L. S. (1980). Identifying the organization of writing processes. In L. W. Gregg, & E. R. Steinberg (Eds.), Cognitive processes in writing (pp. 3e30). Hillsdale, NJ: Erlbaum. Kellogg, R. T. (1988). Attentional overload: effects of rough draft and outline strategies. Journal of Experimental Psychology: Learning, Memory and Cognition, 14, 355e365. Kellogg, R. T. (1990). Effectiveness of prewriting strategies as a function of task demands. American Journal of Psychology, 103, 327e342. Kellogg, R. T. (1994). The psychology of writing. New York, NY: University Press. Kellogg, R. T. (1996). A model of working memory in writing. In C. M. Levy, & S. Ransdell (Eds.), The science of writing: Theories, methods, individual differences, and applications (pp. 57e71). Mahwah, NJ: Erlbaum. Kellogg, R. T. (2008). Training writing skills: a cognitive developmental perspective. Journal of Writing Research, 1(1), 1e26. Kieft, M., Rijlaarsdam, G., & Van den Bergh, H. (2006). Writing as a learning tool: testing the role of students' writing strategies. European Journal of Psychology of Education, 12(1), 17e34. Kozma, R. (1991). The impact of computer-based tools and embedded prompts on writing processes and products of novice and advanced college writers. Cognition and Instruction, 8(1), 1e27. Le Bigot, N., Passerault, J.-M., & Olive, T. (2012). Visuospatial processing in memory for word location in writing. Experimental Psychology, 59(3), 138e146. http://dx.doi.org/ 10.1027/1618-3169/a000136. Leijten, M. (2007). Writing and speech recognition. Observing error correction strategies of professional writers. Unpublished doctoral dissertation. Utrecht, the Netherlands: LOT. Leijten, M., Janssen, D., & Van Waes, L. (2010). Error correction strategies of professional speech recognition users: three profiles. Computers in Human Behavior, 26, 964e975. Leijten, M., & Van Waes, L. (2005). Writing with speech recognition: the adaptation process of professional writers with and without dictating experience. Interacting with Computers, 17, 736e772. Leijten, M., & Van Waes, L. (2006). Inputlog: new perspectives on the logging of on-line writing. In K. P. H. Sullivan, & E. Lindgren (Eds.), Studies in writing: Vol. 18. Computer keystroke logging and writing: Methods and applications (pp. 73e94). Oxford, England: Elsevier. Leijten, M., & Van Waes, L. (2013). Keystroke logging in writing research: using Inputlog to analyze and visualize writing processes. Written Communication, 30(3), 358e392. http://dx.doi.org/10.1177/0741088313491692. Leijten, M., Van Waes, L., Schriver, K., & Hayes, J. R. (2014). Writing in the workplace: constructing documents using multiple digital sources. Journal of Writing Research, 5(3), 285e337. Levy, C. M., & Ransdell, S. E. (1995). Is writing as difficult as it seems? Memory & Cognition, 23, 767e779. http://dx.doi.org/10.3758/BF03200928. Lindgren, E., & Sullivan, K. P. H. (2005). Analysing online revision. In G. Rijlaarsdam (Ed.), Computer keystroke logging and writing: Methods and applications (pp. 157e188). UK/ Nederland: Elsevier. Lindgren, E., & Sullivan, K. P. H. (2006). Writing and the analysis of revision: an overview. In K. P. H. Sullivan, & E. Lindgren (Eds.), Studies in writing: Vol. 18. Computer keystroke logging and writing: Methods and applications (pp. 31e44). Oxford: Elsevier. Lindgren, E., Sullivan, K. P. H., & Spelman Miller, K. (2008). Development of fluency and revision in L1 and L2 writing in Swedish high school years 8 and 9. International Journal of Applied Linguistics, 156, 133e151. Matsuhashi, A. (1982). Explorations in real-time production of written discourse. In M. Nystrand (Ed.), What writers know. The language, process, and structure of written discourse (pp. 269e290). New York, NY: Academic Press. McCutchen, D. (1987). Children's discourse skills: form and modality requirements of schooled writing. Discourse Processes, 10(3), 267e286. McCutchen, D. (1988). Functional automaticity in children's writing: a problem of metacognitive control. Written Communication, 5, 306e324. http://dx.doi.org/10.1177/ 0741088388005003003. McCutchen, D. (2000). Knowledge, processing, and working memory: Implications for a theory of writing. Educational Psychologist, 35(1), 13e23. McCutchen, D., Covill, A., Hoyne, S. H., & Mildes, K. (1994). Individual differences in writing: implications of translating fluency. Journal of Educational Psychology, 86, 256e266. Murray, R. (2011). How to write a thesis. Maidenhead, UK: Open University Press. Olive, T., Kellogg, R. T., & Piolat, A. (2008). Verbal, visual and spatial working memory demands during text composition. Applied Psycholinguistics, 19, 669e687. Olive, T., & Passerault, J.-M. (2012). The visuospatial dimension of writing. Written Communication, 29, 326e344. http://dx.doi.org/10.1177/0741088312451111. Paas, F. G. (1992). Training strategies for attaining transfer of problem-solving skill in statistics: a cognitive-load approach. Journal of Educational Psychology, 84, 429e434. Piolat, A., & Roussey, J.-Y. (1996). Students' drafting strategies and text quality. Learning and Instruction, 6(2), 111e129. Pouit, D., & Golder, C. (2002). Idea retrieval in argumentative text writing by 11e18 year old students. European Journal of Psychology of Education, 17, 309e320. Price, J. (1997). Electronic outlining as a tool for making writing visible. Computers and Composition, 14, 409e427. Quinlan, T., Loncke, M., Leijten, M., & Van Waes, L. (2012). Coordinating the cognitive processes of writing: the role of the monitor. Written Communication, 29(3), 345e368. Reznitskaya, A., Kuo, L., Glina, M., & Anderson, R. C. (2009). Measuring argumentative reasoning: what's behind the number? Learning and Individual Differences, 19, 219e224. Schilperoord, J. (1996). It's about time. Temporal aspects of cognitive processes in text production. Amsterdam/Atlanta: Rodopi. Shaffer, D., Doube, W., & Tuovinen, J. (2003). Applying cognitive load theory to computer science education. In M. Petre, & D. Budgen (Eds.), Proceedings of the Joint Conference for Evaluation and Assessment in Software Engineering and the Psychology of Programming Interest Group 2003 (pp. 333e346).

366

M.J.R. de Smet et al. / Computers & Education 78 (2014) 352e366

Spelman Miller, K. S. (2005). Pausing, productivity and the processing of topic in online writing. In G. Rijlaarsdam (Ed.), Computer keystroke logging and writing: Methods and applications (pp. 131e155). UK/Nederland: Elsevier. €nboer, J. J. G., & Paas, F. G. W. C. (1998). Cognitive architecture and instructional design. Educational Psychology Review, 10(3), 251e296. Sweller, J., Van Merrie Torrance, M., Thomas, G. V., & Robinson, E. J. (1994). The effect of outlining and rough drafting strategies on the quality of short essays. Paper presented at EARLI SIG Writing Conference, Utrecht, the Netherlands. Torrance, M., Thomas, G. V., & Robinson, E. J. (2000). Individual differences in undergraduate essay-writing strategies: a longitudinal study. Higher Education, 39, 181e200. University of Chicago. (2003). The Chicago manual of style (15th ed.). Chicago, IL: University of Chicago Press. U.S. Department of Education. Institute of Education Sciences. National Center for Education Statistics. (2003). The Nation's report card: Writing 2002, NCES 2003e529, by H. R. Persky, M. C. Daane, and Y. Jin. Washington, DC. Van Bruggen, J. M., Kirschner, P. A., & Jochems, W. (2002). External representation of argumentation in CSCLS and the managment of cognitive load. Learning and Instruction, 12(1), 121e138. Van Waes, L., & Leijten, M. (2012, July). Writing fluency revisited. Paper presented at EARLI SIG Writing Conference, Porto, Portugal. Van Waes, L., & Leijten, M. (2013). Vlot schrijven: Een multidimensioneel perspectief op ‘writing fluency’ [Fluent writing: A multidimensional perspective on writing fluency]. Tijdschrift voor Taalbeheersing, 35(2), 160e182. Van Waes, L., Leijten, M., & Quinlan, T. (2010). Reading during sentence composing and error correction: a multilevel analysis of the influences of task complexity. Reading and Writing: An Interdisciplinary Journal, 23(7), 803e834. Van Weijen, D. (2009). Writing processes, text quality, and task effects. Empirical studies in first and second language writing. Unpublished doctoral dissertation. Utrecht, the Netherlands: LOT. Walvoord, B. E., Anderson, V. J., Breihan, J. R., McCarthy, L. P., Robison, S. M., & Sherman, A. K. (1995). Functions of outlining among college students in four disciplines. Research in the Teaching of English, 29, 390e421. Wengelin, A. (2005). Examining pauses in writing: theory, methods and empirical data. In G. Rijlaarsdam (Ed.), Computer keystroke logging and writing: Methods and Applications (pp. 107e130). UK, Nederland: Elsevier.