Variability in reading ability gains as a function of computer-assisted instruction method of presentation

Computers & Education 55 (2010) 209–217

Contents lists available at ScienceDirect

Computers & Education journal homepage: www.elsevier.com/locate/compedu

Variability in reading ability gains as a function of computer-assisted instruction method of presentation Erin Phinney Johnson *, Justin Perry, Haya Shamir Waterford Research Institute, 55 W. 900 S., Salt Lake City, UT 84101, USA

a r t i c l e

i n f o

Article history: Received 16 November 2009 Received in revised form 5 January 2010 Accepted 10 January 2010

Keywords: Interactive learning environments Elementary education

a b s t r a c t This study examines the effects on early reading skills of three different methods of presenting material with computer-assisted instruction (CAI): (1) learner-controlled picture menu, which allows the student to choose activities, (2) linear sequencer, which progresses the students through lessons at a pre-specified pace, and (3) mastery-based adaptive sequencer, which progresses students through lessons based on whether or not the student has mastered the given skill. Preschool- and kindergarten-aged children (n = 183) were randomly assigned to one of the three CAI groups and spent 40 min a week, for 13 weeks, using the software program in a computer lab. An additional control group of students attending typical preschool or kindergarten received no CAI. ANCOVA results examining post-test reading ability sum score, covarying pre-test score, indicated that the mastery-based sequencer group significantly outperformed the learner-control and control groups, but was not statistically different from the linear sequence group. Analysis by task, rather than overall reading score, revealed significantly better performance for the linear sequence group over controls and picture menu group on the Initial Sound Fluency task, while the mastery-based sequencer group outperformed all three other groups on Non Word Fluency. In sum, these results suggest that the use of a sequencer is a very important element in presenting computerized reading content for young children. Ó 2010 Elsevier Ltd. All rights reserved.

1. Introduction Computer technology has changed education by allowing students and educators access to a wider range of resources to suit their needs. Computer-assisted instruction (CAI) in the classroom allows for a dynamic presentation of material, individualized instruction, and a level of engagement in the learning process that may not be possible in a more traditional classroom setting.1 CAI can provide immediate feedback regarding correct responses, reinforcement where appropriate, and modeling when needed. These benefits have been related to substantive student gains in knowledge (Lepper & Gurtner, 1989; Wenglinsky, 1998) and many help eliminate certain impediments to effective intervention among younger students and children at-risk (e.g., Fish et al., 2008). In early classrooms, reading instruction has long been the primary focus of computerized instruction, likely because reading acquisition has been shown to be closely related to later academic achievement in a variety of subjects (National Reading Panel Report, 2000). Not surprisingly, however, not all CAI is created equal. Just as changes to teaching technique can make a huge difference in student growth in the classroom (e.g., Bloom, 1984), the technique used to present CAI material—in addition to the effectiveness of specific programs—should always be considered. While a large number of studies have demonstrated that using CAI can be effective for supporting reading development (Wise, Ring, & Olson, 2000; MacArthur, Ferretti, Okolo, & Cavalier, 2001; Hecht & Close, 2002; Cassady and Smith; 2004) and phonological awareness (Foster, Erickson, Forster, Brinkman, & Torgensen, 1994; Macaruso & Walker, 2008; Mitchell & Fox, 2001; Reitsma & Wesseling, 1998), there is still a great deal of skepticism that computerized systems can provide reading instruction on the level of a human teacher in the classroom (e.g., Blok, Oostdam, Otter, & Overmaat, 2002). It is our opinion that any use of CAI in the classroom must be complemented both by a strong instructional framework and by faculty support. Computer use, as we discuss it here, should be considered as a supplement to the more traditional classroom.

* Corresponding author. Address: Waterford Research Institute, 55 W. 900 S., Salt Lake City, UT 84101, USA. Tel.: +1 (801) 349 2369; fax: +1 (801) 363 1508. E-mail addresses: [email protected] (E.P. Johnson), [email protected] (J. Perry), [email protected] (H. Shamir). 1 CAI is used here as a blanket term to cover any technology use in the classroom. 0360-1315/$ - see front matter Ó 2010 Elsevier Ltd. All rights reserved. doi:10.1016/j.compedu.2010.01.006

210

E.P. Johnson et al. / Computers & Education 55 (2010) 209–217

While the effects of computerized instruction on reading ability have usually been shown to be positive, some studies have not closely examined the effects of using different methods of teaching through computers. The need to distinguish between types of CAI has increased in recent years, too, as computer technology has grown to incorporate a wider range of educational theories and instructional styles. Besides traditional, question-and-answer-based CAI, for instance, there is a great deal of software under the category of what researchers call ‘‘talking books” or ‘‘multimedia stories” (see Verhallen, Bus, & de Jong, 2006). These programs, as Wood, Pillinger, and Jackson (2010) describe, have typically been used by teachers as breaks from the routine of classroom instruction rather than as instructional tools; as a result, less research has focused on such products than their promise has merited (see Mayer & Anderson, 1991, 1992; Wood et al., 2010). Furthermore, because the instructional aspects of ‘‘talking book” software are focused primarily on reading strategies and comprehension, data gathered regarding such programs’ effectiveness is difficult to compare to the more basic skill improvements sought and measured by non-storybook CAI. For these reasons, we chose not to include ‘‘talking book” software in the current study. Most newer research on CAI has focused on programs that seek to combine short reading passages with more direct learner involvement, generally with the goal of emulating the instructional methods of a human tutor. Park and Lee (2007), for instance, have distinguished been three major types of ‘‘tutoring” CAI: (1) macro-level software, which seeks to match instruction based on a student’s goals, general ability, and achievement levels; (2) micro-level software, which diagnoses a student’s learning needs during the course of instruction and guides the learning process accordingly; and (3) Aptitude Treatment Interaction software, which guides instruction based on a student’s identified learner characteristics (Park & Lee, 2007). Within that framework, CAI can be considered an Integrated Learning System (which typically adapts instruction on a macro-level), a Computer-Managed Instructional System (also adapts on a macro-level) or an Intelligent Tutoring System (typically adapts according to ATI or on a micro-level). Aside from these more ‘‘sequence-based” systems, a learner-controlled or hypermedia program gives learners full or partial control over the instruction they receive from the computer (Snow, 1980). These learner-controlled systems differ substantially as well, usually according to the level of independence the learner is given. Levels of learner control include: (1) complete independence and self-direction; (2) imposed tasks, where the learner controls variables like the sequence, scheduling and pace; and (3) fixed tasks, where the learner controls only the pace of instruction. This paper will first discuss these various types of CAI in more detail, providing background information and previous effectiveness studies. Following this analysis, we present an empirical study designed to compare various kinds of CAI on reading gains in the early classroom. The following types of instruction were tested in the current study: (1) a learner-controlled system that imposes tasks but allows the learner to control sequence and pace; (2) a macro-level adaptive system that chooses its sequence based on students’ ability levels; and (3) a second adaptive system that requires mastery of a subject before the student can continue. The final system may fit somewhat with the ‘‘micro-level” designation used by Park and Lee; however, the program is not as flexible as certain other systems, including some Intelligent Tutors. These three system types were chosen both because they represent a broad spectrum of CAI and because all three are currently in use in early-reading classrooms.

2. Variations in CAI 2.1. Learner-controlled systems Self-directed CAI systems are used commonly by older students, usually in environments in which the student explores a topic nonlinearly by following hyperlinks to associated text and media (Scheiter & Gerjets, 2007). Among younger students (grades kindergarten through 2nd), self-directed systems are generally simpler, and typically include some sort of picture menu from which a child chooses the next activity he or she wishes to complete. Picture menus are commonly found in software available online, such as at PBSKids.org, in addition to being included in programs sold to schools (e.g. EducationCity). Learner control allows students to play activities that interest them repeatedly, or to play once or never at all. Because the student is in control of which activity will come next, he or she is usually highly engaged. A number of cognitive researchers have supported the idea that learner control is an effective way for students to learn new material (Merrill, 1975; Reigeluth & Stein, 1983). Cognitive models have proposed that, because the individualization of instruction (i.e., learner control) allows each learner to advance on his or her own initiative (Brown, Collins, & DuGuid, 1989; Merrill, 1975; Reigeluth & Stein, 1983), students are able to process information more deeply and therefore gain a better understanding of the material (Craik & Lockhart, 1972; Tulving & Thompson, 1973). There are several arguments for why learner-controlled environments may lead to better results, including increased interest and motivation (Alexander & Jetton, 2003), adaptation to preferences and cognitive needs (Merrill, 1980), and affordances for constructive and deeper information processing (Patterson, 2000). Some studies have supported these models, indicating that learner control can improve student performance (Campanizzi, 1978; Corbalan, Kester, & van Merrienboer, 2006; Gray, 1987; Kinzie, Sullivan, & Berdel, 1988). Unfortunately, most of the literature in this field has focused on older students, possibly because they are considered to have better computer skills and/or increased cognitive resources. Still, the results of hypermedia research among adults—especially when comparing the use of a self-directed program to a planned sequence—have typically been quite mixed. In fact some studies have rejected the idea that learner-control works better than a planned sequence (Devitt & Palmer, 1999; Shin, Schallert, & Savenye, 1994), while others have not found any difference (Aly, Elen, & Williems, 2005; see Dillon & Gabbard, 1998). One possible cause is that students generally make poor educational choices for themselves (Collins, 1996), and usually experience suboptimal learning rates when they are left in charge. Another possible factor relates to prior knowledge. According to Kintsch’s Construction–Integration Model (Kintsch, 1998), learners with high prior knowledge can benefit from less-coherent information presentation, whereas learners with low prior knowledge are not able to overcome gaps on their own (McNamara & Kintsch, 1996). There is accumulating evidence that only higher-performing students—that is, students with greater levels of prior knowledge and/or greater cognitive skills—can benefit from learner-directed CAI (Boone, Higgins, Notari, & Stump, 1996; Dillon & Gabbard, 1998; Farrell & Moore, 2000; Kopcha & Sullivan, 2008; Kraus, Reed, & Fitzgerald, 2001; Shyu & Brown, 1992, 1995). Because a student must identify what information is needed next and integrate the information into his or her existing knowledge structure, learner-controlled environments are often said to require some prior understanding of the information presented (Gall & Hannafin, 1994).

E.P. Johnson et al. / Computers & Education 55 (2010) 209–217

211

2.2. Adaptive learning systems The studies cited above often compared learner-controlled systems with more traditional CAI, often planned sequence systems. Instead of allowing the student to select which activity he or she will do next, a planned-sequence program enforces a particular domain structure by choosing the order in which information is presented. Such planned sequences are thought to work better for students with lower levels of prior knowledge because the student will not have any knowledge gaps remaining (van Gog, Ericsson, Rikers, & Paas, 2005). Adaptive CAI, on the other hand, uses an approach that has flexibility for student individual differences in learning to be accommodated (Park & Lee, 2007). Such programs adjust both the starting point (if they include pre-assessments) and the path a student takes through the material. As mentioned previously, research examining the effectiveness of adaptive, sequence-based CAI has generally been positive. The use of adaptive learning systems in early education classrooms has resulted in greater reading ability or math ability gains compared to controls for kindergartners living in poverty (Hecht & Close, 2002), English-language-learning kindergartners (Powers & Price-Johnson, 2007), suburban kindergartners (Cassady & Smith, 2004; Macaruso & Walker, 2008), and suburban first graders (Cassady & Smith, 2005; Macaruso, Hook, & McCabe, 2006; Savage, Abrami, Hipps, & Deault, 2009). A pair of studies conducted by the Center for Best Practices in Early Childhood have shown that such technology is successful in teaching young students pre-literacy skills, and that the skills gained through CAI are maintained longitudinally through three years (Hutinger, Bell, Daytner & Johanson, 2005, 2006). Adaptive learning systems have also been demonstrated to significantly increase phoneme awareness in low-performing preschoolers (Mitchell & Fox, 2001), at-risk preschoolers and kindergartners (Lonigan et al., 2003), and typical preschoolers (Foster, Erickson, Forster, Brinkman, and Torgensen, 1994) compared to a control group not utilizing any CAI. Finally, a recent meta-analysis exploring the use of CAI in Turkey found an overall effect size 1.05 (Camnalbur & Erdogan, 2008). Other studies examining technology use in the early grades have not been so positive (e.g., Brooks, Miles, Torgerson, & Torgerson, 2006; Seo & Bryant, 2009). In a meta-analysis examining CAI programs that addressed a single aspect of reading development (such as phonological awareness or reading fluency), Blok, Oostdam, Otter, and Overmaat (2002) found that the average effect size was only .254. Once pre-test effect sizes and native language were accounted for, the overall effect size dropped to .19. It should be noted, however, that the average effect size jumped to .5 when only studies completed among English-speaking countries were considered. Also, the Blok analysis did not include any studies related to products that teach reading acquisition along multiple fronts (e.g., including phonological awareness, vocabulary, phonics, letter recognition, and word reading). A second meta-analysis, examining the effects of CAI in elementary education, found an overall effect size of .342 (Christmann & Badgett, 2003). The researchers noted, however, that some schools were integrating computers into their curricula without the full support and accommodation of the staff, concern for appropriateness of subject matter, or interest in demographical differences. The appearance of these kinds of problems represented, according to the authors, the outdated tendency to consider CAI as a replacement for classroom instruction. 2.3. Mastery learning and CAI Adaptive sequencing software—including the program used in this study—often base their sequencing decisions on a projection related to student ‘‘mastery”. In the case of the current study, such mastery decisions are based on a series of in-program assessments; when a student achieves a certain percent correct on certain objective’s assessment, he or she is judged to have ‘‘mastered” that material and is allowed to proceed. Students who master material more quickly, then, progress more often to the next objective in the sequence. When a student fails an assessment the computer loads a series of remedial activities, then re-assess the student until mastery is obtained. Many mastery-based CAI systems—again, including the system we used—have been developed to be consistent with mastery learning theory in education. First developed in the 1960s, the mastery learning approach focused on the idea that all students can learn as long as they have sufficient time (Carroll, 1963). The idea was expanded most famously by Bloom (1968); its chief instructional prescription was that a student should spend as long as necessary on each objective, mastering each bit of knowledge before he or she could proceed. Mastery learning requires that a student is assessed on an objective standard, receives feedback about the result, and either repeats the material or continues on based on the result of that performance. For a subject area to be considered ‘‘mastered,” the student must have achieved a criterion-referenced standard of performance (Lalley & Gentile, 2009). Research has indicated that mastery learning allows students to learn the information well enough that the subsequent ‘‘forgetting phase” is minimal—and that enough knowledge remains to build upon in the next lesson (Lalley & Gentile, 2009). Early reviews of research (Block & Anderson, 1975; Kulik, Kulik, & Bangert-Downs, 1990; Anderson, 1994) found that mastery learning was beneficial for students despite variations in class size, content area, and class setting. This research noted, too, that such techniques are particularly beneficial to lower-performing students. More recent research has also supported these findings, emphasizing that the mastery learning approach continues to work in the classrooms in which it is implemented (see Zimmerman & Dibenedetto, 2008). One reason mastery learning has been a success in the classroom is because computers allowed the mastery techniques to assume a shape closer to what Bloom originally proposed (Montazemi & Wang, 1995; Motamedi & Sumrall, 2000). These original claims, however, have been described by critics as mere educational rhetoric (Glass & Smith, 1978), and more of a philosophy than a practical, prescriptive theory (Arlin, 1984). One problem with the mastery learning theory is that instruction is designed principally around one variable—time— while achievement is held constant (Park & Lee, 2007). The mastery learning theory posits that every child can learn the information, given enough time, and the curriculum reflects this with a desired achievement the final goal, regardless of the time required to get there. The result is about a 40% increase in classroom learning time, which may be more than a strained educational system like ours could bear (Arlin, 1984). Achievement variance decreases, but teachers may not have enough resources to implement such learner-specific programs. 3. Current study The current study examines the effect of CAI presentation on student reading gains in the early grades. The same reading curriculum was presented to the treatment groups using three different presentation methods: (1) a learner-controlled picture menu, (2) a linear adaptive sequence, (3) and a mastery-based adaptive sequence. The learner-controlled design allowed students to point and click on

212

E.P. Johnson et al. / Computers & Education 55 (2010) 209–217

the preferred activity, and to repeat activities as often as they desired. Scored activities were included in the options a student could choose from, but the child was put under no obligation to complete them and the scores had no impact on subsequent choices. The linear sequence design progressed students linearly through the curriculum. While the students were periodically scored on activities in the sequence, those scores did very little to alter the curriculum. The sequencer was designed to provide enough instruction so that most of the children would be able to learn the material. If a student was found to have not learned the material, the teacher could intervene and have them complete the activity once again. This instructional method is similar to a traditional teaching model that would be found in most classrooms, as well as to prior linear CAI. While the sequence was designed to present the material to students in a particular order, there was no requirement to master the material before moving on. The final sequencer design adapted to the student’s needs based on mastery assessments. Each student was tested before an objective to determine where he or she would begin; if the student was deemed to have mastered the objective already, he or she would see only a few of the lessons associated with the learning objective. In the case of initial failure, the student would see all of the lessons for that objective. Once begun, this version of the software also tested students during the course of instruction; depending on the assessment, ‘‘mastery” was defined as having passed between 80% and 100% of the presented items. If a student did not pass this second assessment, he or she was directed to complete a set of activities similar to those previously seen. When a student was judged to have obtained mastery, he or she was moved to activities focusing on speed and automaticity with the material. The current study was designed to test which method of CAI is most effective at teaching early reading and phonological awareness skills. Our primary goal was to examine differences in reading gains of students using each of these types of CAI, and to compare all three of these instructional groups to a control group of students not using any computer-aided instruction. Reading achievement testing was conducted using the Dynamic Indicators of Basic Early Literacy Skills (DIBELS), a well-known standardized test for emergent English literacy skills created by researchers at the University of Oregon. DIBELS was designed to assess the five key areas defined by the National Reading Panel Report (2000) as integral to reading instruction: phonemic awareness, systematic phonics (i.e., alphabetic principle) instruction, methods to improve fluency, and ways to enhance comprehension. As such, DIBELS includes the following assessments: (1) Initial Sound Fluency, measuring the speed at which a student can recognize which words begin with a given initial sound, (2) Letter Naming Fluency, measuring the speed at which a student can name randomly pictured letters, (3) Phoneme Segmentation Fluency, measuring the speed at which a student can segment given words into their component phonemes, (4) Nonsense Word Fluency, measuring the speed at which a student can sound out or fluently read a series of nonsense words, and (5) Oral Reading Fluency, measuring a student’s speed and accuracy for given paragraphs. Because of the ages of the study participants, no students were given the Oral Reading Fluency test. We predicted that the students using the mastery sequencer would experience larger reading gains than the other three groups, followed by students in the linear sequencer group. Due to the novelty and relative difficulty of early reading acquisition for this age group, students who used the learner-directed picture menu were not expected to do as well as students using the sequenced activities. In addition, we expected that our study would corroborate previous research in demonstrating that students with more advanced pre-test scores perform better on learner-directed software than students with weaker pre-test scores. All three groups using the computer were expected to perform better than controls. 4. Methodology 4.1. Participants All participants were recruited from the Salt Lake City and Provo, Utah areas. Participants in the experimental condition were recruited through fliers posted at libraries and recreation centers, as well as through advertising in the local media. Registered students were broken down into three different groups, each of which was assigned a particular type of CAI: learner-controlled picture menu, linear sequencer, and adaptive, mastery-based sequencer. Initially, 178 students enrolled in the program and were assigned to an instructional method. A total of 129 students completed the 14-week classroom session at the Waterford Institute Community Center (in Salt Lake City) or the Bonneville Community Center (in Provo), with 33 in the picture menu group, 58 in the linear sequencer group, and 38 in the mastery sequencer group. The control group, consisting of an additional 54 participants, was recruited from two Salt Lake City area preschools that also offered before- and after-kindergarten care. Controls were pre-tested and post-tested at the same time as instructional groups but did not spend time on the computer. The resulting four groups were demographically similar in terms of grade, socio-economic status (measured by average parental education level) and ethnicity (Table 1). Gender composition was not equal: the picture menu group had a higher percentage of females than the other three groups. The potential effects of this gender discrepancy are explored below. 4.2. Procedures 4.2.1. Testing In order to accurately gauge changes in the students’ reading proficiency, the kindergarten mid-year benchmark of DIBELS was used for both the pre-test and the post-test. Because DIBELS examines student skill competencies as a student advances, a different version of the Table 1 Group demographics.

Learner-controlled Linear sequencer Mastery sequencer Control

N

Percentage male/female

Average parental education

Percentage preschool/kindergarten

33 58 38 54

41/59 60/40 65/35 55/45

College College College College

69/31 64/36 61/40 69/31

degree degree degree degree

213

E.P. Johnson et al. / Computers & Education 55 (2010) 209–217

test is offered for the beginning, middle, and end points of each grade level. For the current study, we chose to use the mid-year kindergarten assessment for both the pre- and post-test because: (a) using a static, rather than progressive measure provides a more statistically valid and easy-to-interpret set of results, and (b) the mid-year kindergarten test measures Nonsense Word and Phoneme Segmentation fluencies in addition to Letter Naming and Initial Sounds. All four DIBELS subtests are quick to administer and are taken one-on-one with a proctor. Approximately three months passed between testing events, thus minimizing the test/retest effect. 4.2.2. Computer Intervention Participants completed the computer instruction at one of two locations, either the Bonneville Community Center in Provo and the Waterford Institute Community Center in Salt Lake City. The computer lab at the Bonneville Center allows for ten students to be using the product simultaneously, while the Waterford Center allows for fifteen. Computers at both locations were spaced apart so that participants could interact with the computer without interfering with neighboring students. Each computer had headset attached as well, allowing individual students to listen without noise interference from the class. Students came to the community center two days a week for 13 weeks, and spent 20 min per visit using their assigned computerized reading curriculum. The study took place from September to December, 2006. Students were randomly assigned to one of three conditions: learner-controlled picture menu, linear sequence, or mastery-based adaptive sequence. All three conditions used material from the same curriculum, the Waterford Early Reading Program (WERP). WERP includes activities addressing phonics, phonological awareness, vocabulary, comprehension, and print awareness. The activities are presented through a mixture of songs, interactive games, videos, and digital books. In the case of the two sequencers, there are also periodic assessments to track a child’s progress and performance. 4.2.3. Control group Control participants, who came from two early learning centers in the area, underwent a traditional instructional program that did not include any type of CAI. Kindergartners attended the early learning centers chosen for our control institutions either in addition to their public school kindergarten class or for the entire day. Both control schools from which data was collected engage in daily literacy instruction, including activities focused on letter knowledge, rhyme awareness, phonics, and reading comprehension. Such literacy instruction is part of a curriculum focused on ‘‘preparing the children academically, socially, physically, and emotionally” for what lies ahead (Kinderland). As with the instructional groups, all control-group students were administered the kindergarten mid-year DIBELS test at the beginning of the study and again at the end. The interval between pre- and post-tests was the same—13 weeks—as it was for the experimental groups. 5. Results 5.1. Primary analyses Primary analyses concern performance on post-tests from the DIBELS mid-year test, controlling for pre-test performance, among the four different groups (three CAI, one control). The mid-year DIBELS assessment we used included four of the five DIBELS measures: (1) Initial Sound Fluency, (2) Letter Naming Fluency, (3) Segmentation Fluency, and (4) Nonsense Word Fluency. The scores for these four areas were combined for both the pre- (September) and post-test (December) scores. Pre-test scores for the four subtests were found to be highly correlated (Table 2), supporting the combination of tests for the majority of the analyses. Initial analysis of pre-test means indicates no significant differences among the groups [F(3, 179) = 2.0, p = .12], though the combined pre-test mean among control-group students is noticeably lower than the combined mean for the mastery-based group (see Table 3). Analysis of post-test means reveals significant differences between the groups [F(3, 179) = 3.74, p < .05], largely driven by higher post-test scores among the mastery-based group and lower post-test scores among the control group (Table 3). Though we determined there to be no significant differences in pre-test scores among the each of the treatment groups and the control group, an additional analysis of change over time was deemed important in order to account for pre-test variance. Thus, an ANCOVA was performed to determine group effect on post-test score with pre-test score as a covariate. This ANCOVA indicates significant differences in adjusted post-test means among the four groups [F(3, 178) = 3.00, p < .05]. Comparisons between groups differ depending on the inclusion of a conservative Bonferroni estimate. Uncorrected comparisons reveal significantly better performance for the mastery-based group compared to the control group (p < .01) and the picture menu group (p < .01). This effect is visible in Fig. 1. When the ANCOVA is run with a Bonferroni correction, these differences are still strong, but only approach significance (p = .062 in both instances). Interestingly, there is not a significant difference in the gains made by mastery-based and linear sequencer groups (p = .12 uncorrected). Gains made by the control and picture menu groups are also found not to differ significantly (p = .78 uncorrected). When the same analysis is broken down for each of the four subtests included in DIBELS, a different pattern emerges (see Fig. 2). Significant group effects remain for Initial Sound Fluency [F(3, 178) = 3.55, p < .05], where the linear sequence group outperformed both the control group (p < .01, uncorrected; p < .05, corrected) and the picture menu group (p < .05, uncorrected; p = .078, corrected), as well as for Nonsense Word Fluency [F(3, 178) = 2.83, p < .05], where the mastery-based sequencer group outperformed the control group (p < .01,

Table 2 Correlation matrix for DIBELS subtests at pre-test, full group.

ISF LNF PSF

Initial Sound Fluency

Letter Naming Fluency

Phoneme Segmentation Fluency

Nonsense Word Fluency

–

r = .55, p < .001 –

r = .57, p < .001 r = .63, p < .001 –

r = .49, p < .001 r = .68, p < .001 r = .64, p < .001

214

E.P. Johnson et al. / Computers & Education 55 (2010) 209–217

Table 3 Group means and standard deviations on pre-test, post-test, and gains. Mean (SD) for total reading ability

Average Reading Gain

Learner-controlled Linear sequencer Mastery sequencer Control

Pre-test

Post-test

Gain

40.86 49.55 52.05 29.56

61.08 80.13 92.68 49.49

20.22 30.59 40.63 19.93

(38.34) (58.11) (59.11) (41.90)

(55.34) (74.28) (78.23) (56.89)

(28.41) (32.86) (37.71) (24.74)

37.39

40 35

30.59

30 25

20.22

19.93

20 15 10 5 0 Le

arn

Lin

er-

ea

co

ntr

oll

ed

Ma

rS

ste

eq

ue

nc

er

ry

Co

ntr

Se

qu

en

ol

ce

r

Fig. 1. Reading ability mean gains by group. Average reading gain is a computed raw score based on the four subtests in the DIBELS mid-year kindergarten test.

uncorrected; p < .05 corrected), picture menu group (p < .05 uncorrected; p = .28, corrected) and linear sequencer group (p < .05 uncorrected; p = .15 corrected). However, no significant group effect was found for Letter Naming Fluency [F(3, 178) = .92, p = .43]. Segmentation Fluency results approached but did not reach significance [F(3, 178) = 2.44, p = .066]; interestingly, the only significant group difference on this measure is between the mastery-based sequencer and picture menu groups (p < .01, uncorrected; p < .05, corrected). 5.2. Follow-up analyses Because our analyses did not reveal any significant post-test differences between the linear sequence and the mastery-based sequence groups, follow-up analyses combined these two groups to examine the overall educational impact of using a sequence-based software versus picture menu or no CAI. The resulting ANCOVA indicated significant differences among the three groups [F(2, 179) = 3.28, p < .05]. Unadjusted post hoc analyses revealed significantly better performance among the combined sequencer group compared to both the picture menu group (p < .05) and the control group (p < .01), though both of these results are no longer significant after a conservative Bonferroni adjustment. Additional follow-up analyses compared the gains of low and high pre-test performers. A student’s pre-test performance was considered ‘‘low” if his or her overall score was below 20. Using this cut-off point resulted in the following percentage of participants falling into the low group: 42% in the picture menu group, 46% in the linear sequence group, 42% in the mastery sequence group, and 53% in the control group. A comparison of pre- to post-test gains between low- to high-performers reveals a significant effect of pre-test performance on overall gains [F(1, 175) = 25.54, p < .001]: low pre-test performance tends to result in lower gains as well. However, there is no interaction between pre-test performance and CAI group [F(3, 175) = .51, p > .5]. Follow-up t-tests comparing the performance of high- and lower-pretest-score students reveal significantly greater gains for high-starting students in the picture menu (t = 2.35, p < .05), the linear sequencer (t = 3.66, p < .01), and control groups (t = 2.33, p < .05). Gains by high-starting students were not significantly greater within the masterybased sequencer group. 14 12

Learner Controlled Linear Sequencer Mastery Sequencer Control

10 8 6 4 2 0 ISF Gains

LRF Gains

SGF Gains

NWF Gains

Fig. 2. DIBELS subtest raw score mean gains over time by group. ISF = Initial Sound Fluency, LRF = Letter Recognition Fluency, SGF = Segmentation Fluency, NWF = Nonword Reading Fluency.

E.P. Johnson et al. / Computers & Education 55 (2010) 209–217

215

Because the picture menu group included a higher percentage of females than the other groups, an analysis was conducted to ensure that gender differences were not a causal factor in the final results. Results indicated no significant effect of gender [F(1, 153) = .033, p = .86] on adjusted post-test scored and no significant interaction between gender and group [F(3, 153) = .916, p = .44]. Although each group included both preschoolers and kindergartners, an interaction between group and grade level was not expected because the ratio of pre-K to K students was similar between groups. When grade level was added to the model, results indicated a significant effect of grade [F(1, 174) = 19.69, p < .001] as a result of better performance by kindergartners, but no significant interaction with group [F(3, 174) = .29, p = .84]. 6. Discussion The primary goal of the current study was to examine the impact of CAI presentation method on reading ability gains. The expectation was that there would be a linear progression of gains, with the students using the mastery-based sequence making the most progress, followed by the linear sequence group and then the learner-controlled, picture menu group. All three CAI groups were expected to outperform age-matched controls. While the values for the gains appear to follow this linear progression, statistical analyses reveal a significant split between students using a sequencer and students not using a sequencer, with a particular distinction for the mastery-based sequencer over learner-controlled CAI and controls. There are two key findings in this paper. First, in contrast to our hypothesis, students using the mastery-based sequencer had statistically similar gains to students using the linear sequencer. The lack of a significant difference between the groups may be the result of three different factors. First, the previously cited mastery studies examined the impact of mastery learning among older students (the youngest was ‘‘upper elementary” [Kulik et al., 1990]). The effects of mastery learning techniques may not be as strong in the younger grades. Second, the effects of mastery learning through a computer program may be different from a classroom setting. In the classroom, if material has not been learned, the teacher may be able to vary the style of presentation—that is, try something different. With most CAI the style has already been preprogrammed, and even though the child sees some different activities when repeating a failed section, if that child does not learn well from computer presentation of material, the number of times the material is presented may not be able make a difference. In contrast, the lack of a significant difference between these groups may reflect the strength of the linear program. Analysis of the separate subtests revealed that students using the linear sequencer outperformed controls and the picture menu group, but the mastery-based sequencer group did not. If students in a mastery-based program respond well to the amount and the sequence of material presented along the ordinary instructional path, less repetition and remediation will be necessary; in this case, the mastery-based group may end up following a similar path to their peers using the linear sequencer. Unfortunately, we do not have access to the data indicating how far the groups made it through the program on average. Importantly, despite a lack of significant difference between the mastery group and the linear sequence group, only the mastery group performed significantly better on overall reading post-test than controls and the learner-controlled group when the more conservative Bonferroni adjustment was applied to the post hoc analyses. In addition, results for the Nonsense Word Fluency subtest revealed that the mastery-based sequencer group performed significantly better than all three other groups, including the linear sequence group, prior to a bonferroni correction. These results suggest that the mastery-based sequencer may have a greater effect on learning than the linear sequence. The second key finding is that, despite receiving the same instructional information as the students in the sequencer groups, students using the learner-controlled picture menu did not make greater gains than the control group using no CAI. Students in the picture menu group were on a computer for the same amount of time as students in both of the sequencer groups and had the same information accessible to them. However, as classroom instructors in the community center noticed, students were more likely to engage in the same activities that other students around them were engaged in, rather than choosing what might be best for them as an individual. As a result, the students made no more progress than students in a typical preschool or kindergarten, who are also engaging in learning activities focused on letter knowledge, rhyme awareness, phonics, and reading comprehension. Both control preschools from which data was collected engage in daily literacy instruction. Reviews of learner-controlled CAI have sometimes found it to be effective, but outside of the research focused on hypermedia and multimedia (which typically concerns itself with older students) there has not been a sufficient discussion of how learner-control affects student achievement (see MacArthur et al., 2001). Self-directed student learning programs are currently being used in thousands of schools across the nation (EducationCity, for instance, claims to be used by over 11,000 schools) but have not undergone a sufficient evaluation for these younger ages. The current study indicates that the use of a sequencer to guide younger students through CAI material is essential to receiving the full benefit of a computerized program. In addition, a great deal of research has indicated that students who have developed strong ‘‘schemata” (or cognitive structures) for information processing—as well as those who possess greater knowledge of a domain—respond well to the self-determined, non-sequenced learning environment. Lower-performing students, whether as a result of insufficient schemata, domain knowledge, or cognitive load have been generally found to benefit very little, if at all, from learner-controlled software (see Dillon & Gabbard, 1998; Lawless & Brown, 1997; Paas, Renkl, & Sweller, 2003). Preschool- and kindergarten-aged children have very little experience with education environments and reading tutorials on the computer. In our study, most students began instruction with limited alphabetic and phonological awareness knowledge. With so little to build from, the learner-controlled environment may not have had much of an opportunity to work. In addition, while our results did indicate that students in this picture menu group with a higher pre-test score made greater gains than students with a lower pre-test score, these results were replicated in controls and students using the linear, sequenced software. Thus, it appears that the amount of knowledge a student has coming into a program can make a big difference, regardless of the presentation style of the material. 7. Conclusion Parents and educators are barraged with material espousing the merits of one type of educational software over another. The information contained in these products may be beneficial, but if the method of presentation does not fully engage the student in the learning pro-

216

E.P. Johnson et al. / Computers & Education 55 (2010) 209–217

cess, the consumer needs to be aware of what it can and cannot provide. These results suggest that the self-directed learning style may not be an effective CAI tool for this age group and that a mastery-based program, although not significantly better than a linear, sequenced program, may be more beneficial than no CAI or learner-controlled CAI in the classroom. Further research is needed to better understand why the groups using the mastery sequencer and the linear sequencer did not significantly differ in this study.

8. Disclosure statement There is one potential conflict of interest for this paper; the products used in the study are produced by the Waterford Research Institute, the employer of the authors. However, because this article does not examine efficacy of a Waterford product compared to another product and instead examines differences in reading gains associated with method of presentation, this potential conflict is minimized. Additionally, the Waterford Institute played no role in the study design, data collection, data analysis and interpretation, writing of the report, and in the decision to submit the paper for publication.

References Alexander, P. A., & Jetton, T. L. (2003). Learning from traditional and alternative texts: New conceptualization for an information age. In A. Graesser, M. Gernsbacher, & S. Goldman (Eds.), Handbook of discourse processes (pp. 199–241). Mahwah, NJ: Erlbaum. Aly, M., Elen, J., & Williems, G. (2005). Learner-control vs program-control instructional multimedia: A comparison of two interactions when teaching principles of orthodontic appliances. European Journal of Dental Education, 9, 157–163. Anderson, S. A. (1994). Synthesis of Research on Mastery Learning. Northville, MI: Northville Public Schools. (ERIC Document Reproduction Service No. ED382567). Arlin, M. (1984). Time equality, and mastery learning. Review of Educational Research, 54, 65–86. Block, J. H., & Anderson, L. W. (1975). Mastery learning in classroom instruction. New York: Macmillan. Blok, H., Oostdam, R., Otter, M. E., & Overmaat, M. (2002). Computer-assisted instruction in support of beginning reading instructions. Review of Educational Research, 72, 101–130. Bloom, B. S. (1968). Learning for mastery: Instruction and curriculum. Evaluation Comment UCLA-CSIEP, 1, 1–12. Bloom, B. S. (1984). The 2 sigma problem: The search for methods of group instruction as effective as one-to-one tutoring. Educational Researcher, 13(6), 4–16. Boone, R., Higgins, K., Notari, A., & Stump, C. S. (1996). Hypermedia pre-reading lessons: Learner-centered software for kindergarten. Journal of Computing in Childhood Education, 7, 39–69. Brooks, G., Miles, J. N. V., Torgerson, C. J., & Torgerson, D. J. (2006). Is an intervention using computer software effective in literacy learning? A randomised controlled trial. Educational Studies, 32, 133–143. Brown, J. S., Collins, A., & DuGuid, P. (1989). Situated cognition and the culture of learning. Educational Researcher, 18, 32–42. Camnalbur, M., & Erdogan, Y. (2008). A meta analysis on the effectiveness of computer-assisted instruction: Turkey sample. Educational Sciences: Theory and Practice, 8, 497–505. Campanizzi, J. A. (1978). Effects of locus of control and provision of overviews in a computer assisted instruction sequence. AEDS Journal, 12, 21–30. Carroll, J. B. (1963). A model of school learning. Teachers College Record, 64, 723–733. Cassady, J. C., & Smith, L. L. (2004). The impact of a reading-focused integrated learning system on phonological awareness in kindergarten. The Journal of Literacy Research, 35, 947–984. Cassady, J. C., & Smith, L. L. (2005). The impact of a structured integrated learning system on first grade students’ reading gains. Reading and Writing Quarterly, 21, 361–376. Christmann, E. P., & Badgett, J. L. (2003). A meta-analytic comparison of the effects of computer-assisted instruction on elementary students’ academic achievement. Information Technology in Childhood Education, 91, 104. Collins, A. (1996). Design issues for learning environments. In S. Vosniadou, E. De Corte, R. Glaser, & H. Mandl (Eds.), International perspectives on the psychological foundations of technology-based learning environments (pp. 347–361). Hillsdale, NJ: Lawrence Erlbaum. Corbalan, G., Kester, L., & van Merrienboer, J. J. G. (2006). Towards a personalized task selection model with shared instructional control. Instructional Science, 34, 399–422. Craik, F. I. M., & Lockhart, R. S. (1972). Levels of processing: A framework for memory research on interactions. New York: Irvington. Devitt, P., & Palmer, E. (1999). Computer-aided learning: An overvalued educational resource? Medical Education, 33, 136–139. Dillon, A., & Gabbard, R. (1998). Hypermedia as an education technology: A review of the quantitative research literature on learn comprehension, control and style. Review of Educational Research, 68, 322–349. Farrell, I. H., & Moore, D. M. (2000). The effect of navigation tools on learners’ achievement and attitude in a hypermedia environment. Journal of Educational Technology Systems, 29, 169–181. Fish, A. M., Li, X., McCarrick, K., Butler, S. T., Stanton, B., Brumitt, G. A., et al. (2008). Early childhood computer experience and cognitive development among urban low-income preschoolers. Journal of Educational Computing Research, 38, 97–113. Foster, K. C., Erickson, G. C., Forster, D. F., Brinkman, D., & Torgensen, J. K. (1994). Computer administered instruction in phonological awareness: Evaluation of the DaisyQuest program. Journal of Research and Development in Education, 27, 126–137. Gall, J. E., & Hannafin, M. J. (1994). A framework for the study of hypertext. Instructional Science, 22, 207–232. Glass, G. V., & Smith, M. L. (1978). The technology and politics of standards. Educational Technology, 18, 8–12. Gray, S. H. (1987). The effect of sequence control on computer assisted learning. Journal of Computer-Based Instruction, 2, 54–56. Hecht, S. A., & Close, L. (2002). Emergent literacy skills and training time uniquely predict variability in responses to phonemic awareness training in disadvantaged kindergartners. Journal of Experimental Child Psychology, 82, 93–115. Hutinger, P. L., Bell, C., Daytner, G., & Johanson, J. (2005). Disseminating and replicating an effective emerging literacy technology curriculum: A final report. Macomb: Western Illinois University, Center for Best Practices in Early Childhood. (ERIC Document Reproduction Service No. ED469844). Hutinger, P. L., Bell, C., Daytner, G., & Johanson, J. (2006). Establishing and maintaining an early childhood emergent literacy technology. Journal of Special Eduction Technology, 21, 39–54. Kintsch, W. (1998). Comprehension—A paradigm for cognition. Cambridge: Cambridge University Press. Kinzie, M. N., Sullivan, H. J., & Berdel, R. L. (1988). Learner control and achievement in science computer-assisted instruction. Journal of Educational Psychology, 80, 299–303. Kopcha, T. J., & Sullivan, H. (2008). Learner preferences and prior knowledge in learner-controlled computer-based instruction. Education Technology Research Development, 56, 265–286. Kraus, L. A., Reed, W. M., & Fitzgerald, G. E. (2001). The effects of learning style and hypermedia prior experience on behavioral disorders knowledge and time on task: A casebased hypermedia environment. Computers in Human Behavior, 17, 125–140. Kulik, C. C., Kulik, J., & Bangert-Downs, R. L. (1990). Effectiveness of mastery learning programs: A meta-analysis. Review of Educational Research, 60, 265–299. Lalley, J., & Gentile, J. (2009). Classroom assessment and grading to assure mastery. Theory into Practice, 48, 28–35. Lawless, K. A., & Brown, S. W. (1997). Multimedia learning environments: Issues of learner control and navigation. Instructional Science, 25, 117–131. Lepper, M. R., & Gurtner, J. (1989). Children and computers: Approaching the 21st century. American Psychologist, 89, 170–178. Lonigan, C., Driscoll, K., Phillips, B. M., Cantor, B. G., Anthony, J. L., & Goldstein, H. (2003). A computer-assisted instruction phonological sensitivity program for preschool children at-risk for reading problems. Journal of Early Intervention, 25, 248–262. MacArthur, C. A., Ferretti, R. P., Okolo, C. M., & Cavalier, A. R. (2001). Technology applications for students with literacy problems: A critical review. The Elementary School Journal, 101, 273–301. Macaruso, P., Hook, P. E., & McCabe, R. (2006). The efficacy of computer-based supplementary phonics programs for advancing reading skills in at-risk elementary students. Journal of research in Reading, 29, 162–172. Macaruso, P., & Walker, A. (2008). The efficacy of computer assisted instruction for advancing literacy skills in kindergarten children. Reading Psychology, 29, 266–287. Mayer, R. E., & Anderson, R. B. (1991). Animations need narrations: An experimental test of a dual-coding hypothesis. Journal of Educational Psychology, 83, 484–490.

E.P. Johnson et al. / Computers & Education 55 (2010) 209–217

217

Mayer, R. E., & Anderson, R. B. (1992). The instructive animation: Helping students build connections between words and pictures in multimedia learning. Journal of Educational Psychology, 84, 444–452. McNamara, D. S., & Kintsch, W. (1996). Learning from texts: Effects of prior knowledge and text coherence. Discourse Processes, 22, 247–288. Merrill, M. D. (1975). Learner control: Beyond aptitude–treatment interactions. AV Communications Review, 23, 217–226. Merrill, D. M. (1980). Learning control in computer based learning. Computers and Education, 4, 77–95. Mitchell, M. J., & Fox, B. J. (2001). The effects of computer software for developing phonological awareness in low-progress readers. Reading Research and Instruction, 40, 315–332. Montazemi, A. R., & Wang, F. (1995). An empirical investigation of CBI in support of mastery learning. Journal of Educational Computing Research, 13, 185–205. Motamedi, V., & Sumrall, W. J. (2000). Mastery learning and contemporary issues in education. Action in Teacher Education, 22, 32–42. Paas, F., Renkl, A., & Sweller, J. (2003). Cognitive load theory and instructional design: Recent developments. Educational Psychologist, 38(1), 1–4. Park, O., & Lee, J. (2007). Adaptive instructional systems. In J. M. Spector, M. D. Merril, J. J. G. Van Merrienboer, & M. P. Driscoll (Eds.), Handbook of research for educational communications and technology (pp. 634–664). New York: Macmillan. Patterson, N. G. (2000). Hypertext and the changing roles of readers. English Journal, 90, 74–80. Powers, S., & Price-Johnson, C. (2007). Evaluation of the waterford early reading program in kindergarten 2005–2006. Tucson, AZ: Creative Research Associates. Reigeluth, C. M., & Stein, F. S. (1983). The elaboration theory of instruction. In C. M. Reigeluth (Ed.), Instructional design theories and models: An overview of their current status. Hillsdale: Erlbaum. Reitsma, P., & Wesseling, R. (1998). Effects of computer-assisted training of blending skills in kindergartners. Scientific Studies of Reading, 2, 301–320. Savage, R. S., Abrami, P., Hipps, G., & Deault, L. (2009). A randomized controlled trial study of the ABRACADABRA reading intervention program in grade 1. Journal of Educational Psychology, 101, 590–604. Scheiter, K., & Gerjets, P. (2007). Learner control in hypermedia environments. Educational Psychology Review, 19, 285–307. Seo, Y., & Bryant, D. P. (2009). Analysis of studies of the effects of computer-assisted instruction on the mathematics performance of students with learning disabilities. Computers and Education, 53, 913–928. Shin, E. C., Schallert, D. L., & Savenye, W. C. (1994). Effects of learner control, advisement, and prior knowledge on young students’ learning in a hypertext environment. Educational Technology, Research and Development, 42, 33–46. Shyu, H. S., & Brown, S. W. (1992). Learner control versus program control in interactive videodisc instruction: What are the effects in procedural learning? International Journal of Instructional Media, 19, 85–96. Shyu, H. S., & Brown, S. W. (1995). Learner-control: The effects on learning a procedural task during computer-based videodisc instruction. International Journal of Instructional Media, 22, 217–231. Snow, E. R. (1980). Aptitude, learner control, and adaptive instruction. Educational Psychologist, 15, 151–158. Tulving, E., & Thompson, D. M. (1973). Encoding specificity and retrieval processes in episodic memory. Psychological Review, 80, 352–373. van Gog, T., Ericsson, K. A., Rikers, R. M. J. P., & Paas, F. (2005). Instructional design for advanced learners: Establishing connections between the theoretical frameworks of cognitive load and deliberate practice. Educational Technology Research and Design, 53, 73–81. Verhallen, M. J. A. J., Bus, A. G., & de Jong, M. T. (2006). The promise of multimedia stories for kindergarten children at risk. Journal of Educational Psychology, 98(2), 410–419. Wenglinsky, H. (1998). Does it compute? The relationship between educational technology and student achievement in mathematics. The policy information center of the educational testing service. Wise, B. W., Ring, J., & Olson, R. K. (2000). Individual differences in gains from computer-assisted remedial reading. Journal of Experimental Child Psychology, 77, 197–235. Wood, C., Pillinger, C., & Jackson, E. (2010). Understanding the nature and impact of young readers’ literacy interactions with talking books and during adult reading support. Computers and Education, 54, 190–198. Zimmerman, B. J., & Dibenedetto, M. K. (2008). Mastery learning and assessment: Implications for students and teachers in an era of high-stakes testing. Psychology in the Schools, 45, 206–216.