An Empirical Analysis of Using Text-to-Speech Software to Revise First-Year College Students’ Essays

Available online at www.sciencedirect.com

Computers and Composition 26 (2009) 288–301

An Empirical Analysis of Using Text-to-Speech Software to Revise First-Year College Students’ Essays Kevin Garrison ∗ Angelo State University, 2601W. Avenue N, San Angelo, Texas 76909, United States Received 13 May 2008; received in revised form 24 August 2009; accepted 1 September 2009

Abstract Traditionally, composition experts have suggested reading drafts aloud as a means of revising essays; however, the method of reading drafts aloud is severely limited by a single factor: student writers do not always read what is on the page (Hartwell, 1985). Text-to-speech (TTS) software allows students to have their essays read to them so that the limiting factor of reading their own drafts aloud becomes minimized. TTS programs read what is written on the computer screen, and the result is that the students can “hear” the problems of their essays as opposed to simply “seeing” them. Nevertheless, composition researchers have not conducted any empirical studies to determine whether or not TTS is beneficial for “local” and “global” revision, nor have any studies been conducted to determine if TTS is beneficial for students above the fifth grade. This article documents an experimental study conducted at a southwestern university in the United States with fifty-one students to determine whether or not TTS software is useful in the revision process. The results show that users of TTS were as likely as users in the control group to make proofreading changes but less inclined to make local or global changes in the revision process, indicating that TTS possibly works well for proofreading but not necessarily as well for higher-order revision. Further research is recommended to determine TTS’s effectiveness during a longitudinal study as well as for auditory learners and ESL students. © 2009 Elsevier Inc. All rights reserved. Keywords: Text-to-speech or speech synthesis; TTS; Proofreading and/or revision methods; Limitations of “read aloud”; Empirical or quantitative analysis; Computers and writing; Microsoft Word; Writing autonomy; College composition; First year writing (FYW)

1. Introduction Computers have been integrated into public and private life so rapidly over the past twenty years that it has become difficult to keep up with the latest features and technology. One such technology that has undergone expansive development recently is in the field of text-to-speech (TTS) software. TTS programs convert computer text into spoken words using a synthesized computer voice. Tom Atkinson, Jerry Neal, and Marilyn Greches (2003) defined TTS as providing “playback of printed text as spoken words. An internal driver, called a TTS engine, recognizes and speaks written text through a synthesized voice selected from several pregenerated voices” (p. 178). Most TTS software programs can read any text that can be copied and pasted from any text-based program, such as HTML files or Microsoft Word files. ∗ Kevin graduated from Texas Tech University in May 2009 with a Ph.D. in Technical Communication and Rhetoric, and he currently works as a tenure-track assistant professor of Professional Writing at Angelo State University. E-mail address: [email protected].

8755-4615/$ – see front matter © 2009 Elsevier Inc. All rights reserved. doi:10.1016/j.compcom.2009.09.002

K. Garrison / Computers and Composition 26 (2009) 288–301

289

The voices in most TTS software programs range anywhere from female to male, native to non-native, and slower to faster speakers. In the last several years, AT&T (2008) has created some of the most natural voices available, and the voices sound nearly human. TTS software works much like a word processing or a web browser program in that it contains an interface for text to be shown on screen. Developed for multiple reasons, such as reducing eye strain, saving time, and teaching second language students (Natural Reader, 2008), most TTS programs are able to be downloaded and used for free, with upgraded voices costing as little as $49.50 at the Natural Reader web site (2008). Most TTS programs highlight each word as it is being read aloud, thereby allowing a listener to follow along on the screen, and the Natural Reader software also allows for any text file (including web documents, Word documents, emails, and PDFs) to be saved as an MP3 and read through a portable MP3 device, allowing users to listen to text files while away from their computers (Natural Reader, 2008). TTS software reads text aloud, and such software provides a new way for students to experience their essays, potentially facilitating their own revision. Flower, Hayes, Carey, Schriver, and Stratman (1986) suggested that revision constitutes a “reseeing” of an essay, and TTS software may allow students to “resee” their essays through the use of a computerized voice. Sound is increasingly being used in computer composition and has been shown to have positive effects as it reframes our understanding of literacy (Shankar, 2006; Hewett, 2000), and one of the most common methods of proofreading, for example, is to have students read their essays aloud (Eschweiler, 1998; Madraso, 1993; Sobolik, 1975). However, reading aloud has clear limitations because students frequently read what they think they have written as opposed to what they have actually written. As Patrick Hartwell (1985) wrote in his landmark essay on grammar, “[m]ost students, reading their writing aloud, will correct in essence all errors of spelling, grammar, and by intonation, punctuation, but usually without noticing that what they read departs from what they wrote” (p. 121). The alternative method of proofreading, of course, is to have someone else read the draft aloud, and although this method has been shown to be more helpful with students with disabilities (Raskind & Higgins, 1995), recruiting a volunteer to read can become difficult. TTS software, however, requires no incentive to read and does not read what the writer thinks has been written but rather what has actually been written, with fairly accurate intonation, pronunciation, and emotion. So while writers and reading volunteers have limited abilities to catch errors due to cognitive separation from their own writing, a computer using TTS software will read exactly what is on the computer screen, thereby allowing the listener to hear the disjunction between what is believed to have been typed and what has actually been typed. TTS programs may be able to allow students to facilitate their own revision of drafts if they are unable to find a volunteer to read their drafts; however, current composition research is insufficient to determine whether or not TTS has much applicability beyond merely being a novelty. Three reasons have likely contributed to the slowness of research: (a) TTS software has been expensive; (b) the software has been largely inaccessible to the general public; and (c) the voices have generally been difficult to understand. However, these limitations apply to a much lesser degree due to recent improvements in TTS technology. AT&T Natural Voices (2008) are available for the price of commercial brand software, and the quality of the voices mimics human speech somewhat realistically. Furthermore, updated algorithms and programming techniques yield voices that rarely mispronounce words and contain the ability to accent different parts of the words depending on punctuation. Composition researchers are not fully exploring the abilities that this technology has to offer its students. This article documents an empirical study that reveals some of the potentials, as well as the pitfalls, that TTS programs offer both students and teachers of writing. 2. Literature review 2.1. Revising with computers First drafts of writing tend to be “writer-centered” as opposed to “reader-centered” (Flower, 1998), and revision equates to “re-seeing” a draft from the reader’s perspective rather than from the writer’s perspective. In this context, revision is simply a metaphor for “revisioning” the way that the current draft works and making changes accordingly. On a more historical and philosophical level, revision, according to Catherine Haar (2006), often means different things to different people. Haar broke revision into four different categories: 1) “correctness,” which is concerned with revising a text based on the norms of writing, 2) “development and discovery,” which is concerned more with the psychological tendencies of writers as they undergo recursive revision during the process of writing, 3) “rhetorical goalsetting and function,” which is concerned with revision as a text is thrust into a rhetorical situation of writer/reader/and

290

K. Garrison / Computers and Composition 26 (2009) 288–301

context, and 4) “assertion of identity,” which is concerned with how writers construct their identities while revising their texts to be unique and philosophical (2006, pp. 15-19). Perhaps the most useful distinction between types of revision is found in David Wallace and John Hayes’ (1990) study, which drew a distinction between “local and global” issues in revision. This current study will adopt a further distinction between revision as “proofreading” (which is concerned with correctness) and revision as “local” and “global” (which are more concerned with audience awareness). For the purposes of this study, the word “revision” encompasses proofreading changes, local changes, and global changes. This allows the word “revision” to exist on a continuum, with changes on the left-hand side dealing with grammar, punctuation, and areas of “correctness” and moving toward more complex ideas on the right-hand side such as “local” revision (word choice, sentence structure, flow, style) and finally “global” revision (organization, thesis, arguments). Computer technology has largely restructured the way that writers conceptualize revision. As Douglas Eyman and Colleen Reilly (2006) claimed, “the computer has affected the process of writing at every stage, from invention, through revision, to delivery” (p. 102), and as the personal computer continues to be the avenue that revision gravitates toward, research should be conducted to determine whether or not the computer has had a positive impact on the process of revision. Numerous studies on computer revision share both optimism and a level of skepticism toward writing with a computer. The optimism toward computers in studies of revision is due to the flexibility of word processing programs that can offer students a chance to improve their writing through both quantitatively more revisions and qualitatively better revisions (Goldberg, Russell, & Cook, 2003; Bean, 1983). However, such optimism is frequently challenged by authors who have found reasons to be skeptical about computers and revision. For example, computers tend to make an already complex task (writing) even more complex by introducing more tools (Crafton, 1996); computers often lead to a focus on only surface-level changes rather than content changes (New, 2002); and the suggestion that computers lead to more revision or better writing has been called into question (Hawisher, 1987). Oftentimes these studies, however, have a tendency to forget that, as technology studies have shown, a tool (such as a computer) is not intrinsically a reason for optimism or for pessimism; rather, as Gail E. Hawisher and Cynthia L. Selfe (1998) noted, technology and its use must be governed by informed educators and students. As Thomas Reynolds and Curtis Bonk (1996) showed, it is not necessarily that computers themselves encourage revision; however, computerprompted strategies that encourage revision can allow students to revise better. Such research places the onus of revision not on the computer as a tool but on the educator and level of technological and educational literacy. For example, although the use of word processing programs does not necessarily equate to better writing, research has suggested that feedback submitted through computer technologies can sometimes be more effective at encouraging revision than more traditional means. Frank Tuzi (2004), for instance, presented findings that electronic feedback helps second-language writers revise better than traditional oral methods. Such findings suggest that computers are useful for encouraging revision, specifically when guided by pedagogy. 2.2. TTS in business and education Despite competing claims that new technologies may or may not be enhancing the writing process in substantial ways, the same cannot be said for the impact of TTS research in business and education. Much research has been conducted to document the benefits and limits of text-to-speech in disciplines other than composition studies. TTS has been shown to have economic benefits in telephone-based transactions (Oberteuffer, 1995) and has saved businesses millions of dollars in telecommunications by allowing computerized voices to conduct phone transactions (Wilpon, 1995). TTS is also predicted to connect the world in a single language by allowing the computer to do the translations for us, thereby saving time and money on transactions (Kato, 1995). TTS educational researchers have also identified areas in which TTS may prove beneficial to students, such as the use of TTS to help ESL students learn new languages (Williams & Williams, 2000). TTS has been shown much attention in the field of disability studies. Because TTS reads text aloud, it has been hypothesized that TTS software could allow students, such as individuals with severe spelling disabilities, to use their ears to allow them to hear their spelling mistakes. Specialized software, such as Kurzweil 3000 (2007), is designed specifically for students with disabilities, giving researchers a potentially added impetus for documenting some of the possibilities of incorporating TTS into students’ education. The results of disability studies have been very promising. Research has revealed that TTS enables students with learning disabilities to improve word recognition skills (Olson & Wise, 1992; Torgesen & Barker, 1995; Lundberg, 1995); it increases motivation to read and is generally more

K. Garrison / Computers and Composition 26 (2009) 288–301

291

successful with older students who are better able to navigate the software (Montali & Lewandowski, 1996); and it is useful for college students who have reading and writing difficulties (Lees, 1985). Most significantly, perhaps, Raskind et al. (1995) showed that TTS allowed students with disabilities to catch a higher percentage of errors than receiving no assistance or receiving the limited assistance of having someone else read the essays aloud. Even more promising for researchers in composition, a dissertation conducted to determine whether or not synthesized speech aided students with learning disabilities revealed that text written by students using TTS scored significantly higher in a scoring rubric than students who used only paper and pencils or Microsoft Word. Studying 33 students, Zhang and Brooks (1993) recorded that students between the ages of 7.6 and 13.2, with the mean being 9.9 years of age, wrote significantly better short stories about their best friends using a modified TTS program than students who didn’t. Students using TTS used the software as a means of proofreading during the process of writing the stories. Though the study showed TTS helping students who had learning disabilities construct more grammatically correct essays, the study did not reveal which grammar mistakes TTS users identified most often, focusing instead on a holistic scoring approach. Though TTS seems to be a miracle technology for some researchers, several studies have suggested that TTS is actually quite limited. For Rena Lewis (1998), spell-check features worked better than TTS in editing for spelling errors, though Charles MacArthur (1996) demonstrated that while spell-check may be more useful, TTS was still helpful for 9-10 year-olds in correcting their spelling mistakes from 42-75% to 90-100% when combined with word prediction software. Further empirical research connecting TTS to writing was conducted by MacArthur (1999), who tested three students ages 9-10 to see if they wrote better in their journals with TTS than without. Testing students with severe spelling problems, his findings suggested that only one student received much benefit from the program, and the benefit was in the form of spelling better. MacArthur’s study was limited due to the small number of participants, and furthermore, the findings could not be generalized to academic populations due to the students all having severe spelling problems. 2.3. TTS in composition The benefits of TTS can be seen for educators involved with students who are non-traditional or who have disabilities; however, only one such study has been conducted on traditional students without disabilities. Improvements in writing due to the opportunity to hear and see words simultaneously have been suggested by education researchers Karin Borgh and Patrick Dickson (1992), who found that second to fifth graders proofread more carefully and were more motivated to write when they had TTS than when they didn’t. Their test, which included 48 students, asked participants to write stories; the students using TTS were able to make small-level editing changes that enhanced the quality of the writing compared to students without TTS. Moreover, confirming what disability studies have implied, the study showed that less-skilled writers benefited the most from inclusion of TTS. Beyond Borgh and Dickson’s (1992) study of children, research into the benefits of TTS in upper-level composition and rhetoric programs has been limited to research in writing centers. Tammy Conard-Salvo (2004, 2008) has presented preliminary research at the Computers and Writing Conference regarding how TTS relates to writing center tutorials at Purdue University. Her research has suggested that incorporating TTS in a Writing Center could potentially be positive. Additionally, in an attempt to gather information regarding students’ reactions to TTS, I recommended Natural Reader software to my own students and to clients in my university’s writing center. Though word-of-mouth research is limited as a research method, verbal and written feedback from students and clients who claim to have used TTS have suggested that the Natural Reader software is able to help them hear minor-level concerns such as wrong words, repetition of the same word, misspelled words, run-on sentences, fragments, and a number of other editing mistakes. Furthermore, the preliminary discussions with students and clients revealed that TTS might be useful for higher-level revision concerns as evident in the students’ reports that they spent time restructuring paragraphs, adding more content to flesh out places where their writing was lacking, rephrasing awkward sentences, and including more statistical data to enhance their claims. Although this preliminary data offers promising results, no empirical research documenting the effects of TTS on revision in college composition has been done. Borgh and Dickson’s (1992) study worked with children from second to fifth grade at a time when TTS faced technological and economical limitations, and the results suggest that it helps students proofread their essays and also allows them to feel more motivated to write because they can hear what they have written. The literature, then, has revealed two gaps in research: 1) it is unknown if TTS is beneficial for students

292

K. Garrison / Computers and Composition 26 (2009) 288–301

beyond the age of thirteen who do not have disabilities, and 2) it is unknown if TTS is as useful for facilitating local and global revision as it is for proofreading. Therefore, research needs to be done to explore the effectiveness of TTS for college students’ abilities to revise their own essays. 3. Methods 3.1. Participants To determine if TTS software is effective as a revision tool for college students, I conducted both a pilot test and a full-scale experimental study. The pilot test was implemented using a quasi-experimental design where six students were taken from my freshman composition class and randomly assigned to a test group and a control group. The pilot test data was used to refine the original coding scheme and the design of the study; therefore, the pilot test data was removed from the full-scale study. The following semester, a full-scale test was conducted using 52 (30 male and 22 female) students from two of my second-semester, freshmen composition classes. All of these students were required by the university to take at least two first-year writing classes (or an equivalent advanced placement class/test for the first semester’s class). Both classes were taught the same curriculum, and they met during back-to-back time slots (2:00 and 3:30) once a week for in-class instruction. All of them volunteered to be part of the project with no reimbursement for their time. 3.2. Data collection At Texas Tech University, the composition faculty has implemented a hybrid program where all first-semester students (ENGL 1301) and second-semester students (ENGL 1302) attend class once a week and submit all work online. The work is then assessed anonymously by a pool of graders. To emphasize the idea that writing is a process, the students in second-semester composition (1302) write two essays, with several drafts of each. For this study, I asked students to revise the second assignment from the first essay cycle, with the assignment description being to write a persuasive letter to a decision-maker to convince him or her to implement a change. For example, one student wrote a letter to the Texas Tech Traffic and Parking Department attempting to persuade them to create more parking spaces. This assignment was chosen for this project for three reasons. First, after students wrote their initial essays on this draft cycle, they later turned in a revised and expanded version of their essays, and as such, it was beneficial for the students to spend time revising an essay that they would have to revise anyway. Second, the word count criterion for this draft was a manageable 500-800 words—large enough to warrant substantial changes, yet small enough to revise in one session. And finally, the timing of the draft occurred early in the semester, which allowed for me to test students early in the semester rather than when other obligations (i.e., mid-terms and final exams) diverted their attention away from the experiment. After obtaining IRB approval from the university, students from two 1302 classes were randomly assigned into the control group and test group, and both were tested during class time in a computer lab with over 20 computers. Half the class worked on a class activity while the other half was being tested. The assignment criterion was used universally between all students, thereby mostly guaranteeing that each individual within both groups were working on similar drafts. The word count requirement, for instance, was between 500 and 800 words, suggesting that each group was working on a similarly sized draft. Equivalency between groups was assumed due to 1) randomization and 2) the participants being of similar educational backgrounds because the class was a university-required class for all students (except those who tested out, had advanced placement opportunities, were placed in remedial classes, etc.). Both groups were tested in a computer lab, and all individuals were given their own computer and handed a set of instructions. (See Appendix A for test group and Appendix B for control group). The instructions gave steps for how to retrieve their drafts from the online database, which revision ideas to focus on, and how to save their final drafts to the computer. For both groups, I asked the students to specifically revise their essays for two things: proofreading mistakes (such as punctuation, spelling, and other grammar concerns) and more local/global-level concerns (such as word choice, organization, and flow). It was assumed that because most of the students had taken the first-semester writing course (1301) the previous semester, and the subjects of revision were covered during the course of that class, the students would not be confused by any of the composition terms such as “grammar” or “organization.”

K. Garrison / Computers and Composition 26 (2009) 288–301

293

In both the pilot test and the full study, students had written their first drafts of the essay using either a home computer or a school computer. Because Microsoft Word (2000 version) was available as a free download for all students, used campus-wide on school computers and encouraged by the instructor, it was assumed that each student had composed their essay using Microsoft Word and its composing features, such as the spelling and grammar-check function. For the test, I gave students in the control group 20 minutes to spend revising their essays with Microsoft Word. All students in the control group were able to use Microsoft Word with no difficulty, and no restrictions were placed on students’ ability to revise according to Microsoft Word’s spelling and grammar-check. For the test study, the composition staff at Texas Tech University purchased four licensed copies of the Natural Voice Reader Professional Text-to-Speech software, version 4.10. The composition staff also purchased two voices for each of the four copies of the program: a female, AT&T’s Crystal 16HZ, and a male, AT&T’s Mike 16HZ. Both voices speak with an American dialect, and both speak any word that is presented to them, with only idiosyncratic words or phrases, such as “T-storm” for “thunderstorm,” spoken somewhat awkwardly. A test conducted by Automatic Speech Recognition (Text-to-Speech Accuracy Testing, 2005) found that AT&T’s voices were accurate 66.4% of the time out of twelve hundred possible phrases. If the pronunciation of particular words is somewhat off, the software includes the ability to manually manipulate individual words according to the user’s preference, though this was not demonstrated to the participants of the test group due to the 20-minute time limit. The program utilizes an interface similar to a text-editing program with several menu items, such as “cut,” “copy,” and “paste.” To use the software, participants in the software test group pasted a draft of their essay into the Natural Voice Reader’s text-editing interface, highlighted a section of text, and then clicked “Read.” The voice would then read that section of the text, and if the participants wanted to revise a portion of the text or change the speed of the speaker, then they would click “Pause,” change the text or speed, and then click “Resume.” For the test group, I spent several minutes showing participants how to use the Natural Voice Reader Text-to-Speech software, and then I allowed them to practice listening to a sample draft being read aloud. After the subjects felt comfortable navigating the program, I gave them 20 minutes to spend revising their essays. After each subject in the test and control groups finished the revision process, I used Microsoft Word’s “Compare and Merge Document” feature to identify all the changes made. 3.3. Data analysis Changes were coded into the following categories (see Table 1): positive changes (changes that enhanced the quality of the draft), neutral changes (changes that did not affect the quality), and negative changes (changes that Table 1 Coded Variables Positive Proofreading Changes Spelling Changes Punctuation Changes Editing Changes Positive Local/Global Changes Clarity Changes Neutral Changes Negative Proofreading Changes Spelling Changes Punctuation Changes Editing Changes Negative Local/Global Changes Clarity Changes

Examples from Participants’ Drafts Replacing “intrest” with “interest” or “rigging” with “ringing” Replacing “. . .University however I thought” with “. . .University; however, I thought” or “the areas high schools” with “the area’s high schools” Replacing “another thing that been affected” with “another thing that has been affected” or “if would be appreciated” with “it would be appreciated” Replacing “people do make mistakes and it is a fact of life” with “people do make mistakes” or “he won’t really ever use that knowledge again” with “he may not ever use that knowledge again” Replacing “wonderful” for “nice,” “walk” for the word “trek,” or “great” for “very good” Adding the word “attened” instead of “attend” or “reveral” instead of “reversal” Replacing “215,000” with “215000” or “repair their vehicle or their vehicle’s windows” with “repair their vehicle, or their vehicle’s windows” Replacing “to be in the best choir” with “to be in University choir” or “theory of evolution” with “Theory of Evolution” Replacing “There are four main areas in town that should be fixed first—the end of Midland Drive, the intersection at Midland Drive and Wadley, the entire length of Wadley, and Cimmaron Street.” with “There are four main areas in town that should be fixed first. To begin with there is the end of Midland Drive, and then there is the intersection at Midland Drive and Wadley, also the entire length of Wadley, and Cimmaron Street. . .”

294

K. Garrison / Computers and Composition 26 (2009) 288–301

introduced ambiguity or editing errors). The positive and negative changes were further broken down into two subcategories: 1) proofreading variables, which included “spelling” (if a misspelled word was corrected), “punctuation” (if a punctuation error was corrected), and “editing” (if other grammatical mistakes such as missing words were corrected), and 2) local and global revision variables which were denoted by the term “clarity” (if sentences were revised, words replaced with more specific denotations, organization changed, ambiguity eliminated, etc.). Other variables were also coded, such as the number of minutes taken to complete the task and the total number of changes made. The categories were chosen for several different reasons. First, the pilot test revealed that students using TTS were better able to identify spelling errors, were less inclined to make neutral changes, and were able to complete their tasks quicker than students using Microsoft Word. Thus, part of the categorical scheme was derived from an analysis of the pilot study. A second reason for using the coding scheme identified above was to determine if either local or global revision was taking place when using TTS. Because no previous study has been conducted that focused on local or global revision, the word “clarity” was used to denote all local and global revision changes that were made beyond sentence-level constructions because the word “clarity” implies recognition of “purpose, audience, and the overall organization of the text” (Wallace & Hayes, 1990, p. 2). This study also used the term “neutral” to denote local and global changes that did not improve the text because “neutral” implies that what is changed is unlikely to be the result of rhetorical awareness. A third reason for choosing these categories is that the prior studies exploring TTS have not revealed any significant variables for this study to draw upon; therefore, the categories were defined broadly (i.e., “punctuation” as opposed to comma splices, commas in introductory phrases, commas in compound sentences, etc.) in an effort to identify significant variables from which subsequent studies could work. Finally, the relatively smaller number of subjects (52) disallowed for any significant understandings to be determined by limiting the categories by sharper distinctions. 4. Results Coding was done anonymously by two people: myself and a professional editor hired to code each of the drafts. A Pearson r correlation revealed that the coders’ inter-rating reliability was .806, and an intraclass correlation test revealed that the inter-rater reliability was .793 between all coded variables. Lawrence Frey et al. (2000) argued that an inter-rater reliability should be .7 or higher to insure that the results are accurate (p. 115). After averaging the two coders’ rankings, the data of the two coders was analyzed and run through SPSS 15.0 (2007) to determine which variables were significantly impacted by the use of TTS. When analyzed, SPSS determined that the data was parametric; therefore, an analysis of variance (ANOVA) test was used to compare scores across a range of variables to see if there was a significant difference between groups. The ANOVA revealed four variables (see Table 2) to be significant according to the .05 level. First, the number of positive clarity changes in the control group was higher than the number in the test group: F (1, 50) = 8.170, p < .05, indicating that the users in the control group were able to identify local and global changes better than TTS users. Second, the number of total positive changes in the control group was higher than the number in the test group: F (1, 50) = 4.858, p < .05, indicating that users in the control group made more useful changes to their essays than TTS users. Third, the number of neutral changes in the control group was higher than the numbers in the test group: F (1, 50) = 11.764, p < .05, indicating that users in the control group spent more time making neutral changes than their TTS counterparts. And finally, the number of total changes was higher in the control group than the numbers in the TTS group: F (1, 50) = 6.388, p < .05, indicating that Word users made more total changes than TTS users. Based on these results, it can be determined that users in the control group focused on making more changes, both positive revision changes and neutral changes. Table 2 Significant Variables

Mean/Std. Dev. For TTS

Mean/Std. Dev. For Control

Level of Sig.

Total Number of Positive Clarity Changes Total Number of Positive Changes Total Number of Neutral Changes Total Number of Changes

1.935/2.007 4.674/3.661 1.565/1.532 7.957/5.584

3.862/2.692 7.000/3.871 3.966/3.062 12.259/6.469

.006 .032 .001 .015

K. Garrison / Computers and Composition 26 (2009) 288–301

295

5. Discussion The results suggested three major points of discussion: 1) none of the proofreading variables (spelling, punctuation, editing) showed significant difference between the control and test groups, 2) users in the control group made significantly more global changes that enhanced their essays than TTS users, and 3) TTS users made fewer “neutral” changes than users in the control group. Each point is elaborated below. The first significant finding of this study is that none of the proofreading variables showed any significant difference. Because the control group averaged 3.138 positive proofreading changes and the test group averaged 2.739 positive proofreading changes, both TTS and Microsoft Word are somewhat useful for identifying proofreading mistakes, though there was no significant difference between the two groups, F (1, 50) = .328, p > .05. Therefore, contrary to the results presented by Borgh and Dickson (1992), the results of this study showed that the Natural Reader software performed no better than Microsoft Word for helping college students identify proofreading mistakes. All three categories (spelling mistakes, punctuation errors, and editing errors) did not report significant differences between the two groups. Also, there was no significant difference in the amount of time it took for groups to complete the assigned tasks (only one student in the control group actually took all 20 minutes), which further enhanced the finding that students received no proofreading advantage between the two methods. One conflicting variable that should be mentioned is the presence of the spelling and grammar-check in Microsoft Word. Even though students in both the test and control groups likely composed and revised their first drafts on Microsoft Word, the presence of the spelling and grammar-check during the revision process of the second drafts could have possibly influenced decision-making in the control group that the test group did not have available. However, even with the possibility of the spelling and grammar-check influencing the control group, the results show neither group as having an advantage. Such a finding suggests that even if the control group did have an advantage with the spelling and grammar-check, they still performed no better in the statistical analysis than the test group. This finding has two basic significances. First, TTS software did work to help students proofread their essays—an incentive for offering it to students of composition. Even subtracting the total number of negative proofreading errors from the total number of positive proofreading changes reveals that the control group had a net average gain of 2.138 positive changes and the test group had a net average gain of 1.217 positive changes, though there was no significant difference between the two groups, F (1, 50) = 1.790, p > .05. Second, TTS has been shown to improve the proofreading abilities of students with disabilities, such as in the case of MacArthur (1999) who found that it did help a student with severe spelling problems. Based on these findings, it is likely that TTS, broadly distributed to students, would not help students proofread any better or worse than simply suggesting they continue to use a word-processing program or Microsoft Word for proofreading purposes, but it is probable that, as with students with disabilities, TTS may allow certain individuals access to a tool that could possibly allow them to proofread more effectively than simply using a word-processing program or Microsoft Word alone. Further research might show, for instance, that auditory learners might benefit from TTS more than visual learners. A second significant finding of this study is that users in the control group made more local and global changes that enhanced their essays than TTS users. Because the literature review did not reveal any studies that tested higher-level revision as a variable, one of the major questions that this study attempted to answer is whether or not TTS software positively impacts local and global revision. The results suggested that it did, though not as well as using Microsoft Word. TTS users revised, on average, 1.935 places in their essays, indicating that the Natural Reader program allowed for students to “hear” breakdowns in clarity and organization. However, users in the control group averaged 3.862 changes per draft, indicating that Word users are even more inclined toward positive revision than TTS users. For example, one student made a claim in his first essay which said that it would be “convenient to locate recycling bins near the newsstands.” After reading his essay on Microsoft Word, he clarified his assumption and stated two reasons as to why it would be convenient for the recycling bins to be near the newsstands. Such a revision not only increased the cogency of the argument, but it also strengthened the essay. This finding is important for two reasons. First, according to this study, using TTS to facilitate local and global revision, perhaps more complex tasks than proofreading, might be less beneficial than using Microsoft Word. Such a finding implies that students who used TTS were more inclined toward surface-level issues rather than higher-level concerns. Composition instructors, then, should be cautious about advocating TTS for local and global revision; at the same time, however, TTS may be more useful for encouraging students to pay attention to surface-level editing, and if used in conjunction with one another, TTS software and Microsoft Word may work at encouraging substantive

296

K. Garrison / Computers and Composition 26 (2009) 288–301

revision, both local and global. Second, one of the limitations of this study is that a pre-test questionnaire revealed that none of the users of TTS had ever used the Natural Reader software before. Each person was only given five minutes to familiarize themselves with the program. Though it is not necessarily difficult to operate the program (mostly they were just required to highlight a passage and click “Read”), most students listened to their essays being read aloud and then changed text where the computer had a tendency to “hiccup” in its speech. Post-test discussions with students indicated that the quality of the voices in the Natural Reader software was “surprising” in effect, and that when one student, for example, heard the computer read the passage “it can be hard to remember all the prices to check when rigging out” in reference to her letter concerning her job as a cashier, she heard the word “rigging” and immediately replaced it with the word “ringing.” More familiarity and comfort with the TTS software could potentially increase their ability to achieve a higher level of both local and global revision on their own essays. Further studies should be conducted to identify, in a longitudinal format, how students might use the technology. Finally, a third significant finding of this study is that TTS users made fewer “neutral” changes than users in the control group. By neutral changes, this study denoted changes that, based on the context of the usage, ultimately did not influence the quality of the essay in a positive or negative way, and as such, any change that does not influence the quality of the draft is possibly wasted time for the student. To some degree, small changes like changing “very good” to “great” could be construed as beneficial (i.e., elimination of wordiness); however, an audience would be unlikely to pick up on these changes because the changes neither 1) distract the reader with local errors nor 2) help the reader better conceptualize the argument. For example, one student initially wrote that “it would be heart wrenching to watch this team fall apart” in reference to an athletic event at the university, and after re-reading her essay, she changed “wrenching” to “breaking.” Both the phrases “heart wrenching” and “heart breaking” are clichés that carry similar connotative meanings, and from the audience’s perspective, the rhetorical purpose is identical: to use an emotionally-laden term that connects the reader emotionally to the situation. Such an example was coded as “neutral” because while the student has revisited word choice, the student has not revisited her use of the argument, offered the reader a more coherent perspective, or otherwise identified an underlying reason for why the word choice should be revised. On average, TTS users made 1.565 changes that were coded as being “neutral” changes while users in the control group made 3.966 changes per draft. In essence, though users in the control group were able to proofread as well as TTS users and revise locally and globally better than TTS users, they were also prone toward making large numbers of changes that had little to no impact on their essays. The significance of this finding is substantial for two reasons. First, the fourth variable the ANOVA revealed to be significant was that users in the control group made far more changes than TTS users. As seen in Table 2 above, users in the control group were likely to make nearly five more total changes than TTS users. However, because users in the control group were also much more likely to make neutral changes that did not impact their drafts, the significance of the fourth variable was negated when the total number of neutral changes was subtracted from the total number of changes, F (1, 50) = 2.451, p > .05. Second, even though TTS users were less inclined to make changes on their essays, their hesitancy to change parts of their essays perhaps indicated a critical eye toward not simply making changes for changes’ sake but rather making only changes that enhanced or decreased the quality of their essays (or perhaps, once again, it indicated that students were unfamiliar with the software). Or, possibly, the tendency of the control group to make neutral changes might also indicate that users of Microsoft Word were attempting to make more significant changes (such as word choice) but were uncertain about how to do so, or perhaps they were simply more inclined toward revising their essays for personal preference rather than because of audience awareness. Based on these three significant findings, it could be suggested that using TTS in composition pedagogy might allow students to have access to a useful tool for writing essays, mostly for proofreading-level changes. The old standard of “read your draft aloud” may still be quite useful for facilitating proofreading, but more research should be undertaken to determine if sound, for particular users and in other writing contexts, allows for higher-level revision to occur and as well to determine why students were less likely to revise locally, globally, and neutrally after hearing their essays. 6. Conclusion The results of this study show evidence that TTS does work for proofreading, though not better, which suggests that TTS is not more helpful than Microsoft Word, as suggested by Borgh and Dickson (1992). Furthermore, this

K. Garrison / Computers and Composition 26 (2009) 288–301

297

Table 3 Summary of the Most Commonly Studied Composition Variables. Mechanics

Unity/Focus

Voice Style Tone Structure/Language Use Audience Content Commitment Coherence/Competence Creativity

Punctuation Purpose Theme Word Choice/Vocabulary Setting Grammar Characterization Organization Emotion

study’s findings suggest that composition teachers might incorporate TTS software/instruction into their classrooms in order to allow students to potentially maximize their success at achieving writing autonomy. Though not everyone will find the software as beneficial as using only Microsoft Word or another word processor, past studies on students with disabilities (such as people with spelling disabilities) have shown that students do benefit from incorporating TTS software into their writing process. Because this study also supports the generalized finding that TTS software is useful for proofreading and also for local and global revision (though less so), more research should be conducted in order to pinpoint and elaborate on exactly how much more or less effective it is over more traditional methods. TTS’s influence on writing is still largely unknown, but this study confirms what Hawisher and Selfe (1998) noted concerning the indeterminacy of technology. TTS should not be hailed as a revolutionary technology that can cure all students’ revision difficulties, but rather TTS must be guided by an informed pedagogy and an engaged listener. As writing and literacy studies increasingly becomes multimedia-oriented, students should be educated on how to use TTS software, shown the benefits and limits of TTS, and also made aware that local and global revision should continue beyond TTS. This study does show, though, that TTS can be helpful and that students are revising when listening to their text read aloud. Knowing this should give composition instructors yet another computer tool for encouraging students to continue to re-conceptualize their writing, both as process and product. As discussed above, this study only conducted an analysis of a few common variables as it relates to revision, and future studies should increase the number of coded variables. One study that might provide a starting point for future studies is that of Amie Goldberg et al. (2003), who determined the most common variables tested in composition studies, as shown in Table 3. Future studies might include these additional variables to determine the effectiveness of TTS software. Such studies should increase the sample size while also running more tests. For example, knowing if the grades were significantly impacted from one group to another would be worthwhile knowledge; pinpointing even more specific variables, such as auditory vs. visual learners or native vs. ESL speakers, would be useful in determining the effectiveness of TTS as a revision tool. Most importantly, however, longitudinal studies should be conducted to determine if students improve with TTS over a longer period of time. Besides future research, TTS could possibly be used in other contexts as well. For example, Texas Tech uses a locally-produced online program that all students use to submit their drafts in first-year composition courses. It would be relatively cheap to include a TTS interface in the submission procedures that would allow students to click “read” prior to submission in order to have a voice read their essays. Or, in the context of a writing center, tutors could always direct clients toward computers equipped with headphones and TTS programs that would allow clients to revise their essays prior to a discussion between the tutor and client. With iPods or MP3 players, students could save their essays (or course readings) as MP3 files (by utilizing Natural Reader’s ability to save text as sound files), load them onto a player, and then listen to their work as they walk across campus. In any of these cases, TTS can be utilized in innovative ways that would allow students to combine sound, text, listening, writing, and revising into a singular format. Composition instructors are always searching for ways to help students enhance the quality of their writing as well as to help students feel more autonomous in their writing. Because basic TTS software is free or cheap, effective,

298

K. Garrison / Computers and Composition 26 (2009) 288–301

and useful for revision, instructors might enhance a student’s educational experience by taking the time to explain the potential benefits and limitations of this software to students in freshman-level composition and other composition classes. This study showed that TTS is helpful as a proofreading tool, and the findings also suggested that TTS can be helpful as a local and global revision tool, though at a lesser level than Microsoft Word. Sound should be incorporated into composition so that students have the greatest number of opportunities available to them in order to succeed in education and in the workplace. Acknowledgments First and foremost, I would like to thank Dr. Angela Eaton for inspiring me with her enthusiasm. As well, her overseeing of the project, her providing commentary on numerous drafts, and her giving of her time to talk about nuances of my research methods was immensely helpful. This project would not have been started or completed without her. I would also like to thank the Texas Tech composition faculty—Dr. Rich Rice, Dr. Fred Kemp, Dr. Susan Lang, and Dr. Rebecca Rickly—for providing funding for the Natural Reader software and funding for the second coder, and to Dr. Kathleen Gillis for allowing me to use the Writing Center as a hub for preliminary testing. I would also like to thank the anonymous reviewers for their feedback—their insightful commentary reshaped this article in numerous, positive ways. Finally, my wife, Katherine, was instrumental in providing support and in serving as the second coder. Appendix A. Beginning Steps: 1. 2. 3. 4. 5. 6. 7. 8. 9.

Place the headphones on your ears Open Internet Explorer and login to Topic – http://ttopic.english.ttu.edu/ Click “Manage Assignments” Click “1.2 Formal Letter to a Decision Maker” Copy your entire 1.2 draft by highlighting it, and then press “Control + c” Minimize Internet Explorer On the desktop, open the light blue icon that says “Natural Voice Reader Pro” Highlight the opening text and click “Read” – this explains how to use the program Once you are done listening to the instructions, then highlight the introductory text, delete it, and paste your 1.2 draft into this program by pressing “Control + v” 10. Save your draft to the desktop. The filename should be your first and last name with the word “Unedited” attached to the end of the name. So if your name is “John Doe” then save the document as “JohnDoeUnedited.” Intermediate Steps: When I say “begin,” edit and revise your 1.2 draft by listening to your draft being read aloud to you. If at any point you want the voice to stop, click “Stop” and then make the necessary changes. Among your focus should be such concerns as: 1. 2. 3. 4. 5. 6.

Grammar Spelling Punctuation Word choice Organization Flow

You will have up to 20 min to complete this task. Keep in mind that you don’t have to take the entire time. Listen to the draft as many or as few times as you prefer. When you think your draft is edited and revised thoroughly, then proceed to the final steps.

K. Garrison / Computers and Composition 26 (2009) 288–301

299

Final Steps: 1. Click “File” and “Save As” 2. Save your edited document to the desktop as your first and last name with the word “Edited” attached to the end of the name. So if your name is “John Doe” then save the document as “JohnDoeEdited.” 3. Leave Microsoft Word and Internet Explorer open 4. Quietly leave the room Note: If at any point you are confused, then simply raise your hand, and I will help you. Appendix B. Beginning Steps: 1. 2. 3. 4. 5. 6. 7. 8.

Open Internet Explorer and login to Topic – http://ttopic.english.ttu.edu/ Click “Manage Assignments” Click “1.2 Formal Letter to a Decision Maker” Copy your entire 1.2 draft by highlighting it, and then press “Control + c” Minimize Internet Explorer Open Microsoft Word Paste your 1.2 draft into Word by pressing “Control + v” Save your draft to the desktop. The filename should be your first and last name with the word “Unedited” attached to the end of the name. So if your name is “John Doe” then save the document as “JohnDoeUnedited.”

Intermediate Steps: When I say “begin,” edit and revise your 1.2 draft. Among your focus should be such concerns as: 1. 2. 3. 4. 5. 6.

Grammar Spelling Punctuation Word choice Organization Flow

You will have up to 20 min to complete this task. Keep in mind that you don’t have to take the entire time. When you think your draft is edited and revised thoroughly, then proceed to the final steps. Final Steps: 1. Click “File” and “Save As” 2. Save your edited document to the desktop as your first and last name with the word “Edited” attached to the end of the name. So if your name is “John Doe” then save the document as “JohnDoeEdited.” 3. Leave Microsoft Word and Internet Explorer open 4. Quietly leave the room Note: If at any point you are confused, then simply raise your hand, and I will help you. References AT&T Natural Voices. (2008). Retrieved from http://www.naturalvoices.att.com Atkinson, Tom, Neal, Jerry, & Greches, Marilyn. (2003). What works for me: Microsoft Windows XP accessibility features. Intervention in School and Clinic, 38(3), 177–180.

300

K. Garrison / Computers and Composition 26 (2009) 288–301

Bean, John C. (1983). Computerized word-processing as an aid to revision. College Composition and Communication, 34(2), 146–148. Borgh, Karin, & Dickson, W. Patrick. (1992). The effects on children’s writing of adding speech synthesis to a word processor. Journal of Research on Computing in Education, 24(4), 533–544. Conard-Salvo, Tammy. (2004). Beyond disabilities: Text-to-speech software in the writing center. Computers &Writing Conference, Honolulu, Hawaii. Crafton, Robert E. (1996). Promises, promises: Computer-assisted revision and basic writers. Computers and Composition, 13(3), 317–326. Eschweiler, Kristin A. (1998). Proofreading prowess. Women in Business, 50(4), 36–39. Eyman, Douglas, & Reilly, Colleen. (2006). Revising with word processing/technology/document design. In Alice Horning, & Anne Becker (Eds.), Revision: History, theory, and practice (pp. 102–116). West Lafayette, IN: Parlor Press. Flower, Linda. (1998). Problem-solving strategies for writing in college and community. Fort Worth, TX: Harcourt Brace College. Flower, Linda, Hayes, John R., Carey, Linda, Schriver, Karen, & Stratman, James. (1986). Detection, diagnosis, and the strategies of revision. College Composition and Communication, 37(1), 16–55. Frey, Lawrence R., Botan, Carl H., & Kreps, Gary L. (2000). Investigating communication: An introduction to research methods. Needham Heights, MA: Allyn and Bacon. Goldberg, Amiel Russell, Michael, & Cook, Abigail G. (2003). The effect of computers on student writing: A meta-analysis of studies from 1992-2002. Journal of Technology, Learning, and Assessment, 2(1), 1–52. Haar, Catherine. (2006). Definitions and distinctions. In Alice Horning, & Anne Becker (Eds.), Revision: History, theory, and practice (pp. 10–24). West Lafayette, IN: Parlor Press. Hartwell, Patrick. (1985). Grammar, grammars, and the teaching of grammar. College English, 47(2), 105–127. Hawisher, Gail E. (1987). The effects of word processing on the revision strategies of college freshmen. Research in the Teaching of English, 21(2), 145–159. Hawisher, Gail E., & Selfe, Cynthia L. (1998). Reflections on computers and composition studies at the century’s end. In Ilana Snyder (Ed.), Page to screen: Taking literacy into the electronic era (pp. 3–20). Florence, KY: Routledge. Hewett, Beth L. (2000). Characteristics of interactive oral and computer-mediated peer group talk and its influence on revision. Computers and Composition, 17(3), 265–288. Kato, Yasuo. (1995). The future of voice-processing technology in the world of computers and communications. In Proceedings of the National Academy of Sciences of the United States of America, 92, 22 (pp. 10060–10063). Kurzweil 3000. (2007). Kurzweil Educational Systems. Retrieved from http://www.kurzweiledu.com Lees, Elaine O. (1985). Proofreading with the ears: A case study of text-to-voice performance of a student’s writing. Collegiate Microcomputer, 3(4), 339–344. Lewis, Rena B. (1998). Enhancing the writing skills of students with learning disabilities through technology: An investigation of the effects of text entry tools, editing tools, and speech synthesis. Unpublished Report for U.S. Department of Education. Retrieved from http://eric.ed.gov/ERICWebPortal/contentdelivery/servlet/ERICServlet?accno=ED432117 Lundberg, Ingvar. (1995). The computer as a tool of remediation in the education of students with reading disabilities: A theory-based approach. Learning Disabilities Quarterly, 18(2), 89–99. MacArthur, Charles A. (1996). Using technology to enhance the writing processes of students with learning disabilities. Journal of Learning Disabilities, 29(4), 344–354. MacArthur, Charles A. (1999). Word prediction for students with severe spelling problems. Learning Disability Quarterly, 22(3), 158–172. Madraso, Jan. (1993). Proofreading: The skill we’ve neglected to teach. English Journal, 82(2), 32–41. Montali, Julie, & Lewandowski, Lawrence. (1996). Bimodal reading: Benefits of a talking computer for average and less skilled readers. Journal of Learning Disabilities., 29(3), 271–279. Natural Reader. (2008). NaturalSoft. Retrieved from http://www.naturalreaders.com New, Elizabeth. (2002). Computer-aided writing in French as a foreign language: A qualitative and quantitative look at the process of revision. The Modern Language Journal, 83(1), 81–97. Oberteuffer, John A. (1995). Commercial applications of speech interface technology: An industry at the threshold. In Proceedings of the National Academy of Sciences of the United States of America, 92, 22 (pp. 10007–10010). Olson, Richard K., & Wise, Barbara W. (1992). Reading on the computer with orthographic and speech feedback. Reading and Writing: An Interdisciplinary Journal, 4, 107–144. Raskind, Marshall H., & Higgins, Eleanor. (1995). Effects of speech synthesis on the proofreading efficiency of postsecondary students with learning disabilities. Learning Disability Quarterly, 18(2), 141–158. Reynolds, Thomas H., & Bonk, Curtis Jay. (1996). Facilitating college writers’ revisions within a generative-evaluative computerized prompting framework. Computers and Composition, 13(1), 93–108. Shankar, Tara R. (2006). Speaking on the record: A theory of composition. Computers and Composition, 23(3), 374–393. Sobolik, Gayle A. (1975). It pays to be sure—Proofread!. Journal of Business Education, 50(5), 188–189. SPSS 15.0. (2007). Statistical analysis software. Retrieved from http://spss.com/ Text-to-Speech Accuracy Testing. (2005). Automatic speech recognition. Retrieved from http://www.asrnews.com/accuracy.htm Torgesen, Joseph K., & Barker, Theodore A. (1995). Computers as aids in the prevention and remediation of reading disabilities. Learning Disabilities Quarterly, 18(2), 76–87. Tuzi, Frank. (2004). The impact of e-feedback on the revisions of L2 writers in an academic writing course. Computers and Composition, 21(2), 217–235. Wallace, David L., & Hayes, John R. (1990). Redefining revision for freshmen. Occasional Paper, 21, 1–10.

K. Garrison / Computers and Composition 26 (2009) 288–301

301

Williams, Henry S., & Williams, Penny N. (2000). Integrating reading and computers: An approach to improve ESL students [sic] reading skills. Reading Improvement, 37(3), 98–100. Wilpon, Jay G. (1995). Voice-processing technologies: Their application in telecommunications. In Proceedings of the National Academy of Sciences of the United States of America, 92, 22 (pp. 9991–9998). Zhang, Yuehua, & Brooks, David W. (1993). The holistic quality of texts created by elementary students with learning disabilities is improved when appropriate computer-based software is employed as the writing tool. (Doctoral Dissertation, University of Nebraska-Lincoln, 1993). Retrieved from http://dwb4.unl.edu/Diss/YZhang/YZDissPaper.html