- Email: [email protected]
A proactive recommendation system for writing: Helping without disrupting
International Journal of Industrial Ergonomics 39 (2009) 516–523
Contents lists available at ScienceDirect
International Journal of Industrial Ergonomics journal homepage: www.elsevier.com/locate/ergon
A proactive recommendation system for writing: Helping without disrupting ˜ oz Ramos b Mari Carmen Puerta Melguizo a, *, Lou Boves a, Olga Mun a b
Department of Language and Speech, Radboud University, P.O. Box 9103, Erasmus building (8th floor), 6500 HD Nijmegen, The Netherlands Department of Experimental Psychology, Faculty of Psychology – Campus de Cartuja, 18071 Granada, Spain
a r t i c l e i n f o
a b s t r a c t
Article history: Received 18 December 2007 Received in revised form 14 July 2008 Accepted 16 October 2008 Available online 29 November 2008
Writing professional documents are situations in which finding relevant, related information is important. Seeking for information is however not easy. A proactive recommendation system (PRS) retrieves information relevant to the written text, and presents it without user intervention. The information presented can improve the quality of the text but can also interrupt the process of writing. We describe ` Propos. Our goal is to develop a PRS for two experiments performed to optimize the design of the PRS A professional writers that present information non-intrusively and timely so as to minimize disturbing the writing process. The results of our experiments suggest that the tendency to interleave (re)search and writing implies a need for integrating the PRS in a writing environment able to support the task of writing the text itself and the stage of planning the contents and structure of the document.
Keywords: Proactive recommendation system Information seeking Model of the writing process Planning Editing Interruptions
Relevance to industry: While automation has increased the efficiency of repetitive processes, the impact on professional tasks has been minor. Professional writing is a complex task that takes a lot of time. The PRS that we are developing can result in substantial reductions of the cost of writing documents, and in substantial improvements of the quality of the written texts. Ó 2008 Elsevier B.V. All rights reserved.
1. Introduction Writing is a complex task and several computer systems have been developed in order to support writing. Up to now, most systems developed to support writing (or computer writing environments as Britton and Glynn (1989) called them), seem to be designed with the main purpose of supporting the processes of planning and reviewing the written text. In general, systems have been built to help writers with grammatical and spelling errors or to help them formulate external visual representations of their ideas and connect the main issues that should be written in the paper, the ordering of the sections, etc. For example, the Writing Environment (WE) developed by Smith et al. (1987) was one of the first systems developed to support the processes of organization and editing by including a network mode, a tree mode, an editing mode, and a text mode. The whole process of writing a document, however, also requires looking for information and especially with the advent of the Internet, more often than not, writing is now interleaved with searching for new or additional information. The question is whether repeated switching between the tasks of writing and searching is efficient, and whether it tends to result in the best
* Corresponding author. Tel.: þ31 (24) 361 5820; fax: þ31 (24) 361 2907. E-mail addresses: [email protected] (M.C. Puerta Melguizo), [email protected] (L. Boves), [email protected] (O.M. Ramos). 0169-8141/$ – see front matter Ó 2008 Elsevier B.V. All rights reserved. doi:10.1016/j.ergon.2008.10.004
possible quality of the texts. It has been proposed that task switching can be alleviated by a proactive recommendation system that searches and presents relevant information. However, such systems may interrupt the writing process in other ways, and by doing so, they may affect user satisfaction, efficiency and the quality of the resulting documents. This paper presents the results of two experiments designed to investigate the impact of a proactive recommendation system. The experimental results raise the question whether, in order to be effective and efficient, proactive recommendation systems should be integrated in more comprehensive writing support environments. 2. Proactive recommendation systems for writing Proactive recommendation systems (PRSs) retrieve large quantities of documents, decide what available information is most likely relevant to the text being written, and offer that information without user requests. The decision about what information to offer is mainly based on the text that is currently being written, in combination with personal profiles and profiles of the working group to which the person belongs. Only a few PRSs have been specifically developed to support writing in professional settings. For example, the Remembrance Agent (Rhodes, 2000) suggests personal email and documents based on text being written. Watson (Budzik and Hammond, 1999) performs automatic Web searches based on text being written or read. IntelliGentÔ is yet another PRS that proactively submits queries to a potentially large number of
M.C. Puerta Melguizo et al. / International Journal of Industrial Ergonomics 39 (2009) 516–523
517
search engines and presents the retrieved information while the user is writing a document. A serious problem with all of these PRSs is that they are developed as search support tools and do not seem to take into account the specific characteristics of the task at hand. Writing professional documents is a complex and highly demanding task that can be seriously affected by any type of interruption from the environment.
To be helpful the retrieved information has to be precise and highly relevant to the written document, but also it should be presented in a non-intrusive and timely manner. If the way of presenting the information seriously interferes with the main task, users might stop using the system. In the experiments presented in this paper we studied the effects that presenting proactive information by IntelliGentÔ can have on writing performance.
2.1. An IntelliGentÔ proactive recommendation system
3. The effects of interruptions on writing performance
As a tool for research, we are performing experiments with IntelliGentÔ. The system proactively submits queries based on a broadly defined user profile in combination with what the user is currently typing or reading. The system presents the retrieved information to the user proactively and immediately. The results of the search are presented in a semi-transparent window located in the bottom right of the screen (see Fig. 1). The window contains URLs related to what the user is typing. As the user moves the cursor over the references, the URLs become fully visible and active. On clicking the required URL, the user accesses the corresponding document. The information in the window is changed depending upon the text that is being input and new queries are created. The URLs in the transparent window also change as the user moves the cursor while reviewing previously written parts of the document, again on the basis of queries created from the text in the paragraph in which the cursor is located. For more information see Puerta Melguizo et al. (2007).
Presenting information proactively, as IntelliGentÔ does, can provide the user the opportunity to do a better job in less time (Maglio and Campbell, 2000). However, proactive presentation of information can be considered as an interruption that imposes extra-task demands on user’s cognitive resources. The effects of interruptions on the user’s main task have been studied frequently (e.g. Bailey et al., 2000; Piolat et al., 2001; Zijlstra et al., 1999). For example, Bailey et al. (2000) used six tasks with different cognitive load and two interruption tasks. They found an increase of the time spent on the main task when interruptions were present. The authors suggest this is due to the additional time needed to resume the main task after an interrupt. Summarizing, we think that the presentation of information by a PRS may have a detrimental effect, because the additional task (i.e. to check if the offered papers are interesting to the writer) has specific demands and there is the additional need to reorientate and resume the original task at a later moment. This reorientation
Fig. 1. IntelliGentÔ system for proactive information retrieval.
518
M.C. Puerta Melguizo et al. / International Journal of Industrial Ergonomics 39 (2009) 516–523
requires the user to remember the status of the writing (i.e. to complete some argument in the text being written). Therefore, we investigated if the peripheral pop-up of IntelliGentÔ distracts writers from their main task. In addition, we also investigated if IntelliGentÔ can improve the quality of the final paper when it presents relevant information. 4. The cognitive model of writing The need for a PRS might differ from one moment to another during the process of writing a document. Consequently, to optimally support professional document writing through a PRS, it is necessary to explore the stages involved in writing and the different effects that interruptions can have in these processes. Since the beginning of empirical research scholars have agreed that writing involves at least three different cognitive processes, usually called ‘planning’, ‘translation’ and ‘review/editing’. The most widely accepted and influential model of writing is the one proposed by Hayes and Flower (1980). These authors challenged the traditional linear sequence models of writing by proposing that there is a continuous recursive and non-linear interaction between the planning, translation and review/editing processes. Their model defines three main components: the writing process proper (which includes the three processes/stages mentioned above), the task environment and the writer’s long-term memory (see Fig. 2). Writing happens in three stages: Planning, Translating, and Reviewing. During Planning, ideas are generated, arranged into a coherent structure and goals are set. Planning involves retrieving domain knowledge from the writer’s Long-Term memory (LTM) and organizing it into a plan. This plan specifies, among other things, the effects that the writer wants (or needs) the text to have on the prospective readers. During the process of Translating writer’s plans and goals are transformed into sentences. In the Reviewing stage the writer evaluates the relation between the text written so far and the linguistic, semantic and pragmatic aspects that would best serve the writing goal. Reviewing involves reading and editing. Reading allows to detect errors or weaknesses and to evaluate the appropriateness of the written text in relation to the goals established during planning. Editing appears as a system of production rules that result in changes to the text.
The task environment includes everything existing outside the writers’ mind and that can influence the writing task. The main elements included here are the text produced so far and the socalled rhetorical problem (the writing assignment, the specification of topic and the audience). In the writer’s LTM is stored the writer’s knowledge about the topic, the knowledge of sources based on literature search, the writing plans and the knowledge about the audience who will read the work. Hayes (1996) revised the model, emphasizing the role of working memory, as well as socio-cultural and motivational aspects in writing. Another important addition of Hayes is the inclusion of the physical environment as an important factor that influences writing. In the physical environment, together with the text produced so far, the composing medium or tool used to write is included. Variations in the medium seem to lead to differences in the way people carry out the writing task. For example, Haas (1996) found that writers tend to plan more when they write on paper than when using a word processor, possibly because it is easier to sketch, to draw and to interconnect ideas using pen and paper. Haas also found that writers tend to revise documents on a general level (i.e. modifying the structure) when using a pen, and more on a local level (i.e. revision of syntax, semantics, vocabulary) when working on screen. These results suggest that the introduction of computer tools as a medium for writing seems to stimulate users to change the processes they use, rather than making the original processes easier or more effective. 5. Finding relevant information during writing Searching information is a difficult and time consuming task. Nowadays, search engines have become the primary tool for information access in both the Internet and company-internal networks. However, searching and navigating are often not as efficient and easy as users might like. While searching for information, users frequently feel ‘‘lost in hyperspace’’. Actually, feeling lost or the tendency to lose one’s sense of location is the most common problem users experience while navigating and strongly affects user’s performance and satisfaction (Gwizdka and Spence, 2007; Puerta Melguizo et al., 2006). Ironically, the very potential strength of hypertext, namely improving the management of loose
Fig. 2. The model of writing proposed by Hayes and Flower (1980).
M.C. Puerta Melguizo et al. / International Journal of Industrial Ergonomics 39 (2009) 516–523
collections of relatively unstructured information, turned out to be a major potential weakness (Neuwirth and Kaufer, 1989). Furthermore, keyword-based search is still inefficient and relevant information may be missed because the writer did not realize that the information exists and could be looked up. Also considerable time is spent interacting with low-precision search engines. Consequently, the time in which the author is away from creating the document can have a negative impact on the total time spent, and on the eventual quality of the text. 5.1. Writing and seeking for information: an initial exploratory study To explore in which of the writing stages authors are most in need of additional information, Deshpande et al. (2006) performed an exploratory study with IntelliGentÔ. During two months researchers from the department of Language and Speech Technology at Radboud University (The Netherlands) used IntelliGentÔ whenever they were using MS Word. The ScopusÒ database was linked to IntelliGentÔ as the source for information. To investigate if there were different information seeking needs during the different stages of writing, several interviews and questionnaires were done. Also the issues of efficiency, effectiveness, and overall satisfaction working with IntelliGentÔ were explored. The main results of the study show that, as the user shifts between the stages of writing, information requirements differ. In most cases, participants were not aware of doing any planning before starting the translating stage. However, in all cases, they found that the most important moment to search for information is before translating starts. When asked if they also looked for new information during translating and reviewing, participants claimed not to do it frequently and only when it appeared that better justification of their ideas was needed. These results are similar to the ones described by Dansac and Alamargot (1999). Furthermore, participants perceived that the system often presented information at an inopportune moment, interrupting their writing. As a consequence, some users simply ended up stopping using IntelliGentÔ. Based on these results we performed two experiments in which we studied the effects of presenting information by IntelliGentÔ during the stages in which finding new information seem to be more relevant: Planning and Reviewing. 6. Experiment 1: presenting proactive information during planning tasks Planning involves creating and organizing ideas, and setting goals during composition. In order to simulate the stage of Planning, we used a procedure similar to the one described by Berninger et al. (1996). Participants were told that they had to write short essays about different topics but before starting to create actual sentences they had to plan and write an outline of the major points of the text with supporting details and the order in which they would be introduced in the text. The planning task consisted in writing the outlines for each topic. 6.1. Method 6.1.1. Subjects Thirty-two students from the Radboud University (Nijmegen) participated in the experiment. All participants met the following criteria: (a) familiarity using MS Word and Internet Explorer (b) working knowledge of English, and, (c) they have never used IntelliGentÔ. 6.1.2. Design The information seeking conditions were manipulated within subjects. Four different options were presented: (1) without the
519
option of seeking for information, (2) with the option of getting information by actively searching information in the Web (condition of active search), (3) with presentation of proactive relevant information by IntelliGentÔ and, (4) with presentation of proactive but no related information by IntelliGentÔ. In condition 3 participants received proactive information that was helpful in their writing. For example, if their task was to write an outline about the topic ‘‘vaccinations children receive during their first year’’, a proactive relevant document will contain a description of vaccinations and the period they are given. In the last condition participants received proactive information that was not related with the task topic and consequently could not possibly helping in preparing the outline. In conditions 3 and 4, participants were also allowed to perform active search if they considered it necessary. 6.1.3. Procedure Participants were asked to write in a MS Word document a planning outline about different topics. They had to write the major points they wanted to make with supporting details indicating the order in which they would be introduced. Before starting the planning tasks participants were asked about the previous knowledge they had about the different topics they had to write about. The selected topics were about activities or requirements needed in order to reach a specific goal (e.g. what are the requirements to import a car to USA?). None of the participants reported high knowledge about any of the topics. Participants wrote 4 planning outlines, one per condition. The presentation order of the planning tasks was counterbalanced across all subjects. Participants had to finish every condition in less than 10 min, and it was mandatory to write at least four points in every planning. In conditions 3 and 4, and after participants wrote a few words of the outline, IntelliGentÔ appeared in the screen containing titles of documents that could be relevant for the planning task. Participants were asked to check if the presented titles could help them in performing the task (see Fig. 1). If they decided that one of the presented links (with a brief description of the contents of the related hypertext) could contain relevant information to the planning task, they could click the link. Upon clicking the link, the related text appeared and participants were asked to read it. Half of the cases participants were presented with a title that (clearly) was linked with a document containing the answer to perform the editing task properly (condition 3). In the not related conditions participants were interrupted with the presentation of links whose content was not related to the main task (condition 4). Participants were not allowed to copy and paste text from other documents. The experiment sessions were recorded using Camtasia Studio Software. We measured (1) total time on planning, (2) the amount of time spent on searching/checking new information and, (3) the text quality as a function of the number of correct written ideas (activities or requirements).
6.2. Results 6.2.1. Time on planning task The time on planning tasks was measured as the amount of time needed to complete the planning task (in seconds); this measure did not include the time spent on seeking information. There were significant differences in time on planning as a function of the information seeking condition F(3, 90) ¼ 2.71, p < 0.05 Table 1 shows the averages per condition. Pair wise comparisons showed that the average time in the condition in which participants could not search for information was significantly higher than the conditions no relevant and active search (both p < 0.02). A trend with the relevant condition was also found (p < 0.06). No other comparison was significant. Concluding, when no option to search
520
M.C. Puerta Melguizo et al. / International Journal of Industrial Ergonomics 39 (2009) 516–523
Table 1 Average and standard deviation (between parentheses) of time performance of the planning tasks according to the information seeking condition.
Table 3 Average and standard deviation (between parentheses) of quality on task performance as a function of the information seeking condition.
No search
Active search
Relevant
No relevant
No search
Active search
Relevant
No related
400.68(148.6)
338.42(102.8)
341.13(104.9)
336.9(151.7)
4.05(2.2)
4.33(1.7)
6.52(2.1)
4.18(1.6)
is given participants tend to spend more time planning than in the rest of the conditions. 6.2.2. Time on information seeking The amount of time spent searching/checking new information was measured in seconds starting from the moment participants started and active search (condition 2) or when PRS information was presented (conditions 3 and 4), and finishing when participants resumed the planning task. We did not find differences in information seeking time F(2, 54) ¼ 1.67, p ¼ 0.19 (see Table 2). 6.2.3. Quality of the planning The quality of planning was measured by the number of relevant ideas written in the text. There were significant differences between conditions F(3, 93) ¼ 10.46, p < 0.00. Pair wise comparisons showed that the relevant condition was significantly better than the rest (all p < 0.00) and the conditions no search, active search and no related were no significantly different between them Table 3. 6.3. Discussion The presentation of proactive information by the PRS did not seriously impair time performance during planning. Furthermore, when relevant information was presented proactively, the quality of the resulting writing plan was significantly better and participants introduced more information and included more correct ideas in their planning than in any of the other conditions. Concluding, the presentation of proactive relevant information by the PRS improves the quality of the planning process. At the same time, the interrupts caused by presenting newly retrieved information does not disturb the task, since it does not take more time to write a plan in the condition where the PRS presented relevant information, compared to the other conditions. The results of this experiment also show that active search initiated by the user resulted in a lower quality of the information found and the written text. We expected participants to spend less time seeking for new relevant information in the condition in which proactive relevant information was presented, but we did not find significant differences between conditions. The cause of these results can be the fact that in the no related condition a considerable amount of participants (11) did not look for extra information even when they were aware that the information presented by the PRS was not helping to improve their planning. As a result of this behaviour, the quality of their planning in this condition was considerably low. 7. Experiment 2: presenting proactive information during reviewing tasks When reviewing previously composed text, writers read and edit their written text whenever errors or weaknesses are detected
Table 2 Average and standard deviation (between parentheses) of time on information seeking tasks as a function of the information seeking condition. Active search
Relevant
No relevant
233.39(101.7)
200.86(98.3)
185(133.4)
in the text. To explore the effects of presenting proactive information during the reviewing stage, we asked participants to perform two different editing tasks: spelling corrections and filling in factual information in a text. These two tasks are similar to the ones described by Iqbal et al. (2005). 7.1. Method 7.1.1. Subjects Twenty students from the Radboud University (Nijmegen) participated in the experiment. All participants met the following criteria: (a) familiarity using MS Word and Internet Explorer (b) working knowledge of English, and (c) they have never used IntelliGentÔ. 7.1.2. Design Two independent variables were manipulated within subjects: the editing task category and the information seeking conditions. Participants performed two different editing tasks: spelling corrections and filling the blanks. In the spelling correction task, participants corrected texts containing the misspelled names of three Indian personalities. The texts were divided into three paragraphs and presented in MS Word with the misspelled names underlined with coloured wriggles (see Fig. 3). In the filling the blanks task, participants were given texts on various topics. The task consisted in filling in the gaps with the correct word (one gap per each of the three paragraphs included in one text). For example, in the sentence ‘‘the Baby Boomers are often referred to as the ________ generation as they are still in a phase of life where they are not only taking care of their kids but also their parents’’, the correct answer was the word ‘‘Sandwich’’ (see Fig. 4). The information seeking conditions were manipulated within subjects. Three different options were presented: (1) with the option of getting information by actively searching information in the Web (condition of active search), (2) with presentation of proactive relevant information by IntelliGentÔ and (3) with presentation of proactive but no relevant information by IntelliGent. In the condition with relevant proactive information (condition 2) IntelliGentÔ presented texts containing the correct answer for the editing task (i.e. the name correctly written). When proactive but no relevant information was presented by IntelliGentÔ (condition 3), the texts were about a completely different topic and consequently, did not contain the correct answers for the editing tasks. When no proactive information was presented (condition 1) participants were encouraged to do active searches of information in order to perform the editing tasks properly (i.e. using search engines). Since we made sure participants had no previous knowledge of the topics selected for the experiment, we decided to eliminate the condition without the option of seeking for information. Another reason to exclude this condition was to make the situation more similar to the current real situation in which writers continuously intertwine writing and searching information. 7.1.3. Procedure Each subject edited a total of six texts each of them containing three items to edit (one per paragraph). Three texts contained spelling errors and the rest contained gaps to fill in. Prior to starting each editing task, the experimenter gave a verbal description of the
M.C. Puerta Melguizo et al. / International Journal of Industrial Ergonomics 39 (2009) 516–523
521
Fig. 3. Spelling correction task.
task. In each editing condition, participants worked with one text without being interrupted and the other two texts where interrupted by the presentation of proactive information, related or not related. Proactive information appeared as soon as the cursor was placed on the underlined wriggled word or the blank. The presentation order of the editing tasks and the information seeking conditions were counterbalanced across all subjects.
IntelliGentÔ appeared in the right low corner of the screen and participants were asked to click on the link presented. Upon clicking the link, a text appeared and participants were asked to check if the presented titles could help them in performing the task (see Fig. 1). If they decided that one link could contain relevant information to the editing task, they could click the link. Upon clicking the link, a text appeared and participants were asked to
Fig. 4. Filling de blanks task.
522
M.C. Puerta Melguizo et al. / International Journal of Industrial Ergonomics 39 (2009) 516–523
read it. Half of the cases participants were presented with text containing the answer to perform the editing task properly. Half of the cases participants were presented with text containing the answer to perform the editing task properly. In the not related conditions participants were interrupted with the presentation of a text whose content was not related to the main task. The duration of the experiment was about 1 h depending on the subjects reading and information seeking skills. The experiment sessions were recorded using Camtasia Studio Software. The Dependent variables were: (1) time on editing task, (2) time on information seeking and, (3) quality of editing task performance. 7.2. Results
Table 5 Average and standard deviation (between parentheses) of time on information seeking tasks as a function of the main task and the information seeking condition.
Spelling errors Filling blanks
Related
Non related
No interruption
30.0(16.5) 41.08(21.8)
198.2(81.6) 146.49(69.9)
152.6(74.4) 137.0(78.37)
Pair wise comparisons showed significant differences between both editing tasks (p < 0.00). Participants performed the task of editing spelling errors better than the task of filling the blanks. Similar results have been found by Iqbal et al. (2005). The related condition was significantly different to both non related and no interruption conditions (both p < 0.001).
7.2.1. Time on editing task The time on editing tasks was measured as the amount of time needed to complete the editing task (in seconds); this measure did not include the time spent on seeking information. We did not find significant differences in time as a function of the editing task (spelling corrections vs. filling the blanks) F(1, 19) ¼ 1.59, p ¼ 0.22 or as a function of the information seeking condition (related proactive information, non-related proactive information and no information) F(2, 38) ¼ 0.85; p ¼ 0.43. The interaction between type of task and information seeking conditions was also not significant F(2, 38) ¼ 0.69; p ¼ 0.51. Table 1 shows the averages per condition. From these results we can conclude that there are no differences in time between the tasks of filling in the blanks and spelling corrections in any of the manipulated information seeking tasks conditions Table 4.
7.3. Discussion
7.2.2. Time on information seeking The amount of time spent searching/checking new information was measured in seconds starting from the moment participants started and active search (condition 2) or when PRS information was presented, and finishing when participants resumed the editing task. We did not find differences in information seeking time as a function of the editing task F(1, 19) ¼ 2.05, p ¼ 0.17. We found differences as a function of the information seeking condition F(2, 38) ¼ 64.34; p < 0.00. Comparisons showed significant differences between the conditions of no information (average ¼ 144.99) and non related (average ¼ 172.35) with a t(19) ¼ 2.04; p < 0.05. Differences between no information (average ¼ 144.99) and related (average ¼ 35.54) were also significant t(19) ¼ 7.99; p < 0.000. These data indicate that participants took more time seeking information in the non related and no information conditions, and this time was bigger when non-related information was presented by IntelliGentÔ. The interaction between the task type and information seeking condition was not significant F(2, 38) ¼ 2.90; p ¼ 0.07 (see Table 5).
8. Conclusions and suggestions for future research
The main results of this experiment again show that the presentation of proactive information by IntelliGentÔ does not seriously impair the time performance in editing tasks in comparison with the conditions in which the user was not interrupted by the system. Furthermore, the time spent in looking for new relevant information was less when the PRS presented relevant information than in the cases in which participants had to search for the information actively. The information seeking time was even bigger when non-related information was presented proactively. In this case, after assessing that the information by the PRS could not help in performing the editing task, participants started an active search. The quality of the editing tasks was also improved by the presentation of proactive relevant information.
Proactive recommendation systems such as IntelliGentÔ retrieve and decide which available information is most likely relevant to the text being written, and present it without user intervention. However, the structure of the writing process has a large impact on the ways in which a PRS should interact with the user. In this paper we present two experiments in which the effects of presenting proactive information during the tasks of planning and editing are studied. The experiments presented here show that the presentation of proactive information by a PRS does not seriously increase the time needed for the main task and may improve the quality of the product in the editing and planning phases. From these results, we would like to raise several challenges that must be met if we want to design optimal PRS for writers. 8.1. The challenge of interrupting without disrupting
7.2.3. Quality on editing task performance Task performance was measured in one scale from 0 to 1. Tasks finished with the correct answer scored 1. We found significant differences as a function of the editing task F(1, 19) ¼ 23.53, p < 0.00. Also were significant the differences between information seeking conditions F(2, 38) ¼ 10.23; p < 0.00. The interaction between type of task and information seeking conditions was also significant F(2, 38) ¼ 3.38; p ¼ 0.04. Table 6 shows the averages per condition.
We have found that proactive information recommendation does not interrupt the ongoing writing tasks. Furthermore, when the information presented is really relevant, the quality of the written text seems to improve. But it is also important to design the interface and interaction procedure in such a way that it is easy for writers to observe that potentially relevant information has been retrieved, while at the same time, it is easy to ignore the interrupt without losing the information in the moments in which the writer is in a phase of a task that would be difficult to resume after having been interrupted.
Table 4 Average and standard deviation (between parentheses) of time performance of the editing tasks according to the information seeking condition.
Table 6 Average and standard deviation (between parentheses) of quality on task performance as a function of the main task and the information seeking condition.
Spelling errors Filling blanks
Related
Non related
No interruption
37.73(11.05) 46.48(25.05)
38.89(14.23) 46.65(19.84)
46.53(25.74) 46.83(23.21)
Spelling errors Filling blanks
Related
Non related
No interruption
1(0) 0.97(0.10)
0.92(0.18) 0.79(0.25)
0.93(0.14) 0.67(0.26)
M.C. Puerta Melguizo et al. / International Journal of Industrial Ergonomics 39 (2009) 516–523
8.2. The planning challenge Planning is the moment in which seeking for related information to the written text seems to be most relevant. However, most of the planning seems to occur before any substantial writing is done. Although a writing strategy that involves explicit planning is often recommended in formal writing courses, few writers seem to do it. As a consequence, writing support systems such as the Writing Environment developed by Smith et al. (1987) and other related tools in text processing systems such as MS Word are rarely used. The challenge to designers is to integrate the PRS in a more general writing support system and make the combined system so effective and powerful that professional writers experience the added value of adhering to a strategy that involves explicit planning. In other words, a clever combination of outline tools for planning and a PRS should be able to motivate users to change their writing procedures in such a manner that the system can help them to find information in the appropriate moment. The idea is that when using an outline system, users would make their writing plans explicit and then a PRS might be in a much better position to search for potentially relevant information. The benefit to writers would be that they receive recommendations in a proactive manner, shortening considerably their task of seeking for information, and minimizing the risk of missing essential information. 8.3. The challenge of an external long-term memory In this paper we stressed the need to use a model of the writing process to investigate the impact of a PRS on the process and the quality of the writing product. However, we also noted that the introduction of computers may change the conventional process, instead of just making it more efficient and effective. So far, virtually all writing research has been conducted in settings in which the Long-Term Memory from which participants could ‘get information’ was limited to their own brain (e.g. Olive, 2004). However, the advent of extremely powerful search systems is already affecting the way people consider and use LTM. Students are no longer trained to memorize information; rather, they are trained in efficient and effective search techniques. Thus, already now is becoming more important to know how to find information than to memorize information in the first place. However, accessing information in the Internet is not without problems. Knowing less, while searching more makes more difficult to assess the importance of newly found information and to integrate it in a coherent framework. While computer tools may not be able to facilitate the integration process any time soon, they may be able to support the decisions about the relevance of the results returned from a query. Also information retrieved in the form of documents or text snippets may have a different impact on how one decides to organize the information in a coherent text than when the information is retrieved from one’s own LTM. Consequently, we think it is necessary to develop a new model of cognitive writing processes in which the external LTM that the WWW and other databases conforms, needs to be included as an important part of the physical environment. 8.4. The challenge of an integrated model of the cognitive processes of writing and searching We think that the new situation in which text creation and information search are interleaved calls for the need of developing integrated models of writing and searching. This need is especially important since existing authoring tools are based on cognitive models of writing that have been developed in the paper-andpencil era and these models frequently assume that search for
523
information is done before any writing is done. We are convinced that in order to design better tools for writing it is still necessary to develop an integrated cognitive model of the processes of writing and searching, and new authoring tools need to be designed based in this integrated model. The experiments presented in this paper are our first step towards reaching this goal. However more research, i.e. comparing pen and paper writing with computer writing, how and when people search for information, how writers use proactive information, etc., needs to be done. Acknowledgments We would like to thank Frank Hofstede of Search Expertise Centrum (The Netherlands) for providing us with IntelliGentÔ. We would also like to thank IOP-MMI, administered by SenterNovem, for funding this project. References Bailey, B.P., Konstan, J., Carlis, J.V., 2000. Measuring the effects of interruptions on task performance in the user interface. In: IEEE Conferences on Systems, Man and Cybernetics, pp. 752–762. Berninger, V., Whitaker, D., Yuen, F., Swanson, H.L., Abbott, R.D., 1996. Assessment of planning, translating, and revising in junior high writers. Journal of School Psychology, 23–52. Britton, B.K., Glynn, S.M., 1989. Computer Writing Environments: Theory Research and Design. Lawrence Erlbaum Associates, Hillsdale New Jersey. Budzik, J., Hammond, K., 1999. Watson: anticipating and contextualizing information needs. In: Proceedings of the 62nd Annual Meeting American Society for Information Science, pp. 727–740. Dansac, C., Alamargot, D., 1999. Accessing referential information during text composition: when and why? In: Torrance, M., Galbraith, D. (Eds.), Knowing What to Write: Conceptual Processes in Text Production. Amsterdam University Press, pp. 76–97. Deshpande, A., Boves, L., Puerta Melguizo, M.C., 2006. A` propos: pro-active personalization for professional document writing. In: SigWriting, 10th International Conference of the EARLI Special Interest Group on writing. September, 2006, Antwerp, Belgium. Gwizdka, J., Spence, I., 2007. Implicit measures of lostness and success in web navigation. Interacting with Computers 19, 357–369. Haas, C., 1996. Writing Technology Studies on The Materiality of literacy. Lawrence Erlbaum Associates, Hillsdale New Jersey. Hayes, J.R., 1996. A new framework for understanding cognition and affect in writing. In: Levy, C.M., Ransdell, S.E. (Eds.), The Science of Writing: Theories, Methods, Individual Differences, and Applications. Lawrence Erlbaum Associates, Hillsdale New Jersey, pp. 76–97. Hayes, L.S., Flower, J.R., 1980. dentifying the organization of writing processes. In: Gregg, L.W., Steinberg, E.R. (Eds.), Cognitive Processes in Writing. Lawrence Erlbaum, Hillsdale, NJ, pp. 3–30. Iqbal, S.T., Adamczyk, P.D., Zheng, X.S., Bailey, B.P., 2005. Towards and index of opportunity: understanding changes in mental workload during task execution. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 311–320. Maglio, P.P., Campbell, C.S., 2000. Tradeoffs in displaying peripheral information. In: Proceedings of The CHI 2000 Conference on HUMAN Factors In Computing Systems. ACM Press, New York, pp. 241–248. Neuwirth, C.M., Kaufer, D.S., 1989. The role of external representations in the writing process: implications for the design of hypertext-based writing tools. In: Proceedings Hypertext 1989. ACM Press, pp. 319–341. Olive, T., 2004. Working memory in writing: empirical evidence from the dual-task technique. European Psychologist 9, 32–34. Piolat, A., Kellogg, R.T., Farioli, F., 2001. The triple task technique for studying writing processes: on which task is attention focused? Current Psychology Letters: Behaviour, Brain & Cognition 4, 67–83. Puerta Melguizo, M.C., Bogers, T., Boves, L., Deshpande, A., van den Bosch, A., 2007. What a proactive recommendation system needs: relevance, non-intrusiveness, and a new long-term memory. In: Jorge Cardoso, Jose´ Cordeiro, Joaquim Filipe (Eds.), ICEIS 2007: Proceedings of the Ninth International Conference on Enterprise Information Systems, pp. 86–91. Puerta Melguizo, M.C., Lemmert, V.R., van Oostendorp, H., 2006. Lostness, mental models and performance. In: Vicente, P., Guerrero Bote, (Eds.), Current Research in Information Sciences and Technologies: multidisciplinary approaches to global information systems, vol. I., pp. 256–260. Rhodes, B.J., 2000. Just-in-time Information Retrieval, Ph.D. Thesis, MIT. Smith, J.B., Weiss, S.F., Ferguson, G.J., 1987. A hypertext writing environment and its cognitive basis. In: Hypertext 1987. ACM Press, pp. 195–214. Zijlstra, F.R.H., Roe, R.A., Leonora, A.B., Krediet, I., 1999. Temporal factors in mental work: effects of interrupted activities. Journal of Occupational and Organizational Psychology 72, 163–185.
Contents lists available at ScienceDirect
International Journal of Industrial Ergonomics journal homepage: www.elsevier.com/locate/ergon
A proactive recommendation system for writing: Helping without disrupting ˜ oz Ramos b Mari Carmen Puerta Melguizo a, *, Lou Boves a, Olga Mun a b
Department of Language and Speech, Radboud University, P.O. Box 9103, Erasmus building (8th floor), 6500 HD Nijmegen, The Netherlands Department of Experimental Psychology, Faculty of Psychology – Campus de Cartuja, 18071 Granada, Spain
a r t i c l e i n f o
a b s t r a c t
Article history: Received 18 December 2007 Received in revised form 14 July 2008 Accepted 16 October 2008 Available online 29 November 2008
Writing professional documents are situations in which finding relevant, related information is important. Seeking for information is however not easy. A proactive recommendation system (PRS) retrieves information relevant to the written text, and presents it without user intervention. The information presented can improve the quality of the text but can also interrupt the process of writing. We describe ` Propos. Our goal is to develop a PRS for two experiments performed to optimize the design of the PRS A professional writers that present information non-intrusively and timely so as to minimize disturbing the writing process. The results of our experiments suggest that the tendency to interleave (re)search and writing implies a need for integrating the PRS in a writing environment able to support the task of writing the text itself and the stage of planning the contents and structure of the document.
Keywords: Proactive recommendation system Information seeking Model of the writing process Planning Editing Interruptions
Relevance to industry: While automation has increased the efficiency of repetitive processes, the impact on professional tasks has been minor. Professional writing is a complex task that takes a lot of time. The PRS that we are developing can result in substantial reductions of the cost of writing documents, and in substantial improvements of the quality of the written texts. Ó 2008 Elsevier B.V. All rights reserved.
1. Introduction Writing is a complex task and several computer systems have been developed in order to support writing. Up to now, most systems developed to support writing (or computer writing environments as Britton and Glynn (1989) called them), seem to be designed with the main purpose of supporting the processes of planning and reviewing the written text. In general, systems have been built to help writers with grammatical and spelling errors or to help them formulate external visual representations of their ideas and connect the main issues that should be written in the paper, the ordering of the sections, etc. For example, the Writing Environment (WE) developed by Smith et al. (1987) was one of the first systems developed to support the processes of organization and editing by including a network mode, a tree mode, an editing mode, and a text mode. The whole process of writing a document, however, also requires looking for information and especially with the advent of the Internet, more often than not, writing is now interleaved with searching for new or additional information. The question is whether repeated switching between the tasks of writing and searching is efficient, and whether it tends to result in the best
* Corresponding author. Tel.: þ31 (24) 361 5820; fax: þ31 (24) 361 2907. E-mail addresses: [email protected] (M.C. Puerta Melguizo), [email protected] (L. Boves), [email protected] (O.M. Ramos). 0169-8141/$ – see front matter Ó 2008 Elsevier B.V. All rights reserved. doi:10.1016/j.ergon.2008.10.004
possible quality of the texts. It has been proposed that task switching can be alleviated by a proactive recommendation system that searches and presents relevant information. However, such systems may interrupt the writing process in other ways, and by doing so, they may affect user satisfaction, efficiency and the quality of the resulting documents. This paper presents the results of two experiments designed to investigate the impact of a proactive recommendation system. The experimental results raise the question whether, in order to be effective and efficient, proactive recommendation systems should be integrated in more comprehensive writing support environments. 2. Proactive recommendation systems for writing Proactive recommendation systems (PRSs) retrieve large quantities of documents, decide what available information is most likely relevant to the text being written, and offer that information without user requests. The decision about what information to offer is mainly based on the text that is currently being written, in combination with personal profiles and profiles of the working group to which the person belongs. Only a few PRSs have been specifically developed to support writing in professional settings. For example, the Remembrance Agent (Rhodes, 2000) suggests personal email and documents based on text being written. Watson (Budzik and Hammond, 1999) performs automatic Web searches based on text being written or read. IntelliGentÔ is yet another PRS that proactively submits queries to a potentially large number of
M.C. Puerta Melguizo et al. / International Journal of Industrial Ergonomics 39 (2009) 516–523
517
search engines and presents the retrieved information while the user is writing a document. A serious problem with all of these PRSs is that they are developed as search support tools and do not seem to take into account the specific characteristics of the task at hand. Writing professional documents is a complex and highly demanding task that can be seriously affected by any type of interruption from the environment.
To be helpful the retrieved information has to be precise and highly relevant to the written document, but also it should be presented in a non-intrusive and timely manner. If the way of presenting the information seriously interferes with the main task, users might stop using the system. In the experiments presented in this paper we studied the effects that presenting proactive information by IntelliGentÔ can have on writing performance.
2.1. An IntelliGentÔ proactive recommendation system
3. The effects of interruptions on writing performance
As a tool for research, we are performing experiments with IntelliGentÔ. The system proactively submits queries based on a broadly defined user profile in combination with what the user is currently typing or reading. The system presents the retrieved information to the user proactively and immediately. The results of the search are presented in a semi-transparent window located in the bottom right of the screen (see Fig. 1). The window contains URLs related to what the user is typing. As the user moves the cursor over the references, the URLs become fully visible and active. On clicking the required URL, the user accesses the corresponding document. The information in the window is changed depending upon the text that is being input and new queries are created. The URLs in the transparent window also change as the user moves the cursor while reviewing previously written parts of the document, again on the basis of queries created from the text in the paragraph in which the cursor is located. For more information see Puerta Melguizo et al. (2007).
Presenting information proactively, as IntelliGentÔ does, can provide the user the opportunity to do a better job in less time (Maglio and Campbell, 2000). However, proactive presentation of information can be considered as an interruption that imposes extra-task demands on user’s cognitive resources. The effects of interruptions on the user’s main task have been studied frequently (e.g. Bailey et al., 2000; Piolat et al., 2001; Zijlstra et al., 1999). For example, Bailey et al. (2000) used six tasks with different cognitive load and two interruption tasks. They found an increase of the time spent on the main task when interruptions were present. The authors suggest this is due to the additional time needed to resume the main task after an interrupt. Summarizing, we think that the presentation of information by a PRS may have a detrimental effect, because the additional task (i.e. to check if the offered papers are interesting to the writer) has specific demands and there is the additional need to reorientate and resume the original task at a later moment. This reorientation
Fig. 1. IntelliGentÔ system for proactive information retrieval.
518
M.C. Puerta Melguizo et al. / International Journal of Industrial Ergonomics 39 (2009) 516–523
requires the user to remember the status of the writing (i.e. to complete some argument in the text being written). Therefore, we investigated if the peripheral pop-up of IntelliGentÔ distracts writers from their main task. In addition, we also investigated if IntelliGentÔ can improve the quality of the final paper when it presents relevant information. 4. The cognitive model of writing The need for a PRS might differ from one moment to another during the process of writing a document. Consequently, to optimally support professional document writing through a PRS, it is necessary to explore the stages involved in writing and the different effects that interruptions can have in these processes. Since the beginning of empirical research scholars have agreed that writing involves at least three different cognitive processes, usually called ‘planning’, ‘translation’ and ‘review/editing’. The most widely accepted and influential model of writing is the one proposed by Hayes and Flower (1980). These authors challenged the traditional linear sequence models of writing by proposing that there is a continuous recursive and non-linear interaction between the planning, translation and review/editing processes. Their model defines three main components: the writing process proper (which includes the three processes/stages mentioned above), the task environment and the writer’s long-term memory (see Fig. 2). Writing happens in three stages: Planning, Translating, and Reviewing. During Planning, ideas are generated, arranged into a coherent structure and goals are set. Planning involves retrieving domain knowledge from the writer’s Long-Term memory (LTM) and organizing it into a plan. This plan specifies, among other things, the effects that the writer wants (or needs) the text to have on the prospective readers. During the process of Translating writer’s plans and goals are transformed into sentences. In the Reviewing stage the writer evaluates the relation between the text written so far and the linguistic, semantic and pragmatic aspects that would best serve the writing goal. Reviewing involves reading and editing. Reading allows to detect errors or weaknesses and to evaluate the appropriateness of the written text in relation to the goals established during planning. Editing appears as a system of production rules that result in changes to the text.
The task environment includes everything existing outside the writers’ mind and that can influence the writing task. The main elements included here are the text produced so far and the socalled rhetorical problem (the writing assignment, the specification of topic and the audience). In the writer’s LTM is stored the writer’s knowledge about the topic, the knowledge of sources based on literature search, the writing plans and the knowledge about the audience who will read the work. Hayes (1996) revised the model, emphasizing the role of working memory, as well as socio-cultural and motivational aspects in writing. Another important addition of Hayes is the inclusion of the physical environment as an important factor that influences writing. In the physical environment, together with the text produced so far, the composing medium or tool used to write is included. Variations in the medium seem to lead to differences in the way people carry out the writing task. For example, Haas (1996) found that writers tend to plan more when they write on paper than when using a word processor, possibly because it is easier to sketch, to draw and to interconnect ideas using pen and paper. Haas also found that writers tend to revise documents on a general level (i.e. modifying the structure) when using a pen, and more on a local level (i.e. revision of syntax, semantics, vocabulary) when working on screen. These results suggest that the introduction of computer tools as a medium for writing seems to stimulate users to change the processes they use, rather than making the original processes easier or more effective. 5. Finding relevant information during writing Searching information is a difficult and time consuming task. Nowadays, search engines have become the primary tool for information access in both the Internet and company-internal networks. However, searching and navigating are often not as efficient and easy as users might like. While searching for information, users frequently feel ‘‘lost in hyperspace’’. Actually, feeling lost or the tendency to lose one’s sense of location is the most common problem users experience while navigating and strongly affects user’s performance and satisfaction (Gwizdka and Spence, 2007; Puerta Melguizo et al., 2006). Ironically, the very potential strength of hypertext, namely improving the management of loose
Fig. 2. The model of writing proposed by Hayes and Flower (1980).
M.C. Puerta Melguizo et al. / International Journal of Industrial Ergonomics 39 (2009) 516–523
collections of relatively unstructured information, turned out to be a major potential weakness (Neuwirth and Kaufer, 1989). Furthermore, keyword-based search is still inefficient and relevant information may be missed because the writer did not realize that the information exists and could be looked up. Also considerable time is spent interacting with low-precision search engines. Consequently, the time in which the author is away from creating the document can have a negative impact on the total time spent, and on the eventual quality of the text. 5.1. Writing and seeking for information: an initial exploratory study To explore in which of the writing stages authors are most in need of additional information, Deshpande et al. (2006) performed an exploratory study with IntelliGentÔ. During two months researchers from the department of Language and Speech Technology at Radboud University (The Netherlands) used IntelliGentÔ whenever they were using MS Word. The ScopusÒ database was linked to IntelliGentÔ as the source for information. To investigate if there were different information seeking needs during the different stages of writing, several interviews and questionnaires were done. Also the issues of efficiency, effectiveness, and overall satisfaction working with IntelliGentÔ were explored. The main results of the study show that, as the user shifts between the stages of writing, information requirements differ. In most cases, participants were not aware of doing any planning before starting the translating stage. However, in all cases, they found that the most important moment to search for information is before translating starts. When asked if they also looked for new information during translating and reviewing, participants claimed not to do it frequently and only when it appeared that better justification of their ideas was needed. These results are similar to the ones described by Dansac and Alamargot (1999). Furthermore, participants perceived that the system often presented information at an inopportune moment, interrupting their writing. As a consequence, some users simply ended up stopping using IntelliGentÔ. Based on these results we performed two experiments in which we studied the effects of presenting information by IntelliGentÔ during the stages in which finding new information seem to be more relevant: Planning and Reviewing. 6. Experiment 1: presenting proactive information during planning tasks Planning involves creating and organizing ideas, and setting goals during composition. In order to simulate the stage of Planning, we used a procedure similar to the one described by Berninger et al. (1996). Participants were told that they had to write short essays about different topics but before starting to create actual sentences they had to plan and write an outline of the major points of the text with supporting details and the order in which they would be introduced in the text. The planning task consisted in writing the outlines for each topic. 6.1. Method 6.1.1. Subjects Thirty-two students from the Radboud University (Nijmegen) participated in the experiment. All participants met the following criteria: (a) familiarity using MS Word and Internet Explorer (b) working knowledge of English, and, (c) they have never used IntelliGentÔ. 6.1.2. Design The information seeking conditions were manipulated within subjects. Four different options were presented: (1) without the
519
option of seeking for information, (2) with the option of getting information by actively searching information in the Web (condition of active search), (3) with presentation of proactive relevant information by IntelliGentÔ and, (4) with presentation of proactive but no related information by IntelliGentÔ. In condition 3 participants received proactive information that was helpful in their writing. For example, if their task was to write an outline about the topic ‘‘vaccinations children receive during their first year’’, a proactive relevant document will contain a description of vaccinations and the period they are given. In the last condition participants received proactive information that was not related with the task topic and consequently could not possibly helping in preparing the outline. In conditions 3 and 4, participants were also allowed to perform active search if they considered it necessary. 6.1.3. Procedure Participants were asked to write in a MS Word document a planning outline about different topics. They had to write the major points they wanted to make with supporting details indicating the order in which they would be introduced. Before starting the planning tasks participants were asked about the previous knowledge they had about the different topics they had to write about. The selected topics were about activities or requirements needed in order to reach a specific goal (e.g. what are the requirements to import a car to USA?). None of the participants reported high knowledge about any of the topics. Participants wrote 4 planning outlines, one per condition. The presentation order of the planning tasks was counterbalanced across all subjects. Participants had to finish every condition in less than 10 min, and it was mandatory to write at least four points in every planning. In conditions 3 and 4, and after participants wrote a few words of the outline, IntelliGentÔ appeared in the screen containing titles of documents that could be relevant for the planning task. Participants were asked to check if the presented titles could help them in performing the task (see Fig. 1). If they decided that one of the presented links (with a brief description of the contents of the related hypertext) could contain relevant information to the planning task, they could click the link. Upon clicking the link, the related text appeared and participants were asked to read it. Half of the cases participants were presented with a title that (clearly) was linked with a document containing the answer to perform the editing task properly (condition 3). In the not related conditions participants were interrupted with the presentation of links whose content was not related to the main task (condition 4). Participants were not allowed to copy and paste text from other documents. The experiment sessions were recorded using Camtasia Studio Software. We measured (1) total time on planning, (2) the amount of time spent on searching/checking new information and, (3) the text quality as a function of the number of correct written ideas (activities or requirements).
6.2. Results 6.2.1. Time on planning task The time on planning tasks was measured as the amount of time needed to complete the planning task (in seconds); this measure did not include the time spent on seeking information. There were significant differences in time on planning as a function of the information seeking condition F(3, 90) ¼ 2.71, p < 0.05 Table 1 shows the averages per condition. Pair wise comparisons showed that the average time in the condition in which participants could not search for information was significantly higher than the conditions no relevant and active search (both p < 0.02). A trend with the relevant condition was also found (p < 0.06). No other comparison was significant. Concluding, when no option to search
520
M.C. Puerta Melguizo et al. / International Journal of Industrial Ergonomics 39 (2009) 516–523
Table 1 Average and standard deviation (between parentheses) of time performance of the planning tasks according to the information seeking condition.
Table 3 Average and standard deviation (between parentheses) of quality on task performance as a function of the information seeking condition.
No search
Active search
Relevant
No relevant
No search
Active search
Relevant
No related
400.68(148.6)
338.42(102.8)
341.13(104.9)
336.9(151.7)
4.05(2.2)
4.33(1.7)
6.52(2.1)
4.18(1.6)
is given participants tend to spend more time planning than in the rest of the conditions. 6.2.2. Time on information seeking The amount of time spent searching/checking new information was measured in seconds starting from the moment participants started and active search (condition 2) or when PRS information was presented (conditions 3 and 4), and finishing when participants resumed the planning task. We did not find differences in information seeking time F(2, 54) ¼ 1.67, p ¼ 0.19 (see Table 2). 6.2.3. Quality of the planning The quality of planning was measured by the number of relevant ideas written in the text. There were significant differences between conditions F(3, 93) ¼ 10.46, p < 0.00. Pair wise comparisons showed that the relevant condition was significantly better than the rest (all p < 0.00) and the conditions no search, active search and no related were no significantly different between them Table 3. 6.3. Discussion The presentation of proactive information by the PRS did not seriously impair time performance during planning. Furthermore, when relevant information was presented proactively, the quality of the resulting writing plan was significantly better and participants introduced more information and included more correct ideas in their planning than in any of the other conditions. Concluding, the presentation of proactive relevant information by the PRS improves the quality of the planning process. At the same time, the interrupts caused by presenting newly retrieved information does not disturb the task, since it does not take more time to write a plan in the condition where the PRS presented relevant information, compared to the other conditions. The results of this experiment also show that active search initiated by the user resulted in a lower quality of the information found and the written text. We expected participants to spend less time seeking for new relevant information in the condition in which proactive relevant information was presented, but we did not find significant differences between conditions. The cause of these results can be the fact that in the no related condition a considerable amount of participants (11) did not look for extra information even when they were aware that the information presented by the PRS was not helping to improve their planning. As a result of this behaviour, the quality of their planning in this condition was considerably low. 7. Experiment 2: presenting proactive information during reviewing tasks When reviewing previously composed text, writers read and edit their written text whenever errors or weaknesses are detected
Table 2 Average and standard deviation (between parentheses) of time on information seeking tasks as a function of the information seeking condition. Active search
Relevant
No relevant
233.39(101.7)
200.86(98.3)
185(133.4)
in the text. To explore the effects of presenting proactive information during the reviewing stage, we asked participants to perform two different editing tasks: spelling corrections and filling in factual information in a text. These two tasks are similar to the ones described by Iqbal et al. (2005). 7.1. Method 7.1.1. Subjects Twenty students from the Radboud University (Nijmegen) participated in the experiment. All participants met the following criteria: (a) familiarity using MS Word and Internet Explorer (b) working knowledge of English, and (c) they have never used IntelliGentÔ. 7.1.2. Design Two independent variables were manipulated within subjects: the editing task category and the information seeking conditions. Participants performed two different editing tasks: spelling corrections and filling the blanks. In the spelling correction task, participants corrected texts containing the misspelled names of three Indian personalities. The texts were divided into three paragraphs and presented in MS Word with the misspelled names underlined with coloured wriggles (see Fig. 3). In the filling the blanks task, participants were given texts on various topics. The task consisted in filling in the gaps with the correct word (one gap per each of the three paragraphs included in one text). For example, in the sentence ‘‘the Baby Boomers are often referred to as the ________ generation as they are still in a phase of life where they are not only taking care of their kids but also their parents’’, the correct answer was the word ‘‘Sandwich’’ (see Fig. 4). The information seeking conditions were manipulated within subjects. Three different options were presented: (1) with the option of getting information by actively searching information in the Web (condition of active search), (2) with presentation of proactive relevant information by IntelliGentÔ and (3) with presentation of proactive but no relevant information by IntelliGent. In the condition with relevant proactive information (condition 2) IntelliGentÔ presented texts containing the correct answer for the editing task (i.e. the name correctly written). When proactive but no relevant information was presented by IntelliGentÔ (condition 3), the texts were about a completely different topic and consequently, did not contain the correct answers for the editing tasks. When no proactive information was presented (condition 1) participants were encouraged to do active searches of information in order to perform the editing tasks properly (i.e. using search engines). Since we made sure participants had no previous knowledge of the topics selected for the experiment, we decided to eliminate the condition without the option of seeking for information. Another reason to exclude this condition was to make the situation more similar to the current real situation in which writers continuously intertwine writing and searching information. 7.1.3. Procedure Each subject edited a total of six texts each of them containing three items to edit (one per paragraph). Three texts contained spelling errors and the rest contained gaps to fill in. Prior to starting each editing task, the experimenter gave a verbal description of the
M.C. Puerta Melguizo et al. / International Journal of Industrial Ergonomics 39 (2009) 516–523
521
Fig. 3. Spelling correction task.
task. In each editing condition, participants worked with one text without being interrupted and the other two texts where interrupted by the presentation of proactive information, related or not related. Proactive information appeared as soon as the cursor was placed on the underlined wriggled word or the blank. The presentation order of the editing tasks and the information seeking conditions were counterbalanced across all subjects.
IntelliGentÔ appeared in the right low corner of the screen and participants were asked to click on the link presented. Upon clicking the link, a text appeared and participants were asked to check if the presented titles could help them in performing the task (see Fig. 1). If they decided that one link could contain relevant information to the editing task, they could click the link. Upon clicking the link, a text appeared and participants were asked to
Fig. 4. Filling de blanks task.
522
M.C. Puerta Melguizo et al. / International Journal of Industrial Ergonomics 39 (2009) 516–523
read it. Half of the cases participants were presented with text containing the answer to perform the editing task properly. Half of the cases participants were presented with text containing the answer to perform the editing task properly. In the not related conditions participants were interrupted with the presentation of a text whose content was not related to the main task. The duration of the experiment was about 1 h depending on the subjects reading and information seeking skills. The experiment sessions were recorded using Camtasia Studio Software. The Dependent variables were: (1) time on editing task, (2) time on information seeking and, (3) quality of editing task performance. 7.2. Results
Table 5 Average and standard deviation (between parentheses) of time on information seeking tasks as a function of the main task and the information seeking condition.
Spelling errors Filling blanks
Related
Non related
No interruption
30.0(16.5) 41.08(21.8)
198.2(81.6) 146.49(69.9)
152.6(74.4) 137.0(78.37)
Pair wise comparisons showed significant differences between both editing tasks (p < 0.00). Participants performed the task of editing spelling errors better than the task of filling the blanks. Similar results have been found by Iqbal et al. (2005). The related condition was significantly different to both non related and no interruption conditions (both p < 0.001).
7.2.1. Time on editing task The time on editing tasks was measured as the amount of time needed to complete the editing task (in seconds); this measure did not include the time spent on seeking information. We did not find significant differences in time as a function of the editing task (spelling corrections vs. filling the blanks) F(1, 19) ¼ 1.59, p ¼ 0.22 or as a function of the information seeking condition (related proactive information, non-related proactive information and no information) F(2, 38) ¼ 0.85; p ¼ 0.43. The interaction between type of task and information seeking conditions was also not significant F(2, 38) ¼ 0.69; p ¼ 0.51. Table 1 shows the averages per condition. From these results we can conclude that there are no differences in time between the tasks of filling in the blanks and spelling corrections in any of the manipulated information seeking tasks conditions Table 4.
7.3. Discussion
7.2.2. Time on information seeking The amount of time spent searching/checking new information was measured in seconds starting from the moment participants started and active search (condition 2) or when PRS information was presented, and finishing when participants resumed the editing task. We did not find differences in information seeking time as a function of the editing task F(1, 19) ¼ 2.05, p ¼ 0.17. We found differences as a function of the information seeking condition F(2, 38) ¼ 64.34; p < 0.00. Comparisons showed significant differences between the conditions of no information (average ¼ 144.99) and non related (average ¼ 172.35) with a t(19) ¼ 2.04; p < 0.05. Differences between no information (average ¼ 144.99) and related (average ¼ 35.54) were also significant t(19) ¼ 7.99; p < 0.000. These data indicate that participants took more time seeking information in the non related and no information conditions, and this time was bigger when non-related information was presented by IntelliGentÔ. The interaction between the task type and information seeking condition was not significant F(2, 38) ¼ 2.90; p ¼ 0.07 (see Table 5).
8. Conclusions and suggestions for future research
The main results of this experiment again show that the presentation of proactive information by IntelliGentÔ does not seriously impair the time performance in editing tasks in comparison with the conditions in which the user was not interrupted by the system. Furthermore, the time spent in looking for new relevant information was less when the PRS presented relevant information than in the cases in which participants had to search for the information actively. The information seeking time was even bigger when non-related information was presented proactively. In this case, after assessing that the information by the PRS could not help in performing the editing task, participants started an active search. The quality of the editing tasks was also improved by the presentation of proactive relevant information.
Proactive recommendation systems such as IntelliGentÔ retrieve and decide which available information is most likely relevant to the text being written, and present it without user intervention. However, the structure of the writing process has a large impact on the ways in which a PRS should interact with the user. In this paper we present two experiments in which the effects of presenting proactive information during the tasks of planning and editing are studied. The experiments presented here show that the presentation of proactive information by a PRS does not seriously increase the time needed for the main task and may improve the quality of the product in the editing and planning phases. From these results, we would like to raise several challenges that must be met if we want to design optimal PRS for writers. 8.1. The challenge of interrupting without disrupting
7.2.3. Quality on editing task performance Task performance was measured in one scale from 0 to 1. Tasks finished with the correct answer scored 1. We found significant differences as a function of the editing task F(1, 19) ¼ 23.53, p < 0.00. Also were significant the differences between information seeking conditions F(2, 38) ¼ 10.23; p < 0.00. The interaction between type of task and information seeking conditions was also significant F(2, 38) ¼ 3.38; p ¼ 0.04. Table 6 shows the averages per condition.
We have found that proactive information recommendation does not interrupt the ongoing writing tasks. Furthermore, when the information presented is really relevant, the quality of the written text seems to improve. But it is also important to design the interface and interaction procedure in such a way that it is easy for writers to observe that potentially relevant information has been retrieved, while at the same time, it is easy to ignore the interrupt without losing the information in the moments in which the writer is in a phase of a task that would be difficult to resume after having been interrupted.
Table 4 Average and standard deviation (between parentheses) of time performance of the editing tasks according to the information seeking condition.
Table 6 Average and standard deviation (between parentheses) of quality on task performance as a function of the main task and the information seeking condition.
Spelling errors Filling blanks
Related
Non related
No interruption
37.73(11.05) 46.48(25.05)
38.89(14.23) 46.65(19.84)
46.53(25.74) 46.83(23.21)
Spelling errors Filling blanks
Related
Non related
No interruption
1(0) 0.97(0.10)
0.92(0.18) 0.79(0.25)
0.93(0.14) 0.67(0.26)
M.C. Puerta Melguizo et al. / International Journal of Industrial Ergonomics 39 (2009) 516–523
8.2. The planning challenge Planning is the moment in which seeking for related information to the written text seems to be most relevant. However, most of the planning seems to occur before any substantial writing is done. Although a writing strategy that involves explicit planning is often recommended in formal writing courses, few writers seem to do it. As a consequence, writing support systems such as the Writing Environment developed by Smith et al. (1987) and other related tools in text processing systems such as MS Word are rarely used. The challenge to designers is to integrate the PRS in a more general writing support system and make the combined system so effective and powerful that professional writers experience the added value of adhering to a strategy that involves explicit planning. In other words, a clever combination of outline tools for planning and a PRS should be able to motivate users to change their writing procedures in such a manner that the system can help them to find information in the appropriate moment. The idea is that when using an outline system, users would make their writing plans explicit and then a PRS might be in a much better position to search for potentially relevant information. The benefit to writers would be that they receive recommendations in a proactive manner, shortening considerably their task of seeking for information, and minimizing the risk of missing essential information. 8.3. The challenge of an external long-term memory In this paper we stressed the need to use a model of the writing process to investigate the impact of a PRS on the process and the quality of the writing product. However, we also noted that the introduction of computers may change the conventional process, instead of just making it more efficient and effective. So far, virtually all writing research has been conducted in settings in which the Long-Term Memory from which participants could ‘get information’ was limited to their own brain (e.g. Olive, 2004). However, the advent of extremely powerful search systems is already affecting the way people consider and use LTM. Students are no longer trained to memorize information; rather, they are trained in efficient and effective search techniques. Thus, already now is becoming more important to know how to find information than to memorize information in the first place. However, accessing information in the Internet is not without problems. Knowing less, while searching more makes more difficult to assess the importance of newly found information and to integrate it in a coherent framework. While computer tools may not be able to facilitate the integration process any time soon, they may be able to support the decisions about the relevance of the results returned from a query. Also information retrieved in the form of documents or text snippets may have a different impact on how one decides to organize the information in a coherent text than when the information is retrieved from one’s own LTM. Consequently, we think it is necessary to develop a new model of cognitive writing processes in which the external LTM that the WWW and other databases conforms, needs to be included as an important part of the physical environment. 8.4. The challenge of an integrated model of the cognitive processes of writing and searching We think that the new situation in which text creation and information search are interleaved calls for the need of developing integrated models of writing and searching. This need is especially important since existing authoring tools are based on cognitive models of writing that have been developed in the paper-andpencil era and these models frequently assume that search for
523
information is done before any writing is done. We are convinced that in order to design better tools for writing it is still necessary to develop an integrated cognitive model of the processes of writing and searching, and new authoring tools need to be designed based in this integrated model. The experiments presented in this paper are our first step towards reaching this goal. However more research, i.e. comparing pen and paper writing with computer writing, how and when people search for information, how writers use proactive information, etc., needs to be done. Acknowledgments We would like to thank Frank Hofstede of Search Expertise Centrum (The Netherlands) for providing us with IntelliGentÔ. We would also like to thank IOP-MMI, administered by SenterNovem, for funding this project. References Bailey, B.P., Konstan, J., Carlis, J.V., 2000. Measuring the effects of interruptions on task performance in the user interface. In: IEEE Conferences on Systems, Man and Cybernetics, pp. 752–762. Berninger, V., Whitaker, D., Yuen, F., Swanson, H.L., Abbott, R.D., 1996. Assessment of planning, translating, and revising in junior high writers. Journal of School Psychology, 23–52. Britton, B.K., Glynn, S.M., 1989. Computer Writing Environments: Theory Research and Design. Lawrence Erlbaum Associates, Hillsdale New Jersey. Budzik, J., Hammond, K., 1999. Watson: anticipating and contextualizing information needs. In: Proceedings of the 62nd Annual Meeting American Society for Information Science, pp. 727–740. Dansac, C., Alamargot, D., 1999. Accessing referential information during text composition: when and why? In: Torrance, M., Galbraith, D. (Eds.), Knowing What to Write: Conceptual Processes in Text Production. Amsterdam University Press, pp. 76–97. Deshpande, A., Boves, L., Puerta Melguizo, M.C., 2006. A` propos: pro-active personalization for professional document writing. In: SigWriting, 10th International Conference of the EARLI Special Interest Group on writing. September, 2006, Antwerp, Belgium. Gwizdka, J., Spence, I., 2007. Implicit measures of lostness and success in web navigation. Interacting with Computers 19, 357–369. Haas, C., 1996. Writing Technology Studies on The Materiality of literacy. Lawrence Erlbaum Associates, Hillsdale New Jersey. Hayes, J.R., 1996. A new framework for understanding cognition and affect in writing. In: Levy, C.M., Ransdell, S.E. (Eds.), The Science of Writing: Theories, Methods, Individual Differences, and Applications. Lawrence Erlbaum Associates, Hillsdale New Jersey, pp. 76–97. Hayes, L.S., Flower, J.R., 1980. dentifying the organization of writing processes. In: Gregg, L.W., Steinberg, E.R. (Eds.), Cognitive Processes in Writing. Lawrence Erlbaum, Hillsdale, NJ, pp. 3–30. Iqbal, S.T., Adamczyk, P.D., Zheng, X.S., Bailey, B.P., 2005. Towards and index of opportunity: understanding changes in mental workload during task execution. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 311–320. Maglio, P.P., Campbell, C.S., 2000. Tradeoffs in displaying peripheral information. In: Proceedings of The CHI 2000 Conference on HUMAN Factors In Computing Systems. ACM Press, New York, pp. 241–248. Neuwirth, C.M., Kaufer, D.S., 1989. The role of external representations in the writing process: implications for the design of hypertext-based writing tools. In: Proceedings Hypertext 1989. ACM Press, pp. 319–341. Olive, T., 2004. Working memory in writing: empirical evidence from the dual-task technique. European Psychologist 9, 32–34. Piolat, A., Kellogg, R.T., Farioli, F., 2001. The triple task technique for studying writing processes: on which task is attention focused? Current Psychology Letters: Behaviour, Brain & Cognition 4, 67–83. Puerta Melguizo, M.C., Bogers, T., Boves, L., Deshpande, A., van den Bosch, A., 2007. What a proactive recommendation system needs: relevance, non-intrusiveness, and a new long-term memory. In: Jorge Cardoso, Jose´ Cordeiro, Joaquim Filipe (Eds.), ICEIS 2007: Proceedings of the Ninth International Conference on Enterprise Information Systems, pp. 86–91. Puerta Melguizo, M.C., Lemmert, V.R., van Oostendorp, H., 2006. Lostness, mental models and performance. In: Vicente, P., Guerrero Bote, (Eds.), Current Research in Information Sciences and Technologies: multidisciplinary approaches to global information systems, vol. I., pp. 256–260. Rhodes, B.J., 2000. Just-in-time Information Retrieval, Ph.D. Thesis, MIT. Smith, J.B., Weiss, S.F., Ferguson, G.J., 1987. A hypertext writing environment and its cognitive basis. In: Hypertext 1987. ACM Press, pp. 195–214. Zijlstra, F.R.H., Roe, R.A., Leonora, A.B., Krediet, I., 1999. Temporal factors in mental work: effects of interrupted activities. Journal of Occupational and Organizational Psychology 72, 163–185.