The influence of interaction style and experience on user perceptions of software packages

Int. J. Human-Computer Studies (1997) 46, 563—588

The influence of interaction style and experience on user perceptions of software packages SUSAN WIEDENBECK Computer Science Department, University of Nebraska, Lincoln, NE 68588, USA SID DAVIS Faculty of Commerce and Administration, Concordia University, Montre& al, Canada H3G 1M8 (Received 27 February 1996 and accepted in revised form 6 December 1996 ) In recent years, a body of literature has developed which shows that users’ perceptions of software are a key element in its ultimate acceptance and use. We focus on how the interaction style and prior experience with similar software affect users’ perceptions of software packages. In our experiment, direct manipulation, menu-driven and command-driven interfaces were investigated. We studied users’ perceptions of the software in two hands-on training sessions. In the first session, novice users were given initial training with word-processing software, and in the second session the users were trained on a word processor which was functionally equivalent to the prior one, but had a different interaction style. In the initial training session, we found that the interaction style had a reliable but small effect on learners’ perceptions of ease of use. The direct manipulation interface was judged easier to use than the command style. The interaction style, however, did not affect learners’ perceptions of the usefulness of the software. In the second training session, subjects who had used a direct manipulation interface in the first session learned either the menu-based or command-based software. The perceptions of these users were compared to those of learners, who had used the menu or command software in the initial training session. We found that both interaction style and the prior experience with a direct manipulation interface affected perceptions of ease of use. Subjects with prior experience of a direct style interface tended to have very negative attitudes toward a less direct interface style. The interaction style did not affect perceptions of usefulness of the package, but the prior experience did. These results suggest that users’ attitudes toward software are strongly influenced by their past history of usage, including what interaction styles the user has encountered, and this should be considered in the design of software and training programs. ( 1997 Academic Press Limited

1. Introduction We are concerned with the ability of end users to learn software packages initially and to go on to learn other functionally similar packages when required. To date, most work on learning and transfer of software skill has focused on performance issues, i.e. how different conditions affect the speed of performance and correctness (Singley & Anderson, 1989; Bovair, Kieras & Polson, 1990; Carroll, 1990; Benbasat & Todd, 1993; Davis & Bostrom, 1993). In this paper, we focus instead on users’ perceptions. In recent years, a body of literature has developed which shows that learner perceptions of software are a key element in its ultimate acceptance and use (Davis, Bagozzi & Warshaw, 1989; Davis, 1989, 1993; Adams, Nelson & Todd, 1992; Segars & Grover, 1993; Chin & Todd, 563 1071-5819/97/050563#26$25.00/0/hc960106

( 1997 Academic Press Limited

564

S. WIEDENBECK AND S. DAVIS

1995). This relationship of perceptions of software packages to their adoption and use was our motivation for exploring factors which affect users’ perceptions. While it seems clear that many factors may play a role in perceptions, we chose to focus on the interaction style because it is a prominent feature of software which has already been shown to influence performance (e.g. Whiteside, Jones, Levy & Wixon, 1985; Benbasat & Todd, 1993; Davis & Bostrom, 1993). The interaction style may have a strong impact on perceptions of software and ultimately on its use, particularly for users who are not computer professionals and who are characterized by an irregular or less-intense pattern of use (Santhanam & Wiedenbeck, 1993). Systems which are perceived as difficult to learn or use or of marginal usefulness may be rejected by discretionary users, who have the freedom to choose their own software. Even users who do not have a free choice of software may minimize their use of packages that they perceive as being difficult to use or less useful. This sort of avoidance may prevent users from attaining the level of skill which is necessary to achieve their minimal computing goals. Even if they learn to carry out a minimal set of tasks adequately, they may be discouraged by their perceptions of the software from learning efficient methods or from expanding their computer skills to support tasks not initially foreseen. In our experiment three word-processing systems, representing different interaction styles, were investigated. The first was a system incorporating elements of direct manipulation and menu selection. The second was a wholly menu-driven system. The third was a command-based system. We studied attitudes toward the software in two hands-on training sessions. The first session was the user’s initial training with word-processing software. In the second session, the subject was trained on a word processor which was functionally equivalent to the first one but had a different interaction style. Two interaction styles were used in the second session, one relatively similar to the style used in the first session and the other distinctly different. With respect to initial learning, we wanted to determine whether people develop different perceptions about software when they are trained with different interaction styles. Learning a second system implies a situation in which there are similarities between the old and new situations as also differences. In terms of perceptions, the question is how the previous experience with a similar or distinctly different interaction style affects users’ views of the new software and their corresponding willingness to use it. It seems plausible that the users’ previous perceptions may influence future perceptions and that perceptions may be strongly affected by the mode of interaction, which mediates the use of the system. The following section presents a framework for our research on the role of perceptions in computer learning. It then reviews past research relevant to this study. Section 3 presents the research method. The results of the experiment are presented and discussed in Section 4. The implications of this research for design of interfaces and training are discussed in the final section, along with suggestions for future research.

2. Research framework and previous research 2.1. USER PERCEPTIONS OF COMPUTER SYSTEMS

In this paper we approach the study of user perceptions of computer systems through Davis’ Technology Acceptance Model (TAM) (Davis et al., 1989). The model appears in

USERS’ PERCEPTIONS OF SOFTWARE PACKAGES

565

FIGURE 1. Technology acceptance model (Davis et al., 1989).

Figure 1. TAM is an adaptation of the more general Theory of Reasoned Action (Fishbein & Ajzen, 1975; Ajzen & Fishbein, 1980), and its purpose is to model computerusage behavior. Like the Theory of Reasoned Action, it is an intention model. According to TAM, the actual use of a computer system is determined by the strength of an individual’s behavioral intention to use it. While this behavioral intention to use a computer is an internal psychological state of the user, it is determined by external variables such as system design and conditions of use. Therefore, a key purpose of TAM is to evaluate the impact of external factors on internal beliefs and to link that to actual use. According to TAM, two key beliefs are primarily involved in decisions to use computer systems. These beliefs are perceived usefulness (PU) and perceived ease of use (PEU) (Davis et al., 1989). Perceived usefulness is the user’s subjective perception of the extent to which the system or software will aid in job performance. Perceived ease of use is defined as the extent to which the user expects a system or software to be easy to learn and use. As shown in Figure 1, the behavioral intention to use a system is jointly determined by perceived usefulness and the individual’s attitudes toward usage. Attitudes toward usage represent the user’s affect about a computer system. The relationship of attitudes to the behavioral intention to use a system is that, all other things being equal, people will form an intention to use systems about which they have positive attitudes. The relationship of usefulness and intention to use is founded on the idea that, independent of users’ attitudes toward a system, they may form the intention to use it if they perceive it as improving their job performance. In TAM, attitudes toward usage are determined by both perceived usefulness and perceived ease of use (see Figure 1). If a system is perceived as useful, it leads to the creation of positive attitudes about it. If it is perceived as easy to use, it also creates a positive attitude. Because of the link of usefulness to attitudes, beliefs about usefulness have two routes to influence the intention to use, acting directly and also indirectly through attitudes. Perceived ease of use also acts in two ways. While it is a determinant of attitudes toward usage, it also influences the perceived usefulness of a system. In TAM, ease of use is seen as influencing attitudes through the mechanism of self-efficacy (Davis et al., 1989). Self-efficacy is subjective beliefs about one’s ability to perform a certain behavior, and it is believed to underlie intrinsic motivation (Bandura, 1986). Research has shown that self-efficacy beliefs have a strong influence on computer use (Hill, Smith

566

S. WIEDENBECK AND S. DAVIS

& Mann, 1987; Gist, Schowoerer & Rosen, 1989; Compeau & Higgins, 1991, 1995). Davis et al. (1989) and Venkatesh and Davis (1994) argue that a person’s self-efficacy should be increased by a system which is easy to use and increases one’s sense of mastery. The other role of ease of use in TAM is as a contributor to perceived usefulness (Davis et al., 1989). A system which is considered easy to use may be perceived as likely to save time and effort, thus increasing its perceived usefulness, as well. Davis and his colleagues developed and validated TAM through a series of studies (Davis et al., 1989; Davis, 1989, 1993; Venkatesh & Davis, 1994), and their work has been replicated and extended by others (Adams et al., 1992; Chin & Todd, 1995; Hendrickson, Massey & Cronan, 1993; Segars & Grover, 1993). In one study (Davis, 1989), users were trained to use two graphics packages, then the PEU, PU and subjects’ self-predictions of future use of the packages were measured. In this study, the correlation of usefulness with predicted usage was 0.85 (p(0.001) and of ease of use with predicted usage was 0.59 (p(0.001). In another study, reported by Davis (1989, 1993), users of an electronic mail program and a line editor filled out the PEU and PU questionnaires and also a questionnaire about their current usage of the two programs. Here there was a significant correlation of 0.63 (p(0.001) between usefulness with usage and of 0.45 (p(0.001) between ease of use and usage. In both of these studies, the relationship of usefulness with actual or predicted usage was stronger than the relationship of ease of use with usage. Also, there was a high correlation between ease of use and usefulness, 0.56 (p(0.001) in the graphics study and 0.64 (p(0.001) in the electronic mail/editor study. These results were interpreted as evidence that usefulness operates directly on intention to use a system, while ease of use operates more indirectly, at least in part through usefulness. A third study involving the use of a word processor (Davis et al., 1989) measured PEU, PU, intention to use the software and self-reported usage of the software at a later point in time. This study generally supported the results and interpretations of the previous studies. It established that there was a significant correlation of 0.35 (p(0.001) between subjects’ intentions to use the system after training and self-reported use of the system 14 weeks later. The results of measurements of usage at two points in time suggested that the relationship of PEU to behavioral intention to use a system is more direct in the early stages of use and gradually becomes an indirect influence through PU. A possible explanation for this change is that the user is highly influenced by the surface features of the interface in the early stages of use, but becomes more concerned with the functionality of the software as initial difficulties are overcome. In a replication of Davis’ work, Adams et al. (1992) also found that, while usefulness is the better predictor of usage, there are some cases where ease of use predicts quite well. They relate this to the stage of learning of the user and to differences in the software studied.

2.2. EXTERNAL VARIABLES AND USER PERCEPTIONS

Work on the TAM has established that perceived ease of use and perceived usefulness are fundamental determinants of user acceptance of technology (Davis, 1989; Davis et al., 1989; Adams et al., 1992). TAM postulates that external variables influence both PEU and PU (Davis et al., 1989). However, the external variables that are hypothesized to influence PEU and PU have not been studied extensively. External variables include system characteristics and also characteristics of the usage situation. For example,

USERS’ PERCEPTIONS OF SOFTWARE PACKAGES

567

a system which produces results faster, gives more accurate output, or produces more kinds of output may be judged more useful based on those system characteristics. Likewise, a system may be judged easier to use, based on external variables. System features such as menus, icons and mouse input have been argued to increase ease of use (Bewley, Roberts, Schroit & Verplank, 1983; Davis & Bostrom, 1993). Training, documentation, printed or on-line help and user support services may also increase the ease of use of a system (Davis et al., 1989; Carroll, 1990; Davis & Bostrom, 1993). An external variable which is likely to influence outcomes of computer training is the learner’s prior experience with computers in general and, particularly, experience with similar software. We refer to the situation in which the trainee lacks significant prior experience as first-time use and the situation in which the trainee has prior experience with similar software as subsequent use. It seems likely that attitudes toward using a computer system will be affected by the prior relevant computer experience, or lack thereof, of the trainee. Empirical work on TAM has not systematically addressed the influence of prior experience on the formation of attitudes and eventual computer use. However, research on self-efficacy and computer anxiety gives some suggestion that prior experience is a variable worth exploring. Bandura (1986) showed in several studies that prior experience is an important influence on self-efficacy. Heinssen, Glass and Knight (1987) showed that there is a significant but moderate negative correlation between computer anxiety and prior computer experience. The interaction style of the system is another external variable which may have an important influence on the trainee’s attitudes toward use. Three interaction styles are commonly in use: direct manipulation, menu and command. Direct manipulation interfaces (DMIs) allow the user to carry out computer operations as if they were working on the actual objects of interest in the real world. The gap between the user’s intention and the actions necessary to carry it out is small. These two characteristics of direct manipulation are referred to as engagement and distance by Hutchins, Hollan and Norman (1985). They argue that high engagement and small distance lead to a feeling of directness in a system. Many sources may contribute to the feeling of directness: continuous visibility of the objects of interest, representation of objects in a familiar form (often implemented through icons), manipulation of objects by physical actions (pointing, clicking, touching, dragging) rather than complex syntax, rapid incremental operations, reversibility of actions and immediately visible feedback about the result of actions (Shneiderman, 1982). In the menu style of interaction, objects and possible actions are represented by a list of choices, usually presented through text. Menus are similar to direct manipulation in that they provide guidance to the user. Thus, the burden on memory is reduced, the menu may aid the user in structuring the task and many kinds of syntactic errors, such as misspelled command names or illegal command arguments, are impossible (Shneiderman, 1992). However, menu-based systems are, on the whole, less direct than DMIs because the actions of users are mediated by the syntax and semantics embodied in the menus, and pointing devices may be replaced by keyboards. As a result, the user tends to experience a reduced sense of performing actions directly on the objects of interest. Command interfaces are operated by the user typing a command string in the vocabulary and syntax recognized by the system. The burden of remembering the commands is on the user, as is the burden of structuring a sequence of actions correctly to obtain a desired result (Davis & Bostrom, 1993). Interactions are carried out via

568

S. WIEDENBECK AND S. DAVIS

a keyboard, rather than by pointing, clicking and dragging. The results of actions are often not as visible as in direct manipulation or menu-based systems. Shneiderman (1982) argues that in a command-based system there is a relatively large distance between user’s intentions and the actions to carry them out, and the sense of working on the objects themselves is reduced dramatically. With respect to perceptions of software, Davis et al. (1989) suggest that ions and the use of a mouse to manipulate objects and information directly are external variables which have the potential to affect perceptions of a system positively. These particular system features are associated most closely with direct manipulation. So, in keeping with the Davis et al. suggestion, we might expect that systems with a direct manipulation style of interaction would lead to more positive perceptions of software. In an empirical study which addressed this question, Davis and Bostrom (1993) found that perceptions of ease of use were higher for novice computer users trained on a DMI, than for users trained on a command-based system. In the empirical study reported here we address three research questions: (1) is there evidence that the interaction style affects trainees’ perceptions of software?, (2) is there evidence that trainees’ perceptions of software are affected by previous training with functionally similar software? and (3) is there an interaction between previous training with similar software and the interaction style?

3. Research method 3.1. OPERATIONALIZING THE RESEARCH VARIABLES

The domain selected for this study, word processing, offers several advantages for a study of this type. First, since it is a common application domain, the findings of the study should be applicable to a wide variety of users. Second, given the large number of applications available, word processing allows us to select software systems with similar functionality but different interaction styles. Third, while there is a history of research on learning and transfer in word processing and text editing, the only work on perceptions of word processing was done in the development and validation of the TAM (Davis et al., 1989; Adams et al., 1992; Segars & Grover, 1993). In this research, we chose to study direct manipulation, menu and command-based interaction styles. However, the question of how to operationalize the three interface styles for research purposes is a difficult one. Some researchers have tried to create ‘‘pure’’ instantiations of different interaction styles for the purpose of comparison (e.g. Benbasat & Todd, 1993). This approach has the advantage of maximizing the differences between systems and allowing for a clean comparison. Its disadvantage, however, is that it appears to be low in ecological validity (Thomas & Kellogg, 1988) because pure interaction styles, particularly a pure DMI style, are difficult to adapt to complex, multi-function software. In fact, most commercial software, with the exception of some games and educational packages, consists of a combination of styles. We are concerned with training individuals to use realistic software systems. Therefore, we chose to increase ecological validity by carrying out this research using complex commercial software rather than ‘‘toy’’ systems. This led us to do a holistic comparison, as have previous researchers whose goal was to study realistic systems (Whiteside

USERS’ PERCEPTIONS OF SOFTWARE PACKAGES

569

et al., 1985; Davis & Bostrom, 1993). In a holistic comparison, the ensemble of software and hardware making up the system is tested without decomposing into component variables. In fact, Whiteside et al. (1985) argue that a reductionist experimental strategy is futile in investigating realistic systems because differences are due to an inextricably complex interaction of causes. Under these circumstances, the goal of experimentation should be to identify the largest sources of variance and investigate how they affect users, simply accepting that other sources are present and probably play a small role in the observed results. In the current research, we argue on logical and experimental grounds that the important difference experienced by subjects was the interaction styles of the different word-processing systems. Other software and hardware differences were minor or even invisible to subjects and would not realistically be expected to have an important effect on perceptions. In the following paragraphs, we first describe the three wordprocessing systems which we used, and then the other software and hardware differences which we consider to be minor in the context of this research. As a result of choosing complex commercial software, the word-processing systems we used for testing were hybrids, and the differences in interaction styles among them should be understood as a matter of degree. We see them as occupying points on a continuum, where a direct style of interaction would be at one end and indirect style at the other. Viewed in this way, the system that we refer to as direct manipulation is clearly the closest to the direct end of the continuum, even though it uses menus for certain operations. Likewise, the system that we refer to as command-based is closest to the indirect end of the continuum, even though a few operations have an immediately visible effect on the object of interest, which is more characteristic of DMI systems. Macintosh Word was the word-processing system closest to the direct manipulation end of the continuum. The features which led us to classify it as most direct are: (1) continuous visibility of objects being worked on, (2) representation of objects in familiar form (text pages for a document and descriptive icons for operations), (3) most actions carried out directly on the objects of interest, either icons or the document’s text, without the intermediary of a command-language or menu choices, (4) use of a mouse for selection of text and icons, which allows direct actions on objects by pointing, clicking and dragging and (5) immediate feedback on the result of operations by a change in the visible features of screen objects. On the other hand, Macintosh Word makes use of menus for some editing and formatting operations, and this feature is less direct because direct action on text or objects is replaced by the intermediary of the menus. Specifically, in the version of Macintosh Word which we used, text style options (bold, italics) and the copy/paste option were menu-controlled. The other operations taught in the experiment were represented by direct actions on icons. Thus, for a few specific operations the DMI was similar to the menu-based system. Nevertheless, even in these specific operations, the DMI was still distinguished from the menu-based system by its more direct selection using a mouse. Macintosh Word also has command options for some operations, but subjects were not made aware of these commands in the training and they were carefully monitored to be sure that they did not use them. (In fact, our monitoring shows that subjects did not try to use them spontaneously.) In classifying this software as closest to the direct manipulation end of the continuum, it is important to note that the authors do not view direct manipulation as a purely non-linguistic phenomenon. What makes an interface direct manipulation is its ability to give the user the feeling of operating directly

570

S. WIEDENBECK AND S. DAVIS

on the objects of interest. While no commercial interfaces have fully achieved the ideal of a non-linguistic mode of interaction, we argue that some, such as the Macintosh Word used in this study, do impart this greater feeling of directness. The command-driven Vi/TROFF software was chosen to represent the indirect end of the continuum. In our experiment, this software ran on a mainframe and was accessed through a Macintosh acting as a terminal. Vi is a full-screen editor and TROFF is a text-formatting program. The text of a document and formatting commands are typed into a Vi file. Then the file is processed by TROFF, and the formatting commands are applied to the text to produce a file containing the formatted document, ready to be printed. This system is very indirect for several reasons: (1) in both Vi and TROFF, actions are expressed by artificial and often non-mnemonic command codes, (2) references to a document are expressed indirectly (e.g. to select multiple text lines for copying in Vi one must go to the first line of the segment to be operated on, count the number of lines and issue a command to copy that number of lines) and (3) the results of TROFF actions are not immediately visible, since formatting commands are only applied to the document when it is processed by the formatting program. Even though Vi/TROFF is very indirect, it should be pointed out that it too is a hybrid with a few features that can be considered more direct. Two features come to mind. First, since Vi is a full-screen editor, a whole screen of text is visible at once, not just isolated lines, as in line editors or small window electronic typewriters. To correct or add to the text in a full-screen editor, one goes to the location in the text where the changes are to appear and types. Second, when text is moved, copied or deleted in Vi, the result of the operation appears immediately in the text. Nevertheless, we argue that, overall, the features of this software give it a noticeably greater feeling of indirectness than the DMI. Finally, PC Word was used as the menu-drive interface. It represented the middle of our three interfaces on the continuum from direct to indirect interface styles. The version of PC Word used in this experiment runs under the DOS operating system. While versions of PC Word running under the Windows operating system are very similar to Macintosh Word in their incorporation of direct manipulation features, this DOS-based version is strongly menu-driven because all operations are carried out by making choices in hierarchical menus. Compared to Macintosh Word, the actions of users are mediated to a much greater extent by the linguistic representation of operations embodied in the textual menus. Icons are not used in the software. A mouse may be used optionally but was not used in our experimental environment. Rather, the specification of operations was done by using the arrow keys to select operations from lists and using the keyboard to fill in parameters. Also, all text selection in this version was done by using arrow keys to move the cursor over the desired text, not clicking or dragging with a mouse. Because of the linguistic representation of operations and the less direct keyboard selection, we argue that PC Word users experienced less of a sense of directness than subjects using the Macintosh. Nevertheless, several features of PC Word are direct manipulation-like. For example, while selection is done using the keyboard, it does involve action on the text itself, not an indirect reference to the text. The menu choices relieve the user of the burden of remembering commands and provide constraints to guide the user’s actions. Also, the results of most actions are immediately visible. Menu shortcuts, which were more command-like, were present in the software. However, they were not taught to our subjects and our close monitoring of the training assured us that subjects did not use

USERS’ PERCEPTIONS OF SOFTWARE PACKAGES

571

them. Based on the features discussed above, we consider PC Word to occupy the middle place of our three interfaces. Based on the above analysis, we argue that the interactions styles, although differing in degree rather than absolute terms, would be experienced as a major difference by our subjects. Other sources of variance in the total software system, which we argue to be very minor in comparison, are word-processing functionality and operating system software. All three word-processing systems are sophisticated software capable of carrying out a wide range of functions. There are differences among them in functionality. However, these differences are in advanced features of the software. All three systems possessed the set of basic functions that was taught to our subjects. The subjects never dealt with more advanced features and did not encounter any instances in which the functionality of the software systems differed. Thus, this was not a source of variance in the experiment. Likewise, the different operating systems did not affect the subjects. When the experimental session began, subjects were placed immediately in the environment of the word processor, and they stayed there throughout the session. They never had a need to carry out operating system commands or to use any software other than the word processor. With respect to hardware differences, we argue that most of these could not reasonably be expected to have more than a minor effect on subjects. The experiment was set up so that all subjects worked on a small 10 inch monitor in a single-window environment. The Macintosh monitor was in an integrated unit with the computer, while the PC monitor was a separate unit sitting on top of the computer. The Vi subjects used a Macintosh as a terminal to the mainframe and so were identical to the Macintosh subjects with respect to the visual display unit. In all cases, subjects could move the visual display and keyboard to suit themselves (but tilting was not possible in any monitor). The text presentation was designed so that the same amount of text could be displayed on the monitor for each system. Subjects could always see the results of their actions on the screen in the format provided by their word processor. Computer response time was never an issue. Subjects worked slowly, since they were learning to use a word processor. Their interactions involved typing text and invoking formatting operations on a small document, so there were no computing-intensive activities. Response time always appeared to be instantaneous in all three systems. We have collected and analysed verbal protocols of a smaller group of subjects learning to use the three word processors under the same conditions as in the present experiment. There is no indication in the protocols that the subjects experienced difficulties with one hardware configuration more than with the other. Our experimental data also tend to support this interpretation (see Section 4). The keyboard and mouse (where it existed) were the input devices by which subjects carried out interaction and effected all changes. As such, they play a dual role in our view. They are hardware components, on the one hand, but in a sense more relevant to this experiment they are part of the interaction styles that we were comparing. The keyboards were similar across all three systems (identical for the Macintosh, Word and Vi), containing letters, numbers, arrow keys and a few special keys (e.g. escape, delete, control, alt), but no numeric keypad. In addition, the PC keyboard contained a row of function keys that were not used in this experiment, since the PC Word used in the experiment was completely menu-driven. There was no indication in the protocols of

572

S. WIEDENBECK AND S. DAVIS

difficulties with either keyboard, beyond the occasional wrong key press. Only the DMI system used a mouse. (It was removed when the Macintosh was used as a terminal to the mainframe.) While this is a hardware difference, we intentionally included it in our experiment. According to the argument made above and supported by others (Shneiderman, 1982; Hutchins et al., 1985; Benbasat & Todd, 1993), manipulation of objects by physical actions of pointing, clicking and dragging is a fundamental component of the directness of DMI systems. Thus, in the context of this experiment, we consider it to be part of the interaction style we were testing.

3.2. SUBJECTS

One hundred and seventy-three subjects from a large Midwestern university were selected for the study from several sections of an undergraduate computer literacy course. The purpose of the course was to introduce business students to software tools, including operating systems, word processing, spreadsheet and database applications for the PC and the Macintosh. This study served as a substitute for the word-processing section of the course. Participation was voluntary. Those who took part participated in two training sessions and had to attend both sessions in order to receive credit. Students were not told in advance what systems they would be learning. So, they had no expectations about how easy or difficult the applications would be to learn prior to entering the experiment. Each subject was administered a background questionnaire to assess familiarity with computing and word processing. The 173 subjects selected for the study were carefully screened from a pool of about 450 volunteers, and were selected on the basis of having little or no knowledge of computers or word processing. The average age of the subjects was 21.38 years and the average college grade point average was 3.08 on a scale from 0 to 4. There were 106 male subjects and 67 female subjects. On average, the subjects had first been introduced to computers in high school. They had taken an average of one previous course about computers in high school or college and they had taken one previous course in high school or college which involved some computer use (e.g. using a drawing program in a course assignment). The subjects’ self-reported previous use of word processing was 1.82 on a scale from 1 to 5. Thus, it fell between designated usage categories of ‘‘not at all’’ and ‘‘a little.’’ We believe that this low exposure to computers and word processors is important, because it makes it unlikely that subjects had developed specific perceptions of the software prior to the study. To guard further against possible effects from differences in prior perceptions, subjects were randomly assigned to treatments. Comparisons of the treatment groups with respect to the background variables described above showed no differences between groups.

3.3. MATERIALS

Self-study tutorial manuals were prepared for use in the training sessions. The manuals covered basic word-processing tasks, including typing, inserting, modifying, deleting and copying text, character formatting (bold and underline), line and paragraph formatting (centering different units of text and line spacing), indenting paragraph margins, setting tab stops and inserting page breaks. There was one manual for the DMI, another for the

USERS’ PERCEPTIONS OF SOFTWARE PACKAGES

573

menu-based software and a third for the command-based software. The three manuals each were about 23 pages long. All three manuals covered the same material, were structured in the same way and used the same headings. The wording of the text of the manuals was very similar, being adjusted only to reflect the conventions of each word-processing application. For each word-processing skill taught, the manuals presented a brief description, followed by an example, followed by instructions for a handson practice exercise. In the hands-on practice exercises, tasks were given to illustrate the concept under discussion. A computer file was provided in which the subject was supposed to do the practice exercises.

3.4. EXPERIMENTAL PROCEDURE

Training sessions, which included from 10 to 12 subjects each, were held in computertraining facilities at the university. Each interface type (DMI, menu and command) used separate facilities, so subjects using different interfaces were not mixed. The DMI and menu groups used training facilities containing stand-alone machines, while the command group used terminals linked to a mainframe computer. Training sessions 1 and 2 were separated by a time lag of 2—3 days. The time between sessions was kept relatively short so that subjects would not lose interest or forget what they had learnt in the previous session. All subjects were assigned randomly to treatments. In the first session, which represented the first-time use condition, one-half of the subjects were assigned to the DMI and one-quarter each to the menu and command software. In the second session, the DMI subjects from session 1 were randomly assigned to training on the menu-based software or the command-based software. In the second session, the DMI served as these subjects’ prior context in formation of perceptions of the menu or command-based system. Past research has shown that a DMI helps build a strong mental model of systems (Benbasat & Todd, 1993; Davis & Bostrom, 1993). We wanted our subjects to develop some mental model of what a word processor does, and the DMI seemed the best candidate for doing that. The initial mental model then became their framework for developing and evaluating subsequent mental models, based on other interface paradigms. Two experimenters conducted training sessions following a standard script. In the first 10 min of each session, the experimenter presented a brief overview of the activities that would follow, and subjects filled out some short questionnaires. Next, subjects worked through the training manuals on their own. Each word-processing operation in the manual and its related set of hands-on tasks referred to sections of a four-page document which subjects worked with and modified on their computers. They also had at their disposal a paper copy of the document in which tasks were number-keyed to the appropriate sections of the manual. Subjects worked through the training manual from beginning to end. If they encountered any problems, the experimenter referred them to the sections of the manual that offered solutions. If subjects were unable to answer their questions themselves using the manual, the experimenter gave brief hints. Subjects were not allowed to communicate among themselves. The hands-on training lasted 55 min. This was sufficient time for subjects to work through the training manual doing the practice exercises given in it. It is similar to the amount of time that Davis (1989) and Davis et al. (1989) trained subjects in their studies.

574

S. WIEDENBECK AND S. DAVIS

Subjects worked through the manuals independently. If they finished before the end of the training period, they were allowed to review or repeat their work on the wordprocessing features taught in the manual. After having done the exercises, the subjects were familiar with basic word-processing tasks and the procedures for executing them in the given software. However, they had not yet gained full mastery of the software. At this point we assessed attitudes toward the software. Subjects filled out the perceived ease of use and perceived usefulness questionnaires. The PEU and PU questionnaires used were based on the four-item scales used in Davis et al. (1989), with slight modifications to reflect the domain of word processing. The questionnaires appear in Appendix 1.

3.5. DATA ANALYSIS

The perceived ease of use and perceived usefulness instruments were scored on a sevenpoint Likert scale. Each item was stated positively, with a score of 1 indicating strong agreement and a score of 7 strong disagreement. Therefore, low overall scores indicated high perceived ease of use or usefulness, while high scores indicated the opposite. The PEU and PU scales were highly reliable. The Cronbach’s a for the PEU was 0.95 and for the PU 0.94. These reliability figures are similar to those reported in past research. Davis et al. (1989) found a Cronbach’s a of 0.94 for the PEU and 0.91 for the PU. Adams et al. (1992) report similar results for the PEU and PU. Also, Davis and Bostrom (1993) found a Cronbach’s a of 0.92 for the PEU. The data were analysed using path analysis (Kerlinger & Pehazur, 1973; Retherford & Choe, 1993). Path analysis uses multiple regression to analyse causal models in which complex relationships exist. Multiple regression tests for a direct effect of each predictor variable on the response (dependent) variable. Path analysis extends multiple regression to the case in which each predictor variable may affect the response variable directly and also indirectly through intervening variables (Retherford & Choe, 1993). In the model we tested, interface and session were exogenous predictor variables (i.e. variables in the model whose variability is caused by variables outside the model), and we tested their direct effect on the result variables PEU and PU. However, we also tested the direct effect of the variable PEU on PU. PEU is an endogenous variable whose variability is caused by variables in the model, so it also represents an indirect effect of those variables (interface and session) on PU. This is exactly the case that path analysis is meant to handle. The relationships posited in TAM suggest the use of a path-analytic technique. Other researchers studying TAM (Davis et al., 1989; Adams et al., 1992; Segars & Grover, 1993; Chin & Todd, 1995) have used analysis techniques drawn from a broad family of techniques known as structural equation modeling, which also test complex models of direct and indirect causality. Thus, our analysis is consistent with the approaches taken in their work. One analysis was done of first-time users alone and another comparing first-time users to subsequent users. The first-time use design was based on data of all subjects in the first session. The subsequent use design compared the subjects who learnt the DMI in session 1 and then the menu or command-based software in session 2 to the subjects who used the menu or command-based software as first-time users in session 1. Thus, in this design, subjects who used these applications in session 1 served as controls for the subjects who used the same applications in session 2.

575

USERS’ PERCEPTIONS OF SOFTWARE PACKAGES

4. Results 4.1. FIRST-TIME USE OF A WORD-PROCESSING SYSTEM

For first-time use, we tested the influence of the interface type on perceptions. The means and standard deviations of PU and PEU in first-time use are shown in Table 1. The original causal model tested in our path analysis is shown in Figure 2(a). This model is based on expectations from TAM. Interface is an external variable, hypothesized to influence both PU and PEU. PEU is also hypothesized to affect PU, as argued by Davis et al. (1989). The model also includes disturbance terms, D1 and D2, to account for unmeasured influences on the two result variables. (Disturbance terms are calculated as (1!R2)1@2.) Because the interface variable consisted of three levels, DMI, menu and command, we followed the procedure outlined in Kerlinger and Pehazur (1973) and Retherford and

TABLE 1 Means and standard deviations (in parentheses) for firsttime use analysis

PEU† PU‡

DMI

Menu

Command

Mean

9.85 (3.12) 8.52 (3.10)

10.66 (4.10) 8.89 (4.44)

11.82 (3.88) 9.64 (3.75)

10.55 (3.66) 8.90 (3.65)

Note: DMI n"85; menu n"44; command n"44. †Scores range from 4 to 28, greatest ease of use"4. ‡Scores range from 4 to 28, most useful"4.

FIGURE 2. Results of path analysis for first-time use: (a) original model and (b) reduced model.

576

S. WIEDENBECK AND S. DAVIS

Choe (1993) of grouping the levels of the interface variable to create two binary (dummy) variables. One variable indicated whether an observation was menu (coded 1) or not menu (coded 0). The other indicated whether an observation was command (coded 1) or not command (coded 0). In general, k levels of a variable can be compared by k!1 groupings of the levels. The third variable, DMI, was not coded, but was used as a base-level comparison with a regression coefficient equal to zero. Thus, in our case, this analysis allowed us to discover any significant differences among the three interface types by comparing the command and menu binary variables with the base-level DMI variable. The path coefficients are shown in Figure 2(a) along with the R2-values and the disturbance terms for the result variables, PEU and PU. These path coefficients are b-values, or normalized regression coefficients, which represent the command vs. DMI comparisons. The F-test for the regression of the interface variables on PEU indicated that there was a significant difference among the three types of interfaces and this was confirmed by an ANOVA with interface as the independent variable and PEU as the dependent variable [F(2, 172)"4.41, p(0.014]. We were interested, however, in understanding where those differences lay. The t-tests of individual b-values indicated that the commandbased group (b"0.24) was significantly different from the DMI group (p(0.01), while the menu-driven interface was not (b"0.10, p(0.22). In order to find out whether the menu group differed significantly from the DMI group, we re-ran the previous regression substituting a DMI binary variable for the command variable and using command as the base-level comparison variable. The t-test of the menu-interface beta value indicated that the effect of the menu-driven system was not statistically different from the DMI (b"!0.14, p(0.13). Therefore, these results suggest that for first-time users the DMI was perceived as easier to use than the command-based interface. The menu system was rated between these two and was not significantly different from either. These results indicate that the difference between the DMI and the command-driven interface accounted for the positive effect of the interface variable on the PEU. The effect of the interface on PU was not significant (p(0.41). However, PEU did have a significant effect on PU (p(0.01). To confirm these results, we ran an ANCOVA with interface as the independent variable and PU as the dependent variable but using PEU as a covariate. In this analysis, interface was not significant [F(2, 169)"0.34, p(0.713], but the covariate was [F(1, 169)"14.98, p(0.0001]. From this analysis, we can infer the indirect effect of PEU on PU, since interface becomes non-significant when the effect of PEU is subtracted out. Interface accounted for about 5% of the variance in PEU (R2"0.05), and PEU accounted for about 10% of the variance in PU (R2"0.10). Because of the lack of significance of the interface to PU path, we revised our original model. The revised model appears in Figure 2(b). It shows that interface has a direct effect on PEU and also an indirect effect on PU through PEU. This model was validated by recreating the correlation between interface and PU using indirect path coefficients (Kerlinger & Pehazur, 1973). If the revised model is an accurate representation of the relationships of the variables, then the recreated correlation coefficient should be approximately equal to the actual correlation coefficient. A small difference between the recreated vs. actual correlations indicates that the deleted path is not needed in the model. Guidelines for how large a difference is acceptable between the actual r and the recreated r are not firmly established in the literature. Kerlinger and Pehazur (1973)

USERS’ PERCEPTIONS OF SOFTWARE PACKAGES

577

recommend a difference of less than or equal to 0.05 in order to conclude that the recreated and actual correlation coefficients are approximately equal. The calculations for recreating the correlation between interface and PU are shown in Appendix 2. The difference between the actual and the recreated correlation coefficients was 0.047. Since this value is less than 0.05, we conclude that the revised model is a valid representation of the causal relationships among interface style, PEU and PU, and that the direct path from interface to PU is not needed. To return to the discussion of the effect of hardware in a previous section, the results on first-time use tend to support our argument that extraneous hardware differences did not have an appreciable impact on perceptions. As stated earlier, we consider the interface style to consist of the word-processing software along with the interaction devices used for input, i.e. keyboard or keyboard plus mouse here. Thus, the presence or absence of a mouse was part of our experimental manipulation. Beyond that the hardware difference most obvious to the subjects was probably the monitor. As described earlier, the DMI and command subjects both used identical Macintosh monitors, while the menu subjects used a PC monitor. Our results show that the PEU was not significantly different for the DMI and the menu groups. Also, it was not significantly different for the menu and the command groups. These are the two cases in which the monitor differed, but the PEU was unaffected. By contrast, the PEU for the DMI subjects was significantly better than for the command subjects. Here the monitor experienced by the user was the same but the PEU differed. These results lend support to our contention that this hardware difference did not have a large effect on users’ perceptions of ease of use.

4.2. SUBSEQUENT USE OF A DIFFERENT WORD-PROCESSING SYSTEM

In this design we compared the attitudes of subjects, who used the menu- or commandbased interface in their first session, without previous word-processing training, to the attitudes of those who learnt the menu- or command-based interface in their second session, after having had previous word-processing experience with the DMI in session 1. Subjects who learnt the menu- or command-based system in session 1 went through 55 min of training and then their perceptions were measured. Subjects who used the same systems previously in session 2 had 55 min of training on the DMI, then 55 min of training on the menu or command system, after which their perceptions were measured. We did not give the former subjects, a second training session on the menu or command software to try to equalize the amount of training time or training tasks with the other group, because of the danger of boredom and apathy, which could have gravely affected perceptions if the same training were repeated. The subjects learning the menu- or command-based software after the DMI were unlikely to be subject to these dangers in their second session because they were being trained on very different interfaces, which presented new challenges. According to the experimenters’ observations there was no evidence of boredom in these subjects’ second sessions. The subjects were fully engaged in the training and were busy for the whole training period. The means and standard deviations for PU and PEU are shown in Table 2. A path analysis was carried out to measure the effects of interface and previous experience on PEU and PU. The original causal model, based on TAM, is shown in Figure 3(a) with the

578

S. WIEDENBECK AND S. DAVIS

TABLE 2 Means and standard deviations (in parentheses) for subsequent use analysis Session 1 (first-time users)

Session 2 (subsequent users)

Mean firsttime

Mean subsequent

Grand mean

Menu

Command

Menu

Command

PEU†

10.66 (4.10)

11.82 (3.88)

10.21 (4.11)

17.19 (5.70)

11.24 (4.01)

13.74 (6.06)

12.47 (5.26)

PU‡

8.89 (4.44)

9.64 (3.75)

9.95 (4.32)

13.49 (5.91)

9.26 (4.10)

11.74 (5.45)

10.48 (4.96)

Note: Menu first-time users n"44; command first-time users n"44; menu subsequent users n"42; command subsequent users n"43. †Scores range from 4 to 28, greatest ease of use"4. ‡Scores range from 4 to 28, most useful"4.

FIGURE 3. Results of path analysis for subsequent use: (a) original model and (b) reduced model.

path coefficients, R2-values and disturbance terms. In this model interface refers to either menu or command. The interface type is an external variable which is hypothesized to affect both PU and PEU, in accordance with the suggestions of TAM (Davis et al., 1989). Session is also an external variable which incorporates the concept of prior experience with another interface type. Specifically, half the subjects used the menu- or commandbased interface in the first session, without previous training on another interface type. The other half used the menu- or command-based interface in the second session after training on the DMI. Figure 3(a) shows that the direct path from interface to PEU was significant (p(0.01), as was the direct path from session to PEU (p(0.01). The positive sign of the coefficient on the path from interface to PEU indicates that the menu group had significantly more positive perceptions of their software than the command group.

USERS’ PERCEPTIONS OF SOFTWARE PACKAGES

579

The negative sign of the coefficient on the path from session to PEU indicates that the group that used the menu- or command-based interface in the second session had a significantly more negative attitude towards the ease of use of the software than the group that used the same software in session 1. These main effects were confirmed by an ANOVA with interface and session as the independent variables and PEU as the dependent variables [interface: F(1, 172)"34.41, p(0.0001, session: F(1, 172)"18.03, p(0.0001]. Interface type and session accounted for 20% of the variation in PEU (R2"0.20). The direct path from interface to PU was not significant (p(0.81), but the direct path from session to PU was significant (p(0.05). Again, the negative sign of the latter path coefficient indicates that those using the menu or command software in session 2 were more negative about its usefulness than those using it in session 1. PEU also had a significant effect on PU (p(0.01). These results were confirmed by an ANCOVA with interface and session as the independent variables, PEU as the covariate and PU as the dependent variable. In this analysis, the covariate PEU was significant [F(1, 168)"47.99, p(0.0001]. Controlling for the effect of PEU, interface was not significant [F(1, 168)"0.05, p(0.822], but session remained significant [F(1, 168)"3.71, p(0.05]. This is consistent with our path analysis in suggesting that the interface affects PU indirectly through PEU, while session has a direct effect on PU. Interface, session and PEU accounted for 32% of the variance in PU (R2"0.32). Based on these results we pruned the non-significant direct path from interface to PU. The revised model is shown in Figure 3(b). We recreated the correlation between interface and PU to validate this revised model. The calculations are shown in Appendix 2. The difference between the actual r and recreated r was 0.011. This small difference indicates that direct path from interface to PU can be eliminated, supporting the revised model. We were also interested in testing for an interaction between interface and session in our model. Retherford and Choe (1993) recommend investigating interactions by carrying out a separate path analysis for each level of one of the variables in the interaction. We did this by carrying out two path analyses, one for the menu users only and one for the command users only. This allowed us to view the effects of session while holding the interface constant. A comparison of the results of the two analyses allows us to draw conclusions about the interaction of interface and session. Figure 4(a) presents the path analysis results for menu users only (n"86). It shows that neither the direct path from session to PEU was significant (p(0.612), nor was the direct path from session to PU (p(0.10). However, the path from PEU to PU was significant (p(0.01). Figure 4(b) shows the path analysis for command users only (n"87). For command users the direct path from session to PEU was significant (p(0.01), but the direct path from session to PU was non-significant (p(0.17). The path from PEU to PU was, again, significant (p(0.01). The fact that the path coefficients of Figure 4(a) and (b) are approximately the same except for the large difference in the paths from session to PEU suggests that an interface by session interaction exists with regard to PEU. This interaction of interface and session with respect to PEU is confirmed by an ANOVA [interface * condition: F(1, 169)" 18.03, p(0.0001]. The means in Table 2 show that subjects who moved from the DMI to the command-based interface perceived the latter as much more difficult to use than any of the other interface by session groups. This is indicated by the high mean PEU

580

S. WIEDENBECK AND S. DAVIS

FIGURE 4. Path analysis for subsequent use: (a) menu users only and (b) command users only.

score (17.19) for command/subsequent users. It is worth noting that the session-to-PU paths in both interaction models [Figure 4(a) and (b)] were not significant, while this path was significant in the overall subsequent use model [Figure 3(b)]. A possible explanation of this lack of significance in the interaction models is reduced power resulting from the smaller sample sizes. In spite of this, however, it is important to point out that in the interaction models we were not interested in the magnitude of the paths per se. Rather, we were interested in comparing the relative magnitude of the corresponding paths for the menu and command conditions, in order to detect significant differences (i.e. possible interactions). As in the case of first-time use, the statistical results from this portion of the research tend to support our contention that extraneous hardware differences did not play a large role in the results. In this analysis, the PEU for the menu-based system was higher than the PEU of the command-based system. However, in session 1, which used the same hardware, the PEU for these same two systems did not differ. This supports the argument that the difference in session 2 was the result of the subjects’ prior experience with the DMI, not of hardware differences.

5. Discussion The results of the study support the argument that both prior experience and design of the interface affect users’ attitudes toward the use of computer systems. Nevertheless, the effect of these factors was somewhat mixed. Although generally in keeping with expectations based on the model of Davis and his colleagues, there were some interesting deviations from what was predicted by TAM. Also, some results conformed to the model but were surprisingly weak in practical terms. The results are discussed and interpreted below, first for first-time use and then for subsequent use. The question may be asked in both first-time and subsequent use whether PEU and PU are merely a reflection of performance. First, it is not clear from a theoretical viewpoint what the relationship is between perceptions and performance, i.e. one-way or

USERS’ PERCEPTIONS OF SOFTWARE PACKAGES

581

two-way. Also, in many instances perceptions do not match performance. In this work, our intention was to study perceptions, not performance, so we cannot address the directionality of influence between them. However, we feel that it is very unlikely that performance affected perception to an appreciable extent. Our subjects were administered the PEU and PU immediately after completing the training manuals, without the chance for any performance on their own. The tutorial training manuals led them through commands and operations step by step. We did not observe that any group of subjects had special difficulty following the steps in their manual. This was confirmed by a later protocol analysis which we did using the same training materials and procedure. The manuals told subjects what to do if something went wrong, and in rare cases in which subjects still experienced difficulties the experimenter pointed out solutions in the training manual. Subjects did not really ‘‘perform’’, so much as to follow a set of steps, and virtually all subjects were able to complete the steps of training satisfactorily, using the manual as a guide. Furthermore, subjects received no feedback from the experimenters or other subjects about their work on the training tasks. Based on these conditions, the possible effects of performance on PEU and PU were minimized.

5.1. FIRST-TIME USE OF A WORD-PROCESSING SYSTEM

In first-time use, the interface type had a significant effect on subjects’ perceived ease of use of the software. The perceived ease of use, in turn, had a significant effect on the perceived usefulness. This result is predicted by TAM. However, there was no direct influence of interface type on perceived usefulness. From TAM we might expect that external variables, such as interface type, would have a direct effect on PU, as well as an indirect effect through PEU. Why did we fail to find a direct effect of interface on PU? It may be that the effect of external variables on PU is fully mediated by PEU. This seems to make some sense as a hypothesis in the case of interface type. The interface type is a surface feature of the software. It is likely to affect the ease with which users learn the software. On the other hand, users may form their perceptions of usefulness based mostly on the functionality. Even a poor interface may not have much direct influence on perceptions about the functionality of the software, unless it is so extremely poor that it actually obscures the functionality to the user. Nevertheless, the interface type might modify the perception of usefulness indirectly, especially if the interface was thought to be very difficult or very easy to use. Then the ease of use of the interface might be seen as enhancing or detracting from the ability to do useful work with the tool, no matter what the tool’s basic functionality. Table 1 shows that in first-time use the DMI interface was considered the easiest to use of the three interfaces tested. The menu-based interface was intermediate, and the command-based interface was considered hardest to use, although the menu interface did not differ from either of the others statistically. The statistically significant difference between the DMI and command-based systems in perceived ease of use supports the argument made by TAM that interfaces permitting a more direct style of interaction (e.g. using icons, direct actions on objects and a pointing device) enhance ease of use. In general, we may extend these arguments to conclude that systems requiring a greater amount of cognitive effort to understand and use will be perceived as more difficult to

582

S. WIEDENBECK AND S. DAVIS

use. TAM, as discussed previously, predicts that a negative attitude about ease of use will also have a negative effect on perceived usefulness. While the ordering of the three interfaces in perceived ease of use is in accord with TAM, it should be noted that the amount of variance in PEU accounted for by interface type was very low, just 5%. Thus, although significant statistically, the interface type cannot be considered of great practical significance in the formation of attitudes about ease of use for users who are complete novices. Other unknown variables account for most of the variance. We can postulate what some of these variables might be: training method, prior experience of writing and editing in a non-computer environment, motivation and any number of other individual difference variables. The user of word processing, who has very little or no previous experience, may find the situation extremely different from his or her normal way of editing, leaving aside the question of the interface. The difficulties may include using unfamiliar keys, managing new input devices such as a mouse or trackball, understanding unfamiliar terminology, coordinating attention between a printed document and the screen and navigating in an on-line document (Carroll & Mack, 1984). Among all these early difficulties, the difference in interface type may simply be rather subtle. We might expect to find a greater effect of interface type on PEU in a setting where subjects had a bit more experience and had mastered more of the initial, short-term problems of moving to a computerized environment for editing. In the case of PU, the amount of variance accounted for by variables in the model was 10%. This is also rather low and is accounted for almost entirely by the influence of PEU on PU. Again, in first-time use, variables outside our model make up most of the variance. From this we can conclude that the interface type is not very important in first-time users’ judgments about usefulness of software. Other factors which may be more important are the users’ perceptions of the functionality of the software itself, whether it will improve their work, and whether the improvement in their work to be gained will result in a substantial benefit to them. Learners are usually affected by a production bias (Carroll & Rosson, 1987), and the perceived usefulness may also be influenced by their judgments about whether becoming proficient in the system will be worth the short-term loss of productivity while learning to use it. Given these other considerations, it is not surprising that in our experiment a large part of the variance of PU was unaccounted for.

5.2. SUBSEQUENT USE OF A DIFFERENT WORD-PROCESSING SYSTEM

Our study of subsequent use compared subjects who learnt the menu- and commandbased system in their first session to those who learnt the same interfaces in their second session, after having had experience with the DMI system in their first session. Interaction style and session are both external variables which were expected to affect PEU and PU. Considering the menu and command subjects’ results together, we found that groups which used the menu- or command-based system in their first session, had higher perceptions of ease of use than those who used them in their second session. This result is interesting and somewhat surprising. It would seem reasonable that if subjects learnt some basic word-processing concepts and techniques in the first session, then learning another software system in the same domain would make that system easier to use. Based

USERS’ PERCEPTIONS OF SOFTWARE PACKAGES

583

on the assertions of TAM (Davis et al., 1989), this should also affect perceived usefulness positively. To understand this unexpected result better, we investigated the interaction of session and interface, analysing the subjects who used the menu-based software in their second session separately from those who used the command-based software in their second session. Here we found that for the subjects moving to the menu-based word processor session did not have a significant effect on PEU. However, for the subjects moving to the command-based software there was a strong negative effect of session on PEU. This difference indicates that there was indeed an interaction between interface and session. The very negative attitudes toward ease of use of the command-based software in session 2 indicate that the subjects’ perceptions were negatively impacted by the prior experience they had with the DMI software. Given their frame of reference, these subjects judged the command-based software more harshly than did their counterparts who used it in the first session. This result argues in favor of the importance of surface and operating similarities between systems and of the context of system use. In our study, subjects may have already in their first session developed a mental model of a word processor in which all they had to do for most operations was point, click and drag. This directness matched more closely with how they may have thought of performing such operations in a non-computer environment. Subjects who subsequently used the menu-based software may have perceived it as being relatively similar to the DMI software. It appears that the subjects were able to use more of the knowledge they gained in the earlier experience with the DMI to understand how to use the menu system. They did not experience sharp discontinuities in going from one to the other, and reached the same conclusion about its ease of use as subjects who used it in the first session. On the other hand, when subjects encountered the command-based system, they found the directness which they had experienced in the DMI replaced by typing cryptic command codes. This model of word processing apparently did not fit the more direct mode of operating that subjects had already internalized. Therefore, they perceived this system as relatively less useful and more difficult to use. First-time users of the command-based software, lacking the context of direct manipulation and having no a priori internal models of how computer word processing should work, found the command-based systems relatively easy to use by simply following the instructions in the training manuals. These results lead us to make a distinction between intrinsic usability and similarity. First-time users, with no significant previous context of using other word processors, made their judgments of usability based largely on their perceptions of intrinsic usability. Subsequent users were no longer basing their perceptions on intrinsic usability alone but on surface and operating similarities to the previous word processor. These findings add to our understanding of TAM by suggesting that relevant prior computer experience is a very important factor in determining users’ perceptions of the ease of use and usefulness of computer software, and, ultimately, their willingness to use it. However, the relationship is not a straightforward one in which any prior experience has a positive impact on attitudes. Thus, managers and trainers should take into account trainees’ prior relevant experience in selecting software, establishing guidelines for its use in organizations and designing training programs. One way to do this

584

S. WIEDENBECK AND S. DAVIS

would be to point out in training similarities between new and old packages and to emphasize the features of the new package that make it easier to use or potentially more useful. In our subsequent use design, the interface had a stronger effect on PEU than it did in the previous first-time use, accounting for about 15% of the variance in PEU. As argued previously, differences in the interaction style may be more important when the user has some experience than when the user is a complete novice. Having gained some familiarity and comfort with the various dimensions of the task in a computerized environment, the user becomes more sensitive to the design features of the interface. We would expect this to happen if the individual continued to use and gain mastery of the same interface over some medium length time frame. However, over a much longer time ease of use may again become less important, as argued by Davis et al. (1989). As in the first-time use design, the direct effect of interface on PU was not significant. Again, this evidence suggests that not all external variables have a direct effect on attitudes toward usefulness. As argued above, usefulness perceptions may be grounded in the functionality of the system and then modified by the perception of how easy it will be to achieve one’s goals with the software. This interpretation is consistent with our data in which there is a strong relationship between PEU and PU, which leads to an indirect influence of interface on PU.

6. Conclusions The results of this study lead to several conclusions. The first is that interface type does affect perceptions of ease of use in first-time use of computers. A direct manipulation interface appears to lead to more positive perceptions of ease of use than does a command-based interface. High perceptions of ease of use should, in turn, play a positive role in the adoption of computer technology, as suggested by several authors (Davis et al., 1989; Adams et al., 1992; Davis, 1992). Some caution, however, is necessary in making this conclusion. The positive effect of interface on perceived ease of use, though significant, was small. This indicates that other unknown factors play a more important role in the formation of initial perceptions than does the interaction style. Further research should try to isolate those factors. For novices and near-novices of word-processing software, previous computing experience does not play a large role. However, we might hypothesize that relevant factors would include individual factors, such as intrinsic motivation, learning style or visual ability (Davis & Bostrom, 1992, 1994). Further research also needs to determine whether this phenomenon holds true for applications other than word processing or for task domains of varying complexity. Future research should also address whether there are different effects for PEU and PU for novices and experts, what happens to PEU and PU over time and repeated use, and whether changes in PEU and PU are correlated with the learning curve of the software. Another challenging open problem is to synthesize the cognitive complexity model of Kieras and Polson (1985) (see also Bovair et al., 1990), which predicts objective performance, with the perception model of Davis et al. (1989). It is an interesting question whether other external variables would behave in the same way or a different way in first-time use. Davis et al. (1989) mention training method as an external variable that may be important in TAM. Here we might speculate that the effect would be different. A poor training method might arguably have a direct effect on both

USERS’ PERCEPTIONS OF SOFTWARE PACKAGES

585

ease of use and usefulness. It could affect perceptions of ease of use if it failed to provide the instruction and practice necessary for the user to feel efficacious in accomplishing basic operations using the software. Such a lack of confidence could then act to decrease the user’s judgment about the ultimate usefulness of the software. However, the training method could affect perceptions of usefulness directly if the training did a poor job of conveying the software’s capabilities to the learner. Then, independent of how easy the software was to use, the user might judge it to be of limited usefulness. From this argument, we might propose that different external variables have different effects, some affecting PU only through PEU and others being capable of influencing PU directly as well. More research on a variety of external variables is necessary in order to understand better how they influence user perceptions. A second conclusion is that, for situations in which users learn functionally similar applications in a sequential manner, those applications that represent a change to a much less direct style of interaction will be perceived by users as more difficult to use and, thus, will reduce the chances of adoption and use. This is based on our results showing much poorer perceptions of the command-based software in the subsequent use group than in the first-time use group. These results argue in favor of maintaining consistency with respect to directness within this learning context. It is not apparent, however, that this consistency argument would hold for situations in which applications are learnt and used simultaneously. In these cases it may make more sense for the interfaces to have a somewhat different look and feel to reduce the potential for confusion and negative transfer. In practical terms, our findings suggest that when users move from one interface style to another, managers need to be very aware of potential problems that may occur because of their history of past use. Users may resist learning to use the new software, minimize their use or fail to learn efficient methods. Formal training and user support need to be designed to overcome negative perceptions and encourage full use of the new software. It should also be kept in mind that some modern packages provide multiple methods for carrying out operations and thus may incorporate direct manipulation, menus and commands in one package. This raises several questions: whether it is more confusing than helpful to provide multiple ways of accomplishing the same task, if and how users migrate across modes and how their perceptions of the system are influenced by the coexistence of multiple styles of interaction. A third conclusion is that we now have evidence suggesting that different external variables act on user perceptions in different ways. In the first-time use analysis, interface had an effect on PEU, but not on PU. In the subsequent use analysis, session had an effect on both PEU and PU, while interface had an effect on PEU but did not influence PU. Thus, it appears that external variables may act on one or both perceptions. This suggests a need for further study of a variety of external variables to determine exactly how they affect perceptions. Such study would give us a better understanding of the characteristics of the variables affecting perceptions of ease of use and usefulness.

References ADAMS, D. A., NELSON, R. R. & TODD, P. A. (1992). Perceived usefulness, ease of use, and usage of information technology: a replication. MIS Quarterly, 15, 227—247.

586

S. WIEDENBECK AND S. DAVIS

AJZEN, I. & FISHBEIN, M. (1980). Understanding Attitudes and Predicting Social Behavior. Englewood Cliffs, NJ: Prentice-Hall. BANDURA, A. (1986). Social Foundations of Thought and Action: A Social Cognitive ¹heory. Englewood Cliffs, NJ: Prentice-Hall. BENBASAT, I. & TODD, P. (1993). An experimental investigation of interface design alternatives; icon vs. text and direct manipulation vs. menus. International Journal of Man—Machine Studies, 38, 369—402. BEWLEY, W. L., ROBERTS, T. L., SCHROIT, D. & VERPLANK, W. L. (1983). Human factors testing in the design of Xerox’s 8010 ‘‘Star’’ Office Workstations. CHI’83 Proceedings, pp. 72—77. New York: ACM. BOVAIR, S., KIERAS, D. E. & POLSON, P. G. (1990). The acquisition and performance of text-editing skill: a cognitive complexity analysis. Human—Computer Interaction, 5, 1—48. CARROLL, J. M. (1990). The Nurnberg Funnel: Designing Minimalist Instruction for Practical Computer Skill. Cambridge, MA: MIT Press. CARROLL, J. M. & MACK, R. L. (1984). Learning to use a word processor: by doing, by thinking, and by knowing. In J. C. THOMAS & M. L. SCHNEIDER, Eds. Human Factors in Computer Systems, pp. 13—51. Norwood, NJ: Ablex. CARROLL, J. M. & ROSSON, M. B. (1987). Paradox of the active user. In J. M. CARROLL, Ed. Interfacing Thought. Cambridge, MA: MIT. CHIN, W. W. & TODD, P. A. (1995). On the use, usefulness, and ease of use of structural equation modeling in MIS research: a note of caution. MIS Quarterly, 19, 237—246. COMPEAU, D. R. & HIGGINS, C. A. (1991). A social cognitive theory perspective on individual reactions to computing technology. Proceedings of the International Conference on Information Systems: ICIS91, pp. 187—198. New York: ACM. COMPEAU, D. R. & HIGGINS, C. A. (1995). Computer self-efficacy: development of a measure and initial test. MIS Quarterly, 19, 189—211. DAVIS, F. D. (1989). Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Quarterly, 13, 319—340. DAVIS, F. D. (1993). User acceptance of information technology: system characteristics, user perceptions and behavioral impacts. International Journal of Human—Computer Studies, 38, 475—487. DAVIS, S. A. & BOSTROM, R. P. (1992). An experimental investigation of the roles of the computer interface and individual characteristics in the learning of computer systems. International Journal of Human—Computer Interaction, 4, 143—172. DAVIS, S. A. & BOSTROM, R. P. (1993). Training end users: an experimental investigation of the roles of the computer interface and training methods. MIS Quarterly, 17, 61—85. DAVIS, S. A. & BOSTROM, R. P. (1994). The effects of an intrinsically motivating instructional environment on software learning and acceptance. Information Systems Journal, 4, 3—26. DAVIS, F. D., BAGOZZI, R. P. & WARSHAW, P. R. (1989). User acceptance of computer technology: a comparison of two theoretical models. Management Science, 35, 982—1003. FISHBEIN, M. & AJZEN, I. (1975). Belief, Attitude, Intention and Behavior: An Introduction to Theory and Research. Reading, MA: Addison-Wesley. GIST, M. E., SCHOWOERER, C. & ROSEN, B. (1989). Effects of alternative training methods on self-efficacy and performance in computer software training. Journal of Applied Psychology, 74, 884—891. HEINSSEN, R. K., GLASS, C. R. & KNIGHT, L. A. (1987). Assessing computer anxiety: development and validation of the computer anxiety rating scale. Computers in Human Behavior, 3, 49—59. HENDRICKSON, A. R., MASSEY, P. D. & CRONAN, T. P. (1993). On the test-retest reliability of perceived usefulness and perceived ease of use scales. MIS Quarterly, 17, 227—230. HILL, T., SMITH, N. D. & MANN, M. F. (1987). Role of efficacy expectations in predicting the decision to use advanced technologies: the case of computers. Journal of Applied Psychology, 72, 307—313. HUTCHINS, E. L., HOLLAN, J. D. & NORMAN, D. A. (1985). Direct manipulation interfaces. Human—Computer Interaction, 1, 311—338.

587

USERS’ PERCEPTIONS OF SOFTWARE PACKAGES

KERLINGER, F. N. & PEHAZUR, E. J. (1973). Multiple Regression in Behavioral Research. New York: Holt, Reinhart, and Winston. KIERAS, D. E. & POLSON, P. G. (1985). An approach to the formal analysis of user complexity. International Journal of Man—Machine Studies, 22, 365—394. RETHERFORD, R. D. & CHOE, M. K. (1993). Statistical Models for Causal Analysis. New York: Wiley. SANTHANAM, R. & WIEDENBECK, S. (1993). Neither novice nor expert: the discretionary user of software. International Journal of Man—Machine Studies, 38, 201—229. SEGARS, A. H. & GROVER, V. (1993). Re-examining perceived ease of use and usefulness: a confirmatory factor analysis. MIS Quarterly, 17, 517—525. SHNEIDERMAN, B. (1982). The future of interactive systems and the emergence of direct manipulation. Behaviour and Information Technology, 1, 237—256 SHNEIDERMAN, B. (1992). Designing the User Interface: Strategies for Effective Human—Computer Interaction (2nd edn.). Reading, MA: Addison-Wesley. SINGLEY, M. K. & ANDERSON, J. R. (1989). The Transfer of Cognitive Skill. Cambridge, MA: Harvard University Press. THOMAS, J. C. & KELLOGG, W. A. (1988). Minimizing ecological gaps in interface design. IEEE Software, 22, 78—86. VENKATESH, V. & DAVIS, F. D. (1994). Modeling the determinants of perceived ease of use. ICIS’94 Proceedings, pp. 213—228. WHITESIDE, J., JONES, S., LEVY, P. S. & WIXON, D. (1985). User performance with command, menu, and iconic interfaces. CHI’85 Conference Proceedings, pp. 185—191. New York: ACM. Paper accepted for publication by Associate Editor, Dr. D. Boehm-Davis.

Appendix 1: PEU and PU questionnaires used in the study (A) Perceived ease of use

(1) Learning to operate the word processor would be easy for me (2) I would find it easy to get the word processor to do what I want it to do (3) It would be easy for me to become skillful at using the word processor (4) I would find the word processor easy to use (B) Perceived usefulness (5) Using the word processor would improve my performance in my coursework (6) Using the word processor in my coursework would increase my productivity

Strongly agree 1 2

Neutral 3

4

Strongly disagree 5 6 7

1

2

3

4

5

6

7

1

2

3

4

5

6

7

1

2

3

4

5

6

7

1

2

3

4

5

6

7

1

2

3

4

5

6

7

588

S. WIEDENBECK AND S. DAVIS

Strongly agree 1 2

(7) Using the word processor would enhance my effectiveness in my coursework (8) I would find the word processor useful in my coursework

1

2

Neutral 3

4

Strongly disagree 5 6 7

3

4

5

6

7

Appendix 2: Comparison of actual and recreated correlation coefficients (A) First-time use. The actual Pearson’s correlation coefficient between interface and PU (i.e. the direct path) was actual r(interface, PU)"0.1187. The recreated correlation coefficient was the product of the path coefficients from interface to PEU and from PEU to PU (i.e. the indirect path): recreated r(interface, PU)"(0.2355) (0.3038)"0.0715. The difference between the actual and recreated correlation coefficients was diff(actual r,

"0.1187!0.0715"0.0472.

recreated r)

(B) Subsequent use. The actual Pearson’s correlation coefficient between interface and PU (i.e. the direct path) was actual r(interface, PU)"0.2158. The recreated correlation coefficient was the product of the path coefficients from interface to PEU and from PEU to PU (i.e. the indirect path): recreated r(interface, PU)"(0.3828) (0.5242)"0.2007. The difference between the actual and recreated correlation coefficients was diff(actual r,

.

"0.2158!0.2007"0.0151.

recreated r)