- Email: [email protected]
Experimental Studies of Man-Computer Interaction in Financial Accounting Systems
Cop'Tight © IF AC Analvsis. Design an d Evalu at ion of \lan - \1 ac hine Systcms Bade n · Baden . Fede ral Republi c "f Gcrmam' 19R2
EXPERIMENT AL STUDIES OF MAN-COMPUTER INTERACTION IN FINANCIAL ACCOUNTING SYSTEMS B. Luke L ehrstuhl fur Betriebslnformatz'k , Universz'tiit Dortmund, D-4600 Dortmund 50 , Federal Republz'c of Germany
Abstract. Business application systems with online operations o bviously ne ed good man-comput e r interfaces. Experimental studies seem to be a useful approach to face the problem of des igning user-adequa te interf aces. \-li th special regard to the design of dialogues for online accounting, a subset o f an application system was redesigned to give several different featur e s of dialogue design, while all other parameters like hardware a nd applicatio n-bound functions remained identical. The differences in the constructed di a logues can be divided into three categories, which are (A) the presentation of inf o rmation by the system, (B) the handling of input and user erro rs and (C) the mea ns of dialogue operations including control functions ano online documentation. 48 professional accountants carried out a series of tasks, involving the handling of accounts receivable entries, with each of the three systems. All interactions were recorded. Results ar e given for the improvements achieved by the modifications in dial ogue design. The first part co ncerns the behaviour of the subjects with respect t o their different on line experiences. In the second part the general effects of the special design features are shown. Finally the user's own judgements bas e d on the evaluation of a post-test questionnaire are discussed. Keywords. online operation; man-machine systems; interactive systems; o nline accounting; software ergonomics; design of dialogues; human engineering; experimental evaluations.
DESIGN OF DIALCCUES
SOFTWARE INTERFACE FOR BUSINESS APPLICATIONS
Various recommendations have been published by several authors for the design o f dialogues in commercial applications, e.g. Mar tin (1973), Engel and Granda (1975), Stewart (1976), Hebditch (1979), Gaines (1981). The ir recommendations or guidelines for impl e mentation are mainly based upon the designer's experiences and on the more informal feedback from the users. Empirical or experimental validations are given only in some cases (e.g. Hirsch, 1981).
Nowadays more and more people in the business environment are directly faced with computers, working with a display terminal for data-entry, inquiries and/or other kinds of tasks. Video-display-units are typical equipment for c o mmercial applications such as order entry or accounting. They are utilised in conjunction with suitable software packages. The amount of work as a whole is divided up into user's tasks and computer's tasks with a high rate of interaction between user and computer. In this process of dialogue operation the n e ed for a good man-computer interface is obvious. The state of the art in computer technology allows the machines to be adapted to the user in an ergonomically acceptable manner. In the design of workstations, and especially of terminal hardware, much progress has been made during the last years, where definitive guidelines and checklists have been developed (Cakir, Hart and Stewart, 1979). Growing attention is now given to software aspects of the interface. Shackel (1980) uses the term "software ergonomics" for this type of research, while the more psychologically oriented approach is referred to as "cognitive ergonomics".
It has been shown that standard online accounting systems for smaller business computers have quite similar structures and power of functions (Griese, 1980). However, in spite of this similarity, it can be recognized that, with respect to the design of dialogues, there is a wide variety of different products. This may be due to a lack of the des igner' s general knowledge and of thorough and covering guidelines (besides the constraints caused by different hardware and operating systems). It seems to be useful to give some contributions to the augmentation of knowledge about user behaviour in such kinds of interactive systems. Thus, in this study, emphasis is placed on the evaluation of special features in the design of dialogues.
287
B. Liike
288
rESEARCH METHODS Classical forms of ergonomic research are empirical an0 experimental studies. For mancomputer interaction in commercial applications the E'mpirical results of Eason, Damodaran and StE'wart (1974) show the variety and problems wYich arise from the differences in user job types. In other fields of interactive computer usage valuable experimental results have been achieved, e.g. by Ledgard, Singer and Whiteside (1981) who studied the syntax of text editors. They started with the hypothesis that a command language should be as natural and familiar to the user as possible. For the experimental evaluation two editors wel-e used: one already existing editor and another one with a structure and syntax of comnands based on legitimate English phrases avoiding unfamiliar words and the use of special delimiting characters. The results, based upon 24 sessions with student subjects, show that the performance using the second editor was much superior on the basis of three measures (percentage of task completed, erroneous commands and editing efficiency). Many other experiments with various kinds of user tasks have been carried out and are assessed in the current literature, e.g. by Embley and Nagy (1981), Shneiderman (1980) and Reisner (1981).
'rHE EXPERIMENT The general idea of our research project was to let users carry out comparable tasks in different online accounting systems, where the hardware and the specific applicationbound functions remained identical and only the design of dialogues was changed between the systems. In this situation a laboratory experiment seemed to be the most promising approach. Three accounting systems on a small business computer with disk storage and two VDU-workstations were investigated: a supplier-made open item method accounting package and two other systems constructed under the direction of the author. For the sake of a Manageable experimental environment the development efforts were restricted to tasks connected with accounts receivable entries. AMong all frequent operations in accounting these seem to be best suited for our purposes, because they provide reasonable handling problems for the user, and are neither too complex nor too specialized. It was expected that every accountant should be able to understand the contents and procedures of sllch operations. Design Goals Tasks like those operations in accounting can be regarded as rather 'closed tasks' following the classification of Eason (1980). This means, that. input and required output have a rather fix€d structure and vary only within predictabl€ limits. In a simplifying example with accounts receipts we always have an entry of the amount of money for two accounts,
one of these being a cash or bank account and the other being a customer's or eventually a supplier's account, where the existing open items may fit to the payed amount or not. The knowledge about the structure and contents of the task makes it possible to create dialogues with prescribed performance procedures, at least as far as the limited variations of tasks do not require free choices or selection of alternatives by the user him self. These restricted assumptions are help ful by making the area of design problems smaller, but not even the general goals for user-adequate design are completely determined by them alone. For our design purpose we used a set of design goals with some similari ty to those developed by Dzida, Herda and Itzfeld (1978) and Dehning, Essig and Maass ( 1978). Having in mind that the dialogue should fit to the frequent and accustomed user as well as to the less frequent and untrained user we stated three categories of design goals: the dialogue should be (A) adequate for required functions, (B) capable of explanation and (C) secure in utilization. Adeguate for reguired functions means that the form of the dialogues should be suited, as far as possible, to the contents and structure of the tasks. Capable of explanations. Any instruction and information within the dialogue should be understandable and relate to the user's task. The user should see what the system is doing as a result of his operations, it should be clear ·....hat the system is expecting from him, and, finally, the user should be able to recognize the whole set of his possible alternatives. Secure in utilization. During operation mainly on data input - the user should have the lowest possible feeling of uncertainty about the correctness of his actions. This means especially the user-perceived security against unexpected and unwanted reactions by the system. Features of the Three Systems Oriented to those design goals the two new accounting systems were developed. A selection was taken from among those existing techniques and additional features, which in our belief would lead to some improvements. The implementation was carried out with different sets of those selected techniques and features in the new systems. Some constraints on the designer's freedom were set by the known limits in hardware and operating system of the machine to be used. However the major common factor of the three dialogues results from the decision to use the screen in a form-filling mode with both the new versions too. This decision allowed for the fact that all comparable online accounting systems we know use this technique as interaction mode. Taking into account this global common feature, the differences in the constructed dialogues can be classified in three catego-
Man-Computer Intera c tion
~n
ries. These teing (A) the presentation of in formation by the system, (B) the handling of inputs and user errors (C) the means of dialogue operation. Presentation of information. This comprises everything which is displayed on the screen to the user. It includes the type of inputform used, the spacing and grouping of the input fields, and the feedback that is given to the user as information about the current state of the dialogue. In addition, it includes the frequency in changing of complete screen-images and the location of explanatory text. The kind of presentation determines essentially the visible surface of the dialogue. Handling of inputs and user errors. Within each system there is a unique manner of designing all the fields that are used for data input by the user. The differences between the systems are the kinds of preformatting or indicating the length of the input fields, and the availability and visibi 1 i ty of default values. Handling of user errors is done by error messages in a fixed partition at the botto m of the screen, where the reason of the error is indicated and a repetition of the input is required. Means of dialogue operation. All features of the control functions available to the user are categorized here, i.e. backtracking and reset facilities as well as the type and completeness o f online assistance on the syntactical and semantical level. Syntactical means f or control actions are implemented as commands, furlct ion keys or menus. Each of the sys tems can be described as consisting of a join of e lements fram all the three categories. Figures 1 to 3 show the treatment of an identical set of data and dial o gue state by each of the three systems, which are re fe rred to as S1, S2 and S3. A short verbal characterization is given in the following. Features of system S1 (Fig. 1). The dialogue enables the user to carry out the whole task without further written material such as application manuals and tables for coding of input. Special aids for input operations such as visible defaults or abbreviations are not provided. Consecutive input fields are organized column by column instead of line by line. Compared to the other systems we have here the highest flexibility because operations can be controlled by a small set of commands. The dialogue is split into several frames of up to 5 different screen-images for one task. The available special options (e.g. scrolling or other control functions) are always visible at the top of the screen. All further material is put together in a permanently available subsystem for on-line assistance. This help-system can be reached by issuing one of the offered specific commands. A syntactical and semantical explanation of the current input field is then dis-
Financial Accountin g Systems
289
played by default. Guided by a set of menus within the help-system other facilities can be activated, i.e. further explanations, tables of coded items (e.g. account numbers) and finally the end of t he subsystem to continue in the interrupted main dialogue. Features of system S2 (Fig. 2). The main features in dialogue design of this system are the operator instruction facilities and the assistance for input operations. The operator instru ct ions are located at the bottom of the screen and show the pos s ible user actions by explaining the meaning of the function-keys. According to the current input field, up to 7 different actions are offered by short keywords. Visible default values in the input fi el ds allow the r e -use of data items for the current or following tasks. An additional featur e is the use of line numbers instead of the complete coded values in those cases where co des are not known and have to be l ooked up in internal dictionaries (e.g. the code of a c ustome r of wham only the name is known fram the source do cument ). The line number with its one digit causes the entire code to be moved into the input field. The control within the dial ogue is generally handled by the use o f function-keys; accordingly the meaning of single keys may change from one input field to another. However fixed key numbers are used for backtracking and reset functions, which are available with every single step. Features of system S3 (Fig. 3). In this system the dialogue provided by the supplier of the standard package has n ot been changed. Within the dialogue the explanatory texts on the screen are essentially restricted to the descriptions of the input fields. Sometimes a single descr iption is followed by !lOre than one input field. Because all the input fields are packed rather closely together and the explanatory text is quite short, most of the tasks can be carried out without a complete change of the screen-image. For the necessary control functions some o f the function-keys are used, but the facilities differ fron field to field and are not available everywhere. In some situations a short text is is sue d as a hint for the possible control actions. Two of the dictionaries to be used for finding codes of input items consist of printed lists and are not accessible within the dialogue. In addition to the operation manual a printed list of those fields is provided, where backtracking is possible. Subjects For experimental testing 48 professional accountants were available for participation. The majority of them was delegated by their department managers. Only rough information about the desired procedure was given prior to the experimental session itself. The subjects were classified according to the kind of computer-assistance for the accounting system of their own affiliation.
290
B. Luke
1 B U C HEN Z A H L U N G S E I N G A N G E 0 P - A U S G L E I CHI 1--------------------------------------------------------------------------------1 1 FELD ZURUCK : Fl ENDE : F2 INFORMATION: *H VORBLATT~RN: + ZURUCKBL. : 1 1--------------------------------------------------------------------------------1 1 16 NEU-OP/SKONTO (N/S)7 1 17 REF ERENZ-NUMMER . 1 18 AUSGLEICHSBETRAG . 1 19 ANDERUNG OK (J/N)7
_ _______ ___________ _
2736.50 ZAHLBETRAG 54.82 SKONTOBETRAG 2018.29 RESTBETRAG
1 1
1
1
1---------------_·_---------------------------------------------------------------1 1 1
REF-NR
RE-DATUM
FALLIG
ZU ZAHLEN
SKONTO
NOCH OFFEN
1
17420 17438 17512 17580 17700 17782 17783 17922
78.07.05 78.07.07 78.07.09 78.07.11 78.07.13 78.07.15 78.07.17 78.07.19
78.10.05 78.10.07 78.10.09 78.10.11 78.08.03 78.10.15 78.10.17 78.10.19
732.46 2777.77 9.19 299.66 1402.11 0.00 2061.43 902.66
12.98 138.89 0.46 0.00 0.00 0.00 103.07 45.13
AUSGEGLICHEN
1 1
I 1 1 1
I
1 I
1 1 Fig. 1
Treatment of open items in system 8 1
BUCHUNG VON ZAHLUNGSEINGAENGEN FUER
1
BLECHALBRECHT GMBH
11 ============================================================================== UNTERNEHMEN/~ERK 101 SACHKONTONUMMER 1130000 BELEGNUMMER 19438 1 DEBITORENNUMMER
1432600
BERHERMANN GMBH
ZAHLBETRAG
2736.50
1------------------------------------------------------------------------------1 LISTE DER OFFENEN POSTEN 1
1 1
1 1 1 1 1
1
INDEX 1 2 3 4 5 6 7 8
REFNR 17420 17438 17512 17580 17700 17782 18113 18422
RE-DATUM 78.07.05 78.07.07 78.07.09 78.07.11 78.07.13 78.07.15 78.08.12 78.08.14
OP-BETRAG 732.46 2777.77 9.19
SKTOBETRAG 12.98 138.89 0.46 0.00 0.00 2.00 103.07 45.13
299.~6
1402.11 100.00 2131.43 902.66
ZAHLBETRAG
1------------------------------------------------------------------------------1 INDEX I OP-NR +1 + RESTBETRAG 2736.50 1=============================================================================== 1 F3 = VORBLAETTERN F4 = RUECKBLAETTERN F5 = OP-NEUANLAGE 1 FA = ABB BUCHUNG FB = ABB OP F2 = ZURUECK BUSCHLUE ETB = INDEXAUSWAHL 1=============================================================================== Fi g . 2 IUNT/~ERK
101
Trea tment of op e n items in system 82
BUCHERZEICHEN BI
BUCHUNGSMONAT 7803
1
1BELEGDIHUM
ISTORNO-KZ IRESTSUMME IZAHLBETRAG 1STEUER 1
78.08.30 0 4189.20 2736.50
BELEGNR 35267 SACHKONTONR. 1000000 KONTOBER/-NR. 1 1432600 SKTO/STEUER 0.00 STEUER2
I~ECHSELDATUM
IREST-AUFTBET
1 1 1 1 1 1 1 1 1 1 1 1
2736.50
REF-NR. 17420 17438 17512 17580 17700 17782 18113 18422 18631
RE-DATUM 78.07.05 78.07.07 78.07.09 78.07.11 78.07.13 78.07.15 78.08.02 78.08.04 78.08.09
OP-SALDO 732.46 2777.77 9.19 299.66 1402.11 100.00 2061.43 902.66 1087.40
17420
78.07.05
732.46
Fig. 3
BELEGSUMME
4189.20
SCHLUSSEL GEGENWERT TEXT ZE 1
31 2
SKONTO ZAHLBETR. REST-OP 12.98 (3.00%) 83.33 (3.00%) 0.28 (3.00%) 0.00 0.00 2.95 (3.00%) 61.84 (3.00%) 27.08 (3.00%) 32.62 (3.00%) 12.98
719.48
Treatment of open items in system 83
1
1 1 1
0.00
Han-Computer Interaction ~n Financial Accounting Systems
Group 1, termed the 'manual'-gr oup, consisted of 6 subjects, who do all accounting without computer-assistance at all. Group 2, terned the ' batch' -gr o up, consisted of 18 subjects, who use batch systems by filling coding- sheets as input and receiving printed lists as output. Group 3, terned the ' on line' -gr oup , consisted of 18 subjects , ''''ho use online accounting systems by themselves (generally several hours per day). These subjects had no prior knowledge about any of the three systems used in the experiment. Group 4, tern 'ed the 'S3'-gr oup , consisted of 6 subjects , Hho use the standard package referred here ~s S3 for daily work. Procedure Each session started with an interactive tutorial at th~ workstation. This tutorial consisted of im;tructions and practices about the use of tl' e keyboard, especially the locations and meanings of the function-keys . It included at its end a multiple-choice questionnaire about the characteristics of the subjects ' cl"rical work. After that, a description of the first system to be used was given to the suject . It was about 7 page~; long and contained a summarized presentation of the structure of the implemented nialogue. After the reading of the matecial was cnmpleted the system was started and the set of prepared tasks was given to the subject. This set was composed of 26 tasks with a fixed order . Each task consisted of the " original " source documents of a single accounts receivable entry as received from the bank or the cashier's office. I\dditional written remarks were given if necessary for the understanding of the documents. 1\ member of the experimental staff was present during the whole session, giving advice for the first two tasks but restricting his further communication with the subject to exceptional cases. When the set of tasks was finished it was offered to the subject to have a break without prescribed time l imits . The second and the third system were treated in the same manner. The order of sequence between the three systems was randomly determined so that the six possible sequences were balanced within in each group of subjects. After a subject had completed all tasks in all systems , individual judgements about the different features in the dialogues were collected via a post - test questionnaire. The end of the session was normally reached in the early afterr.oon with a time taken for the whole session ranging from 4 up to 9 hours (breaks incluned).
29 1
Measurement of Interactions All user input was recorded by a monitor program especially implemented for that purpose. For every interaction (i.e. one dialogue step represented by the input o f the user into one field) a record was created containing the following data: field identificatio n, the input itself, the end-key for completion of data entry , the user time and a code for indication of errors or brand"les in connection with that step. I\ccumulation to the overall and the task level resulted in a set of three variables for each of the systems. These being (A) the performance time, (B) the number of errors, (C) the overhead on steps. Performance time. This is the time the system has to wait for the end of the user's input action. It covers the time fron the moment the cursor is moved to the input field by the system until the user presses an end -key for transmission of the field. Measurement is in units of 0,15 seconds. The performance time for a single step may be very short and even zero, because the workstation allows some characters to be typed in advance. Otherwise performance time can exceed one minute or more, if the user has to l ook at manuals or other additional material (or is just thinking about what to do next ). Number of errors. By the monitor program only those errors were logged, wh ich could be found by a syntactical and semantical check of the single input items. Other errors (e.g. the use of a wrong but valid account number) were evaluated by manual procedures . If a user finds and corrects an error by himself, this action counts to the overhead of steps. Overhead on steps. For each task the minimum number of necessary steps was calculated as the best possible solution. The overhead on steps is then defined as the percentage of steps additional ly performed by the user within one task or at the overall level.
RESULTS FROM MEASUREMENT OF DITERAcrIONS Performance times The total performance times for the three systems are shown in table 1. It should be mentioned again, that these times do not cover the processing time taken by the systems themselves, so that performance time is always lower than the whole session time. The processing ti,ne does not differ much fron system to system and is ordinarily so short, that it does not cause the user to wait for conpletion of processing . Therefore processing time is neglected in further eva luation . The average performance times required for the use of system 51 or S2 are considerably shorter than those needed for the use of S3. Subdivision into the predefined user groups shows that
B. Luke
292
TABLE 1 Average of Performance Times at the Overall Level
system 51 52 53
all subjects
'manual' group
'batch' group
97,8 88% 90,7 81% 111,5 100%
94,9 90% 90,0 86% 105,2 100%
115,2 81% 108,3 76% 142,9 100%
'online' group
all subjects exc. '53'-group
'53 ' group
90,4 89% 84,1 83% 101,3 100%
70,2 58,6 54,4
101 ,7 95,3 119,7
129% 108% 100%
85% 80% 100%
The first number of each column is the performance time scaled with the value of 100 for the overall average of performance times. The second number is the percentage of performance time of system 53. this is not true for the '53'-group, where the lowest performance time was recorded 't,ith the known system (i.e. 53), while the new systems required more time. However all times for this user group remain far lower than those for the other groups. If the subjects of the '53'-group are excluded, the average of improvement of total performance time by systems 51 and 52 over system 53 is about 15 % and 20 % respectively. Further comparisons of the values between the groups show unexpectedly low times for the 'manual'-group. These times are not much higher than those for the 'online'-group whose subjects could be expected to have some advantage due to their previous experiences with that kind of online environment. Analysis of errors The average for the number of errors made by the subjects has the best value with system 52 (see table 2), the averages for the other two systems are up to 50% higher. With respect to these number of errors it has to be taken in consideration, that the possibilities of making errors were somewhat higher in systems 51 and 52, where the additional features have replaced the use of printed material, partially or completely, by equivalent online operation. This is especially true in 51. TABLE 2 Averages for Number of Errors and OVerhead on Steps
system 51 52 53
number of errors
overhead on steps
20,4 13,4 18,6
29,5 % 18,7 % 13,6 %
A more sophisticated analysis of the total number of errors shows that in all three systems about 70 % of the errors are concentrated upon only 5 of the more than 20 input fields. The most frequent error in 51 and 53 results frorr the interconnection of two input fields: the user has first to select a code to distinguish between customers, suppliers and cash receipts, an then, in the next step, the proper code number from the chart of
accounts is required as input. The selection code is not always required but is copied automatically from one task to another. Moreover sometimes the same code applies for several tasks in sequence. Thus it happens quite frequently, that the correctness of the select ion code for the actual task is not carefully enough controlled and causes errors in the interpretation of the code number in the next input field. Another frequent error in 51 occurs at the point where a return from the help-system to the main dialogue is intended but the input is not a valid selection from the menu table. The two most frequent errors in 52 are wrong inputs during the described one-digit selection from internal dict ionaries. It is of course true that a user learns to avoid errors, and that errors are an unavoidable consequence of trying to ope ra te wi th an unknown system. Thus a system's designer has to worry more about errors that a user makes after some time of experience than about those errors made at the very beginning of the operation. For the analysis of persistence of errors in our study three sections of tasks near the beginning, the middle and the end of the operation with the systems were formed. Each section consists of an equal number of tasks. The first section contains comparably simple tasks, the second sect ion contains rather complica ted tasks which cannot be completely carried out with the experience gairoed in the first section, and the third section contains only tasks with no additional difficulties. The average for the number of errors per section is shown in table 3. Due to the addi tional onE ne operation that is required in the use of 51 and 52 the corresponding values in these systems are at first higher than those for 53 TABLE 3 Average for Number of Errors in three Sections of a Session
system 51 52 53
first section simple tasks 4,4 3,2 3,0
second section complicated tasks 5,3 2,6 6,2
third section known tasks 1,9 1,0 2,5
Man-Computer Interaction ~n Financial Accounting Systems
but the situation changes in the second and third section. Thus it seems especially with system S2 to be possible to overcome more complicated tasks with fewer problems. Learning It was intenoed to give special emphasis to the progress that users make by learning during operation. In order to measure learning progress, the performance of the subjects was tested when they carried out a series of four pairs of tas} s each of which had identical structures of documents. These four pairs were fairly evenly distributed within the complete set of the 26 tasks and had the numbers 3/4, 11/12, 18/19 and 25/26. The results from evaluation of the performance times for those selected tasks are shown in Fig. 4. The thick lines indicate the average performance times measured for all subjects, while the thin lines ahove and below indicate the standard deviation between the subjects. It can be seen that the progress made by learning is similarly structured in all the three systems. The highest differences between the systems occur in the first period of measurement. These differences become smaller from period to period. In the second period the level of performance time is reduced to about one half of the value of the first period. The further reduction of time is much slower. That means that the subjects seem to have remembered quite well the structure of a task that they have performed once, since most of the progress for execution of the identically structured tasks was achieved already with the first repetition.
PERFORMANCE TIME \
\
81
\
82 83
100
50
3/4 Fig. 4
11/12
18/19 25/26 PAIR CF TASKS
Performance times for selected sets of tasks (times are scaled with the value of 100 for the overall ~verage of time per 2 tasks)
293
Comparison between the four user groups shows that the structure of the curves is similar for all groups, although the level of their performance times is different. The subjects of the 'S3'-group perform much faster than the subjects of the other groups, especially those of the 'batch' -group.
RESULTS FROM POST-TEST QUESTIONNAIRE Asking users for their opinion about features in des~gn of dialogues is often difficult. These users normally have at most the experience with their own system as a background for discussion, and they treat the user interface, especially on the software aspects, as a fact and do not discuss it. Such users have overcome the initial problems of handling and have adapted themselves to which ever kind of dialogue they are involved with. They do not perceive that possibly something in the design could be better, and sometimes they work with a subset of functions without complaint and without looking for more convenient procedures. While working with our systems the subjects have g:iined experience with some alternatives, and, with that background, the individual judgement could be expected to be somewhat sharper, though in our case certainly restricted to the bandwidth of variation between the three systems. Some distinct features The subjects were asked to to give ratings for some distinct features of dialogue design in the three systems. They had to assign a coded number per feature to each of the systems. The code numbers ranged from 1 to 5, code 1 meaning very good and code 5 meaning very bad. The average ratings for some of the features are given in table 4. Best ratings for the organization of input fields were give to system S1, in which the distance from one input field to another is minimized by the column by column org:inization. The lower ratings for S3 indicate that the inconsistencies against the line-by-line principle are well noticed and criticized. Comparable low ratings were given for the missing preformatting of S3, while the other two systems are rated somewhat better. It seems to be most useful to have preformatting as contents of the whole field (such as the lines in S1) instead of merely marking the left and right limit. The ratings in reg:ird to the intensity of changing of screen-images indicate that such changes should be used quite restrictively by the designer. Although even in 81 the changes only occurred in accordance with changes in requirements during the dialogue (e.g. looking for account numbers in the dictionary instead of typing the number directly), this system received somewhat lower ratings than the other two.
13. Luke
294
TABLE 4
Average Ratings for Distinct Features in Dialogue Design
Features
average ratings
Organization of input fields S1 column by column S2 line bv line S3 line by line with some inconsistencies Kind of preformatting used S1 Line at the bottom of the field S2 '+'-symbol at the limits of the field S3 none Intensity in changing of screen-images S1 high S2 medium S3 low
routine operation in the business environment whose functions are others than just accounting have comparable structure in the kind of on line operation and can be treated somewhat similarly with regard to the design of dialogues. REFERENCES
2,2 2,5 3,5
2,0 2,6 3,6
3,1 2,4 2,5
Achievement of goals Finally the subjects were asked to give ratings for the achievement of the predefined goals in the design of the dialogues (adequate for required functions, capable of explanations, secure in utilization). On the average the ratings for the systems S1 and S2 are nearly the same, for the second and the third goal they are each more than a whole point on the scale higher than those for the compared system S3. The subjects of the 'S3'-group judge their own system to achieve most successfully the first and the third goal, which should bedue to their previous knowledge. Long practice helped them to 'convert' the source documents to dialogue steps immediately and made them secure in utilization, whereas with the two new systems they could only use some knowledge about the basic structure and the practice in handling of the hardware environment.
CONCLUSION The results of the experiment prove that the design of dialogues in such applications like online accounting systems leads to a measurable impact upon the behaviour and attitudes of users performing given tasks at a display terminal. The subjects perceived the redesigned systems as more adequate for required functions, more capable of explanations and more secure in utilization. Deeper analysis of the results shows advantages and problems with specific features of the three evaluated dialogues. Users' problems tend to be concentrated upon distinct situations that arise during operation. Of course the scope of contribution to the research in software ergonomics is limited by the special application environment and the extent of variation of dialogue structure. However some systems for
Cakir, A., D.J. Hart, and T.F.M. Stewart (1979). The VDT Manual. IFRA-Institut, Darmstadt. Dehning, W., H. Essig, and S. Maass (1981). The Adaption of Virtual Man-Computer Interfaces to User Reguirements in Dialogs. Springer, Berlin Heidelberg New York. Dzida, W., S. Herda, and \-I.D. Itzfeldt (1978). User-perceived quality of interactive systems. IEEE Trans. Software Eng., 1., 270-276. Eason, K.D. (1980). Dialogue design implications of task allocation between man and computer. Ergonomics, £1., 881-891. Eason, K.D., L. Damodaran, and T.F.M. Stewart (1974). MICA Survey: A Report of a Survey of Man-Computer Interaction in Commercial Applications. Department of Human Sciences, University of Technology,Loughborcugh Embley, D.W., and G. Nagy (1981). Behavioral aspects of text editors. ACM Computing Surveys, .ll, 33-70. Engel, S.E., and R.E. Granda (1975). Guidelines for Man/Display Interfaces. IBM Poughkeepsie Laboratory Technical Report TR 00.2720. Gaines, B.R. (1981). The technology of interaction - dialogue programming rules. Int. J. Man-Mach. Stud., .11, 133-150. Griese, J. (1980). Vergleich des Leistungsumfangs von Softwaresysternen zur OnlineFinanzbuchhaltung. In P. Stahlknecht (Ed.), Online-Systeme im Finanz- und Rechnungswesen, GI-Anwendergesprach Berlin April 1980. Springer, Berlin Heidelberg New York. pp. 229-239. Hebditch, D. (1979). Design of dialogues for interactive commercial applications. In B. Shackel (Ed.), Infotech State of the Art Report Man/Computer Communication, Vol. 2, Invited Papers. Infotech International, Maidenhead. pp. 171-192. Hirsch, R.S. (1981). Procedures of the human factors center at San Jose. IBM Syst. J., lQ, 123-171. Ledgard, H., A. Singer, and J. Wniteside (1981). Directions in Human Factors for Interactive Systems. Springer, Berlin Heidelberg New York. Martin, J. (1973). Design of Man-Computer Dialogues. Prentice Hall,Englewood Cliffs. Reisner, P. (1981). Human factors studies of da tabase query languages: a survey and assessment. ACM Computing Surveys,.ll,13-31 Shackel, B. (1980). Dialogues and language can computer ergonomics help? Ergonomics, .£1., 857-880. Shneiderman, B. (1980). Software Psychology. Winthrop, Cambridge, Massachusetts. Stewart, T.F.M. (1976). Displays and the software interface. Applied Ergonomics, J., 137-146.
EXPERIMENT AL STUDIES OF MAN-COMPUTER INTERACTION IN FINANCIAL ACCOUNTING SYSTEMS B. Luke L ehrstuhl fur Betriebslnformatz'k , Universz'tiit Dortmund, D-4600 Dortmund 50 , Federal Republz'c of Germany
Abstract. Business application systems with online operations o bviously ne ed good man-comput e r interfaces. Experimental studies seem to be a useful approach to face the problem of des igning user-adequa te interf aces. \-li th special regard to the design of dialogues for online accounting, a subset o f an application system was redesigned to give several different featur e s of dialogue design, while all other parameters like hardware a nd applicatio n-bound functions remained identical. The differences in the constructed di a logues can be divided into three categories, which are (A) the presentation of inf o rmation by the system, (B) the handling of input and user erro rs and (C) the mea ns of dialogue operations including control functions ano online documentation. 48 professional accountants carried out a series of tasks, involving the handling of accounts receivable entries, with each of the three systems. All interactions were recorded. Results ar e given for the improvements achieved by the modifications in dial ogue design. The first part co ncerns the behaviour of the subjects with respect t o their different on line experiences. In the second part the general effects of the special design features are shown. Finally the user's own judgements bas e d on the evaluation of a post-test questionnaire are discussed. Keywords. online operation; man-machine systems; interactive systems; o nline accounting; software ergonomics; design of dialogues; human engineering; experimental evaluations.
DESIGN OF DIALCCUES
SOFTWARE INTERFACE FOR BUSINESS APPLICATIONS
Various recommendations have been published by several authors for the design o f dialogues in commercial applications, e.g. Mar tin (1973), Engel and Granda (1975), Stewart (1976), Hebditch (1979), Gaines (1981). The ir recommendations or guidelines for impl e mentation are mainly based upon the designer's experiences and on the more informal feedback from the users. Empirical or experimental validations are given only in some cases (e.g. Hirsch, 1981).
Nowadays more and more people in the business environment are directly faced with computers, working with a display terminal for data-entry, inquiries and/or other kinds of tasks. Video-display-units are typical equipment for c o mmercial applications such as order entry or accounting. They are utilised in conjunction with suitable software packages. The amount of work as a whole is divided up into user's tasks and computer's tasks with a high rate of interaction between user and computer. In this process of dialogue operation the n e ed for a good man-computer interface is obvious. The state of the art in computer technology allows the machines to be adapted to the user in an ergonomically acceptable manner. In the design of workstations, and especially of terminal hardware, much progress has been made during the last years, where definitive guidelines and checklists have been developed (Cakir, Hart and Stewart, 1979). Growing attention is now given to software aspects of the interface. Shackel (1980) uses the term "software ergonomics" for this type of research, while the more psychologically oriented approach is referred to as "cognitive ergonomics".
It has been shown that standard online accounting systems for smaller business computers have quite similar structures and power of functions (Griese, 1980). However, in spite of this similarity, it can be recognized that, with respect to the design of dialogues, there is a wide variety of different products. This may be due to a lack of the des igner' s general knowledge and of thorough and covering guidelines (besides the constraints caused by different hardware and operating systems). It seems to be useful to give some contributions to the augmentation of knowledge about user behaviour in such kinds of interactive systems. Thus, in this study, emphasis is placed on the evaluation of special features in the design of dialogues.
287
B. Liike
288
rESEARCH METHODS Classical forms of ergonomic research are empirical an0 experimental studies. For mancomputer interaction in commercial applications the E'mpirical results of Eason, Damodaran and StE'wart (1974) show the variety and problems wYich arise from the differences in user job types. In other fields of interactive computer usage valuable experimental results have been achieved, e.g. by Ledgard, Singer and Whiteside (1981) who studied the syntax of text editors. They started with the hypothesis that a command language should be as natural and familiar to the user as possible. For the experimental evaluation two editors wel-e used: one already existing editor and another one with a structure and syntax of comnands based on legitimate English phrases avoiding unfamiliar words and the use of special delimiting characters. The results, based upon 24 sessions with student subjects, show that the performance using the second editor was much superior on the basis of three measures (percentage of task completed, erroneous commands and editing efficiency). Many other experiments with various kinds of user tasks have been carried out and are assessed in the current literature, e.g. by Embley and Nagy (1981), Shneiderman (1980) and Reisner (1981).
'rHE EXPERIMENT The general idea of our research project was to let users carry out comparable tasks in different online accounting systems, where the hardware and the specific applicationbound functions remained identical and only the design of dialogues was changed between the systems. In this situation a laboratory experiment seemed to be the most promising approach. Three accounting systems on a small business computer with disk storage and two VDU-workstations were investigated: a supplier-made open item method accounting package and two other systems constructed under the direction of the author. For the sake of a Manageable experimental environment the development efforts were restricted to tasks connected with accounts receivable entries. AMong all frequent operations in accounting these seem to be best suited for our purposes, because they provide reasonable handling problems for the user, and are neither too complex nor too specialized. It was expected that every accountant should be able to understand the contents and procedures of sllch operations. Design Goals Tasks like those operations in accounting can be regarded as rather 'closed tasks' following the classification of Eason (1980). This means, that. input and required output have a rather fix€d structure and vary only within predictabl€ limits. In a simplifying example with accounts receipts we always have an entry of the amount of money for two accounts,
one of these being a cash or bank account and the other being a customer's or eventually a supplier's account, where the existing open items may fit to the payed amount or not. The knowledge about the structure and contents of the task makes it possible to create dialogues with prescribed performance procedures, at least as far as the limited variations of tasks do not require free choices or selection of alternatives by the user him self. These restricted assumptions are help ful by making the area of design problems smaller, but not even the general goals for user-adequate design are completely determined by them alone. For our design purpose we used a set of design goals with some similari ty to those developed by Dzida, Herda and Itzfeld (1978) and Dehning, Essig and Maass ( 1978). Having in mind that the dialogue should fit to the frequent and accustomed user as well as to the less frequent and untrained user we stated three categories of design goals: the dialogue should be (A) adequate for required functions, (B) capable of explanation and (C) secure in utilization. Adeguate for reguired functions means that the form of the dialogues should be suited, as far as possible, to the contents and structure of the tasks. Capable of explanations. Any instruction and information within the dialogue should be understandable and relate to the user's task. The user should see what the system is doing as a result of his operations, it should be clear ·....hat the system is expecting from him, and, finally, the user should be able to recognize the whole set of his possible alternatives. Secure in utilization. During operation mainly on data input - the user should have the lowest possible feeling of uncertainty about the correctness of his actions. This means especially the user-perceived security against unexpected and unwanted reactions by the system. Features of the Three Systems Oriented to those design goals the two new accounting systems were developed. A selection was taken from among those existing techniques and additional features, which in our belief would lead to some improvements. The implementation was carried out with different sets of those selected techniques and features in the new systems. Some constraints on the designer's freedom were set by the known limits in hardware and operating system of the machine to be used. However the major common factor of the three dialogues results from the decision to use the screen in a form-filling mode with both the new versions too. This decision allowed for the fact that all comparable online accounting systems we know use this technique as interaction mode. Taking into account this global common feature, the differences in the constructed dialogues can be classified in three catego-
Man-Computer Intera c tion
~n
ries. These teing (A) the presentation of in formation by the system, (B) the handling of inputs and user errors (C) the means of dialogue operation. Presentation of information. This comprises everything which is displayed on the screen to the user. It includes the type of inputform used, the spacing and grouping of the input fields, and the feedback that is given to the user as information about the current state of the dialogue. In addition, it includes the frequency in changing of complete screen-images and the location of explanatory text. The kind of presentation determines essentially the visible surface of the dialogue. Handling of inputs and user errors. Within each system there is a unique manner of designing all the fields that are used for data input by the user. The differences between the systems are the kinds of preformatting or indicating the length of the input fields, and the availability and visibi 1 i ty of default values. Handling of user errors is done by error messages in a fixed partition at the botto m of the screen, where the reason of the error is indicated and a repetition of the input is required. Means of dialogue operation. All features of the control functions available to the user are categorized here, i.e. backtracking and reset facilities as well as the type and completeness o f online assistance on the syntactical and semantical level. Syntactical means f or control actions are implemented as commands, furlct ion keys or menus. Each of the sys tems can be described as consisting of a join of e lements fram all the three categories. Figures 1 to 3 show the treatment of an identical set of data and dial o gue state by each of the three systems, which are re fe rred to as S1, S2 and S3. A short verbal characterization is given in the following. Features of system S1 (Fig. 1). The dialogue enables the user to carry out the whole task without further written material such as application manuals and tables for coding of input. Special aids for input operations such as visible defaults or abbreviations are not provided. Consecutive input fields are organized column by column instead of line by line. Compared to the other systems we have here the highest flexibility because operations can be controlled by a small set of commands. The dialogue is split into several frames of up to 5 different screen-images for one task. The available special options (e.g. scrolling or other control functions) are always visible at the top of the screen. All further material is put together in a permanently available subsystem for on-line assistance. This help-system can be reached by issuing one of the offered specific commands. A syntactical and semantical explanation of the current input field is then dis-
Financial Accountin g Systems
289
played by default. Guided by a set of menus within the help-system other facilities can be activated, i.e. further explanations, tables of coded items (e.g. account numbers) and finally the end of t he subsystem to continue in the interrupted main dialogue. Features of system S2 (Fig. 2). The main features in dialogue design of this system are the operator instruction facilities and the assistance for input operations. The operator instru ct ions are located at the bottom of the screen and show the pos s ible user actions by explaining the meaning of the function-keys. According to the current input field, up to 7 different actions are offered by short keywords. Visible default values in the input fi el ds allow the r e -use of data items for the current or following tasks. An additional featur e is the use of line numbers instead of the complete coded values in those cases where co des are not known and have to be l ooked up in internal dictionaries (e.g. the code of a c ustome r of wham only the name is known fram the source do cument ). The line number with its one digit causes the entire code to be moved into the input field. The control within the dial ogue is generally handled by the use o f function-keys; accordingly the meaning of single keys may change from one input field to another. However fixed key numbers are used for backtracking and reset functions, which are available with every single step. Features of system S3 (Fig. 3). In this system the dialogue provided by the supplier of the standard package has n ot been changed. Within the dialogue the explanatory texts on the screen are essentially restricted to the descriptions of the input fields. Sometimes a single descr iption is followed by !lOre than one input field. Because all the input fields are packed rather closely together and the explanatory text is quite short, most of the tasks can be carried out without a complete change of the screen-image. For the necessary control functions some o f the function-keys are used, but the facilities differ fron field to field and are not available everywhere. In some situations a short text is is sue d as a hint for the possible control actions. Two of the dictionaries to be used for finding codes of input items consist of printed lists and are not accessible within the dialogue. In addition to the operation manual a printed list of those fields is provided, where backtracking is possible. Subjects For experimental testing 48 professional accountants were available for participation. The majority of them was delegated by their department managers. Only rough information about the desired procedure was given prior to the experimental session itself. The subjects were classified according to the kind of computer-assistance for the accounting system of their own affiliation.
290
B. Luke
1 B U C HEN Z A H L U N G S E I N G A N G E 0 P - A U S G L E I CHI 1--------------------------------------------------------------------------------1 1 FELD ZURUCK : Fl ENDE : F2 INFORMATION: *H VORBLATT~RN: + ZURUCKBL. : 1 1--------------------------------------------------------------------------------1 1 16 NEU-OP/SKONTO (N/S)7 1 17 REF ERENZ-NUMMER . 1 18 AUSGLEICHSBETRAG . 1 19 ANDERUNG OK (J/N)7
_ _______ ___________ _
2736.50 ZAHLBETRAG 54.82 SKONTOBETRAG 2018.29 RESTBETRAG
1 1
1
1
1---------------_·_---------------------------------------------------------------1 1 1
REF-NR
RE-DATUM
FALLIG
ZU ZAHLEN
SKONTO
NOCH OFFEN
1
17420 17438 17512 17580 17700 17782 17783 17922
78.07.05 78.07.07 78.07.09 78.07.11 78.07.13 78.07.15 78.07.17 78.07.19
78.10.05 78.10.07 78.10.09 78.10.11 78.08.03 78.10.15 78.10.17 78.10.19
732.46 2777.77 9.19 299.66 1402.11 0.00 2061.43 902.66
12.98 138.89 0.46 0.00 0.00 0.00 103.07 45.13
AUSGEGLICHEN
1 1
I 1 1 1
I
1 I
1 1 Fig. 1
Treatment of open items in system 8 1
BUCHUNG VON ZAHLUNGSEINGAENGEN FUER
1
BLECHALBRECHT GMBH
11 ============================================================================== UNTERNEHMEN/~ERK 101 SACHKONTONUMMER 1130000 BELEGNUMMER 19438 1 DEBITORENNUMMER
1432600
BERHERMANN GMBH
ZAHLBETRAG
2736.50
1------------------------------------------------------------------------------1 LISTE DER OFFENEN POSTEN 1
1 1
1 1 1 1 1
1
INDEX 1 2 3 4 5 6 7 8
REFNR 17420 17438 17512 17580 17700 17782 18113 18422
RE-DATUM 78.07.05 78.07.07 78.07.09 78.07.11 78.07.13 78.07.15 78.08.12 78.08.14
OP-BETRAG 732.46 2777.77 9.19
SKTOBETRAG 12.98 138.89 0.46 0.00 0.00 2.00 103.07 45.13
299.~6
1402.11 100.00 2131.43 902.66
ZAHLBETRAG
1------------------------------------------------------------------------------1 INDEX I OP-NR +1 + RESTBETRAG 2736.50 1=============================================================================== 1 F3 = VORBLAETTERN F4 = RUECKBLAETTERN F5 = OP-NEUANLAGE 1 FA = ABB BUCHUNG FB = ABB OP F2 = ZURUECK BUSCHLUE ETB = INDEXAUSWAHL 1=============================================================================== Fi g . 2 IUNT/~ERK
101
Trea tment of op e n items in system 82
BUCHERZEICHEN BI
BUCHUNGSMONAT 7803
1
1BELEGDIHUM
ISTORNO-KZ IRESTSUMME IZAHLBETRAG 1STEUER 1
78.08.30 0 4189.20 2736.50
BELEGNR 35267 SACHKONTONR. 1000000 KONTOBER/-NR. 1 1432600 SKTO/STEUER 0.00 STEUER2
I~ECHSELDATUM
IREST-AUFTBET
1 1 1 1 1 1 1 1 1 1 1 1
2736.50
REF-NR. 17420 17438 17512 17580 17700 17782 18113 18422 18631
RE-DATUM 78.07.05 78.07.07 78.07.09 78.07.11 78.07.13 78.07.15 78.08.02 78.08.04 78.08.09
OP-SALDO 732.46 2777.77 9.19 299.66 1402.11 100.00 2061.43 902.66 1087.40
17420
78.07.05
732.46
Fig. 3
BELEGSUMME
4189.20
SCHLUSSEL GEGENWERT TEXT ZE 1
31 2
SKONTO ZAHLBETR. REST-OP 12.98 (3.00%) 83.33 (3.00%) 0.28 (3.00%) 0.00 0.00 2.95 (3.00%) 61.84 (3.00%) 27.08 (3.00%) 32.62 (3.00%) 12.98
719.48
Treatment of open items in system 83
1
1 1 1
0.00
Han-Computer Interaction ~n Financial Accounting Systems
Group 1, termed the 'manual'-gr oup, consisted of 6 subjects, who do all accounting without computer-assistance at all. Group 2, terned the ' batch' -gr o up, consisted of 18 subjects, who use batch systems by filling coding- sheets as input and receiving printed lists as output. Group 3, terned the ' on line' -gr oup , consisted of 18 subjects , ''''ho use online accounting systems by themselves (generally several hours per day). These subjects had no prior knowledge about any of the three systems used in the experiment. Group 4, tern 'ed the 'S3'-gr oup , consisted of 6 subjects , Hho use the standard package referred here ~s S3 for daily work. Procedure Each session started with an interactive tutorial at th~ workstation. This tutorial consisted of im;tructions and practices about the use of tl' e keyboard, especially the locations and meanings of the function-keys . It included at its end a multiple-choice questionnaire about the characteristics of the subjects ' cl"rical work. After that, a description of the first system to be used was given to the suject . It was about 7 page~; long and contained a summarized presentation of the structure of the implemented nialogue. After the reading of the matecial was cnmpleted the system was started and the set of prepared tasks was given to the subject. This set was composed of 26 tasks with a fixed order . Each task consisted of the " original " source documents of a single accounts receivable entry as received from the bank or the cashier's office. I\dditional written remarks were given if necessary for the understanding of the documents. 1\ member of the experimental staff was present during the whole session, giving advice for the first two tasks but restricting his further communication with the subject to exceptional cases. When the set of tasks was finished it was offered to the subject to have a break without prescribed time l imits . The second and the third system were treated in the same manner. The order of sequence between the three systems was randomly determined so that the six possible sequences were balanced within in each group of subjects. After a subject had completed all tasks in all systems , individual judgements about the different features in the dialogues were collected via a post - test questionnaire. The end of the session was normally reached in the early afterr.oon with a time taken for the whole session ranging from 4 up to 9 hours (breaks incluned).
29 1
Measurement of Interactions All user input was recorded by a monitor program especially implemented for that purpose. For every interaction (i.e. one dialogue step represented by the input o f the user into one field) a record was created containing the following data: field identificatio n, the input itself, the end-key for completion of data entry , the user time and a code for indication of errors or brand"les in connection with that step. I\ccumulation to the overall and the task level resulted in a set of three variables for each of the systems. These being (A) the performance time, (B) the number of errors, (C) the overhead on steps. Performance time. This is the time the system has to wait for the end of the user's input action. It covers the time fron the moment the cursor is moved to the input field by the system until the user presses an end -key for transmission of the field. Measurement is in units of 0,15 seconds. The performance time for a single step may be very short and even zero, because the workstation allows some characters to be typed in advance. Otherwise performance time can exceed one minute or more, if the user has to l ook at manuals or other additional material (or is just thinking about what to do next ). Number of errors. By the monitor program only those errors were logged, wh ich could be found by a syntactical and semantical check of the single input items. Other errors (e.g. the use of a wrong but valid account number) were evaluated by manual procedures . If a user finds and corrects an error by himself, this action counts to the overhead of steps. Overhead on steps. For each task the minimum number of necessary steps was calculated as the best possible solution. The overhead on steps is then defined as the percentage of steps additional ly performed by the user within one task or at the overall level.
RESULTS FROM MEASUREMENT OF DITERAcrIONS Performance times The total performance times for the three systems are shown in table 1. It should be mentioned again, that these times do not cover the processing time taken by the systems themselves, so that performance time is always lower than the whole session time. The processing ti,ne does not differ much fron system to system and is ordinarily so short, that it does not cause the user to wait for conpletion of processing . Therefore processing time is neglected in further eva luation . The average performance times required for the use of system 51 or S2 are considerably shorter than those needed for the use of S3. Subdivision into the predefined user groups shows that
B. Luke
292
TABLE 1 Average of Performance Times at the Overall Level
system 51 52 53
all subjects
'manual' group
'batch' group
97,8 88% 90,7 81% 111,5 100%
94,9 90% 90,0 86% 105,2 100%
115,2 81% 108,3 76% 142,9 100%
'online' group
all subjects exc. '53'-group
'53 ' group
90,4 89% 84,1 83% 101,3 100%
70,2 58,6 54,4
101 ,7 95,3 119,7
129% 108% 100%
85% 80% 100%
The first number of each column is the performance time scaled with the value of 100 for the overall average of performance times. The second number is the percentage of performance time of system 53. this is not true for the '53'-group, where the lowest performance time was recorded 't,ith the known system (i.e. 53), while the new systems required more time. However all times for this user group remain far lower than those for the other groups. If the subjects of the '53'-group are excluded, the average of improvement of total performance time by systems 51 and 52 over system 53 is about 15 % and 20 % respectively. Further comparisons of the values between the groups show unexpectedly low times for the 'manual'-group. These times are not much higher than those for the 'online'-group whose subjects could be expected to have some advantage due to their previous experiences with that kind of online environment. Analysis of errors The average for the number of errors made by the subjects has the best value with system 52 (see table 2), the averages for the other two systems are up to 50% higher. With respect to these number of errors it has to be taken in consideration, that the possibilities of making errors were somewhat higher in systems 51 and 52, where the additional features have replaced the use of printed material, partially or completely, by equivalent online operation. This is especially true in 51. TABLE 2 Averages for Number of Errors and OVerhead on Steps
system 51 52 53
number of errors
overhead on steps
20,4 13,4 18,6
29,5 % 18,7 % 13,6 %
A more sophisticated analysis of the total number of errors shows that in all three systems about 70 % of the errors are concentrated upon only 5 of the more than 20 input fields. The most frequent error in 51 and 53 results frorr the interconnection of two input fields: the user has first to select a code to distinguish between customers, suppliers and cash receipts, an then, in the next step, the proper code number from the chart of
accounts is required as input. The selection code is not always required but is copied automatically from one task to another. Moreover sometimes the same code applies for several tasks in sequence. Thus it happens quite frequently, that the correctness of the select ion code for the actual task is not carefully enough controlled and causes errors in the interpretation of the code number in the next input field. Another frequent error in 51 occurs at the point where a return from the help-system to the main dialogue is intended but the input is not a valid selection from the menu table. The two most frequent errors in 52 are wrong inputs during the described one-digit selection from internal dict ionaries. It is of course true that a user learns to avoid errors, and that errors are an unavoidable consequence of trying to ope ra te wi th an unknown system. Thus a system's designer has to worry more about errors that a user makes after some time of experience than about those errors made at the very beginning of the operation. For the analysis of persistence of errors in our study three sections of tasks near the beginning, the middle and the end of the operation with the systems were formed. Each section consists of an equal number of tasks. The first section contains comparably simple tasks, the second sect ion contains rather complica ted tasks which cannot be completely carried out with the experience gairoed in the first section, and the third section contains only tasks with no additional difficulties. The average for the number of errors per section is shown in table 3. Due to the addi tional onE ne operation that is required in the use of 51 and 52 the corresponding values in these systems are at first higher than those for 53 TABLE 3 Average for Number of Errors in three Sections of a Session
system 51 52 53
first section simple tasks 4,4 3,2 3,0
second section complicated tasks 5,3 2,6 6,2
third section known tasks 1,9 1,0 2,5
Man-Computer Interaction ~n Financial Accounting Systems
but the situation changes in the second and third section. Thus it seems especially with system S2 to be possible to overcome more complicated tasks with fewer problems. Learning It was intenoed to give special emphasis to the progress that users make by learning during operation. In order to measure learning progress, the performance of the subjects was tested when they carried out a series of four pairs of tas} s each of which had identical structures of documents. These four pairs were fairly evenly distributed within the complete set of the 26 tasks and had the numbers 3/4, 11/12, 18/19 and 25/26. The results from evaluation of the performance times for those selected tasks are shown in Fig. 4. The thick lines indicate the average performance times measured for all subjects, while the thin lines ahove and below indicate the standard deviation between the subjects. It can be seen that the progress made by learning is similarly structured in all the three systems. The highest differences between the systems occur in the first period of measurement. These differences become smaller from period to period. In the second period the level of performance time is reduced to about one half of the value of the first period. The further reduction of time is much slower. That means that the subjects seem to have remembered quite well the structure of a task that they have performed once, since most of the progress for execution of the identically structured tasks was achieved already with the first repetition.
PERFORMANCE TIME \
\
81
\
82 83
100
50
3/4 Fig. 4
11/12
18/19 25/26 PAIR CF TASKS
Performance times for selected sets of tasks (times are scaled with the value of 100 for the overall ~verage of time per 2 tasks)
293
Comparison between the four user groups shows that the structure of the curves is similar for all groups, although the level of their performance times is different. The subjects of the 'S3'-group perform much faster than the subjects of the other groups, especially those of the 'batch' -group.
RESULTS FROM POST-TEST QUESTIONNAIRE Asking users for their opinion about features in des~gn of dialogues is often difficult. These users normally have at most the experience with their own system as a background for discussion, and they treat the user interface, especially on the software aspects, as a fact and do not discuss it. Such users have overcome the initial problems of handling and have adapted themselves to which ever kind of dialogue they are involved with. They do not perceive that possibly something in the design could be better, and sometimes they work with a subset of functions without complaint and without looking for more convenient procedures. While working with our systems the subjects have g:iined experience with some alternatives, and, with that background, the individual judgement could be expected to be somewhat sharper, though in our case certainly restricted to the bandwidth of variation between the three systems. Some distinct features The subjects were asked to to give ratings for some distinct features of dialogue design in the three systems. They had to assign a coded number per feature to each of the systems. The code numbers ranged from 1 to 5, code 1 meaning very good and code 5 meaning very bad. The average ratings for some of the features are given in table 4. Best ratings for the organization of input fields were give to system S1, in which the distance from one input field to another is minimized by the column by column org:inization. The lower ratings for S3 indicate that the inconsistencies against the line-by-line principle are well noticed and criticized. Comparable low ratings were given for the missing preformatting of S3, while the other two systems are rated somewhat better. It seems to be most useful to have preformatting as contents of the whole field (such as the lines in S1) instead of merely marking the left and right limit. The ratings in reg:ird to the intensity of changing of screen-images indicate that such changes should be used quite restrictively by the designer. Although even in 81 the changes only occurred in accordance with changes in requirements during the dialogue (e.g. looking for account numbers in the dictionary instead of typing the number directly), this system received somewhat lower ratings than the other two.
13. Luke
294
TABLE 4
Average Ratings for Distinct Features in Dialogue Design
Features
average ratings
Organization of input fields S1 column by column S2 line bv line S3 line by line with some inconsistencies Kind of preformatting used S1 Line at the bottom of the field S2 '+'-symbol at the limits of the field S3 none Intensity in changing of screen-images S1 high S2 medium S3 low
routine operation in the business environment whose functions are others than just accounting have comparable structure in the kind of on line operation and can be treated somewhat similarly with regard to the design of dialogues. REFERENCES
2,2 2,5 3,5
2,0 2,6 3,6
3,1 2,4 2,5
Achievement of goals Finally the subjects were asked to give ratings for the achievement of the predefined goals in the design of the dialogues (adequate for required functions, capable of explanations, secure in utilization). On the average the ratings for the systems S1 and S2 are nearly the same, for the second and the third goal they are each more than a whole point on the scale higher than those for the compared system S3. The subjects of the 'S3'-group judge their own system to achieve most successfully the first and the third goal, which should bedue to their previous knowledge. Long practice helped them to 'convert' the source documents to dialogue steps immediately and made them secure in utilization, whereas with the two new systems they could only use some knowledge about the basic structure and the practice in handling of the hardware environment.
CONCLUSION The results of the experiment prove that the design of dialogues in such applications like online accounting systems leads to a measurable impact upon the behaviour and attitudes of users performing given tasks at a display terminal. The subjects perceived the redesigned systems as more adequate for required functions, more capable of explanations and more secure in utilization. Deeper analysis of the results shows advantages and problems with specific features of the three evaluated dialogues. Users' problems tend to be concentrated upon distinct situations that arise during operation. Of course the scope of contribution to the research in software ergonomics is limited by the special application environment and the extent of variation of dialogue structure. However some systems for
Cakir, A., D.J. Hart, and T.F.M. Stewart (1979). The VDT Manual. IFRA-Institut, Darmstadt. Dehning, W., H. Essig, and S. Maass (1981). The Adaption of Virtual Man-Computer Interfaces to User Reguirements in Dialogs. Springer, Berlin Heidelberg New York. Dzida, W., S. Herda, and \-I.D. Itzfeldt (1978). User-perceived quality of interactive systems. IEEE Trans. Software Eng., 1., 270-276. Eason, K.D. (1980). Dialogue design implications of task allocation between man and computer. Ergonomics, £1., 881-891. Eason, K.D., L. Damodaran, and T.F.M. Stewart (1974). MICA Survey: A Report of a Survey of Man-Computer Interaction in Commercial Applications. Department of Human Sciences, University of Technology,Loughborcugh Embley, D.W., and G. Nagy (1981). Behavioral aspects of text editors. ACM Computing Surveys, .ll, 33-70. Engel, S.E., and R.E. Granda (1975). Guidelines for Man/Display Interfaces. IBM Poughkeepsie Laboratory Technical Report TR 00.2720. Gaines, B.R. (1981). The technology of interaction - dialogue programming rules. Int. J. Man-Mach. Stud., .11, 133-150. Griese, J. (1980). Vergleich des Leistungsumfangs von Softwaresysternen zur OnlineFinanzbuchhaltung. In P. Stahlknecht (Ed.), Online-Systeme im Finanz- und Rechnungswesen, GI-Anwendergesprach Berlin April 1980. Springer, Berlin Heidelberg New York. pp. 229-239. Hebditch, D. (1979). Design of dialogues for interactive commercial applications. In B. Shackel (Ed.), Infotech State of the Art Report Man/Computer Communication, Vol. 2, Invited Papers. Infotech International, Maidenhead. pp. 171-192. Hirsch, R.S. (1981). Procedures of the human factors center at San Jose. IBM Syst. J., lQ, 123-171. Ledgard, H., A. Singer, and J. Wniteside (1981). Directions in Human Factors for Interactive Systems. Springer, Berlin Heidelberg New York. Martin, J. (1973). Design of Man-Computer Dialogues. Prentice Hall,Englewood Cliffs. Reisner, P. (1981). Human factors studies of da tabase query languages: a survey and assessment. ACM Computing Surveys,.ll,13-31 Shackel, B. (1980). Dialogues and language can computer ergonomics help? Ergonomics, .£1., 857-880. Shneiderman, B. (1980). Software Psychology. Winthrop, Cambridge, Massachusetts. Stewart, T.F.M. (1976). Displays and the software interface. Applied Ergonomics, J., 137-146.