- Email: [email protected]
Experimental comparison between automatic and manual menu interface design methods
interacting
with Computers
vol 7 no 2 (1995) 73-89
Experimental comparison between automatic and manual menu interface design methods Peretz Shoval
An experimental comparison between two methods of designing a menu-tree interface for an information system is described. The ‘automatic’ method is based on the ADISSA methodology (architectural design of information systems based on structured analysis) according to which the interface is derived automatically from dataflow diagrams (DFD) as a by-product of the system analysis stage. In the ‘manual’ (‘conventional’) method the designer constructs an appropriate interface by applying common principles of interface design. The objective of the experiment is to find out which design method yields a better initial interface, before it is given to users for further improvements. The user’s viewpoint is adopted i.e., the interface was evaluated subjectively. The hypothesis is that menutrees designed automatically are as good as those designed by people. The experiment included six information system development projects. One menu-tree interface was designed automatically for each system. Four others were designed manually by different designers who were assigned randomly to the systems. The five different interfaces of each system were given to 16 potential users for evaluation and comparison. Analysis of the results revealed no significant difference between the mean scores of the two types of interface. Noting that the automatic menus are obtained as a by-product of the system analysis stage, and that they can be improved by users during prototyping, it is concluded that the method provides a good start for the interface design. Keywords: design
human-computer
interaction,
system analysis, interface
Introduction The design of a human-computer
interface is an important
Information Systems Program, Department of Industrial 84105, Gurion University of the Negev, Beer-Sheva [email protected] Paper submitted:
0953-5438/95
17 May 1994; accepted:
Elsevier Science B.V.
30 November
Engineering Israel. Fax:
activity in informaand Management, Ben972 7 280776. E-mail:
1994
73
tion systems development, and a great deal has been written on this subject. Many studies concentrate on the attributes of ‘good’ interfaces and ‘humanengineering’ aspects, e.g., user friendliness, learnability, simplicity, consistency, flexibility to changes, etc. Some literature suggests guidelines and rules for designing the interface, e.g., identifying the user’s needs and the system’s tasks, and ways of grouping tasks before the design begins (see, for example, Gould and Lewis, 1985; Hartson and Hix, 1989; Kiger, 1984; Lustman et al., 1985; Morland, 1983; Rissland, 1984; Shneiderman, 1987; Smith and Mosier, 1986; Thimbleby, 1984; Yound and Hull, 1982). It has been well established that the user interface is a major factor in determining user satisfaction, and that interface design activity must centre on users and their tasks (Grudin, 1989). Moreover, for the interface to fit the users’ expectations and needs, they must be involved in the system analysis and design process (Gould and Lewis, 1985; Mohammad and Wayne, 1988; Norman and Draper, 1986; Thimbleby, 1984). In this light, it is surprising that much of the research on interface design is done without the framework of the whole information system development process. Although the need to view user interface design as an integral part of system design has been acknowledged (Betta et al., 1987; Draper and Norman, 1985; Hartson and Hix, 1989) most of the prevailing analysis and design methodologies totally ingore interface design. According to Hartson and Hix (1989), only two methodologies deal with interface design as part of system design: USE (Wasserman et al., 1986) and DMS (Hix and Hartson, 1986). Even the more recent, object-oriented methodologies, hardly deal with interface design, including popular methodologies such as those described by Martin and Ode11 (1992), Rumbaugh et al. (1991), Shlaer and Mellor (1992), Wirfs-Brock et al. (1990) and others. One exception is Coad and Yourdon (1991), who deal with the “human-interaction component (HIC)“, and propose a strategy to design this component. Their strategy consists mainly of classifying the humans (users), designing the command hierarchy, and designing the HIC classes. To design the command hierarchy they propose: l l
l
to study the existing user interaction metaphors and guidelines; to establish an initial command hierarchy consisting of the system ‘services’ (methods) presented on menu-screens; to refine the command hierarchy (including ordering of the services within each part of the hierarchy, creating whole/part chunks, and considering breadth versus depth, with the objective to avoid overloading user memory and minimize the number of steps.
Another, process oriented methodology, which includes interface design and provides a structured procedure to derive a menu-tree user interface from dataflow diagrams is ADISSA (Shoval, 1988; 1990). A menu-tree interface presents its users with options to choose from and tasks the system can perform. It consists of a hierarchy of menus, where a menu includes several lines (items). A line can be a ‘selection’ type if it presents a lower-level (child) menu from 74
Interacting
with Computers
vol 7 no 1 (2995)
which the user can select more specific lines. Alternatively, it can be a ‘trigger’ type if it activates a transactions, i.e., a system process that performs a task. (Other kinds of menu options, which are not related to the functionality of the system, e.g., help, are not considered here.) A user interacts with the menu-tree interface in a top-down fashion. Beginning at the root-menu, he or she browses the menus and searches down a certain path until a desired transaction is found and activated. At this stage the user leaves the interface and enters the transaction. i.e., the actual application. A specific transaction may (and usually does) involve further interaction with the user, say when the user is expected to input data, or get information from the system, but this type of interaction is not of concern in this study. Our concern is with designing the system interface that will enable its users to find what they need to do with the system. The focus of this study is on the functional aspects of interface design. We are not concerned with technical aspects such as screen layout and formatting, windowing techniques, graphics, use of colour, response time, and other system considerations. Our concern is with the contents and structure of menus - considering such questions as: l l l
l l
Which menu lines will comprise the interface? Which lines will be grouped in separate menus? What is a proper hierarchy of menus to enable objectives? How many levels will there be in the tree? How many lines will be within the menus?
users
to realize
their
In essence, we are concerned with the extent to which a menu-tree interface reflects the information system and its functions, as perceived and preferred by the users. The creation of an interface (including a menu-tree) is an engineering problem that forces trade-offs among many factors (Grudin, 1989). As already noted, there are many principles and guidelines for constructing menus. For example, there are guidelines about the depth of a menu’s hierarchy and the breadth (i.e., the number of options) of a menu. These guidelines suggest that neither factor should typically be greater or less than a certain number. As with many other design problems, however, there are different ways to carry out the task. Much depends on the designer’s knowledge, creativity, imagination, experience, and skill. Hence, the two problems that concern us are firstly, why interface design is not done as part of system analysis and design; and secondly why it is done intuitively in an unstructured manner. A remedy may be found if the process of interface design becomes an integral part of a structure system analysis and design methodology, which will also enable generation of the menus in a well-structured fashion. This approach has been adopted in the ADISSA methodology for analysis, design, and prototyping. According to this CASE-supported methodology, the menu-tree of a system is derived automatically (i.e. algorithmically) from the set of hierarchical dataflow diagrams (DFDs) that are constructed during system analysis. The methodology and its supporting tools, including the method for automatic menu-tree design, have already Shoval
75
been described in Shoval(l990) and Shoval and Manor (1990). In this study we compare the automatic method with an alternative, unstructured method, which we term ‘manual’, as it relies on the human designer. Any menu-tree, whether designed manually by a human designer or automatically by following an algorithm, can and should be given to users to experiment with and improve. In this study we do not deal with such interactive improvements. We are concerned with the initial design produced by either method, before it is improved by the users. The objective is to find out if the automatic method, which is incorporated as an integral part of the whole system analysis and design process, can provide an initial menu-tree interface which would be a good start for further improvements. In the following section we describe the two design methods. In the third section we describe the experiment we conducted to evaluate the resulting menu-trees. The section that follows presents the results, and in the fifth section we discuss the implications of the results, considering the strengths and weaknesses of the experiment, and suggest further research.
Two methods of menu-tree design The automatic method The automatic method is part of ADISSA which is a CASE-supported methodology for systems analysis, design, and prototyping (Shoval, 1988; Shoval and Manor, 1990). The methodology supports the following stages: l
l
l l l
l
functional analysis in which the system analyst produces a hierarchy of enhanced DFDs; transactions (processes) design in which transaction diagrams and their structure descriptions are created using structured programming techniques (e.g. pseudo-code); interface design, where a menu-tree is generated (as will be detailed); database design, which provides a normalized database schema; input/output design, which results in layouts of input and output screens and reports; prototyping, which produces a structured-prototype of the system that is based on the earlier products.
ADISSA provides a functional menu-tree design method according to which the interface is derived from hierarchical DFDs which are produced at the functional analysis stage. Functional analysis is a comprehensive process, involving analysts and users. In an iterative way, users and analysts identify the functions of the system and their subfunctions, and group them in a meaningful way. The final DFDs are approved by users and management as per goals, requirements and semantic design. Hence, the hierarchical DFDs, which are the basis for the system structure and for interface design, actually portray the semantic organization that users have, i.e., they reprsent the user model of the system. Thus, the menu-tree that is derived from the DFDs reflects the functional structure of 76
Interacting
with Computers
vol 7 no 1 (1995)
the system as perceived by the users who take part in the analysis. The method follows Betts et al. (1987), Grudin (1989), Shneiderman (1987) and others, who claim that the interface must be designed according to the user’s tasks. Moreover, the ADISSA approach supports Fischer’s (1989) observation that: “Users do not know what they want, and designers do not understand what users need or will accept. The only viable strategy. . . is incremental, evolutionary development: initial systems must be built to give users something concrete to react to.” Similarly, ADISSA produces an initial menu-tree automatically from the DFDs that can subsequently be improved by its users. The toolsupported method is described herein (see further details in Shoval, 1990). The initial menu-tree is created as follows. The DFD tree is examined by a search for functions that are linked to user entities, because they signify data input or output. For each DFD having such links, a menu is defined, which includes a menu-line for each function linked to a user entity. (At this stage the lines are identified by the numbers and labels of the respective functions.) A line is considered as a ‘selection’ if it originates from a general function, i.e., a function that is decomposed and described in a separate child DFD. Upon its selection at run-time, the interface will present a child menu. It is considered as a ‘trigger’ if it originates from, or terminates on, an elementary function. Upon its selection it will trigger the transaction to which the respective function belongs. (A transaction is an independent task or activity the system can perform. In the DFD, a transaction consists of one or more elementary functions which are linked to each other via dataflows, and of other DFD elements that are linked to those functions, i.e. data stores and external entities. A transaction has to include at least one external entity, to enable its triggering.) The result of this step is a tree of menus that is a subset of the tree of DFDs. In the next step, a pass is made through the menus to eliminate ‘degenerate’ menus (a degenerate menu includes one line only, offering no selections, and has been created because the DFD from which it originates includes only one function linked to the user-entity), and to unite ‘trigger’ lines that belong to the same transaction (such duplicate lines have been created if more than one function belonging to the same transaction is linked to a user entity). Since a trigger line activates the whole transaction to which it belongs, there is no need for duplicate triggers. (The identification number of the united line is a concatenation of the original numbers; a new label for the united line has to be specified by the analyst or user.) An example that explains the method (taken from Shoval, 1990) follows. Figures l-3 show three schematic DFDs of a system. Note that the two general functions 1 and 4 in DFD-0 are decomposed and described in separate DFDs. Figure 4 shows the initial menu-tree that is derived from those DFDs in two steps: in Figure 4 (a) we see the initial menu-tree after step 1. Each of its three menus consists of lines that correspond to general or elementary functions connected to user entities. Note the multiple lines for the same transaction in menu number 1 (lines 1.1 and 1.2, as well as lines 1.3 and 1.5), and also a ‘degenerate’ menu number 4. In Figure 4 (b) we see the menu-tree after line unification (lines 1.1/2 and 1.3/5) and elimination of the degenerate menu. This terminates the stage of initial menu-tree design. Later users can interact Shoval
77
User entity 1
Figure 1. DFD-0 I
user entity 1
-I
User entity 1
Figure 2. DFD-1 78
Interacting
with Computers zwl 7 no 2 (1995)
!,I User
nl
entity 1
Figure 3. DFD-4
Menu number: 0 Name Code General function No. 1 1 Elementatv function No. 2 2 General fhction No. 4 4 Menu number: 1 Name Code 1.1 Elementary function No. 1.1 1.2 Elementary function No. 1.2 Elementary function No. 1.3 1.3 Elementary function No. 1.5 1.5 Menu number: 4 Code Name 4.2 Elementary function No. 4.2
Figure 4(a). initial menu-tree
-
Shoval
-
Transaction
number
Type Terminal Terminal Terminal Terminal
Transaction 2 2 3 3
number
Type Terminal
Transaction 4
number
Type Selection Terminal Quasi-terminal
Transaction
number
Type Terminal Terminal
Transaction number 2 3
1
step 1
Menu number: 0 Code Name General function No. 1 1 Elementary function No. 2 2 General function No. 4 4 Menu number: 1 Name Code United lines 1.1 & 1.2 I.112 United lines 1.3 & 1.5 I.315
Figure 4(b). Initial menu-tree
Type Selection Terminal Selection
1 4
step 2 79
with this interface and improve it. For example, they can embed (small) menus within their parent menus (i.e., a menu replaces its parent ‘selection’ line), thereby controlling the depth and breadth of the menu-tree. They can also change the numbers and titles of menu lines to elucidate the meaning of the ‘selection’ and ‘trigger’ lines, and they can add ‘help’ lines, browsing options, etc. The above options for improving a menu-tree are possible with any design method, automatic or manual. Therefore, we are not concerned with them in this study. Our purpose is to find out if the initial, automatically-produced menu-tree is a good start for improvements. If so, it has the advantages of being a by-product of the system analysis stage, and by not relying on HCI (human-computer interaction) experts, it saves human-designer effort, and may avoid errors.
The manual method In the manual (or conventional) method, a human designer initiates and conducts the design and determines the structure and contents of the menus. To create the design, we have to assume the system has already been analysed and its objectives and functions (transactions) have been specified. Otherwise, we cannot expect a worthwhile interface. This method is unstructured, as no specific, algorithmic guidelines are given to the designer. The designer is assumed to know the principles and guidelines for interface design, as provided by the human-machine interfaces and the HCI literature. (Examples for such principles are simplicity, consistency, familiarity, informativeness, and flexibility. For more detail see Gould and Lewis (1985), Hartson and Hix (1989), Lustman et al. (1985), Morland (1983), Rissland (1984), Shneiderman (1987), Smith and Mosier (1986), and Thimbleby (1984). Overall, the design has to satisfy the users who should easily find what they are looking for. Therefore, the human designers apply principles and guidelines for interface design while reviewing the descriptions of the system, its objectives, users, and functions. The result is a ‘manual’ menu-tree. As with the alternative automated method, the menu-tree can (and should) subsequently be improved by interaction with the user, but this stage is not examined here. The question is which of the two design methods provides better results before user improvements? Specifically, we want to find out if the automatic method, which is based on formal rules of functional decomposition, and is a by-product of the analysis stage, can be a basis for menu design. To provide a reliable answer to the above questions, one has to define criteria for creating both a good interface and for comparing and measuring differences between interfaces. This task is not easy, as there is no ‘gold standard’ for an interface. There are different, sometimes conflicting criteria, and the same interface may be good for certain users and bad for others. This raises an additional question: who should assess and decide which is better? Should it be professional designers, managers, or users? These philosophical questions are beyond the scope of this paper. We have restricted ourselves to a comparison between the two methods by means of a controlled experiment in which potential users of information systems evaluate and score alternative menu-tree interfaces that 80
Interacting
with Computers
vol 7 no I (199.5)
have been designed difference: Ho: u(automatic) The alternative
by means of the two methods.
there is no
= u(manua1)
hypothesis
Hi: p(automatic)
We hypothesize
-
is that they are different:
l$manual)
=
1d 1
The experiment The experiment involved two stages. First, several information systems were analysed and menu-tree interfaces were designed by the two methods. These interfaces were then given to potential users for comparative evaluation. Stage 1: interface design The experiment was based on six information system development projects. The analysis of each system was conducted before the experiment by a pair of students as part of their graduation project of the Information Systems Program at the university. Each project was supervised by an instructor and assistant, and approved by its client. (The analysts did not take part in the experiment.) The product of the analysis stage of each system was a set of hierarchical DFDs. The DFDs have been checked and approved by the clients’ representatives and the instructors. The six information systems include: l l l l l l
management management management management management management
of of of of of of
a video-movies store (20); a small transportation company a dental clinic (15); a students’ club (15); students’ dormitories (21); a mini-market (20).
(18);
The number of transactions in each system is given in parentheses to indicate their size. The systems range from small to medium (but are by no means trivial). The implication of system size with respect to validity of results will be discussed later. For each of the six systems, an initial menu-tree was derived automatically from its DFDs according to the ADISSA methodology, as described earlier. The only changes we made were in the menu line numbers (we gave sequential ID numbers instead of the original function numbers or concatenation of numbers) and in some of their labels, in cases where the original function labels were abbreviated or when duplicate lines were united (when it was necessary to specify a new label for a united line). Recall that another product of ADISSA methodology is the definition and description of the system transactions, and each trigger line in the menus triggers a certain transaction. The transactions of each system were labelled in accordance with their respective triggering lines. In order that manual menu-trees for the same systems be designed, it was necessary to provide appropriate descriptions of those systems. We did not Shoval
81
provide the DFDs because the DFD format might bias the manual results. Instead, for each system we prepared a narrative description of its objectives, users, and tasks, and a description of each of its transactions, including the inputs, outputs, database retrievals and updates, and process logic. (These descriptions were obtained from the system design stage.) Each transaction was given an arbitrary identification number and a label, identical to the label of the respective trigger line in the automatic method. These procedures guaranteed that the designers obtained equivalent information on the system, only in a different format. The descriptions of each system were given to four different designers who were asked to create their own, independent menu-trees. This amounted to 24 different designers who were randomly assigned to the six systems. We preferred to have four different designs for each system so that results would not depend on a single designer. The designers were graduates of the Information Systems Program. During their course of studies in the Information Systems Program, they studied and used various types of human-machine interfaces, including menus, and principles for interface design. One major source of study on these topics was Shneiderman (1987). This does not make them HCI experts, but certainly more than naive designers. (The implication of designer expertise on the validity of results will be discussed later). The designers were instructed to review the descriptions of the system carefully and apply the principles of interface design to create appropriate menu-tree interfaces. They were instructed to make sure all the transactions of their system could be accessed from their menu-tree, and to use the labels of the transactions as labels of the respective trigger lines. (Note that the labels of selection lines are determined by the designers.) At the end of this stage we had five different menu-trees for each of the six systems one designed automatically and four manually. To avoid the possibility that an automatic menu-tree could be distinguished from the manual, all menus were prepared (edited) in a similar format: the menu lines in a set were numbered sequentially, and all menus were surrounded by line frames. Space considerations preclude showing the menu-trees of the systems. Instead, we show some of the major characteristics in Table 1. For each of the six systems and its five menu-trees we show: l l l l
the total number of menus, the maximal depth, the breadth (range of lines per menu in the tree), maximum number of child menus (twins) per parent menu.
(The different menu-trees are numbered arbitrarily; the automatic menus are marked by asterisks.) We tried to find differences between the menu-trees of each system along these dimensions. Particularly, we wanted to see if there were significant differences between the automatic and manual menu-trees with respect to the total number of menus in a set, the depth, breadth and twins. No significant differences were found between the two types of menu sets along these dimensions. 82
interacting with Computers vol 7 no 1 (2995)
Table 1. Characteristics of the menu-trees 1
Video movies store Number of menus Maximal depth Lines per menu Maximum twins
2
1* 9 3 24 3
Small transportation company
6
d
Number of menus Maximal depth Lines per menu Maximum twins
5 2 3-7 4
3
Dental clinic
t C
11 FJ d
Number of Maximal depth menus Lines per menu Maximum twins
4
Students’
C
73 24 3
club 16
:
Maximal of Number depth menus
‘d 5
The five menu-trees 2 3 9 3 24 4
9 3 2-5 4
The five menu-trees 7 8 5 3 2-7 3
11 5 2-4 3
The five menu-trees 12’ 13 63 2-5 3
83 2-5 4
The five menu-trees 17 18
of the system 4
5
6 2 3-5 5
6 2 2-8 5
of the system 9’
10
7 3 2-7 4
8 3 24 4
of the system 14
15
62 2-5 5
83 2-5 5
of the system 19
20’
52
52
42
63
53
Maximum Lines per menu twins
3-4 4
2-5 4
3-6 3
243
3-4 3
Students‘ dormitories
21
of the system 24
25
:
depth Maximal Number of menus
:
Maximum Lines per menu twins
6
Mini market
93
83
37
62
83
2-5 3
3-5 4
2-7 3
2-6 5
2-6 5
of the system 29
30*
26 :
Maximal of menus Number depth
:
Maximum Lines per menu twins
The five menu-trees 22 23’
The five menu-trees 27 28
93
52
37
36
46
2-6 3
4-6 4
3-5 4
2-7 4
2-7 3
Stage 2: subjective evaluation The evaluation of the menu-trees
was conducted by students who acted as users. The students were at the final stage of studies in the Information Systems
Shod
83
Table 2. Summary
Information system
Interface ID number
of results
Number of users who assigned this number of points to the interface 1 2 3 4 5
Average score (n=16)
4 1* 5 3 2
1 4 3 3 4
4 3 3 4 4
2 4 5 3 1
2 1 3 5 5
7 4 2 1 2
3.63 2.88 2.88 2.81 2.81
2 Small transportation company
10 9* 6 7 8
0 0 0 4 7
1 2 6 4 5
4 4 4 3 2
4 8 2 2 2
7 2 4 3 0
4.06 3.63 3.25 2.75 1.94
3 Dental clinic
13 14 11 15 12’
1 1 3 6 4
2 5 4 0 5
6 3 3 5 3
1 3 2 3 4
6 4 4 2 0
3.56 3.25 3.00 2.69 2.63
4 Students’ club
16 19 17 20’ 18
0 4 2 6 6
1 2 4 4 4
1 3 3 2 4
5 2 5 4 2
9 5 2 0 0
4.38 3.13 3.06 2.25 2.13
5 Students’ dormitories
23* 22 24 21 25
2 1 5 4 5
2 5 2 3 5
5 3 1 3 2
4 3 4 1 2
3 4 4 5 2
3.25 3.25 3.00 3.00 2.44
6 Mini market
27 28 29 30% 26
2 1 0 4 9
1 2 2 5 5
1 5 10 2 0
5 5 2 1 2
7 3 2 4 0
3.88 3.44 3.25 2.75 1.69
1 Video movies store
Program. At this stage, students are already familiar and experienced with information systems and can play the role of prospective users. The five sets of menu-trees for each of the six systems were given to 16 different users who were unfamiliar with the systems and their menus. Assignment of menu sets to users was random. (Each user evaluated the menu sets of two systems, but no two users evaluated the same pair of systems. Altogether we had 48 different evaluators and 96 evaluations). 84
Interacting
with Computers
vol 7 no 1 (1995)
5.00
3.00
2.00
I .oo
0.00
I
kiden
Store
Monwl
Transportation
Dental
Clinic
Students’
Club
e.3rmotorks
Minimrkzt
method
Figure 5. Average
scores of the menu systems
To conduct the evaluation, each user was given the five menu-trees designed for the systems (each identified by an arbitrary number) and the description of the system and its transactions (the descriptions were the same that were given to the designers in the first stage). As there was no real, working system behind the menus, it was necessary to provide the descriptions as a substitute. Each user was asked to study the system descriptions and then to review the five menu-trees and rate them on a 5-point scale, from 1 (for the worst) to 5 (for the best). The user was allowed to assign the same number of points to more than one interface. No order of evaluation was pre-defined, i.e., the order was determined randomly by the users. Also, no specific criteria were imposed on the users; only an overall evaluation from a user’s viewpoint was asked. At the end of this stage we had 16 independent evaluations of the five menu-trees of each of the six systems.
Results Table 2 summarizes the results of the evaluations. For each of the six systems it lists the five menu-trees, identified by their arbitrary numbers. The automatic menus are marked by asterisks. The next five columns indicate the 5-point scale, and the values in the cells indicate how many users assigned the respective number of points to any menu-tree. For example, in system 1 (management of a video-movies store), menu number 4 received 5 points (best) Shoval
85
from seven users, 4 points from two users, 3 points from two users, 2 points from four users, and 1 point (worst) from one user. We then computed the average score of each menu-tree as the sum of multiplications of each point on the scale by the number of users who assigned that number of points, divided by the number of evaluators (16). Thus, the average scores range from 1 to 5. The five menu-trees within each system in the table are listed in descending order of their weighted scores, i.e., from best to worst. Intuitively, we can see the automatic menus are distributed in different places in the six systems, ranging from best to worst position. The graph in Figure 5 supplements these findings by showing the score of its automatic menu and the average score of its four manual menus for each of the six systems. To test whether there is a signifcant difference between the mean scores of the automatic and manual menus, we conducted two statistical tests. One is a two-factor analysis of variance with repeated measures. The two factors were the six systems, and the two design methods. The repeated measures were the 16 scores assigned to the menus of each system. The F test of difference between the menu-trees (i.e., the methods) yielded F,i,is,=1.28 (at a=0.05 the value of F in the F-table is 4.54, a much higher value.) The power of the test for the alternative hypothesis (Hi) for 1 d ( at least 0.6 is 0.95 (p=O.OS). This means there is no significant difference between the menu sets. Similarly, the test revealed there is neither a significant difference between the systems nor a significant interaction between the factors. Hence, the null hypothesis, which claims that the automatic, ADISSA-based method will produce initial menutrees that are as good as manual menu-trees designed by human-designers cannot be rejected. To supplement these findings, we also conducted the Wilcoxon matchedpairs rank test, a non-parametric test, to find out whether the two populations (i.e., the menus of the two types) are different. The z value of the test is z= -0.41 and +(z)=O.34, which again means (at a=0.05) the hypothesis that the two populations are not different cannot be rejected.
Discussion and conclusions We found no significant difference between menu-tree interfaces designed automatically according to the ADISSA method and those designed by human designers. This result is meaningful because it supports the claim that the automatic method is valid. Noting that the automatic menu-tree is an integral component and by-product of the stage of system analysis, and that it can and should be improved by users during prototyping, we conclude that the method provides a good start for interface design. This result is not surprising since the automatic menu-tree is not created ‘out of the blue’ - it is a direct outcome of what has been invested by analysts in co-operation with users, in the stage of system analysis. This reaffirms how essential it is to invest in analysis to ensure successful completion of the development process. This is particularly true for the ADISSA methodology, which combines system analysis, design, and prototyping into a unified, integrated process and transforms the results of one stage into input for the next 86
Interacting
with Computers
ml 7 no I (1995)
stage (including the generation of a menu-tree interface). Our conclusions are based on the described experiment, which like other ‘pseudo’ laboratory experiments, has weaknesses relating to internal and external validity. The absence of a ‘result’ (i.e., no significant difference between the menu sets) does not prove that there is no difference in real situations. We cannot rule out the possibility that there are differences which are not evident because of the experimental conditions. One limitation which compromises the internal validity of the result is that the nature of the experimental task required the manual menu designers to work from narrative descriptions of the system, a more ‘primitive’ position when compared to the sets of DFDs used to generate the automatic menus. This is because, as already mentioned, providing the same DFDs to the designers would have biased the results. It should be noted, however, that the task assigned to the designers is not unrealistic: many systems are analysed/designed from narrative (even oral) descriptions of user needs - without the use of DFDs or other diagrammatic techniques. With respect to external validity, one limitation of the experiment, as with many pseudo laboratory experiments, is the size of the systems on which the results are based. It is not the same to analyse systems and design interfaces for systems with only 15-21 transactions as those having a larger number of transactions. This limitation, however, works in favour of both methods: for the automatic method, smaller systems facilitate the creation of ‘good’ DFDs (from which the interface is then derived); for the manual method, smaller systems facilitate the design of the interface. Although smallness is advantageous to both methods, we cannot predict whether the results would differ for larger systems without further experimentation. Another limitation of the experiment is that analysis and design, as well as evaluation, were conducted by students. Even though our students were at the stage of graduating from the Information Systems Program, and the systems have been analysed under supervision and approved by the clients, we cannot say they are experts. But expertise and experience are influential in both methods. For the automatic method, more experienced analysts would produce better DFDs in the analysis stage, thus producing better automatic menu-trees. For the manual method, it may be assumed that more experienced designers would do a better job at interface design. The effect of experience can only be determined by future studies. Concerning the evaluation of the interfaces, we confined ourselves to evaluation by potential users. Although HCI experts may be better evaluators than users, the disadvantage of using students as users is not substantial since the same participants evaluated all interfaces, and there is no reason to assume bias. Moreover, in practical situations it is more likely that users rather than experts will evaluate their information systems/interfaces. The main limitation is that our participants were not real users of real systems. If they were, they would probably provide more reliable evaluations. We are convinced, however, that the evaluations provided by the students are reliable. Another deficiency of the subjective evaluation is that it was based on a single, general measure (overall evaluation). The use of a single measure does not offer insights on the Shoval
87
strengths and weaknesses of the design. This limitation can be rectified in further research that will address specific evaluation criteria of the menu-trees. One aspect of the user interface design that has not been addressed is the quality and variation of (human) designers. The literature indicates that HCI experts design differently than novices. We used students as designers, and controlled the variables of designer-quality by randomly assigning the 24 designers to the design tasks. We did not compare the automatic design to the performance of HCI experts and more experienced designers. We have no doubt that experts can do better, but experts are not always available, and our intent was to examine if the automatic method can provide a good initial design, that can be further improved. Comparison of the automatic method to the performance of experts is left for further experiments. Another issue is the effect of the quality of DFDs on automatic interfaces. For each project we had just one set of DFDs, hence one automatic menu-tree, as opposed to four different designer-made interfaces. It is reasonable to assume that different analysts will produce different DFDs for the same system. Thus, different (initial) menu-trees would be derived. We could not address this problem in the current experiment because different DFDs could also yield different transactions and systems, and therefore there would not have been a common basis for comparing the two interface design methods. We have compensated for the limitation of a single set of DFDs per system by conducting the experiment with six different systems (recall that the two-factor ANOVA test revealed no significant differences between the systems). Still, the quality of DFDs as a tool for systems analysis has not been examined. This is definitely an issue for separate study. It is worth mentioning that criticism on weaknesses of the functional approach (on which DFDs are based) for certain kinds of systems, led to the development of object-oriented (O-O) analysis and design methods. There are many such methods, but - as mentioned in the introduction to this paper - they hardly treat interface design. However, as the O-O methods develop, it is likely that they will treat interface design differently than the manual or automatic methods. It is our plan to incorporate interface design within O-O analysis and design, and conduct additional experiments with the interface design methods.
Acknowledgments I thank Arad Friedman for This research Research and
Kopp for assisting me in conducting the experiment, and Dr Lea helping in the statistical analysis of the results. is partially supported by the Paul Ivanier Center for Robotics Production Management, Ben-Gurion University of the Negev.
References Betts, B., Borlingane, D., Fischer, D., Foley, J., Green, M., Karis, D., Kerr, S., Olsen, D. and Thomas, J. (1987) ‘Goals and objectives for user-interface software’ Comput. Graph. 21, 73-78 Coad, P. and Yourdon,
E. (1991) Object-Oriented
Design Yourdon Press, Prentice-Hall
Draper, S. and Norman, D. (1985) ‘Software engineering 88
for user interfaces’. IEEE Trans.
Interacting with Computers vol
7 no 1 (199.5)
Softw.
Eng. SE-11, 252-258
Fischer, G. (1989) ‘Human-computer ahead’. 1EEE Softw. 6, 44-52
interaction
Gould, J. and Lewis, C. (1985) ‘Designing designers think’ Comm ACM 28, 30&311
software:
for usability
Grudin, J. (1989) ‘The case against user interface consistency’ Hartson, H. and Hix, D. (1989) ‘Human-computer systems’ ACM Computing Surv. 21, 592
lessons
learned,
- key principles
challenges and what
Comm. ACM 32,1164-1173
interface development:
concepts and
Hix, D. and Hartson, H. (1986) ‘An interactive environment for dialogue development: its design, use and evaluation’ in Proc. CH1’86: Conf. Human Factors in Computing Systems ACM Press, 228-234 Kiger, J. (1984) ‘The depth/breadth lnt. J. Man-Mach. Stud. 20
trade-off in the design of men driven user interfaces’.
Lustman, F., Mercier, P. and Gratton, P. (1985) ‘A dialogue-based interactive information systems’ Data Base 16, l&24 Martin, J. and Odell, J. (1992) Object-Oriented Mohhamad, U. and Wayne, developing user interfaces‘
architecture
Analysis and Design Prentice-Hall
D. (1988) ‘A survey of formal tools and methods lnt. 1. Man-Mach. Stud. 29, 479-496
Morland, V. (1983) ‘Human factors guidelines 26, 384-494
for
for
for terminal interface design’ Comm. ACM
Norman, D. and Draper, S. teds) (1986) User Centered System Design Lawrence Erlbaum Associates Rissland,
E. (1984) ‘Ingredients
of intelligent
user interfaces’
lnt. J. Man-Mach.
Rumba@, J., Blaha, M., Premerlani, W., Eddy, F. and Lorensen, Oriented Modeling and Design Prentice-Hall Shlaer, S. and Mellor, Press, Prentice-Hall
S. (1992) Object Lifecycles:
Modeling
Shneiderman, B. (1987) Designing the User Interface Computer interaction Addison-Wesley Shoval, P. (1988) ‘ADISSA: architectural design structured analysis’ info. Sys. 13, 19%210
Stud. 21
W. (1991) Object-
the World in States Yourdon
Strategies
of information
for
Effective systems
Humanbased
Shoval, P. (1990) ‘Functional design of a menu-tree interface within structured development’ lnt. 1. Man-Mach. Stud. 33, 537-556
on
system
Shoval, P. and Manor, 0. (1990) ‘Software engineering tools supporting ADISSA methodology for systems analysis and design’ lnfor. Softw. Technol. 32, 357-360 Smith, S. and Mosier, J. (1986) ‘Guidelines for designing user interface software’ Report 7 MTR-20090, Eds-Tr-86-278 Mitre Corp, Bedford, MA, USA Thimbleby, H. (1984) ‘User interface design: generative user engineering principles’ in Monk, A. (ed) Fundamentals of Human-Computer Interaction Academic Press, 165-180 Wasserman, A., Pircher, P., Shewmake, D. and Kersten, M. (1986) ‘Developing interactive information systems with the user software engineering methodology’ 1EEE Trans. Softw. Eng. SE-12, 326-345 Wirfs-Brock, R., Wilkerson, wure, Prentice-Hall
B. and Wiener,
L. (1990) Designing
Object-Oriented
Soft-
Yound, R. and Hull, A. (1982) ‘Cognitive aspects of the selection of viewdata options by casual users’ in Pathways to the Information Society: Proc. 6th Int. Conf. Computer Communications 571-576 89
interacting with Computers vol 7 no 1 (1995)
with Computers
vol 7 no 2 (1995) 73-89
Experimental comparison between automatic and manual menu interface design methods Peretz Shoval
An experimental comparison between two methods of designing a menu-tree interface for an information system is described. The ‘automatic’ method is based on the ADISSA methodology (architectural design of information systems based on structured analysis) according to which the interface is derived automatically from dataflow diagrams (DFD) as a by-product of the system analysis stage. In the ‘manual’ (‘conventional’) method the designer constructs an appropriate interface by applying common principles of interface design. The objective of the experiment is to find out which design method yields a better initial interface, before it is given to users for further improvements. The user’s viewpoint is adopted i.e., the interface was evaluated subjectively. The hypothesis is that menutrees designed automatically are as good as those designed by people. The experiment included six information system development projects. One menu-tree interface was designed automatically for each system. Four others were designed manually by different designers who were assigned randomly to the systems. The five different interfaces of each system were given to 16 potential users for evaluation and comparison. Analysis of the results revealed no significant difference between the mean scores of the two types of interface. Noting that the automatic menus are obtained as a by-product of the system analysis stage, and that they can be improved by users during prototyping, it is concluded that the method provides a good start for the interface design. Keywords: design
human-computer
interaction,
system analysis, interface
Introduction The design of a human-computer
interface is an important
Information Systems Program, Department of Industrial 84105, Gurion University of the Negev, Beer-Sheva [email protected] Paper submitted:
0953-5438/95
17 May 1994; accepted:
Elsevier Science B.V.
30 November
Engineering Israel. Fax:
activity in informaand Management, Ben972 7 280776. E-mail:
1994
73
tion systems development, and a great deal has been written on this subject. Many studies concentrate on the attributes of ‘good’ interfaces and ‘humanengineering’ aspects, e.g., user friendliness, learnability, simplicity, consistency, flexibility to changes, etc. Some literature suggests guidelines and rules for designing the interface, e.g., identifying the user’s needs and the system’s tasks, and ways of grouping tasks before the design begins (see, for example, Gould and Lewis, 1985; Hartson and Hix, 1989; Kiger, 1984; Lustman et al., 1985; Morland, 1983; Rissland, 1984; Shneiderman, 1987; Smith and Mosier, 1986; Thimbleby, 1984; Yound and Hull, 1982). It has been well established that the user interface is a major factor in determining user satisfaction, and that interface design activity must centre on users and their tasks (Grudin, 1989). Moreover, for the interface to fit the users’ expectations and needs, they must be involved in the system analysis and design process (Gould and Lewis, 1985; Mohammad and Wayne, 1988; Norman and Draper, 1986; Thimbleby, 1984). In this light, it is surprising that much of the research on interface design is done without the framework of the whole information system development process. Although the need to view user interface design as an integral part of system design has been acknowledged (Betta et al., 1987; Draper and Norman, 1985; Hartson and Hix, 1989) most of the prevailing analysis and design methodologies totally ingore interface design. According to Hartson and Hix (1989), only two methodologies deal with interface design as part of system design: USE (Wasserman et al., 1986) and DMS (Hix and Hartson, 1986). Even the more recent, object-oriented methodologies, hardly deal with interface design, including popular methodologies such as those described by Martin and Ode11 (1992), Rumbaugh et al. (1991), Shlaer and Mellor (1992), Wirfs-Brock et al. (1990) and others. One exception is Coad and Yourdon (1991), who deal with the “human-interaction component (HIC)“, and propose a strategy to design this component. Their strategy consists mainly of classifying the humans (users), designing the command hierarchy, and designing the HIC classes. To design the command hierarchy they propose: l l
l
to study the existing user interaction metaphors and guidelines; to establish an initial command hierarchy consisting of the system ‘services’ (methods) presented on menu-screens; to refine the command hierarchy (including ordering of the services within each part of the hierarchy, creating whole/part chunks, and considering breadth versus depth, with the objective to avoid overloading user memory and minimize the number of steps.
Another, process oriented methodology, which includes interface design and provides a structured procedure to derive a menu-tree user interface from dataflow diagrams is ADISSA (Shoval, 1988; 1990). A menu-tree interface presents its users with options to choose from and tasks the system can perform. It consists of a hierarchy of menus, where a menu includes several lines (items). A line can be a ‘selection’ type if it presents a lower-level (child) menu from 74
Interacting
with Computers
vol 7 no 1 (2995)
which the user can select more specific lines. Alternatively, it can be a ‘trigger’ type if it activates a transactions, i.e., a system process that performs a task. (Other kinds of menu options, which are not related to the functionality of the system, e.g., help, are not considered here.) A user interacts with the menu-tree interface in a top-down fashion. Beginning at the root-menu, he or she browses the menus and searches down a certain path until a desired transaction is found and activated. At this stage the user leaves the interface and enters the transaction. i.e., the actual application. A specific transaction may (and usually does) involve further interaction with the user, say when the user is expected to input data, or get information from the system, but this type of interaction is not of concern in this study. Our concern is with designing the system interface that will enable its users to find what they need to do with the system. The focus of this study is on the functional aspects of interface design. We are not concerned with technical aspects such as screen layout and formatting, windowing techniques, graphics, use of colour, response time, and other system considerations. Our concern is with the contents and structure of menus - considering such questions as: l l l
l l
Which menu lines will comprise the interface? Which lines will be grouped in separate menus? What is a proper hierarchy of menus to enable objectives? How many levels will there be in the tree? How many lines will be within the menus?
users
to realize
their
In essence, we are concerned with the extent to which a menu-tree interface reflects the information system and its functions, as perceived and preferred by the users. The creation of an interface (including a menu-tree) is an engineering problem that forces trade-offs among many factors (Grudin, 1989). As already noted, there are many principles and guidelines for constructing menus. For example, there are guidelines about the depth of a menu’s hierarchy and the breadth (i.e., the number of options) of a menu. These guidelines suggest that neither factor should typically be greater or less than a certain number. As with many other design problems, however, there are different ways to carry out the task. Much depends on the designer’s knowledge, creativity, imagination, experience, and skill. Hence, the two problems that concern us are firstly, why interface design is not done as part of system analysis and design; and secondly why it is done intuitively in an unstructured manner. A remedy may be found if the process of interface design becomes an integral part of a structure system analysis and design methodology, which will also enable generation of the menus in a well-structured fashion. This approach has been adopted in the ADISSA methodology for analysis, design, and prototyping. According to this CASE-supported methodology, the menu-tree of a system is derived automatically (i.e. algorithmically) from the set of hierarchical dataflow diagrams (DFDs) that are constructed during system analysis. The methodology and its supporting tools, including the method for automatic menu-tree design, have already Shoval
75
been described in Shoval(l990) and Shoval and Manor (1990). In this study we compare the automatic method with an alternative, unstructured method, which we term ‘manual’, as it relies on the human designer. Any menu-tree, whether designed manually by a human designer or automatically by following an algorithm, can and should be given to users to experiment with and improve. In this study we do not deal with such interactive improvements. We are concerned with the initial design produced by either method, before it is improved by the users. The objective is to find out if the automatic method, which is incorporated as an integral part of the whole system analysis and design process, can provide an initial menu-tree interface which would be a good start for further improvements. In the following section we describe the two design methods. In the third section we describe the experiment we conducted to evaluate the resulting menu-trees. The section that follows presents the results, and in the fifth section we discuss the implications of the results, considering the strengths and weaknesses of the experiment, and suggest further research.
Two methods of menu-tree design The automatic method The automatic method is part of ADISSA which is a CASE-supported methodology for systems analysis, design, and prototyping (Shoval, 1988; Shoval and Manor, 1990). The methodology supports the following stages: l
l
l l l
l
functional analysis in which the system analyst produces a hierarchy of enhanced DFDs; transactions (processes) design in which transaction diagrams and their structure descriptions are created using structured programming techniques (e.g. pseudo-code); interface design, where a menu-tree is generated (as will be detailed); database design, which provides a normalized database schema; input/output design, which results in layouts of input and output screens and reports; prototyping, which produces a structured-prototype of the system that is based on the earlier products.
ADISSA provides a functional menu-tree design method according to which the interface is derived from hierarchical DFDs which are produced at the functional analysis stage. Functional analysis is a comprehensive process, involving analysts and users. In an iterative way, users and analysts identify the functions of the system and their subfunctions, and group them in a meaningful way. The final DFDs are approved by users and management as per goals, requirements and semantic design. Hence, the hierarchical DFDs, which are the basis for the system structure and for interface design, actually portray the semantic organization that users have, i.e., they reprsent the user model of the system. Thus, the menu-tree that is derived from the DFDs reflects the functional structure of 76
Interacting
with Computers
vol 7 no 1 (1995)
the system as perceived by the users who take part in the analysis. The method follows Betts et al. (1987), Grudin (1989), Shneiderman (1987) and others, who claim that the interface must be designed according to the user’s tasks. Moreover, the ADISSA approach supports Fischer’s (1989) observation that: “Users do not know what they want, and designers do not understand what users need or will accept. The only viable strategy. . . is incremental, evolutionary development: initial systems must be built to give users something concrete to react to.” Similarly, ADISSA produces an initial menu-tree automatically from the DFDs that can subsequently be improved by its users. The toolsupported method is described herein (see further details in Shoval, 1990). The initial menu-tree is created as follows. The DFD tree is examined by a search for functions that are linked to user entities, because they signify data input or output. For each DFD having such links, a menu is defined, which includes a menu-line for each function linked to a user entity. (At this stage the lines are identified by the numbers and labels of the respective functions.) A line is considered as a ‘selection’ if it originates from a general function, i.e., a function that is decomposed and described in a separate child DFD. Upon its selection at run-time, the interface will present a child menu. It is considered as a ‘trigger’ if it originates from, or terminates on, an elementary function. Upon its selection it will trigger the transaction to which the respective function belongs. (A transaction is an independent task or activity the system can perform. In the DFD, a transaction consists of one or more elementary functions which are linked to each other via dataflows, and of other DFD elements that are linked to those functions, i.e. data stores and external entities. A transaction has to include at least one external entity, to enable its triggering.) The result of this step is a tree of menus that is a subset of the tree of DFDs. In the next step, a pass is made through the menus to eliminate ‘degenerate’ menus (a degenerate menu includes one line only, offering no selections, and has been created because the DFD from which it originates includes only one function linked to the user-entity), and to unite ‘trigger’ lines that belong to the same transaction (such duplicate lines have been created if more than one function belonging to the same transaction is linked to a user entity). Since a trigger line activates the whole transaction to which it belongs, there is no need for duplicate triggers. (The identification number of the united line is a concatenation of the original numbers; a new label for the united line has to be specified by the analyst or user.) An example that explains the method (taken from Shoval, 1990) follows. Figures l-3 show three schematic DFDs of a system. Note that the two general functions 1 and 4 in DFD-0 are decomposed and described in separate DFDs. Figure 4 shows the initial menu-tree that is derived from those DFDs in two steps: in Figure 4 (a) we see the initial menu-tree after step 1. Each of its three menus consists of lines that correspond to general or elementary functions connected to user entities. Note the multiple lines for the same transaction in menu number 1 (lines 1.1 and 1.2, as well as lines 1.3 and 1.5), and also a ‘degenerate’ menu number 4. In Figure 4 (b) we see the menu-tree after line unification (lines 1.1/2 and 1.3/5) and elimination of the degenerate menu. This terminates the stage of initial menu-tree design. Later users can interact Shoval
77
User entity 1
Figure 1. DFD-0 I
user entity 1
-I
User entity 1
Figure 2. DFD-1 78
Interacting
with Computers zwl 7 no 2 (1995)
!,I User
nl
entity 1
Figure 3. DFD-4
Menu number: 0 Name Code General function No. 1 1 Elementatv function No. 2 2 General fhction No. 4 4 Menu number: 1 Name Code 1.1 Elementary function No. 1.1 1.2 Elementary function No. 1.2 Elementary function No. 1.3 1.3 Elementary function No. 1.5 1.5 Menu number: 4 Code Name 4.2 Elementary function No. 4.2
Figure 4(a). initial menu-tree
-
Shoval
-
Transaction
number
Type Terminal Terminal Terminal Terminal
Transaction 2 2 3 3
number
Type Terminal
Transaction 4
number
Type Selection Terminal Quasi-terminal
Transaction
number
Type Terminal Terminal
Transaction number 2 3
1
step 1
Menu number: 0 Code Name General function No. 1 1 Elementary function No. 2 2 General function No. 4 4 Menu number: 1 Name Code United lines 1.1 & 1.2 I.112 United lines 1.3 & 1.5 I.315
Figure 4(b). Initial menu-tree
Type Selection Terminal Selection
1 4
step 2 79
with this interface and improve it. For example, they can embed (small) menus within their parent menus (i.e., a menu replaces its parent ‘selection’ line), thereby controlling the depth and breadth of the menu-tree. They can also change the numbers and titles of menu lines to elucidate the meaning of the ‘selection’ and ‘trigger’ lines, and they can add ‘help’ lines, browsing options, etc. The above options for improving a menu-tree are possible with any design method, automatic or manual. Therefore, we are not concerned with them in this study. Our purpose is to find out if the initial, automatically-produced menu-tree is a good start for improvements. If so, it has the advantages of being a by-product of the system analysis stage, and by not relying on HCI (human-computer interaction) experts, it saves human-designer effort, and may avoid errors.
The manual method In the manual (or conventional) method, a human designer initiates and conducts the design and determines the structure and contents of the menus. To create the design, we have to assume the system has already been analysed and its objectives and functions (transactions) have been specified. Otherwise, we cannot expect a worthwhile interface. This method is unstructured, as no specific, algorithmic guidelines are given to the designer. The designer is assumed to know the principles and guidelines for interface design, as provided by the human-machine interfaces and the HCI literature. (Examples for such principles are simplicity, consistency, familiarity, informativeness, and flexibility. For more detail see Gould and Lewis (1985), Hartson and Hix (1989), Lustman et al. (1985), Morland (1983), Rissland (1984), Shneiderman (1987), Smith and Mosier (1986), and Thimbleby (1984). Overall, the design has to satisfy the users who should easily find what they are looking for. Therefore, the human designers apply principles and guidelines for interface design while reviewing the descriptions of the system, its objectives, users, and functions. The result is a ‘manual’ menu-tree. As with the alternative automated method, the menu-tree can (and should) subsequently be improved by interaction with the user, but this stage is not examined here. The question is which of the two design methods provides better results before user improvements? Specifically, we want to find out if the automatic method, which is based on formal rules of functional decomposition, and is a by-product of the analysis stage, can be a basis for menu design. To provide a reliable answer to the above questions, one has to define criteria for creating both a good interface and for comparing and measuring differences between interfaces. This task is not easy, as there is no ‘gold standard’ for an interface. There are different, sometimes conflicting criteria, and the same interface may be good for certain users and bad for others. This raises an additional question: who should assess and decide which is better? Should it be professional designers, managers, or users? These philosophical questions are beyond the scope of this paper. We have restricted ourselves to a comparison between the two methods by means of a controlled experiment in which potential users of information systems evaluate and score alternative menu-tree interfaces that 80
Interacting
with Computers
vol 7 no I (199.5)
have been designed difference: Ho: u(automatic) The alternative
by means of the two methods.
there is no
= u(manua1)
hypothesis
Hi: p(automatic)
We hypothesize
-
is that they are different:
l$manual)
=
1d 1
The experiment The experiment involved two stages. First, several information systems were analysed and menu-tree interfaces were designed by the two methods. These interfaces were then given to potential users for comparative evaluation. Stage 1: interface design The experiment was based on six information system development projects. The analysis of each system was conducted before the experiment by a pair of students as part of their graduation project of the Information Systems Program at the university. Each project was supervised by an instructor and assistant, and approved by its client. (The analysts did not take part in the experiment.) The product of the analysis stage of each system was a set of hierarchical DFDs. The DFDs have been checked and approved by the clients’ representatives and the instructors. The six information systems include: l l l l l l
management management management management management management
of of of of of of
a video-movies store (20); a small transportation company a dental clinic (15); a students’ club (15); students’ dormitories (21); a mini-market (20).
(18);
The number of transactions in each system is given in parentheses to indicate their size. The systems range from small to medium (but are by no means trivial). The implication of system size with respect to validity of results will be discussed later. For each of the six systems, an initial menu-tree was derived automatically from its DFDs according to the ADISSA methodology, as described earlier. The only changes we made were in the menu line numbers (we gave sequential ID numbers instead of the original function numbers or concatenation of numbers) and in some of their labels, in cases where the original function labels were abbreviated or when duplicate lines were united (when it was necessary to specify a new label for a united line). Recall that another product of ADISSA methodology is the definition and description of the system transactions, and each trigger line in the menus triggers a certain transaction. The transactions of each system were labelled in accordance with their respective triggering lines. In order that manual menu-trees for the same systems be designed, it was necessary to provide appropriate descriptions of those systems. We did not Shoval
81
provide the DFDs because the DFD format might bias the manual results. Instead, for each system we prepared a narrative description of its objectives, users, and tasks, and a description of each of its transactions, including the inputs, outputs, database retrievals and updates, and process logic. (These descriptions were obtained from the system design stage.) Each transaction was given an arbitrary identification number and a label, identical to the label of the respective trigger line in the automatic method. These procedures guaranteed that the designers obtained equivalent information on the system, only in a different format. The descriptions of each system were given to four different designers who were asked to create their own, independent menu-trees. This amounted to 24 different designers who were randomly assigned to the six systems. We preferred to have four different designs for each system so that results would not depend on a single designer. The designers were graduates of the Information Systems Program. During their course of studies in the Information Systems Program, they studied and used various types of human-machine interfaces, including menus, and principles for interface design. One major source of study on these topics was Shneiderman (1987). This does not make them HCI experts, but certainly more than naive designers. (The implication of designer expertise on the validity of results will be discussed later). The designers were instructed to review the descriptions of the system carefully and apply the principles of interface design to create appropriate menu-tree interfaces. They were instructed to make sure all the transactions of their system could be accessed from their menu-tree, and to use the labels of the transactions as labels of the respective trigger lines. (Note that the labels of selection lines are determined by the designers.) At the end of this stage we had five different menu-trees for each of the six systems one designed automatically and four manually. To avoid the possibility that an automatic menu-tree could be distinguished from the manual, all menus were prepared (edited) in a similar format: the menu lines in a set were numbered sequentially, and all menus were surrounded by line frames. Space considerations preclude showing the menu-trees of the systems. Instead, we show some of the major characteristics in Table 1. For each of the six systems and its five menu-trees we show: l l l l
the total number of menus, the maximal depth, the breadth (range of lines per menu in the tree), maximum number of child menus (twins) per parent menu.
(The different menu-trees are numbered arbitrarily; the automatic menus are marked by asterisks.) We tried to find differences between the menu-trees of each system along these dimensions. Particularly, we wanted to see if there were significant differences between the automatic and manual menu-trees with respect to the total number of menus in a set, the depth, breadth and twins. No significant differences were found between the two types of menu sets along these dimensions. 82
interacting with Computers vol 7 no 1 (2995)
Table 1. Characteristics of the menu-trees 1
Video movies store Number of menus Maximal depth Lines per menu Maximum twins
2
1* 9 3 24 3
Small transportation company
6
d
Number of menus Maximal depth Lines per menu Maximum twins
5 2 3-7 4
3
Dental clinic
t C
11 FJ d
Number of Maximal depth menus Lines per menu Maximum twins
4
Students’
C
73 24 3
club 16
:
Maximal of Number depth menus
‘d 5
The five menu-trees 2 3 9 3 24 4
9 3 2-5 4
The five menu-trees 7 8 5 3 2-7 3
11 5 2-4 3
The five menu-trees 12’ 13 63 2-5 3
83 2-5 4
The five menu-trees 17 18
of the system 4
5
6 2 3-5 5
6 2 2-8 5
of the system 9’
10
7 3 2-7 4
8 3 24 4
of the system 14
15
62 2-5 5
83 2-5 5
of the system 19
20’
52
52
42
63
53
Maximum Lines per menu twins
3-4 4
2-5 4
3-6 3
243
3-4 3
Students‘ dormitories
21
of the system 24
25
:
depth Maximal Number of menus
:
Maximum Lines per menu twins
6
Mini market
93
83
37
62
83
2-5 3
3-5 4
2-7 3
2-6 5
2-6 5
of the system 29
30*
26 :
Maximal of menus Number depth
:
Maximum Lines per menu twins
The five menu-trees 22 23’
The five menu-trees 27 28
93
52
37
36
46
2-6 3
4-6 4
3-5 4
2-7 4
2-7 3
Stage 2: subjective evaluation The evaluation of the menu-trees
was conducted by students who acted as users. The students were at the final stage of studies in the Information Systems
Shod
83
Table 2. Summary
Information system
Interface ID number
of results
Number of users who assigned this number of points to the interface 1 2 3 4 5
Average score (n=16)
4 1* 5 3 2
1 4 3 3 4
4 3 3 4 4
2 4 5 3 1
2 1 3 5 5
7 4 2 1 2
3.63 2.88 2.88 2.81 2.81
2 Small transportation company
10 9* 6 7 8
0 0 0 4 7
1 2 6 4 5
4 4 4 3 2
4 8 2 2 2
7 2 4 3 0
4.06 3.63 3.25 2.75 1.94
3 Dental clinic
13 14 11 15 12’
1 1 3 6 4
2 5 4 0 5
6 3 3 5 3
1 3 2 3 4
6 4 4 2 0
3.56 3.25 3.00 2.69 2.63
4 Students’ club
16 19 17 20’ 18
0 4 2 6 6
1 2 4 4 4
1 3 3 2 4
5 2 5 4 2
9 5 2 0 0
4.38 3.13 3.06 2.25 2.13
5 Students’ dormitories
23* 22 24 21 25
2 1 5 4 5
2 5 2 3 5
5 3 1 3 2
4 3 4 1 2
3 4 4 5 2
3.25 3.25 3.00 3.00 2.44
6 Mini market
27 28 29 30% 26
2 1 0 4 9
1 2 2 5 5
1 5 10 2 0
5 5 2 1 2
7 3 2 4 0
3.88 3.44 3.25 2.75 1.69
1 Video movies store
Program. At this stage, students are already familiar and experienced with information systems and can play the role of prospective users. The five sets of menu-trees for each of the six systems were given to 16 different users who were unfamiliar with the systems and their menus. Assignment of menu sets to users was random. (Each user evaluated the menu sets of two systems, but no two users evaluated the same pair of systems. Altogether we had 48 different evaluators and 96 evaluations). 84
Interacting
with Computers
vol 7 no 1 (1995)
5.00
3.00
2.00
I .oo
0.00
I
kiden
Store
Monwl
Transportation
Dental
Clinic
Students’
Club
e.3rmotorks
Minimrkzt
method
Figure 5. Average
scores of the menu systems
To conduct the evaluation, each user was given the five menu-trees designed for the systems (each identified by an arbitrary number) and the description of the system and its transactions (the descriptions were the same that were given to the designers in the first stage). As there was no real, working system behind the menus, it was necessary to provide the descriptions as a substitute. Each user was asked to study the system descriptions and then to review the five menu-trees and rate them on a 5-point scale, from 1 (for the worst) to 5 (for the best). The user was allowed to assign the same number of points to more than one interface. No order of evaluation was pre-defined, i.e., the order was determined randomly by the users. Also, no specific criteria were imposed on the users; only an overall evaluation from a user’s viewpoint was asked. At the end of this stage we had 16 independent evaluations of the five menu-trees of each of the six systems.
Results Table 2 summarizes the results of the evaluations. For each of the six systems it lists the five menu-trees, identified by their arbitrary numbers. The automatic menus are marked by asterisks. The next five columns indicate the 5-point scale, and the values in the cells indicate how many users assigned the respective number of points to any menu-tree. For example, in system 1 (management of a video-movies store), menu number 4 received 5 points (best) Shoval
85
from seven users, 4 points from two users, 3 points from two users, 2 points from four users, and 1 point (worst) from one user. We then computed the average score of each menu-tree as the sum of multiplications of each point on the scale by the number of users who assigned that number of points, divided by the number of evaluators (16). Thus, the average scores range from 1 to 5. The five menu-trees within each system in the table are listed in descending order of their weighted scores, i.e., from best to worst. Intuitively, we can see the automatic menus are distributed in different places in the six systems, ranging from best to worst position. The graph in Figure 5 supplements these findings by showing the score of its automatic menu and the average score of its four manual menus for each of the six systems. To test whether there is a signifcant difference between the mean scores of the automatic and manual menus, we conducted two statistical tests. One is a two-factor analysis of variance with repeated measures. The two factors were the six systems, and the two design methods. The repeated measures were the 16 scores assigned to the menus of each system. The F test of difference between the menu-trees (i.e., the methods) yielded F,i,is,=1.28 (at a=0.05 the value of F in the F-table is 4.54, a much higher value.) The power of the test for the alternative hypothesis (Hi) for 1 d ( at least 0.6 is 0.95 (p=O.OS). This means there is no significant difference between the menu sets. Similarly, the test revealed there is neither a significant difference between the systems nor a significant interaction between the factors. Hence, the null hypothesis, which claims that the automatic, ADISSA-based method will produce initial menutrees that are as good as manual menu-trees designed by human-designers cannot be rejected. To supplement these findings, we also conducted the Wilcoxon matchedpairs rank test, a non-parametric test, to find out whether the two populations (i.e., the menus of the two types) are different. The z value of the test is z= -0.41 and +(z)=O.34, which again means (at a=0.05) the hypothesis that the two populations are not different cannot be rejected.
Discussion and conclusions We found no significant difference between menu-tree interfaces designed automatically according to the ADISSA method and those designed by human designers. This result is meaningful because it supports the claim that the automatic method is valid. Noting that the automatic menu-tree is an integral component and by-product of the stage of system analysis, and that it can and should be improved by users during prototyping, we conclude that the method provides a good start for interface design. This result is not surprising since the automatic menu-tree is not created ‘out of the blue’ - it is a direct outcome of what has been invested by analysts in co-operation with users, in the stage of system analysis. This reaffirms how essential it is to invest in analysis to ensure successful completion of the development process. This is particularly true for the ADISSA methodology, which combines system analysis, design, and prototyping into a unified, integrated process and transforms the results of one stage into input for the next 86
Interacting
with Computers
ml 7 no I (1995)
stage (including the generation of a menu-tree interface). Our conclusions are based on the described experiment, which like other ‘pseudo’ laboratory experiments, has weaknesses relating to internal and external validity. The absence of a ‘result’ (i.e., no significant difference between the menu sets) does not prove that there is no difference in real situations. We cannot rule out the possibility that there are differences which are not evident because of the experimental conditions. One limitation which compromises the internal validity of the result is that the nature of the experimental task required the manual menu designers to work from narrative descriptions of the system, a more ‘primitive’ position when compared to the sets of DFDs used to generate the automatic menus. This is because, as already mentioned, providing the same DFDs to the designers would have biased the results. It should be noted, however, that the task assigned to the designers is not unrealistic: many systems are analysed/designed from narrative (even oral) descriptions of user needs - without the use of DFDs or other diagrammatic techniques. With respect to external validity, one limitation of the experiment, as with many pseudo laboratory experiments, is the size of the systems on which the results are based. It is not the same to analyse systems and design interfaces for systems with only 15-21 transactions as those having a larger number of transactions. This limitation, however, works in favour of both methods: for the automatic method, smaller systems facilitate the creation of ‘good’ DFDs (from which the interface is then derived); for the manual method, smaller systems facilitate the design of the interface. Although smallness is advantageous to both methods, we cannot predict whether the results would differ for larger systems without further experimentation. Another limitation of the experiment is that analysis and design, as well as evaluation, were conducted by students. Even though our students were at the stage of graduating from the Information Systems Program, and the systems have been analysed under supervision and approved by the clients, we cannot say they are experts. But expertise and experience are influential in both methods. For the automatic method, more experienced analysts would produce better DFDs in the analysis stage, thus producing better automatic menu-trees. For the manual method, it may be assumed that more experienced designers would do a better job at interface design. The effect of experience can only be determined by future studies. Concerning the evaluation of the interfaces, we confined ourselves to evaluation by potential users. Although HCI experts may be better evaluators than users, the disadvantage of using students as users is not substantial since the same participants evaluated all interfaces, and there is no reason to assume bias. Moreover, in practical situations it is more likely that users rather than experts will evaluate their information systems/interfaces. The main limitation is that our participants were not real users of real systems. If they were, they would probably provide more reliable evaluations. We are convinced, however, that the evaluations provided by the students are reliable. Another deficiency of the subjective evaluation is that it was based on a single, general measure (overall evaluation). The use of a single measure does not offer insights on the Shoval
87
strengths and weaknesses of the design. This limitation can be rectified in further research that will address specific evaluation criteria of the menu-trees. One aspect of the user interface design that has not been addressed is the quality and variation of (human) designers. The literature indicates that HCI experts design differently than novices. We used students as designers, and controlled the variables of designer-quality by randomly assigning the 24 designers to the design tasks. We did not compare the automatic design to the performance of HCI experts and more experienced designers. We have no doubt that experts can do better, but experts are not always available, and our intent was to examine if the automatic method can provide a good initial design, that can be further improved. Comparison of the automatic method to the performance of experts is left for further experiments. Another issue is the effect of the quality of DFDs on automatic interfaces. For each project we had just one set of DFDs, hence one automatic menu-tree, as opposed to four different designer-made interfaces. It is reasonable to assume that different analysts will produce different DFDs for the same system. Thus, different (initial) menu-trees would be derived. We could not address this problem in the current experiment because different DFDs could also yield different transactions and systems, and therefore there would not have been a common basis for comparing the two interface design methods. We have compensated for the limitation of a single set of DFDs per system by conducting the experiment with six different systems (recall that the two-factor ANOVA test revealed no significant differences between the systems). Still, the quality of DFDs as a tool for systems analysis has not been examined. This is definitely an issue for separate study. It is worth mentioning that criticism on weaknesses of the functional approach (on which DFDs are based) for certain kinds of systems, led to the development of object-oriented (O-O) analysis and design methods. There are many such methods, but - as mentioned in the introduction to this paper - they hardly treat interface design. However, as the O-O methods develop, it is likely that they will treat interface design differently than the manual or automatic methods. It is our plan to incorporate interface design within O-O analysis and design, and conduct additional experiments with the interface design methods.
Acknowledgments I thank Arad Friedman for This research Research and
Kopp for assisting me in conducting the experiment, and Dr Lea helping in the statistical analysis of the results. is partially supported by the Paul Ivanier Center for Robotics Production Management, Ben-Gurion University of the Negev.
References Betts, B., Borlingane, D., Fischer, D., Foley, J., Green, M., Karis, D., Kerr, S., Olsen, D. and Thomas, J. (1987) ‘Goals and objectives for user-interface software’ Comput. Graph. 21, 73-78 Coad, P. and Yourdon,
E. (1991) Object-Oriented
Design Yourdon Press, Prentice-Hall
Draper, S. and Norman, D. (1985) ‘Software engineering 88
for user interfaces’. IEEE Trans.
Interacting with Computers vol
7 no 1 (199.5)
Softw.
Eng. SE-11, 252-258
Fischer, G. (1989) ‘Human-computer ahead’. 1EEE Softw. 6, 44-52
interaction
Gould, J. and Lewis, C. (1985) ‘Designing designers think’ Comm ACM 28, 30&311
software:
for usability
Grudin, J. (1989) ‘The case against user interface consistency’ Hartson, H. and Hix, D. (1989) ‘Human-computer systems’ ACM Computing Surv. 21, 592
lessons
learned,
- key principles
challenges and what
Comm. ACM 32,1164-1173
interface development:
concepts and
Hix, D. and Hartson, H. (1986) ‘An interactive environment for dialogue development: its design, use and evaluation’ in Proc. CH1’86: Conf. Human Factors in Computing Systems ACM Press, 228-234 Kiger, J. (1984) ‘The depth/breadth lnt. J. Man-Mach. Stud. 20
trade-off in the design of men driven user interfaces’.
Lustman, F., Mercier, P. and Gratton, P. (1985) ‘A dialogue-based interactive information systems’ Data Base 16, l&24 Martin, J. and Odell, J. (1992) Object-Oriented Mohhamad, U. and Wayne, developing user interfaces‘
architecture
Analysis and Design Prentice-Hall
D. (1988) ‘A survey of formal tools and methods lnt. 1. Man-Mach. Stud. 29, 479-496
Morland, V. (1983) ‘Human factors guidelines 26, 384-494
for
for
for terminal interface design’ Comm. ACM
Norman, D. and Draper, S. teds) (1986) User Centered System Design Lawrence Erlbaum Associates Rissland,
E. (1984) ‘Ingredients
of intelligent
user interfaces’
lnt. J. Man-Mach.
Rumba@, J., Blaha, M., Premerlani, W., Eddy, F. and Lorensen, Oriented Modeling and Design Prentice-Hall Shlaer, S. and Mellor, Press, Prentice-Hall
S. (1992) Object Lifecycles:
Modeling
Shneiderman, B. (1987) Designing the User Interface Computer interaction Addison-Wesley Shoval, P. (1988) ‘ADISSA: architectural design structured analysis’ info. Sys. 13, 19%210
Stud. 21
W. (1991) Object-
the World in States Yourdon
Strategies
of information
for
Effective systems
Humanbased
Shoval, P. (1990) ‘Functional design of a menu-tree interface within structured development’ lnt. 1. Man-Mach. Stud. 33, 537-556
on
system
Shoval, P. and Manor, 0. (1990) ‘Software engineering tools supporting ADISSA methodology for systems analysis and design’ lnfor. Softw. Technol. 32, 357-360 Smith, S. and Mosier, J. (1986) ‘Guidelines for designing user interface software’ Report 7 MTR-20090, Eds-Tr-86-278 Mitre Corp, Bedford, MA, USA Thimbleby, H. (1984) ‘User interface design: generative user engineering principles’ in Monk, A. (ed) Fundamentals of Human-Computer Interaction Academic Press, 165-180 Wasserman, A., Pircher, P., Shewmake, D. and Kersten, M. (1986) ‘Developing interactive information systems with the user software engineering methodology’ 1EEE Trans. Softw. Eng. SE-12, 326-345 Wirfs-Brock, R., Wilkerson, wure, Prentice-Hall
B. and Wiener,
L. (1990) Designing
Object-Oriented
Soft-
Yound, R. and Hull, A. (1982) ‘Cognitive aspects of the selection of viewdata options by casual users’ in Pathways to the Information Society: Proc. 6th Int. Conf. Computer Communications 571-576 89
interacting with Computers vol 7 no 1 (1995)