- Email: [email protected]
Human-computer interfaces: Modelling and evaluation
Computers ind. Engng Vol. 21, Nos 1--4,pp. 577-581, 1991 Printed in Great Britain. All fights reserved
0360-8352/91 $3.00+ 0.00 Copyright © 1991 PergamonPress pie
HUMAN-COMPUTER INTERFACES: MODELLING AND EVALUATION
Chin H. Lee Noemi M. Paz Department of Industrial Engineering University of Central Florida
Abstract
User Modellin~
The human-computer interface is the area of contact between people and computers. Within this area, the interface component of interest is the computer software that determines the conceptual and perceptual interactions that users of the computer perform. It is generally accepted that poor interface design can lead users to experience stress, lower work rates, decreased job satisfaction and even absenteeism (Booth, 1989).
A user model is a representation of the user used by a designer to make inferences and decisions about the design. The simplest and most limited model of a user is of a naive or novice user. With this model the design assumes the user has no knowledge of the computer system. However, it is usually implicit in the naive model that the user is not a novice at the task or application to be performed.
Designers are not trained in human factors concepts. They may not understand or consider the user's needs or working environment. A poor interface design can lead to misuse or lack of use of the computer system. The system may include many capabilities without including the functionality needed by the user. Many times a user in this situation will discover a way to perform the needed task, even though it may be clumsy and undesirable from the user's point of view.
Three types of user models are described in human computer interface research (Norman, 1986): 1. The user model or Design Model. This is the model the system designer has of the entire design. This is also called the conceptual model of the system to be built. Included in this model is a model of the typical user. 2. The user's model or Mental Model. This is the user's internal model of the system and is an individual interpretation of the system. 3. The user model or System Image. This is the system's model of the user. It consists of all that every user sees, hears and feels. It may be individualized if the system adapts to the particular user's needs or to the user's knowledge of the system.
Research in the area of human computer interface is needed. In the past, when the primary objective of software design was compilers, operating systems, and other development tools, designers of computer systems were the users or were typical of the computer users. Resultant computer systems were more likely to meet the designers' personal needs since their needs and style were used to model user requirements. Today computer users no longer are the computer professionals but are mostly discretionary users. The software designers are no longer typical of the ultimate user but are rather unique and are unrepresentative of the user (Shackel, 1985).
A user forms the Mental Model from the System Image as the user interacts with the system. The closer the System Image is to the Design Model the closer the Mental Model will be to the Design Model. Also, different users' Mental Models will be closer to each other. If a System Image does not accurately reflect the Design Model, then the system is likely to be misused, under-used, or abused.
Tools, techniques, design practices, and methodologies are required in order to build a model of how users will behave at an interface and of what users require from a system. This paper discusses two topics in this area: user modelling and task analysis, and the evaluation of user interfaces.
Knowledge about the user is crucial to the designer. This knowledge needs to be about the ultimate typical user. There are several approaches to obtaining knowledge about the user population:
577
578
Proceedings of the 13th Annual Conference on Computers and Industrial Engineering
1. Observe the users of a system similar to the one being designed. This approach is best when there is likely to be a large user population. 2. Perform a systems analysis which will define the tasks performed, the necessary user abilities, and the user interface requirements. This approach works when the system being built is handling a specific task for a group of known users. The designer then has prior knowledge of both the task and the capabilities of the user population. 3. Simulate the "final system by building a prototype and observe the system in use by a set of users that are representative the final user group. 4. Observe the task being designed. This approach studies a representative group of people doing the task and analyzes how they do it. This approach is necessary when the system being designed requires particular abilities from the user and the system itself has a unique design. Each of the above approaches requires the designer to have knowledge of the user population. Such information can be gathered by direct observation, indirect observation, or by conducting a survey. Task Analysis For task analysis it is important that users are defined with respect to the tasks. The designer needs to know the tasks that users will perform and how the system will enable the user to perform those tasks. This is not to say that the system designer designs for the current task. Most computer systems are utilized for the original task plus other things that the designer had not intended or predicted. This natural extension of a software product arises from the fact that the new substitute for the old technology is more powerful that it's predecessor. It could do the task better, or more accurately, or faster, or more efficiently, or cheaper or any combination of the above. The designer must seriously consider these facts. The technique used to obtain a description of the tasks is called requirements analysis or task analysis. The two most frequently used forms of task analysis employed in human communications research are (Rubin, 1988): Task analysis that occurs after some design and specification has already taken place. This technique can be used as a means of early evaluation of the system. Also it is used to make behavioral predictions based on the system specification. These predictions can highlight consequences of design decisions. Predictions require that the design is specified in a very detailed form.
2. Task analysis of the old technology before the new technology has made any changes to the performance of the task. Here, the task analysis technique considers an existing task in detail in an attempt to produce a description that will be useful for the design of the new, substitute technology (Buckley and Johnson, 1987). Both of these forms of task analysis have elements in common. They each must provide a means of defining and then classifying the tasks. Just how much specification or depth of analysis is require for the definition of the task is case dependent. Along with doing the task analysis the designer needs to consider the expertise of the user. This is not the application expertise but how expert the user is with the computerized technology. Users can be divided into three categories (James, 1981): expert, regular user, and inexperienced user. For example, a systems programmer would be considered an expert and a scientist might be the regular user. Users who need the computer infrequently or as a necessary tool about which they know little about the technology are the largest group, the inexperienced users. Regardless of the user category, the tasks users need to perform in order to achieve their objectives are often very similar (Rubin, 1988). Four attributes of human computer interaction that are listed as important for the naive user (Eason and Damordaran, 1981) are: task a tool relationship, expertise in computer technology, ease of use, and user support and training. These attributes are general enough that they extend to the other user categories. Evaluatin~ User Interface Designs Two themes for evaluating the user interface design are: where in the design process should evaluation occur and how thorough should the evaluation be? The idea behind these two themes is that evaluation will impact the final product differently depending on when and how involved it is. Formative and summative evaluation (Hewett, 1986) are two approaches to evaluation that embody these two themes. Formative evaluation provides feedback into the design process in order to refine and form the design. This type of evaluation is more likely to require qualitative information in order to pinpoint design flaws. The second approach, summative evaluation, seeks to provide an overall assessment of the final product. This approach finds quantitative intbrmation more useful than qualitative information. The merger of these two themes has become necessary as the design of products continue even
Lee and Paz: Human-Computer Interfaces
579
after their introduction into the user market. Formative evaluation provides feedback into the design process, but . so does summative evaluation when it is used to analyze a product for further refinement. As it becomes more prevalent to develop a computerized product in stages, more'and more systems continue to be enhanced long after users are using the product. Many of these refinements are to the user interface. Hence the evaluation methodology needs to assess a product with a range of different approaches that can be carried out at different stages in the continuing design process. This will allow more frequent, more rapid, and earlier evaluations of the design.
breakdowns is to study the interactive dialogue between the user and the system.
Evaluation Objectives
Hierarchical Level Model
There are three general objectives for evaluating the user interface (Rubin, 1988) regardless of the type of interface or the stage in the design. These are assessment of the capabilities of the design, assessment of the impacts of design decisions, and the diagnosis of problems with the design.
Evaluation of the user interface requires the development of a formal model. The task analysis done for development of the user model can be used to select a task that is representative of the tasks that the system will be used for. The task analysis is followed by identification of the methods or actions the user will be required to perform in order to accomplish the representative task. These methods can be organized into a hierarchical level model (Moran, 1981). The hierarchical level model is divided into three components. At the top is the conceptual component with the task and semantic levels. The next component is the communication component and consists of the syntactic and interaction levels. The last is the physical component which contains the spatial layout and~ device levels.
The first objective, assessing the capabilities of the design, can be handled with two approaches. One is from the view point of the user, and the other is from the view point of the system. Using a user point of view, evaluation assesses what tasks the user can perform using the system through the interface and whether the possible tasks meet the user's task requirements. From the system's point of view, evaluation assesses the features of the system as accessed through the interface and whether these match the features required by the user. The user approach to evaluation is more appropriate for a variety of reasons: 1. the task approach is user centered, 2. the user and the evaluator need to be familiar only with the task desired and not with what features might be possible, 3. the evaluation can be compared to specifications of user requirement at any stage of the design. The second objective of assessing the impacts of the interface design on the user looks for an evaluation of intended and unintended effects. Unintended effects can arise as side effects of implementation of some design feature or even from the design of some feature. However, most of the focus of impacts of the interface deal with the effort a user must expend to use the system capabilities. The third objective is diagnostic in approach. Evaluation here looks for how the system breaks down. In this sense the evaluation determines where the system fails and how those failures are handled. A frequent technique for finding
The three objectives describe what knowledge is to be gathered by evaluating the user interface. The impact of this knowledge will depend on when in the design-evaluation life cycle it is collected. It is generally accepted that evaluation must be done early enough to ensure that the system will not deviate significantly from the specifications (see Booth, 1989; Rubin, 1988). The results of an evaluation feed back into the original specification and if needed, lead to a modification of the design.
Evaluation of the modelled interface can proceed with formal methods of evaluation or with empirical methods. In a top-down approach to design and evaluation, the conceptual level is evaluated first. This allows evaluation at a every early stage in the design, before the more detailed work on the interface has been down. A bottom-up approach to evaluation can also occur independently of the top-down evaluation and then in relationship to one another.
Criteria Measures In order to measure the quality of the user interface design there must be criteria for evaluation. The criteria for evaluation should be specified early in the design process. The criteria can also be modified or refined as the project proceeds and changes are made in the design. Criteria measures are categorized into two groups: functionality or usability. Functionality refers to the tasks the user can perform when using the system while usability relates to the ease of use of the interface. These measures can be performance measures or subjective measures. Since the lower levels of models are more adequately
580
Proceedings of the 13th Annual Conference on Computersand Industrial Engineering
specified there exist various performance measures for the device and interaction levels (see Kellogg, 1987). However, some work has begun on specifying measures to evaluate conceptual consistency (see Kieras and Polson, 1985; Kellogg, 1987; Poison, 1987). 5. The issue of usability is more important in evaluation of the user interface than is the issue of how many functions the interface can perform. A product may have many useful functions but if they are confusing and frustrating to use then the product will not be used. The evaluation criteria used to check for usability must recognize if the design will lead to a product too complex to be used. Evaluation Methods Once the objective of the evaluation has been chosen and the performance measures have been agreed upon by the analyst then the methods of evaluation can be considered. The following is a partial list of methods that are available for evaluation: 1. Concept test or paper-and-pencil test: the ideas and concepts of the system are presented to the user in the form of cards or storyboards. The user can identify acceptable and confusing concepts. 2. Friendly users: Choose from the user population a group of users that can be considered friendly. They will consist of people who have some technical knowledge and hence will be able to provide feedback and suggestions for altering the design of the system. The disadvantage here is that this group of users may miss aspects of the system that make the system difficult to use for naive users. Molich and Nielsen (1990) produced a study on designing dialogues for the user interface which concluded that "many designers and programmers are not sufficiently aware of the importance of designing dialogues in a way that would either prevent of tolerate errors". A system that cannot or does not provide carefully phrased informative messages in situations where the user may need help will effect the naive user the most. 3. Hostile users: Hostile users differ from naive or friendly users in the fact that they do not care if the system fails or not. This group of users is more likely to test the system limitations and provide information that will lead to the identification of inconsistencies and flaws. 4. Simulation of users: Progress of a group of naive users can be studied and then simulated by the analyst. The analyst
6.
7.
8.
executes the system using the same path as the simulated user. In this way the analyst may view the system using the path of the naive user and may encounter problems within the system that the analyst would not have seen. Simulation trials: The designer simulates parts or all of the interface. With this method evaluation can be done in advance of the implementation of the design. The simulation can be in a written form which allows the analyst to ask the users to describe how they would perform various activities or users can be present users with a prototype of the system that has enough fidelity to the ultimate system so users can interact with the system for an entire task sequence. The usefulness of an evaluation made from a simulation will depend upon how well the simulation captures the essential features of the intended system. Iterative informal laboratory experiments: This approach builds on the simulation trials. Execution of prototypes of the system by experts or other knowledge users provides the necessary feedback. Formal laboratory experiments: This approach is used when quantitative data is required in order to confirm or refute a hypotheses. The experiments take place under a control environment. This will reduce the design team bias affect on the results. However, experiments of this kind are not done within the work environment and evaluation results may not generalize to results that hold outside the laboratory. Fields trials: The system is placed into the actual work environment prior to its formal release. Data collected this way is time consuming, but can be useful as a measure of the potential success and usability of the product.
CONCLUSIONS This paper is concerned with the humancomputer interface. This interface is a window through which the user views the contents of the system and which in turn the system views the user. The building and evaluation of a good window depends upon many things. However, there are two important and central issues: 1. knowledge of the user: the designer needs to know the user's prior knowledge, outlook, experience and familiarity with the information the system can provide; 2. knowledge of the task: the designer needs have a task analysis that specifies the task step by step.
Lee and Paz: Human--Computer Interfaces
Any design that is to be used by people should be created with a user-centered approach. Users are able to provide information which is critical to the success of the system. Consideration of the characteristics and requirements of the user can prevent a system from being built that will ultimately not be used due to the fact that users experience stress, lower work rates, or decreased job satisfaction. The view of how the user sees the system is often complex and may be at odds with the design. What can make this more complex is that the user's view of the system will change as the user becomes more knowledgeable with the system. The interface must be designed for or adapted to the needs of each specific user or group of users. This will require knowledge of the job the user will perform. Task analysis is important as it will place demands upon the computer system and upon the interfaces to it. A well designed system should not only focus on the users and their tasks, but should do so from the start. Ideally the system should be built using an iterative design philosophy. The design phase is a design - build prototype - evaluate cycle. Evaluation of the user interface should also be done with a user-centered approach. Perhaps, the most important issue in the evaluation is the issue of usability. Usability is simply defined as: a system is usable if it is used. A product may be useful and have may useful functions but if it is confusing or frustrating to users then the product will not be used.
Booth, P. (1989) An Introduction to HumanComvuter Interaction Lawrence Erlbaum Associates Ltd., Hillsdale. Buckley, P. and P. Johnson (1987) "Analysis of Communication Tasks for the Design of a Structured Messaging System" In Peovle and Computers III. Proceedin,,s of the Third Conference of the British C~omDuter Society Human-Comouter Interaction Soecialist Grouo. University of Exeter (Eds. D. Diaper and R. Winder) 7-11 September, Cambridge University Press, pp. 29-40. Eason, K. D. and L. Damordaran (1981) "The Needs of the Commercial User" Comoutinfl Skills and the User Interface (Eds. M. J. Coombs and J. L A l t y ) Academic Press, New York, pp. 115-139.
581
Hewett, T. T. (1986) 'The Role of Iterative Evaluation in Designing Systems for Usability" Peoole and Comouters: Designing for Usability-Proceedings of the Second Conference of the BCS HCI Soecialist Group (Eds. M. D. Harrison and A. F. Monk) Cambridge: Cambridge University Press, pp. 196-214. James, (1981) "The User Interface: How We May Compute" Computing Skills and the User Interface (Eds. M. J. Coombs and J. I_. Alty) Academic Press, New York, pp. 337-371. Kellogg, W. A. (1987) "Conceptual Consistency in the User Interface: Effects on User Performance" In Proceedings of the Second IFIP Conference on Human-Comouter Interaction - INTERACT '87 (Eds. H. J. Bullinger and B. Shackel) Stuttgart, Federal Republic of Germany, 1-4 September, London: North-Holland. Kieras, D. E. and P. G. Poison (1985) "An Approach to the Formal Analysis of User Complexity" International Journal of ManMachine Studies Vol. 22, pp. 365-394. Molich and Nielsen (1990) "Improving a HumanComputer Dialogue" Communications of the ACM March, Vol. 33, No. 3, pp. 338-348. Moran, T. P. (1981) "The Command Language Grammar: A Representation for the User Interface of Interactive Computer Systems" International Journal of Man-Machine Studies Vol. 15, pp. 3-50. Norman, D. A. (1986) "Cognitive Engineering" User Centered Svstem Design New Perspectives on Human-Computer Interaction (Eds. D. A. Norman and S. W. Draper) Lawrence Erlbaum Associates Ltd. Hillsdale, N J, pp. 31-62. Polson, P. G. (1987) "A Quantitative Theory of Human-Computer Interaction" Interfacin~ Thought Cotrnitive Asoects of HumanComnuter Interaction (Ed. J. M. Carroll) Bradford Book, Mit Press, Cambridge, pp. 184-235. Rubin, T. (1988) User Interface Design for Comnuter Systems Ellis Horwood Limited, Chichester. Shackel, B. (1988) "Interface Design for Usability" User Interfaces (Ed. T. Bernold) Elsevier Science Publishers B. V., North-Holland, pp. 59-70. Shackel, B. (1985) "Ergonomics in Information Technology in Europe--A Review" Behaviour and Information Technolo~ Vol. 4, No. 4, pp. 263-287.
0360-8352/91 $3.00+ 0.00 Copyright © 1991 PergamonPress pie
HUMAN-COMPUTER INTERFACES: MODELLING AND EVALUATION
Chin H. Lee Noemi M. Paz Department of Industrial Engineering University of Central Florida
Abstract
User Modellin~
The human-computer interface is the area of contact between people and computers. Within this area, the interface component of interest is the computer software that determines the conceptual and perceptual interactions that users of the computer perform. It is generally accepted that poor interface design can lead users to experience stress, lower work rates, decreased job satisfaction and even absenteeism (Booth, 1989).
A user model is a representation of the user used by a designer to make inferences and decisions about the design. The simplest and most limited model of a user is of a naive or novice user. With this model the design assumes the user has no knowledge of the computer system. However, it is usually implicit in the naive model that the user is not a novice at the task or application to be performed.
Designers are not trained in human factors concepts. They may not understand or consider the user's needs or working environment. A poor interface design can lead to misuse or lack of use of the computer system. The system may include many capabilities without including the functionality needed by the user. Many times a user in this situation will discover a way to perform the needed task, even though it may be clumsy and undesirable from the user's point of view.
Three types of user models are described in human computer interface research (Norman, 1986): 1. The user model or Design Model. This is the model the system designer has of the entire design. This is also called the conceptual model of the system to be built. Included in this model is a model of the typical user. 2. The user's model or Mental Model. This is the user's internal model of the system and is an individual interpretation of the system. 3. The user model or System Image. This is the system's model of the user. It consists of all that every user sees, hears and feels. It may be individualized if the system adapts to the particular user's needs or to the user's knowledge of the system.
Research in the area of human computer interface is needed. In the past, when the primary objective of software design was compilers, operating systems, and other development tools, designers of computer systems were the users or were typical of the computer users. Resultant computer systems were more likely to meet the designers' personal needs since their needs and style were used to model user requirements. Today computer users no longer are the computer professionals but are mostly discretionary users. The software designers are no longer typical of the ultimate user but are rather unique and are unrepresentative of the user (Shackel, 1985).
A user forms the Mental Model from the System Image as the user interacts with the system. The closer the System Image is to the Design Model the closer the Mental Model will be to the Design Model. Also, different users' Mental Models will be closer to each other. If a System Image does not accurately reflect the Design Model, then the system is likely to be misused, under-used, or abused.
Tools, techniques, design practices, and methodologies are required in order to build a model of how users will behave at an interface and of what users require from a system. This paper discusses two topics in this area: user modelling and task analysis, and the evaluation of user interfaces.
Knowledge about the user is crucial to the designer. This knowledge needs to be about the ultimate typical user. There are several approaches to obtaining knowledge about the user population:
577
578
Proceedings of the 13th Annual Conference on Computers and Industrial Engineering
1. Observe the users of a system similar to the one being designed. This approach is best when there is likely to be a large user population. 2. Perform a systems analysis which will define the tasks performed, the necessary user abilities, and the user interface requirements. This approach works when the system being built is handling a specific task for a group of known users. The designer then has prior knowledge of both the task and the capabilities of the user population. 3. Simulate the "final system by building a prototype and observe the system in use by a set of users that are representative the final user group. 4. Observe the task being designed. This approach studies a representative group of people doing the task and analyzes how they do it. This approach is necessary when the system being designed requires particular abilities from the user and the system itself has a unique design. Each of the above approaches requires the designer to have knowledge of the user population. Such information can be gathered by direct observation, indirect observation, or by conducting a survey. Task Analysis For task analysis it is important that users are defined with respect to the tasks. The designer needs to know the tasks that users will perform and how the system will enable the user to perform those tasks. This is not to say that the system designer designs for the current task. Most computer systems are utilized for the original task plus other things that the designer had not intended or predicted. This natural extension of a software product arises from the fact that the new substitute for the old technology is more powerful that it's predecessor. It could do the task better, or more accurately, or faster, or more efficiently, or cheaper or any combination of the above. The designer must seriously consider these facts. The technique used to obtain a description of the tasks is called requirements analysis or task analysis. The two most frequently used forms of task analysis employed in human communications research are (Rubin, 1988): Task analysis that occurs after some design and specification has already taken place. This technique can be used as a means of early evaluation of the system. Also it is used to make behavioral predictions based on the system specification. These predictions can highlight consequences of design decisions. Predictions require that the design is specified in a very detailed form.
2. Task analysis of the old technology before the new technology has made any changes to the performance of the task. Here, the task analysis technique considers an existing task in detail in an attempt to produce a description that will be useful for the design of the new, substitute technology (Buckley and Johnson, 1987). Both of these forms of task analysis have elements in common. They each must provide a means of defining and then classifying the tasks. Just how much specification or depth of analysis is require for the definition of the task is case dependent. Along with doing the task analysis the designer needs to consider the expertise of the user. This is not the application expertise but how expert the user is with the computerized technology. Users can be divided into three categories (James, 1981): expert, regular user, and inexperienced user. For example, a systems programmer would be considered an expert and a scientist might be the regular user. Users who need the computer infrequently or as a necessary tool about which they know little about the technology are the largest group, the inexperienced users. Regardless of the user category, the tasks users need to perform in order to achieve their objectives are often very similar (Rubin, 1988). Four attributes of human computer interaction that are listed as important for the naive user (Eason and Damordaran, 1981) are: task a tool relationship, expertise in computer technology, ease of use, and user support and training. These attributes are general enough that they extend to the other user categories. Evaluatin~ User Interface Designs Two themes for evaluating the user interface design are: where in the design process should evaluation occur and how thorough should the evaluation be? The idea behind these two themes is that evaluation will impact the final product differently depending on when and how involved it is. Formative and summative evaluation (Hewett, 1986) are two approaches to evaluation that embody these two themes. Formative evaluation provides feedback into the design process in order to refine and form the design. This type of evaluation is more likely to require qualitative information in order to pinpoint design flaws. The second approach, summative evaluation, seeks to provide an overall assessment of the final product. This approach finds quantitative intbrmation more useful than qualitative information. The merger of these two themes has become necessary as the design of products continue even
Lee and Paz: Human-Computer Interfaces
579
after their introduction into the user market. Formative evaluation provides feedback into the design process, but . so does summative evaluation when it is used to analyze a product for further refinement. As it becomes more prevalent to develop a computerized product in stages, more'and more systems continue to be enhanced long after users are using the product. Many of these refinements are to the user interface. Hence the evaluation methodology needs to assess a product with a range of different approaches that can be carried out at different stages in the continuing design process. This will allow more frequent, more rapid, and earlier evaluations of the design.
breakdowns is to study the interactive dialogue between the user and the system.
Evaluation Objectives
Hierarchical Level Model
There are three general objectives for evaluating the user interface (Rubin, 1988) regardless of the type of interface or the stage in the design. These are assessment of the capabilities of the design, assessment of the impacts of design decisions, and the diagnosis of problems with the design.
Evaluation of the user interface requires the development of a formal model. The task analysis done for development of the user model can be used to select a task that is representative of the tasks that the system will be used for. The task analysis is followed by identification of the methods or actions the user will be required to perform in order to accomplish the representative task. These methods can be organized into a hierarchical level model (Moran, 1981). The hierarchical level model is divided into three components. At the top is the conceptual component with the task and semantic levels. The next component is the communication component and consists of the syntactic and interaction levels. The last is the physical component which contains the spatial layout and~ device levels.
The first objective, assessing the capabilities of the design, can be handled with two approaches. One is from the view point of the user, and the other is from the view point of the system. Using a user point of view, evaluation assesses what tasks the user can perform using the system through the interface and whether the possible tasks meet the user's task requirements. From the system's point of view, evaluation assesses the features of the system as accessed through the interface and whether these match the features required by the user. The user approach to evaluation is more appropriate for a variety of reasons: 1. the task approach is user centered, 2. the user and the evaluator need to be familiar only with the task desired and not with what features might be possible, 3. the evaluation can be compared to specifications of user requirement at any stage of the design. The second objective of assessing the impacts of the interface design on the user looks for an evaluation of intended and unintended effects. Unintended effects can arise as side effects of implementation of some design feature or even from the design of some feature. However, most of the focus of impacts of the interface deal with the effort a user must expend to use the system capabilities. The third objective is diagnostic in approach. Evaluation here looks for how the system breaks down. In this sense the evaluation determines where the system fails and how those failures are handled. A frequent technique for finding
The three objectives describe what knowledge is to be gathered by evaluating the user interface. The impact of this knowledge will depend on when in the design-evaluation life cycle it is collected. It is generally accepted that evaluation must be done early enough to ensure that the system will not deviate significantly from the specifications (see Booth, 1989; Rubin, 1988). The results of an evaluation feed back into the original specification and if needed, lead to a modification of the design.
Evaluation of the modelled interface can proceed with formal methods of evaluation or with empirical methods. In a top-down approach to design and evaluation, the conceptual level is evaluated first. This allows evaluation at a every early stage in the design, before the more detailed work on the interface has been down. A bottom-up approach to evaluation can also occur independently of the top-down evaluation and then in relationship to one another.
Criteria Measures In order to measure the quality of the user interface design there must be criteria for evaluation. The criteria for evaluation should be specified early in the design process. The criteria can also be modified or refined as the project proceeds and changes are made in the design. Criteria measures are categorized into two groups: functionality or usability. Functionality refers to the tasks the user can perform when using the system while usability relates to the ease of use of the interface. These measures can be performance measures or subjective measures. Since the lower levels of models are more adequately
580
Proceedings of the 13th Annual Conference on Computersand Industrial Engineering
specified there exist various performance measures for the device and interaction levels (see Kellogg, 1987). However, some work has begun on specifying measures to evaluate conceptual consistency (see Kieras and Polson, 1985; Kellogg, 1987; Poison, 1987). 5. The issue of usability is more important in evaluation of the user interface than is the issue of how many functions the interface can perform. A product may have many useful functions but if they are confusing and frustrating to use then the product will not be used. The evaluation criteria used to check for usability must recognize if the design will lead to a product too complex to be used. Evaluation Methods Once the objective of the evaluation has been chosen and the performance measures have been agreed upon by the analyst then the methods of evaluation can be considered. The following is a partial list of methods that are available for evaluation: 1. Concept test or paper-and-pencil test: the ideas and concepts of the system are presented to the user in the form of cards or storyboards. The user can identify acceptable and confusing concepts. 2. Friendly users: Choose from the user population a group of users that can be considered friendly. They will consist of people who have some technical knowledge and hence will be able to provide feedback and suggestions for altering the design of the system. The disadvantage here is that this group of users may miss aspects of the system that make the system difficult to use for naive users. Molich and Nielsen (1990) produced a study on designing dialogues for the user interface which concluded that "many designers and programmers are not sufficiently aware of the importance of designing dialogues in a way that would either prevent of tolerate errors". A system that cannot or does not provide carefully phrased informative messages in situations where the user may need help will effect the naive user the most. 3. Hostile users: Hostile users differ from naive or friendly users in the fact that they do not care if the system fails or not. This group of users is more likely to test the system limitations and provide information that will lead to the identification of inconsistencies and flaws. 4. Simulation of users: Progress of a group of naive users can be studied and then simulated by the analyst. The analyst
6.
7.
8.
executes the system using the same path as the simulated user. In this way the analyst may view the system using the path of the naive user and may encounter problems within the system that the analyst would not have seen. Simulation trials: The designer simulates parts or all of the interface. With this method evaluation can be done in advance of the implementation of the design. The simulation can be in a written form which allows the analyst to ask the users to describe how they would perform various activities or users can be present users with a prototype of the system that has enough fidelity to the ultimate system so users can interact with the system for an entire task sequence. The usefulness of an evaluation made from a simulation will depend upon how well the simulation captures the essential features of the intended system. Iterative informal laboratory experiments: This approach builds on the simulation trials. Execution of prototypes of the system by experts or other knowledge users provides the necessary feedback. Formal laboratory experiments: This approach is used when quantitative data is required in order to confirm or refute a hypotheses. The experiments take place under a control environment. This will reduce the design team bias affect on the results. However, experiments of this kind are not done within the work environment and evaluation results may not generalize to results that hold outside the laboratory. Fields trials: The system is placed into the actual work environment prior to its formal release. Data collected this way is time consuming, but can be useful as a measure of the potential success and usability of the product.
CONCLUSIONS This paper is concerned with the humancomputer interface. This interface is a window through which the user views the contents of the system and which in turn the system views the user. The building and evaluation of a good window depends upon many things. However, there are two important and central issues: 1. knowledge of the user: the designer needs to know the user's prior knowledge, outlook, experience and familiarity with the information the system can provide; 2. knowledge of the task: the designer needs have a task analysis that specifies the task step by step.
Lee and Paz: Human--Computer Interfaces
Any design that is to be used by people should be created with a user-centered approach. Users are able to provide information which is critical to the success of the system. Consideration of the characteristics and requirements of the user can prevent a system from being built that will ultimately not be used due to the fact that users experience stress, lower work rates, or decreased job satisfaction. The view of how the user sees the system is often complex and may be at odds with the design. What can make this more complex is that the user's view of the system will change as the user becomes more knowledgeable with the system. The interface must be designed for or adapted to the needs of each specific user or group of users. This will require knowledge of the job the user will perform. Task analysis is important as it will place demands upon the computer system and upon the interfaces to it. A well designed system should not only focus on the users and their tasks, but should do so from the start. Ideally the system should be built using an iterative design philosophy. The design phase is a design - build prototype - evaluate cycle. Evaluation of the user interface should also be done with a user-centered approach. Perhaps, the most important issue in the evaluation is the issue of usability. Usability is simply defined as: a system is usable if it is used. A product may be useful and have may useful functions but if it is confusing or frustrating to users then the product will not be used.
Booth, P. (1989) An Introduction to HumanComvuter Interaction Lawrence Erlbaum Associates Ltd., Hillsdale. Buckley, P. and P. Johnson (1987) "Analysis of Communication Tasks for the Design of a Structured Messaging System" In Peovle and Computers III. Proceedin,,s of the Third Conference of the British C~omDuter Society Human-Comouter Interaction Soecialist Grouo. University of Exeter (Eds. D. Diaper and R. Winder) 7-11 September, Cambridge University Press, pp. 29-40. Eason, K. D. and L. Damordaran (1981) "The Needs of the Commercial User" Comoutinfl Skills and the User Interface (Eds. M. J. Coombs and J. L A l t y ) Academic Press, New York, pp. 115-139.
581
Hewett, T. T. (1986) 'The Role of Iterative Evaluation in Designing Systems for Usability" Peoole and Comouters: Designing for Usability-Proceedings of the Second Conference of the BCS HCI Soecialist Group (Eds. M. D. Harrison and A. F. Monk) Cambridge: Cambridge University Press, pp. 196-214. James, (1981) "The User Interface: How We May Compute" Computing Skills and the User Interface (Eds. M. J. Coombs and J. I_. Alty) Academic Press, New York, pp. 337-371. Kellogg, W. A. (1987) "Conceptual Consistency in the User Interface: Effects on User Performance" In Proceedings of the Second IFIP Conference on Human-Comouter Interaction - INTERACT '87 (Eds. H. J. Bullinger and B. Shackel) Stuttgart, Federal Republic of Germany, 1-4 September, London: North-Holland. Kieras, D. E. and P. G. Poison (1985) "An Approach to the Formal Analysis of User Complexity" International Journal of ManMachine Studies Vol. 22, pp. 365-394. Molich and Nielsen (1990) "Improving a HumanComputer Dialogue" Communications of the ACM March, Vol. 33, No. 3, pp. 338-348. Moran, T. P. (1981) "The Command Language Grammar: A Representation for the User Interface of Interactive Computer Systems" International Journal of Man-Machine Studies Vol. 15, pp. 3-50. Norman, D. A. (1986) "Cognitive Engineering" User Centered Svstem Design New Perspectives on Human-Computer Interaction (Eds. D. A. Norman and S. W. Draper) Lawrence Erlbaum Associates Ltd. Hillsdale, N J, pp. 31-62. Polson, P. G. (1987) "A Quantitative Theory of Human-Computer Interaction" Interfacin~ Thought Cotrnitive Asoects of HumanComnuter Interaction (Ed. J. M. Carroll) Bradford Book, Mit Press, Cambridge, pp. 184-235. Rubin, T. (1988) User Interface Design for Comnuter Systems Ellis Horwood Limited, Chichester. Shackel, B. (1988) "Interface Design for Usability" User Interfaces (Ed. T. Bernold) Elsevier Science Publishers B. V., North-Holland, pp. 59-70. Shackel, B. (1985) "Ergonomics in Information Technology in Europe--A Review" Behaviour and Information Technolo~ Vol. 4, No. 4, pp. 263-287.