- Email: [email protected]
Cognitive evaluation: How to assess the usability of information technology in healthcare
ELSEVIER
Computer Methods and Programs in Biomedicine 54 (1997) 19-28
Cognitive evaluation:
euscart-Zkphir a Laboratoire ’ CERIM.
FacultC
-
How to assess the usability of information technology in healthcare a, J. Brender b, R. Beuscart ‘,*, I. Mknager-Depriester
de Psychologie de Mt!decine,
’
Cogniliue. UniuersitC Charles de Gaulle, Lille 3, France b M.I.L., Copeuhagcn. Denmark Unioersite de Lille, 1, Place Verdum, 59041 Lille Cedeex. France
Abstract As the adoption of information technology has increased, so too has the demands that these systems become more easier. The adapted to the physicians and nurses environments,, to make access and management of information developers of information systems in Healthcare must use quality management techniques to ensu.re that their product will satisfy given requirements. This underlines the importance of the preliminary phase where Users Requirements are elicited. Some methodologies, such as KAVAS (E.M.S. Van Gennip. F. Grimy, Med. Inform. 18, 1993, 179) chose to use a continuous assessment protocol as a key strategy for quality management. At each stage of the conception and development of a prototype, the assessment checks that it conforms to the expectation of the users’ requirements. The methodology of evaluation is then seen as a dynamic process which is able to improve the design and development of a dedicated system. The purpose of this paper is to demonstrate the necessity to include a cognitive evaluation phase in the process of evaluation by: (1) evaluating the integration (usability) of the I.T. in the activity of the users; and (2) understanding the motives underlying their management of information. This will help the necessary integration of information management in the workload of the healthcare professionals and the compatibility of the prototypes with the daily activity of the users. 0 1997 Elsevier Science Ireland Ltd. Keywords:
Information
technology;
Medical
informatics;
1. Ttmtro~duction The increasing accessibility of information technology (IT) systemsduring recent years has had a significant effect upon the healthcare field. Many healthcare establishments now operate heteroge* Corresponding author.
Assessment;
Cognitive
evaluation
neous IT environments with equipment ranging from stand-alone PCs to minicomputers and mainframe installations. The influence of information systems can now be seen in most areas of healthcare operation, with an ever increasing number and variety of medical applications. Evaluation methodologies [l-3] are required to assessthe successor failure of these systems. The
0169..260’7/97j$17.00 Q 1997 Elsevier Science Ireland Ltd. ,411rights Feserved. PII SOl69-2607(97)00030-8
20
d4.C.
Beuscart-Zkphir
et al. /Computer
Methods
KAVAS (AlOX) and KAVAS-2 (A2019) [4] projects have developed a methodology for evaluation of information systems. Although it was originally designed for the assessment of knowledge-based systems in medicine, this methodology proves to be generally adaptable to medical information technology. This methodology uses a reference model based upon the Users’ Requirements Analysis. Thus, elicitation of users’ requirements becomes a critical phase as they constitute the reference for the whole assessment methodolog:y as well as for the whole quality management methodology. Based upon the Users’ Requirements Analysis,, the KAVAS methodology, as others [5,6] takes into account the explicit users needs obtained after interviews, wallcharting, groupworks, and other consulting techniques [7]. As a consequence, the evaluation will be oriented to the technical validation of the software versus the technical specifications, to the usability and usage of the application versus the functional specifications. And this. assessment methodology will preferably point out at technical bugs,, human computer interfaces friendliness [S], daily usage of the information s,ystem. But, in our experience, some difficulties and failures are not explicated by technical bugs, ergonomic problems or usability inadequacy. They can be related to the interaction between users and their IT tools. Therefore it is also necessary t’o understand how the IT will be integrated in the users activity, when a physician or a nurse is performing a task where information is necessary as a support [9,10]. This is the objective of cogn:ltive evaluation (Fig. 1).
and Programs
in Biomedicine
2.1. Hospital
injhmation
54 (1997)
19-28
system
The University Hospital of Lille is developing an Integrated Hospital Information System (HIS) to improve the management of information within the medical units and to facilitate the communication of data between these units. The functionalities of the HIS were defined after a complete analysis of the data flows and information flows. This phase of the project was realised using an exhaustive Users’ Requirements Analysis [7]. As the HIS is developing, new needs appear. The physicians, after the first benefit of the basic procedures, want to develop or to acquire complementary applications that could help them in their daily work, to improve the diagnosis or treatment processes.
2.2. The European BAR project The aim of the project is to integrate 6 AIM prototypes on the HIS platform in the University Hospital. These prototypes were elicited from six European consortiums, able to provide pre-competitive products. To integrate the prototypes within the HIS, we defined an architecture based upon an ‘intelligent server’ allowing a safe exchange of data between the two parts, through APIs [11,12]. This architecture was effective in most cases and, using HL-7 as a standard for communication protocols, it was possible to use the prototype workstations in connection with the services offered by the HIS [11,12].
Users
f&a&
2. Background The KAVAS methodology was applied in the assessment framework of the ISAR project and we will illustrate the potentials of this method with real-life examples from this experience. Th:ls work was conducted in the University Hospital of Lille, France, with the collaboration of Aalborg University, Denmark.
Specifications
Fig. 1. Four corresponding
Verification
steps for Users Requirements (left side) and the four steps of evaluation (right side).
M.C.
Beuscart-Z&phir
et al. /Computer
Methods
After the technical integration of the prototypes with the HIS, we had to analyse if a prototype developed in another region, in another country, could ble installed, integrated and used in a new, completely different environment. In short, in this work, we study how the local constraints (workload, work organisation, local habits) can interfere with the potentials of medical computers. The issues, when introducing these new information systems in a medical department, were: e How can the information system be anchored to the clinical processes in order to support them. in an efficacious, effective and efficient way? e How can the information system adapt to the changes in the clinical processes? In this paper, we will take the example of the evaluation of the Anaesthesia Mobile System (AM§) integration with the Hospital Information System of the hospital. This AMS was designed in Ihe TA‘NIT (AIM A 2036) project and developlzd to collect information during the pre-anaesthesia consultation and to give the anaesthetists the possibility of having an understandable summary of the patients status during anaesthesia. The AMS was integrated with the HIS to import administrative and medical history data (stored in the HIS) and to export the anaesthesia report after consultation. A User Requirements Analysis was performed using the usual methods [7], validated the existing functions and asked for the improvements of some of them. Obtained from the Users’ Requirements Analysis, the computerised medical record model (Fjg. 2) of the pre-anaesthesia consultation, focuses on data flow and data repositories. In this model, the activity of the anaesthetist during the consultation is described by the users themselves as a ‘filling the form’ activity. We can notice that the data collected during the standard Users’ Requirements Elicitation say nothing about the underlying medical activity of the physicians, that is medical reasoning, clinical auscultation and so on. Users never referred to this type of activity during the ‘UK Analysis phase. That is why it is necessary to introduce a new step of ‘activity analysis and modelling’ and it’s corresponding cognitive evaluation step.
and Programs
m Biornedicilze
Fig. 2. The computerised the Users’ Requirements sultation. This model repositories.
54 (1997)
19-28
21
medical record model, obtained from Analysis of the pre-anaesthesia confocuses on the data flows and data
3. The KAVAS evaluation methodology KAVAS quality assessment framework
and
This Evaluation Methodology [4,5] suggests an intimate relation between the development and the evaluation throughout the entire system life cycle (Fig. 3). The main idea is to use the users requirements as a norm, and to check at each step of the lifecycle if the development of the tool is consistent with the initial objectives, as defined by the Users Requirements. In this sense, KAVAS assessment methodology can be seen as a quality management framework [4,13], as it assesses any divergence between the effective development of the tool and the stated requirements expressed in the planned development. For example, user’s requirements are translated in a detailed specification document allowing the development of the software. As soon as a first prototype is created, it must be assessed to see if it conforms, in all its aspects, to the Users Requirements expressed in the specification document. That is called ‘Validity phase’ or ‘Technical Verification’. In the same way, when the prototype has been installed, a new assessment phase takes place to check if, in real use, the tool satisfies Users Requirements exhaus-
M.C.
Beuscurt-ZJphir
et al. / Compuler
Merhods and Programs in Biomedicine 54 (1997) 19-3
:cess to the catalogue/list
To close Fig. 3. The KAVAS
Assessment
tively. This is called ‘Functi.onality Assessment’
4. Cognitive evaluation 4.1. The knits of the current methodology As a quality management methodology, th,e KAVAS framework grants that the final tool is consistent with stated users’ requirements. But as the main objective of the project is to build an information system, users’ requirements are defined according to an Information Flow point of view. Then elicitation and formalism of the Users’ rizquirements focus on the management of informarion, neglecting the work’s organisation, the analysis of the activity and the human factors. This may have bad consequencesfor the development of the tool as well as for the evaluation of the process itself. The IWVAS’s Evaluation Methodology allows detection of cognitive and ergonomic problems by means of the Functionality AssessmentMethodology (FAM). For instance, if there is a cognitive problem in a prototype, the FAM will detect it :m terms of deviations of the following kind: errors, appearance of changed and/or new compensatory procedures or activities. The whole idea about FAM is to detect problems in the work organisation with and around the IT-prototype. The dis-
the form
framework
from
Brender
et al. [5]
advantage with FAM is that it - to perform optimally - requires near-stable conditions and real practice, and that it is not dedicated to cognitive problems and therefore have a coarse granularity in this respect. One limitation with K.AVAS’s methodology is that it does not point ai: activities to provide the necessary frame of reference for cognitive assessment. But neither does it provide methods for the other types of assessmentthroughout the life-cycle, this has to blemade operational, when the decision on system development approach has been taken and in accordance with its assumptions and conditions. Thus, although the quality assessmentmethodology has been carefully followed, this may result in a tool with poor usability and hence poor acceptance. An important reason for failure is then neglected [8]. 4.2. Objectives:
assessment
of human ,factors
by
ergonomic and cognitive evaluation Our hypothesis is that the whole assessment process must take into account human faci.ors, i.e. the users’ tasks and activity underlying the information management activities supported by the new tool. As Users Requirements are the norm for the evaluation process, their content has to be extended: Users Requirements must include an expression of fundamental constraints linked to main features of users activity and tasks.
M.C.
Beuscart-Zphir
et al. ,/ Computer
Methods
and Programs
in Biomedicine
54 (1997)
1928
23
In many cases, there is a strong interaction between the medical activity of the professionals (nurses and/or physicians) and the management of administrative and medical information. A better knowledge of this interaction will lead to an improvement in the conception, design, development and installation of IT applications. Therefore, the main objective of this phase is the observation, analysis and modelling of the users’ activity when confronted with the IT application. But am activity is not performed for itself. An activity is goal-oriented in order to achieve a diagnosis or therapy procedure, or to organise a complex chain of tasks devoted to the management of the patient. So, the activity must be observeId not only as the description of the behaviour of the agents but as a series of actio.ns oriented towards an objective that must be identified so as to understand which strategies the agems develop to achieve their tasks.
the various phases of their activity. For each user, we identify sequences of mental (cognitive) and/or behavioural actions, and the underlying motives (aims) guiding these sequences of actions. This is summarised in a hierarchical formalism, describing tasks, sub-tasks, and sub-sub-tasks, until we reach the lower level of activity constituted by elementary actions. It is thus possible to identify the constraints linked to the tasks or goals aimed by users, constraints that the IT application should respect.
S. Methods
6. Results
The methods are essentially based upon the observations of the actors in the place of their medical activity.
As previously described, we applied the complete KAVAS methodology framework to perform the evaluation of the integration of foreign prototypes in the ISAR project [11,12]. Here, we take the example of the evaluation of the Anaesthesia Mobile System (AMS) integration with the Hospital Information System (HIS) of the hospital and we will focus on the pre-anaesthesia consultation information recording. The AMS was integrated with the HIS to import administrative and medical history data (stored in the HIS) and to export the anaesthesia report after consultation.
5.1. Obsemztions
md records
On-site observation of the users activity, before the installation of the application, is indispensable The scope of this observation is defined from the Users’ Requirements Analysis, the description of the a.pplications, and the characteristics of the prototype. The observation must focus on the part of the agents activity concerned by the IT application to be installed. This observation is supported by video and verbal protocols recorded during the task itself or afterwards while autofacing the videotapes. Interviews and questionnaires are directed to confirm the interpretation of these video and verbal protocols. 5.2. Modeelling This is a schematic description of the activity Iof the different users and of their interactions during
S.3. Ergonomic
and cognitive ev&uborz
This evaluation itself is ‘biphasic’. One part is realised in the laboratory environment, but with the users, The second part is realised in the unit, when the application is faced with the necessities and constraints of the real activity of the agents, and the interaction with the patients.
6.1. Technicul and HCI
evaluation
A complete technical verification of the prototype was realised to verify the absence of bugs, the presence of the items, screens and functions as they were described in the Technical Specifications Report obtained after the Users’ Requirements Analysis. Moreover, a study of the quality of Human Computer Interfaces @ICI) was realised according to the methods currently em-
M.C.
Belrscart-Z6pkir
et al. /Computer
Fig. 4. The
Task Model,
Methods
rind Programs
obtained
from
ployed in our hospital and referring principally to the INRlA Methodology [14,15]. At the end of this phase, the technical verification was considered as satisfying, even if some remarks were made about the quality of the human computer interfaces and the rapidity of the software. Thus, the cognitive analysis could be performed before introducing the device in the medical unit, to verify its usability in the daily activity of the physician. 6.2. Obsrrvation
of the users’ activities
Before introducing the AMS in the clinical unit, we observed the way anaesthetists were performing a pre-anaesthesia consultation. This observation demonstrated that the physician was involved in a number of simultaneous activities: 1. Interview of the patient to collect the antecedents of the patient and the history of the current disease; 2. Examination and auscultation of the patient (behavioural activity); 3. Planilication of the collection of critical information regarding anaesthesia and selection of the relevant data (cognitive activity); 4. Writing down the information (behavioural activity), and 5. Underlining the important data that could be essential in case of trouble during the anaesthesia. (cognitive activity). These observations permitted to build an elementary ‘Task Model’ (Fig. 4), enlightening the different aspects of the physicians activity and not only those related to the collection of data. The
bl Biomedicine
the observation
54 (1997)
19-28
protocols.
task model shows that the major aim of the an.aesthetist performing pre-anaesthesia consultation is to categorise the patient (anaesthesia without any risk; medium risk; high risk) in order to decide whether or not he will be able to perform the anaesthesia on this patient and which protocol to choose. The physician thus plans and organises the subtasks (questioning the patient; auscultation; selection of relevant information,. . .,) in order to be able to categorise and decide; all along this cognitive and behavioural medical activity, he writes down and eventually underlines relevant data, so ‘fulfilling the form’ which will be used on the theatre. Only this last and more visible layout of the activity has been identified during Users’ Requirements Analysis (Fig. 2). It is therefore essential to clarify immediately whether or not the tool’s integration within the subject’s activities will cause problems. If there is a risk of this nature, it is necessary to perform a sufficiently detailed description of this activity to be able to anticipate the interaction between this activity and the use of the future tool. This description must enable constraints to be defined: (1) for the detailed specifications; (2) where necessary for the functional specifications; and in any case, (3) for ergonomic choices concerning the interface. When data reading/writing activities supported by the computer tool are parallel or compatible with the other (mental/behavioural) activities within which they are integrated, no real problem will occur. On the contrary, the introduction of new information technologies could disturb the entire healthcare process.
M.C.
Beuscart-Z&phir
et al. /Computer
Methods
6.3. Motivations and goals orienting data acquisition and management
Another key dimension of the tool’s integration in the user’s daily activities concerns the motives underlying these data writing/handling/reading activities. Each user has his own representation Iof his task in general, and of each sub-task, sub-subtask, in particular. These representations enable him to derive plans to organise and finalise his activities. The specific data reading/writing activities are a part of these general plans, are als,o finalised, and can be put to use to perform other activities or sub-activities. In the AMS, the consultation involves the writing of medical data in order to pinpoint specifc risk factors and condition the anaesthesia to avoid catastrophes. The objective is not to build an exhaustive medical record. A strategy for selecting and underlining some of the data is therefore essential. The general objective is that, on the basis of this record, the anaesthesiologist (who might bc someone else other than the one whlo carried out the consultation) makes a good interpretation and organises the anaesthesia in consequence. Simply entering the data, however completely, is not adequate: if the record is (too) complete, the information is too crowded and this will be contrary to the objective. The AMS evolved to offer a ‘Summary’ function, following the users intuition that whatever the number of pages of the file, these would not enable the objective of the consultation to be reached, inasmuch as this is tied up with the anaesthesiologist’s activities. The expressed user requirem8ents never actually provide any information on the objectives pursued by the users when handling (entering,. .,) data. Therefore, it is not only the users’ usual activities that need to be described, but also the finalities of these activities, and particularly of the data handling activities. Tasks must be described. 6.4. Evaluation of the prototype consultation process
in the
After the previous validation steps and the observations, some physicians agreed, after training,
and Program
in Biomedicirze
54 (1997)
19-28
25
to use the prototype to manage the information during the pre-anaesthesia consultation. Seven physicians performed this experience. But the experience was a failure due to three main factors: o Duration of the collection of data: with the AMS, the duration for collecting data was more than 25 min, versus 10 nun with paper and pencil. o Ability to find the right data in the catalogues: even after training, the use of catalogues and repositories is too complex to authorise the collection of data and their recording during the consultation itself. o Cognitive problems. The arborescent architecture of the data base does not allow easy links between different sheets. It is not so easy to collect simultaneously the nature of a disease and its treatment as they are recorded in different parts and screens of the data base. In fact, these results iYere prediaable from the observations of the professionals activities and from the explication of the motivation of the physicians objectives. The means for this prediction is the building of a ‘Normative model’ which merges both the necessities of the computerised data collection and the constraints of the physicians activities. 6.5. The normative model
It is therefore possible to determine the functional normative model from the Users’ Requirements Analysis, from the observations protocols and from the explication of the users motivations. This normative model describes the activities of the professional and, as part of thds, the particular activity of collecting data on a computerised record, using the prototype to be assessed. If the medical record model, obtained from the Users’ Requirements Analysis (Fig. 2) is too different form the ‘Normative model’ (Fig. 5), and/ or if the task model (Fig. 4) is too different from the normative model (Fig. 5) then the probability of failure is very important. On the contrary, if the three models are close, then the probability of success will be high, since the introduction of the computers will not deeply modify the users activities.
I lnitiaiisation _. ..*.-.--“___.._-. _--ll Interrogation --. .-. ----
/ Questionning
1 of the file .-----Li j
1 /CATEG~R~~AT~~N I Auscuitation
7 _j
I ~~cis10~
/
IV To iselect the information
1 Storage
--
1 To underline
.^ important
data i
To choose the heading
To fill the field
t-L
4
To valida!e
“--I
To close the form
Fig. 5. The Normative Model, elaborated by confrontation between the Users’ Requirements Anaiysis and the observation protocols. During the pre-anaesthesia consultation, it is difficult for the physisican simultaneously to perform: the clinical examination, to extract the relevant information to be recorded, and to b’rowse through screens and catalogues to store and type the information using checklists.
In our case, the three models are profoundly different and the failure could be foreseen, even if the anaesthetists themselves were convinced of the opposite. Comparison between computerised medical record model and normative model (Fig. 2 vs. Fig. 4): the computerised medical record model takes no account of the usual medical mental/ behavioural activity of the physicians performing pre-anaesthesia consultation. This is a reason for failure of the integration of the new tool in the daily activity of the anaesthetists. Comparison between the task model and the normative model (Fig. 4 vs. Fig. 5): in the task model, the writing activities are deeply integrated in the other activities and need no additional mental or behavioural resources from the physician. On the contrary, the normative model shows the capture, coding and storage of data as an additional activity, which will inevitably take time and will compete with the medical activity. In other words, the physician will have to handle medical activities on the one hand and capture of data on the other hand. The new software tool will deeply mod-
ify the usual task of the anaesthetists. The difference between the stated task model and the observed task model is striking and frightening. It is frightening, because the traditional approach for preparation of IJser Requireme:nts is based upon the assumption that users are able to express their own way of operating. They are experts at their own mode of operation. However, the conclusion of our study is that they are not able to explicate their knowledge, which is in line with similar observations from other domains, of which knowledge engineering is the prime example. For instance, this phenomenon has been observed also from the domain of information retrieval [16], as well as from system development (Naur in [17]). This points to a paradox in today’s contractual approach for system development [18] and the expectations from system developers, as a detailed specification of the future system is required at an early stage. So the failure of the prototype could have been foreseen. But on the AMS case study, the activity analysis and modelling took place after the elaboration of the prototype. We think that it would be fruitful to perform such an analysis before the development of the prototype, to make it consis-
M.C.
Beuscart-Ziphir
tent with the characteristics which it will be integrated.
‘7. Discussion
et al. /Computer
Methods
of the activity
in
and conclusion
To perform the cognitive evaluation, related to the users activity and objectives, it is therefo:re necessary to follow a rigorous approach in four steps: !. The first step is the identification of goals or tasks and ‘subtasks’, and the analysis of the strategies the agents develop to perform these tasks (and reach their goals). 2. The second step consists of modelling the activity of the different users inasmuch as this activity is concerned by the installation of the new tool. This model must take into accou:nt behavioural as well as cognitive facets of the activity. 3. The third step is to create a normative model which describes the future activity supported by tlhe new tool. 4. The fourth step is to compare the actual and future model to the constraints the new tool must respect in terms of activity and tasks. These constraints constitute the additional layer of the users’ requirements. In this paper, we limited our example to the pre-analesthesia consultation but each phase of the anaesthsesia was equally concerned with this approach. We can give some examples: D During the induction of anaesthesia, the physician frequently glances at the pre-anaesthes.ia record. When using a paper record, he may put it down anywhere in the theatre, even on the stomach of the patient. How will this be possible with a computer? ID In the operating theatre: two or three vital elements of information underlined (or writtlen in red colour) on the sheet (of the medical record) are read while the syringe of hypnotic medication i.s being pushed. During the anaesthesia induction, it will also be impossible to push a syringe and read two or three screen pages of data on the computer, if the screen is easy and comfortable to read.
atzd Programs
in Biomedicine
54 (1997)
19-S
21
o Information must be easily reachable in every place of the operating theatre, and the record is frequently moved from one place to another, following the anaesthetist’s work. With the development of IT in the Healthcare domain, the necessity of quality management beclomes obvious. More often, the quality plan considers only the technical aspects of software engineering, debugging, performances: transferability [4,8,19]. In some cases the HCI [5,14,15] are also evaluated on subjective scales to identify their quality or the users’ workload. However, more and more the users are involved in the development of information applications. They are involved in the Users’ Requirements Analysis at the start of the project, and there is now a crucial need to involve them in the assessment also, even if one part of the evaluation is realised in a laboratory environment before installation. The KAVAS Methodology Framework provided us with a coherent four phase methodology taking into account the technical, functional, and impact evaluation. We have d.emonstrated that even this coherent methodology had to be completed by a cognitive approach to enable a better understanding of the connections between data management and the activities of the users with the goals they wish to attain. These two axes of evaluation are proving the usability of the system, that means the potentials of the application that actually could be used by the healthcare agents.
Acknowledgements This work its Project Telematics ‘Ganymede’ the Region
was supported by the Isar-Telemat(HC 1027) from the IEU Healthcare Programme and by the CPER for Advanced Communication from Nord-Pas de Calais, France.
References [l]
E.M.S. Van Gennip, F. GrCmy, Challenges and opportunities for technology assessment in medical informatics, Med. Inform. 18 (1993) 179-184.
28
M.C.
Beuscart-Zt!phir
et al. /Computer
Melhods
[2] F. Grtmy, P. Degoulet, Assessment of health information technology: which questions for which systems? Proposal for a taxonomy, Med. Inform. 18 (1993) 185-194. [3] P. Degoulet, L. Lucas, MC. Laurent, F.C. Jean, D. Sauquet, M. Lavril, An approach for the evaluation of software engineering environments in medicine, Med. Inform. IS (1993) 1955208. [4] J. Brender, Quality Assurance and Validation of large Information Systems - As viewed from the user perspective. Master Thesis in Computer Science, Report no. 89-l-22, Copenhagen University, March. 1989. [S] J. Brender, J. Talmon, J.P. Nykanen, P. McNair, M. Demeester, R. Beuscart, On the evaluation of system integration, in: E.M.S.J. Van Gennip, J.L. Talmon (Eds.), Assessment and Evaluation of Information Technologies in Medicine, 10s Press, Amsterdam, 1995, pp. 1899208. 161 G, Boloix, P.N. Robillard, A software system evaluation framework, Computer 28 (1995) 17-26. [7] R. Beuscart, J.L. Salomez, M.C. Nuttens, A.G. Pars, h. Purro, Participative methods for the definition of the hospital information system of the University Hospital in Lille, in: Medinfo 92, North Holland, Amsterdam, 1992, pp. 2188219. [S] P.C. Tang, V.L. Patel, Major Issues in user interface design for health professional workstations: a summary and recommendations, Int. J. Biomed. Comput. 34 (1994.) 1399148. [9] A.W. Kushniruk, D.R. Kaufman, V.L. Patel, Y. Levesque, P. Lottin. Assessment of a computerized patient record system: a cognitive approach to evaluating medic;%1 technology, M.D. Computing 3 (1996) 406-415. [lo] M.C. Beuscart-Zephir, J. Brender, N. Souf, I. MenagerDepriester, Cognitive Evaluation for the assessment of Information Technology, in: Healthcare, Proc. 13th Int.
and
Programs
[ll]
[12]
[13] [14]
[15]
[16]
[17]
[18]
[19]
in Biotnedicine
54 (1997)
19-28
Congress Med. Inform. Europe, IOS Press, Copenhagen, 19-22 August 1996. R. Beuscart, M. Demeester, B. Bossart, A. Dubos, B. Modjeddi, Integration Architecture: The ISAR Project, in: Medinfo 95, North Holland, Amsterdam, 1995, p. 981. R. Beuscart, B. Modjeddi, M. Demeester, ISAR: Integration System Architecture, in: M.F. Laires, M.J. Ladeira, J.P. Christensen (Eds.), Health in the New Communication Age, 10s Press, Amsterdam, 1995, pp. 3922403. T. Drake, Measuring Software Quality: A Case-study, Computer 11 (1996) 78-87. B. Senach, Evaluation ergonomique des interfaces homme-machine: une revue de la litttrature. Rapport de recherche INRIA 1180 (Paris). M. Sweeney, M. Maguire, B. Schakel. Evaluating usercomputer interaction: a framework, Int. J. Man-Machine Studies 38 (1993) 6899711. C.A. Barry, Critical issues in evaluating the impact of IT on information activity in academic research: Developing a qualitative research solution, Library Information Sci. Res. 17 (1995) 107-134. J.P. Bansler, E. Havn, The Nature of Software Work Systems Development as a Labor Process, in: Van den Besselaar et al. (Eds.), Information System, Work and Organization Design. Proceedings of the IFIP TC9/ WG9.1 Working Conference on Information System, Work and Organization Design, Amsterdam, Elsevier Science Publishers B.V.. 1991 pp. 1455153. J. Brender, Methodology for assessment of medical ITbased solutions, in: Studies in Health Technology and Information, Vol. 42, IOS Press, Amsterdam, 1997. E.M.S. Van Gennip, A.R. Bakker. Challenges and opportunities for technology assessment in medical informatics. Case Study: PACS, Med. Inform. 18 (1993) 2099218.
Computer Methods and Programs in Biomedicine 54 (1997) 19-28
Cognitive evaluation:
euscart-Zkphir a Laboratoire ’ CERIM.
FacultC
-
How to assess the usability of information technology in healthcare a, J. Brender b, R. Beuscart ‘,*, I. Mknager-Depriester
de Psychologie de Mt!decine,
’
Cogniliue. UniuersitC Charles de Gaulle, Lille 3, France b M.I.L., Copeuhagcn. Denmark Unioersite de Lille, 1, Place Verdum, 59041 Lille Cedeex. France
Abstract As the adoption of information technology has increased, so too has the demands that these systems become more easier. The adapted to the physicians and nurses environments,, to make access and management of information developers of information systems in Healthcare must use quality management techniques to ensu.re that their product will satisfy given requirements. This underlines the importance of the preliminary phase where Users Requirements are elicited. Some methodologies, such as KAVAS (E.M.S. Van Gennip. F. Grimy, Med. Inform. 18, 1993, 179) chose to use a continuous assessment protocol as a key strategy for quality management. At each stage of the conception and development of a prototype, the assessment checks that it conforms to the expectation of the users’ requirements. The methodology of evaluation is then seen as a dynamic process which is able to improve the design and development of a dedicated system. The purpose of this paper is to demonstrate the necessity to include a cognitive evaluation phase in the process of evaluation by: (1) evaluating the integration (usability) of the I.T. in the activity of the users; and (2) understanding the motives underlying their management of information. This will help the necessary integration of information management in the workload of the healthcare professionals and the compatibility of the prototypes with the daily activity of the users. 0 1997 Elsevier Science Ireland Ltd. Keywords:
Information
technology;
Medical
informatics;
1. Ttmtro~duction The increasing accessibility of information technology (IT) systemsduring recent years has had a significant effect upon the healthcare field. Many healthcare establishments now operate heteroge* Corresponding author.
Assessment;
Cognitive
evaluation
neous IT environments with equipment ranging from stand-alone PCs to minicomputers and mainframe installations. The influence of information systems can now be seen in most areas of healthcare operation, with an ever increasing number and variety of medical applications. Evaluation methodologies [l-3] are required to assessthe successor failure of these systems. The
0169..260’7/97j$17.00 Q 1997 Elsevier Science Ireland Ltd. ,411rights Feserved. PII SOl69-2607(97)00030-8
20
d4.C.
Beuscart-Zkphir
et al. /Computer
Methods
KAVAS (AlOX) and KAVAS-2 (A2019) [4] projects have developed a methodology for evaluation of information systems. Although it was originally designed for the assessment of knowledge-based systems in medicine, this methodology proves to be generally adaptable to medical information technology. This methodology uses a reference model based upon the Users’ Requirements Analysis. Thus, elicitation of users’ requirements becomes a critical phase as they constitute the reference for the whole assessment methodolog:y as well as for the whole quality management methodology. Based upon the Users’ Requirements Analysis,, the KAVAS methodology, as others [5,6] takes into account the explicit users needs obtained after interviews, wallcharting, groupworks, and other consulting techniques [7]. As a consequence, the evaluation will be oriented to the technical validation of the software versus the technical specifications, to the usability and usage of the application versus the functional specifications. And this. assessment methodology will preferably point out at technical bugs,, human computer interfaces friendliness [S], daily usage of the information s,ystem. But, in our experience, some difficulties and failures are not explicated by technical bugs, ergonomic problems or usability inadequacy. They can be related to the interaction between users and their IT tools. Therefore it is also necessary t’o understand how the IT will be integrated in the users activity, when a physician or a nurse is performing a task where information is necessary as a support [9,10]. This is the objective of cogn:ltive evaluation (Fig. 1).
and Programs
in Biomedicine
2.1. Hospital
injhmation
54 (1997)
19-28
system
The University Hospital of Lille is developing an Integrated Hospital Information System (HIS) to improve the management of information within the medical units and to facilitate the communication of data between these units. The functionalities of the HIS were defined after a complete analysis of the data flows and information flows. This phase of the project was realised using an exhaustive Users’ Requirements Analysis [7]. As the HIS is developing, new needs appear. The physicians, after the first benefit of the basic procedures, want to develop or to acquire complementary applications that could help them in their daily work, to improve the diagnosis or treatment processes.
2.2. The European BAR project The aim of the project is to integrate 6 AIM prototypes on the HIS platform in the University Hospital. These prototypes were elicited from six European consortiums, able to provide pre-competitive products. To integrate the prototypes within the HIS, we defined an architecture based upon an ‘intelligent server’ allowing a safe exchange of data between the two parts, through APIs [11,12]. This architecture was effective in most cases and, using HL-7 as a standard for communication protocols, it was possible to use the prototype workstations in connection with the services offered by the HIS [11,12].
Users
f&a&
2. Background The KAVAS methodology was applied in the assessment framework of the ISAR project and we will illustrate the potentials of this method with real-life examples from this experience. Th:ls work was conducted in the University Hospital of Lille, France, with the collaboration of Aalborg University, Denmark.
Specifications
Fig. 1. Four corresponding
Verification
steps for Users Requirements (left side) and the four steps of evaluation (right side).
M.C.
Beuscart-Z&phir
et al. /Computer
Methods
After the technical integration of the prototypes with the HIS, we had to analyse if a prototype developed in another region, in another country, could ble installed, integrated and used in a new, completely different environment. In short, in this work, we study how the local constraints (workload, work organisation, local habits) can interfere with the potentials of medical computers. The issues, when introducing these new information systems in a medical department, were: e How can the information system be anchored to the clinical processes in order to support them. in an efficacious, effective and efficient way? e How can the information system adapt to the changes in the clinical processes? In this paper, we will take the example of the evaluation of the Anaesthesia Mobile System (AM§) integration with the Hospital Information System of the hospital. This AMS was designed in Ihe TA‘NIT (AIM A 2036) project and developlzd to collect information during the pre-anaesthesia consultation and to give the anaesthetists the possibility of having an understandable summary of the patients status during anaesthesia. The AMS was integrated with the HIS to import administrative and medical history data (stored in the HIS) and to export the anaesthesia report after consultation. A User Requirements Analysis was performed using the usual methods [7], validated the existing functions and asked for the improvements of some of them. Obtained from the Users’ Requirements Analysis, the computerised medical record model (Fjg. 2) of the pre-anaesthesia consultation, focuses on data flow and data repositories. In this model, the activity of the anaesthetist during the consultation is described by the users themselves as a ‘filling the form’ activity. We can notice that the data collected during the standard Users’ Requirements Elicitation say nothing about the underlying medical activity of the physicians, that is medical reasoning, clinical auscultation and so on. Users never referred to this type of activity during the ‘UK Analysis phase. That is why it is necessary to introduce a new step of ‘activity analysis and modelling’ and it’s corresponding cognitive evaluation step.
and Programs
m Biornedicilze
Fig. 2. The computerised the Users’ Requirements sultation. This model repositories.
54 (1997)
19-28
21
medical record model, obtained from Analysis of the pre-anaesthesia confocuses on the data flows and data
3. The KAVAS evaluation methodology KAVAS quality assessment framework
and
This Evaluation Methodology [4,5] suggests an intimate relation between the development and the evaluation throughout the entire system life cycle (Fig. 3). The main idea is to use the users requirements as a norm, and to check at each step of the lifecycle if the development of the tool is consistent with the initial objectives, as defined by the Users Requirements. In this sense, KAVAS assessment methodology can be seen as a quality management framework [4,13], as it assesses any divergence between the effective development of the tool and the stated requirements expressed in the planned development. For example, user’s requirements are translated in a detailed specification document allowing the development of the software. As soon as a first prototype is created, it must be assessed to see if it conforms, in all its aspects, to the Users Requirements expressed in the specification document. That is called ‘Validity phase’ or ‘Technical Verification’. In the same way, when the prototype has been installed, a new assessment phase takes place to check if, in real use, the tool satisfies Users Requirements exhaus-
M.C.
Beuscurt-ZJphir
et al. / Compuler
Merhods and Programs in Biomedicine 54 (1997) 19-3
:cess to the catalogue/list
To close Fig. 3. The KAVAS
Assessment
tively. This is called ‘Functi.onality Assessment’
4. Cognitive evaluation 4.1. The knits of the current methodology As a quality management methodology, th,e KAVAS framework grants that the final tool is consistent with stated users’ requirements. But as the main objective of the project is to build an information system, users’ requirements are defined according to an Information Flow point of view. Then elicitation and formalism of the Users’ rizquirements focus on the management of informarion, neglecting the work’s organisation, the analysis of the activity and the human factors. This may have bad consequencesfor the development of the tool as well as for the evaluation of the process itself. The IWVAS’s Evaluation Methodology allows detection of cognitive and ergonomic problems by means of the Functionality AssessmentMethodology (FAM). For instance, if there is a cognitive problem in a prototype, the FAM will detect it :m terms of deviations of the following kind: errors, appearance of changed and/or new compensatory procedures or activities. The whole idea about FAM is to detect problems in the work organisation with and around the IT-prototype. The dis-
the form
framework
from
Brender
et al. [5]
advantage with FAM is that it - to perform optimally - requires near-stable conditions and real practice, and that it is not dedicated to cognitive problems and therefore have a coarse granularity in this respect. One limitation with K.AVAS’s methodology is that it does not point ai: activities to provide the necessary frame of reference for cognitive assessment. But neither does it provide methods for the other types of assessmentthroughout the life-cycle, this has to blemade operational, when the decision on system development approach has been taken and in accordance with its assumptions and conditions. Thus, although the quality assessmentmethodology has been carefully followed, this may result in a tool with poor usability and hence poor acceptance. An important reason for failure is then neglected [8]. 4.2. Objectives:
assessment
of human ,factors
by
ergonomic and cognitive evaluation Our hypothesis is that the whole assessment process must take into account human faci.ors, i.e. the users’ tasks and activity underlying the information management activities supported by the new tool. As Users Requirements are the norm for the evaluation process, their content has to be extended: Users Requirements must include an expression of fundamental constraints linked to main features of users activity and tasks.
M.C.
Beuscart-Zphir
et al. ,/ Computer
Methods
and Programs
in Biomedicine
54 (1997)
1928
23
In many cases, there is a strong interaction between the medical activity of the professionals (nurses and/or physicians) and the management of administrative and medical information. A better knowledge of this interaction will lead to an improvement in the conception, design, development and installation of IT applications. Therefore, the main objective of this phase is the observation, analysis and modelling of the users’ activity when confronted with the IT application. But am activity is not performed for itself. An activity is goal-oriented in order to achieve a diagnosis or therapy procedure, or to organise a complex chain of tasks devoted to the management of the patient. So, the activity must be observeId not only as the description of the behaviour of the agents but as a series of actio.ns oriented towards an objective that must be identified so as to understand which strategies the agems develop to achieve their tasks.
the various phases of their activity. For each user, we identify sequences of mental (cognitive) and/or behavioural actions, and the underlying motives (aims) guiding these sequences of actions. This is summarised in a hierarchical formalism, describing tasks, sub-tasks, and sub-sub-tasks, until we reach the lower level of activity constituted by elementary actions. It is thus possible to identify the constraints linked to the tasks or goals aimed by users, constraints that the IT application should respect.
S. Methods
6. Results
The methods are essentially based upon the observations of the actors in the place of their medical activity.
As previously described, we applied the complete KAVAS methodology framework to perform the evaluation of the integration of foreign prototypes in the ISAR project [11,12]. Here, we take the example of the evaluation of the Anaesthesia Mobile System (AMS) integration with the Hospital Information System (HIS) of the hospital and we will focus on the pre-anaesthesia consultation information recording. The AMS was integrated with the HIS to import administrative and medical history data (stored in the HIS) and to export the anaesthesia report after consultation.
5.1. Obsemztions
md records
On-site observation of the users activity, before the installation of the application, is indispensable The scope of this observation is defined from the Users’ Requirements Analysis, the description of the a.pplications, and the characteristics of the prototype. The observation must focus on the part of the agents activity concerned by the IT application to be installed. This observation is supported by video and verbal protocols recorded during the task itself or afterwards while autofacing the videotapes. Interviews and questionnaires are directed to confirm the interpretation of these video and verbal protocols. 5.2. Modeelling This is a schematic description of the activity Iof the different users and of their interactions during
S.3. Ergonomic
and cognitive ev&uborz
This evaluation itself is ‘biphasic’. One part is realised in the laboratory environment, but with the users, The second part is realised in the unit, when the application is faced with the necessities and constraints of the real activity of the agents, and the interaction with the patients.
6.1. Technicul and HCI
evaluation
A complete technical verification of the prototype was realised to verify the absence of bugs, the presence of the items, screens and functions as they were described in the Technical Specifications Report obtained after the Users’ Requirements Analysis. Moreover, a study of the quality of Human Computer Interfaces @ICI) was realised according to the methods currently em-
M.C.
Belrscart-Z6pkir
et al. /Computer
Fig. 4. The
Task Model,
Methods
rind Programs
obtained
from
ployed in our hospital and referring principally to the INRlA Methodology [14,15]. At the end of this phase, the technical verification was considered as satisfying, even if some remarks were made about the quality of the human computer interfaces and the rapidity of the software. Thus, the cognitive analysis could be performed before introducing the device in the medical unit, to verify its usability in the daily activity of the physician. 6.2. Obsrrvation
of the users’ activities
Before introducing the AMS in the clinical unit, we observed the way anaesthetists were performing a pre-anaesthesia consultation. This observation demonstrated that the physician was involved in a number of simultaneous activities: 1. Interview of the patient to collect the antecedents of the patient and the history of the current disease; 2. Examination and auscultation of the patient (behavioural activity); 3. Planilication of the collection of critical information regarding anaesthesia and selection of the relevant data (cognitive activity); 4. Writing down the information (behavioural activity), and 5. Underlining the important data that could be essential in case of trouble during the anaesthesia. (cognitive activity). These observations permitted to build an elementary ‘Task Model’ (Fig. 4), enlightening the different aspects of the physicians activity and not only those related to the collection of data. The
bl Biomedicine
the observation
54 (1997)
19-28
protocols.
task model shows that the major aim of the an.aesthetist performing pre-anaesthesia consultation is to categorise the patient (anaesthesia without any risk; medium risk; high risk) in order to decide whether or not he will be able to perform the anaesthesia on this patient and which protocol to choose. The physician thus plans and organises the subtasks (questioning the patient; auscultation; selection of relevant information,. . .,) in order to be able to categorise and decide; all along this cognitive and behavioural medical activity, he writes down and eventually underlines relevant data, so ‘fulfilling the form’ which will be used on the theatre. Only this last and more visible layout of the activity has been identified during Users’ Requirements Analysis (Fig. 2). It is therefore essential to clarify immediately whether or not the tool’s integration within the subject’s activities will cause problems. If there is a risk of this nature, it is necessary to perform a sufficiently detailed description of this activity to be able to anticipate the interaction between this activity and the use of the future tool. This description must enable constraints to be defined: (1) for the detailed specifications; (2) where necessary for the functional specifications; and in any case, (3) for ergonomic choices concerning the interface. When data reading/writing activities supported by the computer tool are parallel or compatible with the other (mental/behavioural) activities within which they are integrated, no real problem will occur. On the contrary, the introduction of new information technologies could disturb the entire healthcare process.
M.C.
Beuscart-Z&phir
et al. /Computer
Methods
6.3. Motivations and goals orienting data acquisition and management
Another key dimension of the tool’s integration in the user’s daily activities concerns the motives underlying these data writing/handling/reading activities. Each user has his own representation Iof his task in general, and of each sub-task, sub-subtask, in particular. These representations enable him to derive plans to organise and finalise his activities. The specific data reading/writing activities are a part of these general plans, are als,o finalised, and can be put to use to perform other activities or sub-activities. In the AMS, the consultation involves the writing of medical data in order to pinpoint specifc risk factors and condition the anaesthesia to avoid catastrophes. The objective is not to build an exhaustive medical record. A strategy for selecting and underlining some of the data is therefore essential. The general objective is that, on the basis of this record, the anaesthesiologist (who might bc someone else other than the one whlo carried out the consultation) makes a good interpretation and organises the anaesthesia in consequence. Simply entering the data, however completely, is not adequate: if the record is (too) complete, the information is too crowded and this will be contrary to the objective. The AMS evolved to offer a ‘Summary’ function, following the users intuition that whatever the number of pages of the file, these would not enable the objective of the consultation to be reached, inasmuch as this is tied up with the anaesthesiologist’s activities. The expressed user requirem8ents never actually provide any information on the objectives pursued by the users when handling (entering,. .,) data. Therefore, it is not only the users’ usual activities that need to be described, but also the finalities of these activities, and particularly of the data handling activities. Tasks must be described. 6.4. Evaluation of the prototype consultation process
in the
After the previous validation steps and the observations, some physicians agreed, after training,
and Program
in Biomedicirze
54 (1997)
19-28
25
to use the prototype to manage the information during the pre-anaesthesia consultation. Seven physicians performed this experience. But the experience was a failure due to three main factors: o Duration of the collection of data: with the AMS, the duration for collecting data was more than 25 min, versus 10 nun with paper and pencil. o Ability to find the right data in the catalogues: even after training, the use of catalogues and repositories is too complex to authorise the collection of data and their recording during the consultation itself. o Cognitive problems. The arborescent architecture of the data base does not allow easy links between different sheets. It is not so easy to collect simultaneously the nature of a disease and its treatment as they are recorded in different parts and screens of the data base. In fact, these results iYere prediaable from the observations of the professionals activities and from the explication of the motivation of the physicians objectives. The means for this prediction is the building of a ‘Normative model’ which merges both the necessities of the computerised data collection and the constraints of the physicians activities. 6.5. The normative model
It is therefore possible to determine the functional normative model from the Users’ Requirements Analysis, from the observations protocols and from the explication of the users motivations. This normative model describes the activities of the professional and, as part of thds, the particular activity of collecting data on a computerised record, using the prototype to be assessed. If the medical record model, obtained from the Users’ Requirements Analysis (Fig. 2) is too different form the ‘Normative model’ (Fig. 5), and/ or if the task model (Fig. 4) is too different from the normative model (Fig. 5) then the probability of failure is very important. On the contrary, if the three models are close, then the probability of success will be high, since the introduction of the computers will not deeply modify the users activities.
I lnitiaiisation _. ..*.-.--“___.._-. _--ll Interrogation --. .-. ----
/ Questionning
1 of the file .-----Li j
1 /CATEG~R~~AT~~N I Auscuitation
7 _j
I ~~cis10~
/
IV To iselect the information
1 Storage
--
1 To underline
.^ important
data i
To choose the heading
To fill the field
t-L
4
To valida!e
“--I
To close the form
Fig. 5. The Normative Model, elaborated by confrontation between the Users’ Requirements Anaiysis and the observation protocols. During the pre-anaesthesia consultation, it is difficult for the physisican simultaneously to perform: the clinical examination, to extract the relevant information to be recorded, and to b’rowse through screens and catalogues to store and type the information using checklists.
In our case, the three models are profoundly different and the failure could be foreseen, even if the anaesthetists themselves were convinced of the opposite. Comparison between computerised medical record model and normative model (Fig. 2 vs. Fig. 4): the computerised medical record model takes no account of the usual medical mental/ behavioural activity of the physicians performing pre-anaesthesia consultation. This is a reason for failure of the integration of the new tool in the daily activity of the anaesthetists. Comparison between the task model and the normative model (Fig. 4 vs. Fig. 5): in the task model, the writing activities are deeply integrated in the other activities and need no additional mental or behavioural resources from the physician. On the contrary, the normative model shows the capture, coding and storage of data as an additional activity, which will inevitably take time and will compete with the medical activity. In other words, the physician will have to handle medical activities on the one hand and capture of data on the other hand. The new software tool will deeply mod-
ify the usual task of the anaesthetists. The difference between the stated task model and the observed task model is striking and frightening. It is frightening, because the traditional approach for preparation of IJser Requireme:nts is based upon the assumption that users are able to express their own way of operating. They are experts at their own mode of operation. However, the conclusion of our study is that they are not able to explicate their knowledge, which is in line with similar observations from other domains, of which knowledge engineering is the prime example. For instance, this phenomenon has been observed also from the domain of information retrieval [16], as well as from system development (Naur in [17]). This points to a paradox in today’s contractual approach for system development [18] and the expectations from system developers, as a detailed specification of the future system is required at an early stage. So the failure of the prototype could have been foreseen. But on the AMS case study, the activity analysis and modelling took place after the elaboration of the prototype. We think that it would be fruitful to perform such an analysis before the development of the prototype, to make it consis-
M.C.
Beuscart-Ziphir
tent with the characteristics which it will be integrated.
‘7. Discussion
et al. /Computer
Methods
of the activity
in
and conclusion
To perform the cognitive evaluation, related to the users activity and objectives, it is therefo:re necessary to follow a rigorous approach in four steps: !. The first step is the identification of goals or tasks and ‘subtasks’, and the analysis of the strategies the agents develop to perform these tasks (and reach their goals). 2. The second step consists of modelling the activity of the different users inasmuch as this activity is concerned by the installation of the new tool. This model must take into accou:nt behavioural as well as cognitive facets of the activity. 3. The third step is to create a normative model which describes the future activity supported by tlhe new tool. 4. The fourth step is to compare the actual and future model to the constraints the new tool must respect in terms of activity and tasks. These constraints constitute the additional layer of the users’ requirements. In this paper, we limited our example to the pre-analesthesia consultation but each phase of the anaesthsesia was equally concerned with this approach. We can give some examples: D During the induction of anaesthesia, the physician frequently glances at the pre-anaesthes.ia record. When using a paper record, he may put it down anywhere in the theatre, even on the stomach of the patient. How will this be possible with a computer? ID In the operating theatre: two or three vital elements of information underlined (or writtlen in red colour) on the sheet (of the medical record) are read while the syringe of hypnotic medication i.s being pushed. During the anaesthesia induction, it will also be impossible to push a syringe and read two or three screen pages of data on the computer, if the screen is easy and comfortable to read.
atzd Programs
in Biomedicine
54 (1997)
19-S
21
o Information must be easily reachable in every place of the operating theatre, and the record is frequently moved from one place to another, following the anaesthetist’s work. With the development of IT in the Healthcare domain, the necessity of quality management beclomes obvious. More often, the quality plan considers only the technical aspects of software engineering, debugging, performances: transferability [4,8,19]. In some cases the HCI [5,14,15] are also evaluated on subjective scales to identify their quality or the users’ workload. However, more and more the users are involved in the development of information applications. They are involved in the Users’ Requirements Analysis at the start of the project, and there is now a crucial need to involve them in the assessment also, even if one part of the evaluation is realised in a laboratory environment before installation. The KAVAS Methodology Framework provided us with a coherent four phase methodology taking into account the technical, functional, and impact evaluation. We have d.emonstrated that even this coherent methodology had to be completed by a cognitive approach to enable a better understanding of the connections between data management and the activities of the users with the goals they wish to attain. These two axes of evaluation are proving the usability of the system, that means the potentials of the application that actually could be used by the healthcare agents.
Acknowledgements This work its Project Telematics ‘Ganymede’ the Region
was supported by the Isar-Telemat(HC 1027) from the IEU Healthcare Programme and by the CPER for Advanced Communication from Nord-Pas de Calais, France.
References [l]
E.M.S. Van Gennip, F. GrCmy, Challenges and opportunities for technology assessment in medical informatics, Med. Inform. 18 (1993) 179-184.
28
M.C.
Beuscart-Zt!phir
et al. /Computer
Melhods
[2] F. Grtmy, P. Degoulet, Assessment of health information technology: which questions for which systems? Proposal for a taxonomy, Med. Inform. 18 (1993) 185-194. [3] P. Degoulet, L. Lucas, MC. Laurent, F.C. Jean, D. Sauquet, M. Lavril, An approach for the evaluation of software engineering environments in medicine, Med. Inform. IS (1993) 1955208. [4] J. Brender, Quality Assurance and Validation of large Information Systems - As viewed from the user perspective. Master Thesis in Computer Science, Report no. 89-l-22, Copenhagen University, March. 1989. [S] J. Brender, J. Talmon, J.P. Nykanen, P. McNair, M. Demeester, R. Beuscart, On the evaluation of system integration, in: E.M.S.J. Van Gennip, J.L. Talmon (Eds.), Assessment and Evaluation of Information Technologies in Medicine, 10s Press, Amsterdam, 1995, pp. 1899208. 161 G, Boloix, P.N. Robillard, A software system evaluation framework, Computer 28 (1995) 17-26. [7] R. Beuscart, J.L. Salomez, M.C. Nuttens, A.G. Pars, h. Purro, Participative methods for the definition of the hospital information system of the University Hospital in Lille, in: Medinfo 92, North Holland, Amsterdam, 1992, pp. 2188219. [S] P.C. Tang, V.L. Patel, Major Issues in user interface design for health professional workstations: a summary and recommendations, Int. J. Biomed. Comput. 34 (1994.) 1399148. [9] A.W. Kushniruk, D.R. Kaufman, V.L. Patel, Y. Levesque, P. Lottin. Assessment of a computerized patient record system: a cognitive approach to evaluating medic;%1 technology, M.D. Computing 3 (1996) 406-415. [lo] M.C. Beuscart-Zephir, J. Brender, N. Souf, I. MenagerDepriester, Cognitive Evaluation for the assessment of Information Technology, in: Healthcare, Proc. 13th Int.
and
Programs
[ll]
[12]
[13] [14]
[15]
[16]
[17]
[18]
[19]
in Biotnedicine
54 (1997)
19-28
Congress Med. Inform. Europe, IOS Press, Copenhagen, 19-22 August 1996. R. Beuscart, M. Demeester, B. Bossart, A. Dubos, B. Modjeddi, Integration Architecture: The ISAR Project, in: Medinfo 95, North Holland, Amsterdam, 1995, p. 981. R. Beuscart, B. Modjeddi, M. Demeester, ISAR: Integration System Architecture, in: M.F. Laires, M.J. Ladeira, J.P. Christensen (Eds.), Health in the New Communication Age, 10s Press, Amsterdam, 1995, pp. 3922403. T. Drake, Measuring Software Quality: A Case-study, Computer 11 (1996) 78-87. B. Senach, Evaluation ergonomique des interfaces homme-machine: une revue de la litttrature. Rapport de recherche INRIA 1180 (Paris). M. Sweeney, M. Maguire, B. Schakel. Evaluating usercomputer interaction: a framework, Int. J. Man-Machine Studies 38 (1993) 6899711. C.A. Barry, Critical issues in evaluating the impact of IT on information activity in academic research: Developing a qualitative research solution, Library Information Sci. Res. 17 (1995) 107-134. J.P. Bansler, E. Havn, The Nature of Software Work Systems Development as a Labor Process, in: Van den Besselaar et al. (Eds.), Information System, Work and Organization Design. Proceedings of the IFIP TC9/ WG9.1 Working Conference on Information System, Work and Organization Design, Amsterdam, Elsevier Science Publishers B.V.. 1991 pp. 1455153. J. Brender, Methodology for assessment of medical ITbased solutions, in: Studies in Health Technology and Information, Vol. 42, IOS Press, Amsterdam, 1997. E.M.S. Van Gennip, A.R. Bakker. Challenges and opportunities for technology assessment in medical informatics. Case Study: PACS, Med. Inform. 18 (1993) 2099218.